Choosing the Right Experts for the Right Case

Imagine a tumor board. There is a hard case, but instead of one radiologist solving the problem, there are multiple specialists in the room. We have a radiologist, an oncologist, or maybe a pathologist in the room. Each one brings a different interpretation. Now imagine an AI model is also present in the room. The model will not give a final prediction, but it will provide everyone a list of possible answers: "the answer is almost certainly one of these three options." Who among the specialists should actually get a vote on this case, and how do you combine their opinions once they've voted?

That's the question at the center of our paper Conformal Set-Based Human-AI Complementarity with Multiple Experts, a paper selected as an Oral at AAMAS 2025 (the International Conference on Autonomous Agents and Multiagent Systems). Most human-AI complementarity approaches assumes a dual team consisting of one model and one person. This paper asks what changes once there's a multitude of people or experts; it uses a statistical tool called conformal prediction to determine who in that room we should listen to.

What's a conformal prediction set?

Before getting to the multi-expert part, it helps to know what the AI is actually contributing. A conformal predictor doesn't just provide "the answer is class 7." Instead, using a held-out calibration dataset, it outputs a set of plausible classes: "the answer is almost certainly A, B, or C". Now one could imagine, if the conformal predictor is uncertain, the size of the set could be large. However, when it is sure, it could only provide set of small size, such as one or two. This output comes with a mathematical guarantee: for a chosen tolerance level (call it alpha), the true label falls inside that set at least (1 − alpha) percent of the time, no matter what the underlying classifier looks like. Lower alpha means a stronger guarantee, but also, unavoidably, a bigger set. Push alpha too close to zero and the "helpful shortlist" grows into something close to the entire list of classes. This is technically safe but practically useless for the experts.

Adding more people to the room

Prior work on conformal sets for decision support (notably Straitouri et al.'s 2023 ICML paper) looked at a single human working with a single conformal set. Our contribution is to ask: if there are several human experts available, do we really need all of them, for every single case?

Their intuition is that different experts are good at different things, and a case that's easy for the AI to narrow down might only need one confident expert, while an ambiguous case might benefit from more perspectives or from a completely different subset of people than the last case did. So rather than always polling everyone, the paper frames this as a subset selection problem. For each individual instance, pick the subset of available experts whose combined judgment is most likely to land on the right answer, and only then have that subset vote (typically by majority rule) on the final prediction.

This picks up a thread from a 2023 AAMAS paper by Singh, Jain, and Jha, which explored subset selection among multiple humans but without involving conformal sets. The contribution here is combining the two ideas together by using the conformal set itself as the signal for deciding who should be included.

How the algorithm performs expert selection

Details of the algorithm are going to be discussed here and will answer the question: "how do we decide during inference which experts will be selected even if we do not have access to the ground-truth label?"

Here's the idea:

1. Each expert is modeled as a confusion matrix. Given the historical data, the paper uses maximum likelihhod to estimate how often each expert's prediction matches the true label. There is a matrix associated to each expert which describes the chances that the expert will pick the right class or will mistaken it for others.

2. Pseudo-label selection THe algorithm selects from within the conformal set the class that is most likely to be correct. It will then treat this as the pseudo-label.

3. Each expert is assessed based on the pseudo-label If an expert's estimated probability of getting the pseudo-label right is greater than 0.50, then it is ideal to select this expert. If the probability is below that threshold, these experts are ignored.

4. Combine the survivors by majority vote to produce the final class prediction.

4. Using majority voting, we combine the selected experts to produce the final class prediction.

Provably, utilizing the conformal set rather than the full label is better. A lower bound on classification accuracy is derived by the paper. It shows that the bound gets tighter when the pseudo-label and the selection criterion are restricted to the conformal set. The whole method only loops over the available humans the (ideally small) conformal set, hence the running time scales linearly in both. This is cheap enough to run the method for each instance during test.

Testing it: CIFAR-10H and ImageNet-16H

Rather than running a live study with a fresh cohort of human annotators, the experiments lean on two existing datasets of real human judgments: CIFAR-10H and ImageNet-16H. These datasets already contain recorded predictions from many annotators on the same images. The confusion matrices are estimated from this real behavioral data, and everything downstream (the conformal sets, the subset selection, the combination step) is then run as a simulation on top of it. Across these multiclass classification tasks, the greedy subset-selection method comes close to the best possible subset and beats naive baselines like "just use everyone" or "pick a fixed group regardless of the instance."

Reading between the lines

A few things are worth sitting with rather than glossing over, especially if you're thinking about applying this kind of framework somewhere real.

The "one confusion matrix" simplification. The whole pitch is that different experts bring different, complementary perspectives — that's the entire justification for needing subset selection in the first place. Yet in practice, expert diversity has to be encoded somewhere mathematically, and the natural place is in each expert's confusion matrix. If those matrices end up looking similar across experts (which can happen depending on how they're estimated from shared datasets), the "diversity" the method is supposedly exploiting becomes harder to see or verify. It's a reasonable place to start, but it's worth asking, for any real deployment, how distinct the experts' error patterns genuinely are and whether the framework's benefits hold up if they aren't very distinct.

Simulation stands in for the real thing. Nobody in these experiments is a live doctor looking at a shortlist and changing their mind. The "experts" are historical annotation records replayed through a confusion-matrix model. That's a completely reasonable first step for validating an algorithm, and CIFAR-10H/ImageNet-16H are standard benchmarks in this literature precisely because they have real human labels to draw on. But both datasets are also comparatively easy, low-stakes classification problems (everyday objects, animals), which is a long way from the medicine-and-law scenarios the paper's motivation leans on. Whether expert subset selection matters as much when the images are ambiguous specialist cases rather than fairly distinguishable everyday photos is an open question the simulation can't fully answer.

The pseudo-label is a clever patch, but a patch nonetheless. Needing a stand-in for the ground truth at test time is unavoidable — you genuinely don't have the answer yet. Choosing that stand-in from within the conformal set, rather than from the full label space, is a sensible design choice, and it's the source of the tighter theoretical bound. But it does mean the "test" that decides which experts get included is itself only as reliable as this proxy label, and how that proxy's accuracy affects downstream system accuracy isn't something the lower-bound theory alone settles.

The alpha balancing act shows up again. The paper's own analysis shows that pushing alpha toward zero technically maximizes the theoretical bound, but concretely risks conformal sets so large they stop narrowing anything down, which was the entire point of using conformal prediction in the first place. In the reported experiments, a well-calibrated setup keeps sets small even at a low alpha, but that's a property of having an accurate underlying classifier and well-distributed calibration data. This is not something guaranteed in a messier real-world deployment where the pretrained model might be shakier or calibration data scarcer.

Standing on specific shoulders. The framework's two main ingredients — conformal-set-guided decision support, and subset selection among multiple humans. Each already exist separately in the literature (Straitouri et al.; Singh et al.). The genuine contribution is combining them and proving the combination is provably better than either alone. That's a legitimate and non-trivial result, but it's also worth being precise about what's new: it's the combination and the tighter bound, not the individual pieces.

Why it's still worth paying attention to

The core idea is that not every case needs every expert, and that a statistical tool originally designed to help a single human can also be repurposed as a selection signal across many humans; it is a genuinely useful reframing. It turns "how do we combine multiple experts?" into a cleaner, more tractable question: "given what the AI already tells us about the space of plausible answers, which experts does this specific case actually need?"

Whether that reframing survives contact with real specialists making real decisions, on genuinely hard cases, with genuinely diverse expertise, is the natural next experiment. But as a theoretically grounded first step, it's a solid foundation to build that next experiment on human-AI interaction.

References

- Paat, H., & Shen, G. (2025). Conformal Set-Based Human-AI Complementarity with Multiple Experts. In Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2025), 1576–1585.

- Straitouri, E., Wang, L., Okati, N., & Gomez Rodriguez, M. (2023). Improving Expert Predictions with Conformal Prediction. International Conference on Machine Learning (ICML).

- Singh, S., Jain, S., & Jha, S. S. (2023). On Subset Selection of Multiple Humans to Improve Human-AI Team Accuracy. AAMAS 2023.