Choosing the Right Experts for the Right Case

Picture a tumor board. A hard case comes in, and instead of one radiologist staring at a scan, three or four specialists weigh in — a radiologist, an oncologist, maybe a pathologist. Each one brings a different read. Now imagine an AI model sits in the room too, not to make the call, but to hand everyone a shortlist: "the answer is almost certainly one of these three options." Who among the specialists should actually get a vote on this case, and how do you combine their opinions once they've voted?

That's the question at the center of our paper Conformal Set-Based Human-AI Complementarity with Multiple Experts, a paper selected as an Oral at AAMAS 2025 (the International Conference on Autonomous Agents and Multiagent Systems). Most human-AI collaboration research has quietly assumed a team of exactly two: one model, one person. This paper asks what changes once there's a room full of people, and it uses a statistical tool called conformal prediction to decide who in that room should be listened to.

What's a conformal prediction set?

Before getting to the multi-expert part, it helps to know what the AI is actually contributing. A conformal predictor doesn't just output "the answer is class 7." Instead, using a held-out calibration dataset, it outputs a set of plausible classes: "the answer is almost certainly A, B, or C". This output comes with a mathematical guarantee: for a chosen tolerance level (call it alpha), the true label falls inside that set at least (1 − alpha) percent of the time, no matter what the underlying classifier looks like. Lower alpha means a stronger guarantee, but also, unavoidably, a bigger set. Push alpha too close to zero and the "helpful shortlist" balloons into something close to the entire list of classes — technically safe, practically useless.

Adding more people to the room

Prior work on conformal sets for decision support (notably Straitouri et al.'s 2023 ICML paper) looked at a single human working with a single conformal set. Our contribution is to ask: if there are several human experts available, do we really need all of them, for every single case?

Their intuition is that different experts are good at different things, and a case that's easy for the AI to narrow down might only need one confident expert, while an ambiguous case might benefit from more perspectives — or from a completely different subset of people than the last case did. So rather than always polling everyone, the paper frames this as a subset selection problem: for each individual instance, pick the subset of available experts whose combined judgment is most likely to land on the right answer, and only then have that subset vote (typically by majority rule) on the final prediction.

This picks up a thread from a 2023 AAMAS paper by Singh, Jain, and Jha, which explored subset selection among multiple humans but without involving conformal sets. The contribution here is stitching the two ideas together: use the conformal set itself as the signal for deciding who should be included.

How the algorithm decides who gets a vote

The mechanics are worth walking through, because the cleverness is in how the paper works around a problem you'd hit immediately: you can't know, at test time, which experts will get the case right, because you don't have the ground-truth label yet.

Here's the workaround:

1. Model each expert as a confusion matrix. Using historical data, the paper estimates (via maximum likelihood) how often each expert's guess matches the true label — essentially a personalized track record, one matrix per expert, describing their tendencies to get things right or confuse specific classes for others.

2. Since the true label is unknown at test time, substitute a pseudo-label. The algorithm picks, from within the conformal set (not the full universe of classes), whichever class currently looks most likely to be correct, and treats that as a stand-in for the truth.

3. Score each expert against that pseudo-label. For a given expert, if the estimated probability of matching the pseudo-label exceeds 0.5, the math works out that including them can only help the aggregate estimate; below that threshold, they get left out.

4. Combine the survivors by majority vote to produce the final class prediction.

The appeal of tying this to the conformal set rather than the full label space is that it's provably better: the paper derives a lower bound on classification accuracy and shows that bound gets tighter when the pseudo-label and the selection criterion are restricted to the conformal set, compared to searching over every possible class. And because the whole procedure only ever loops over the humans available and the (typically small) conformal set, the running time scales linearly in both — cheap enough to run per-instance without being a bottleneck.

Testing it: CIFAR-10H and ImageNet-16H

Rather than running a live study with a fresh cohort of human annotators, the experiments lean on two existing datasets of real human judgments: CIFAR-10H and ImageNet-16H. These datasets already contain recorded predictions from many annotators on the same images. The confusion matrices are estimated from this real behavioral data, and everything downstream (the conformal sets, the subset selection, the combination step) is then run as a simulation on top of it. Across these multiclass classification tasks, the greedy subset-selection method comes close to the best possible subset and beats naive baselines like "just use everyone" or "pick a fixed group regardless of the instance."

Reading between the lines

A few things are worth sitting with rather than glossing over, especially if you're thinking about applying this kind of framework somewhere real.

The "one confusion matrix" simplification. The whole pitch is that different experts bring different, complementary perspectives — that's the entire justification for needing subset selection in the first place. Yet in practice, expert diversity has to be encoded somewhere mathematically, and the natural place is in each expert's confusion matrix. If those matrices end up looking similar across experts (which can happen depending on how they're estimated from shared datasets), the "diversity" the method is supposedly exploiting becomes harder to see or verify. It's a reasonable place to start, but it's worth asking, for any real deployment, how distinct the experts' error patterns genuinely are and whether the framework's benefits hold up if they aren't very distinct.

Simulation stands in for the real thing. Nobody in these experiments is a live doctor looking at a shortlist and changing their mind. The "experts" are historical annotation records replayed through a confusion-matrix model. That's a completely reasonable first step for validating an algorithm, and CIFAR-10H/ImageNet-16H are standard benchmarks in this literature precisely because they have real human labels to draw on. But both datasets are also comparatively easy, low-stakes classification problems (everyday objects, animals), which is a long way from the medicine-and-law scenarios the paper's motivation leans on. Whether expert subset selection matters as much when the images are ambiguous specialist cases rather than fairly distinguishable everyday photos is an open question the simulation can't fully answer.

The pseudo-label is a clever patch, but a patch nonetheless. Needing a stand-in for the ground truth at test time is unavoidable — you genuinely don't have the answer yet. Choosing that stand-in from within the conformal set, rather than from the full label space, is a sensible design choice, and it's the source of the tighter theoretical bound. But it does mean the "test" that decides which experts get included is itself only as reliable as this proxy label, and how that proxy's accuracy affects downstream system accuracy isn't something the lower-bound theory alone settles.

The alpha balancing act shows up again. The paper's own analysis shows that pushing alpha toward zero technically maximizes the theoretical bound, but concretely risks conformal sets so large they stop narrowing anything down, which was the entire point of using conformal prediction in the first place. In the reported experiments, a well-calibrated setup keeps sets small even at a low alpha, but that's a property of having an accurate underlying classifier and well-distributed calibration data. This is not something guaranteed in a messier real-world deployment where the pretrained model might be shakier or calibration data scarcer.

Standing on specific shoulders. The framework's two main ingredients — conformal-set-guided decision support, and subset selection among multiple humans. Each already exist separately in the literature (Straitouri et al.; Singh et al.). The genuine contribution is combining them and proving the combination is provably better than either alone. That's a legitimate and non-trivial result, but it's also worth being precise about what's new: it's the combination and the tighter bound, not the individual pieces.

Why it's still worth paying attention to

The core idea is that not every case needs every expert, and that a statistical tool originally designed to help a single human can also be repurposed as a selection signal across many humans; it is a genuinely useful reframing. It turns "how do we combine multiple experts?" into a cleaner, more tractable question: "given what the AI already tells us about the space of plausible answers, which experts does this specific case actually need?"

Whether that reframing survives contact with real specialists making real decisions, on genuinely hard cases, with genuinely diverse expertise, is the natural next experiment. But as a theoretically grounded first step, it's a solid foundation to build that next experiment on human-AI interaction.

References

- Paat, H., & Shen, G. (2025). Conformal Set-Based Human-AI Complementarity with Multiple Experts. In Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2025), 1576–1585.

- Straitouri, E., Wang, L., Okati, N., & Gomez Rodriguez, M. (2023). Improving Expert Predictions with Conformal Prediction. International Conference on Machine Learning (ICML).

- Singh, S., Jain, S., & Jha, S. S. (2023). On Subset Selection of Multiple Humans to Improve Human-AI Team Accuracy. AAMAS 2023.