Build a Human–AI Team
Compare human and AI decisions, full review and selective review. Spend a review budget where it can help.
When does adding human review improve the result?
A team does not automatically outperform either member. Review can correct or introduce errors, and the allocation of attention changes both outcomes and costs.
Make a prediction.
Then move a control.
20 invented support cases · 8 urgent, 12 routine. “Human alone” is the initial human judgment. Review uses a separate judgment after additional attention; that judgment can also be wrong.
| Strategy | Accuracy | Urgent misses | False alerts | Human cases | Total cost |
|---|---|---|---|---|---|
| Human alone | 75% | 3 | 2 | 20 | 180 |
| AI alone | 65% | 3 | 4 | 0 | 170 |
| Review every case | 85% | 1 | 2 | 20 | 80 |
| Selective review | 85% | 1 | 2 | 6 | 66 |
Inspect the decisions and routing
Context flags describe information available before the decision. The reference label is shown here to evaluate outcomes, not to choose which cases receive review.
| Case | Reference | AI score | Initial human | Review judgment | Context flag | Routed? | Selective decision | Effect of review |
|---|
Compare the team with both human alone and AI alone on the same cases. A helpful override repairs an error; a harmful override introduces one. Agreement can still be wrong. Total cost = missed urgent cases × miss cost + false alerts × alert cost + human cases × work cost.
Changing only the costs changes the evaluation, while the decisions and accuracy stay fixed. Score proximity is a routing heuristic, not verified model uncertainty.
An educational experiment with illustrative data. The controls run locally in your browser.
The idea,
in Python.
A compact implementation of the core concept. Open the code when you are ready to connect the visual to the algorithm.
Open the core implementation Python ↗
import numpy as np
reference = np.array([1, 0, 1, 0, 1, 0])
score = np.array([0.9, 0.7, 0.45, 0.2, 0.6, 0.1])
initial_human = np.array([1, 0, 1, 0, 0, 0])
review = np.array([1, 0, 1, 0, 1, 1]) # separate, fallible review judgment
ai = (score >= 0.5).astype(int)
# Illustrative routing: evaluate it on representative validation cases.
route = np.abs(score - 0.5) <= 0.15
selective = np.where(route, review, ai)
helpful = route & (review != ai) & (review == reference)
harmful = route & (review != ai) & (ai == reference)
print("Selective decisions:", selective)
print("Helpful / harmful:", helpful.sum(), harmful.sum())Explain the change.
Test your reasoning.
- Compare all four strategies before changing any setting.
- Increase the cost of an urgent miss. Does accuracy change when the decisions stay fixed?
- With a limited review budget, inspect helpful and harmful overrides. Which cases deserve attention?