The Expert You Thought Was Irreplaceable?

A deep dive into Harvard research on the future of complex decision-making

Good judgment is supposed to be one of the last human advantages. We trust experienced people to read between the lines, understand context, weigh imperfect information, and make nuanced calls that machines cannot. It’s a reassuring idea, but it’s also one that new research has put under serious pressure. In “Who Is a Better Matchmaker? Human vs. Algorithmic Judge Assignment in a High-Stakes Startup Competition,” a team including Jacqueline Lane of the HBS AI Institute, asked a pointed question: could an AI algorithm assign expert judges to startup ventures as well as a seasoned human administrator? To find out, they built one, deployed it in an actual competition, and measured its performance. As the results show, the balance between human expertise and algorithmic scale may finally be tipping away from humans. 

Key Insight: The Complexity of Expert Selection

“Can algorithmic judge assignments match the quality of expert human judgment in such high-stakes entrepreneurial evaluation settings?” [1]

The study took place in the Harvard President’s Innovation Challenge, the university’s premier venture competition, where more than $500,000 is distributed to student and alumni startups. For the 2025 competition, administrators were tasked with assigning 231 high-profile judges, including experienced investors and senior scientists, to 101 semifinalist ventures across tracks such as healthcare, fintech, consumer products, and social impact. Each venture requires exactly 12 judge assignments, and each judge can evaluate no more than seven startups. From 2021 to 2024, that task fell to a single program administrator working from informal heuristics, consuming roughly a full work week. The best-performing algorithms already in existence for a task like this depend on citation networks and publication histories missing in this kind of judging pool. As the competition and complexity grew over the years, the case for a new AI solution became more and more attractive.

Key Insight: AI Fine-Tuned for the Business Context

“The HLSE method is an ensemble model that integrates multiple similarity computation models as base learners.” [2]

To tackle the matchmaking problem, the researchers developed the Hybrid Lexical-Semantic Similarity Ensemble (HLSE), a sophisticated model that combines three distinct types of text analysis. The first uses traditional “lexical” matching, meaning it looks for exact keyword overlaps. For example, if a startup mentions “fintech,” the model looks for judges who also use that specific word. The second component utilizes “dense transformer-based embeddings,” which convert text into numerical representations to capture the deeper semantic meaning of words. This helps the model recognize that “digital banking” and “peer-to-peer lending” may be related even when the words differ. A third hybrid component combines both approaches by giving extra weight to important terms while still preserving broader semantic meaning. This design matters because startup judging depends on business, technical, and market vocabulary that might be missing from academic sources used as the basis for traditional matching systems.

Key Insight: No Significant Difference – and That’s the Whole Point

“These results indicate no statistically significant differences in match quality between the assignment approaches.” [3]

To prove the algorithm’s effectiveness, the researchers conducted a head-to-head experiment. The algorithmic process had two parts: HLSE first estimated how well each judge’s background matched each venture, then the researchers used PeerReview4All (an assignment algorithm) to generate the actual judge assignments. The researchers then compared those algorithmic assignments with ones made independently by human experts. Judges were not told whether a match came from a person or the algorithm, and they rated each assignment on a 5-point scale based on how well the venture matched their background and expertise. Human expert assignments averaged 3.94 out of 5; algorithmic assignments averaged 3.90, so the two approaches were effectively indistinguishable. But the time difference was substantial: algorithmic data collection and assignment together took roughly five to seven hours, compared with the full week for the manual process. The authors are careful to frame these findings not as a case for replacing human administrators but as a blueprint for collaboration, where AI handles scale and speed, and humans provide oversight and refinement where it adds the most value. 

Why This Matters

Any organization that routinely matches people to problems, such as assigning consultants to client engagements, routing complex customer cases to the right specialists, evaluating grant applicants, or even pairing mentors with early-career employees, faces a structurally similar opportunity as in the Innovation Challenge. This study suggests that a well-designed AI, especially if it is trained for its specific domain and evaluated against real-world outcomes, can match expert human performance while dramatically reducing the time and coordination costs involved. For business leaders and executives, this shows a clear AI use case to make high-volume judgment processes faster and more consistent without sacrificing quality.

Bonus

If AI can match experts in assignment problems, can it also be trusted to conduct the research those experts rely on? For a look at a new benchmark for evaluating how leading AI research tools perform on real-world production tasks, check out The AI Deep Research Race Has a New Leaderboard.

References

[1] Xi, Yang, et al., “Who is a Better Matchmaker? Human Vs. Algorithmic Judge Assignment in a High-Stakes Startup Competition,” Proceedings of the AAAI Conference on Artificial Intelligence 40 (47) (March 14, 2026): 40138. https://doi.org/10.1609/aaai.v40i47.41449 

[2] Xi et al., “Who is a Better Matchmaker?” 40141.

[3] Xi et al., “Who is a Better Matchmaker?” 40142.

Meet the Authors

Watch a video version of the Insight Article here.