This series introduces Harvard Business School AI Institute Associates Program projects which aim to answer important questions at the intersection of artificial intelligence and digital technologies in business and society.
This article shares insights from Hannah Riley Bowles, Roy E. Larsen Senior Lecturer in Public Policy and Management, Harvard Kennedy School of Government and Nadja Born, PhD Candidate, Harvard Kennedy School of Government who are pursuing research on behavioral science and artificial intelligence in organizations
1. What drew you to this area of research and how did you first become involved in this work?
We have been experimenting for years with ways to leverage technology to support leadership education, particularly negotiation advice and coaching for professionals in early, mid-career, and senior executive positions. Recent advances in AI have made our tech-based coaching tools meaningfully more engaging and useful by integrating evidence-based advice with situation-specific data on negotiating norms and standards. The accelerating growth of AI tools in leadership and career coaching has focused our attention on maximizing their potential and minimizing their perils.
If AI tools can be trained to provide evidence-based leadership coaching and advice, they could enhance the impact of professional education and democratize access to its lessons. However, this aspiration raises numerous challenging questions. For instance, as with the varied quality of human leadership coaching and advice, how will users distinguish evidence-based AI guidance from less rigorous messaging with intuitive appeal? What factors influence professionals’ productive engagement with AI coaching tools? Many organizations are actively experimenting with AI coaching approaches, but we still do not know under what conditions these technologies enhance performance. The tension between this unprecedented opportunity for scalability and the challenges of evidence-based, human-centered implementation is compelling.
2. What are some common misconceptions or barriers around the problem you’re working to solve?
We commonly observe two misconceptions that create barriers to scaling evidence-based AI coaching tools. People assume that, because large language models (LLMs) have been trained on enormous amounts of information, their advice will naturally reflect the best available evidence. People rely on AI coaching for solving problems, in contrast to supporting problem-solving. These working assumptions make AI models, particularly those prone to sycophancy, highly appealing. The associated risks are that these misconceptions distance users from rigorous evidence and weaken their critical skills by providing answers rather than sharpening behavioral competencies and analytic abilities.
Our research addresses three specific barriers to achieving the potential benefits of AI coaching tools.
- How to reduce biases in advice provided by AI, whether by training the AI or its human users?
- What types of interventions would support the usage of evidence-based AI coaching tools? For instance, we have observed high initial engagement in social settings, followed by diminishing independent usage. We do not yet understand how evidence-based AI tools become part of regular workflows.
- How should organizations assess the usefulness of AI coaching tools (e.g., subjective and objective performance indicators relative to costs)?
We are addressing this first question through lab-based experimental research. In relation to these second and third questions, we are collaborating with organizational partners on their implementation of AI coaching tools and strategies for experimentally testing their usefulness. We also remain cognizant of privacy and ethical concerns associated with reliance on AI as a personal tool in a work setting.
3. What research is being done on this topic and how is your approach or perspective unique?
Behavioral science has accumulated decades of robust knowledge about judgment, decision-making, motivation, learning, bias, and interpersonal behavior. AI creates an extraordinary opportunity to translate that knowledge into tools that people can actually use in their everyday decisions. There is an emergent body of scholarly research on leadership-related AI coaching, specifically in the context of negotiation simulations and collaborative work in experimental settings. At the same time, many organizations are actively testing out the usefulness of AI coaching tools, though not systematically evaluating their performance. We aim to bridge our ongoing lab-based experimental research on AI coaching with field experiments in organizational settings. In this way, we aspire to produce rigorous and practically useful insights for organizations interested in where AI coaching succeeds or fails and then use those insights to develop more effective forms of AI-supported coaching.
4. What excites you most about this work and its potential impact?
The overarching question that really excites us is how we can make AI tools enhance productivity and human development. Evidence-based AI coaching tools could help people prepare more mindfully for collaborative work, recognize possibilities they might otherwise overlook, question their assumptions, and make wiser individual and collective decisions. Absent an evidential basis and genuine coaching orientation, AI coaching could be a cost center with little benefit or even contribute to a depletion of critical thinking and increased dependence on AI agents. These costs could also fall differentially on some types of professionals, for instance, undermining early career development while enhancing the performance of experienced professionals. We are motivated to collaborate with scholars and leaders in practice to illuminate potential benefits and pitfalls of AI coaching tools and practices.
5. How do you hope working with the HBS AI Institute will amplify the impact of your work?
The team at the HBS AI Institute combines a culture of excellence with remarkable proactivity, intellectual openness, and support for collaboration. In doing so, the HBS AI Institute provides an exceptional learning environment for connecting rigorous scholarship with experimentation at the frontier of practice. We are especially excited about learning from both scholarly innovators and organizational leaders who are actively grappling with how to deploy AI responsibly and productively. Ultimately, we hope the AI Institute will help us build a tighter bridge between behavioral science and AI development, so that insights about human judgment, learning, negotiation, and bias inform how AI tools are designed and deployed at scale.
6. What changes do you hope to see in your field as a result of the work being done in this area?
We hope to contribute to an expansion of scholarship on the promise and perils of AI coaching tools. This would include behavioral scientists testing the effectiveness of AI support for human decision-making, collaboration, and conflict management; organizational scientists investigating the effectiveness of initiatives to integrate AI tools into work practices; and psychologists examining when AI tools enhance or deplete human learning potential and well-being.
7. What’s an essential area in which AI and digital technologies will reshape the way business or society operate in the long run that we may not be considering?
One of the most consequential questions is how to capture the productivity benefits of AI without undermining the development of the human capabilities on which long-term productivity depends? The central risk may not be that AI “outwits” humans. A more immediate concern is that people become so accustomed to outsourcing difficult cognitive work that they have fewer opportunities to develop the judgment, expertise, creativity, and critical-thinking skills that enable them to evaluate what AI produces. This creates an important distinction between human performance and development. AI may improve our performance on a task today while simultaneously reducing the learning that would make us better at that task tomorrow. Conversely, thoughtfully designed AI could do both: help us perform better in the moment while prompting us to reason more deeply, consider alternatives, and develop expertise.
The Harvard Business School AI Institute Associates Program supports and accelerates faculty research into the ways AI and digital technologies are reshaping companies, organizations, society, and practice.