The Vision: AI That Learns From You
At Laguna Health, our AI philosophy is built on a virtuous cycle of support and analysis. It begins with Laguna Companion, our ambient intelligence tool that assists clinicians in real-time. As a care manager speaks with a patient, Companion listens in the background, providing in-the-moment guidance, automating documentation, and surfacing critical information to help them deliver the best possible care. While it actively supports the clinician, it also passively gathers the data that powers our hallmark analytics product, Laguna Insight.
The initial version of Insight, which was honored as one of TIME’s Best Inventions of 2024, delivered powerful out-of-the-box analytics. It used a set of pre-defined clinical and compliance-based skills, developed by our own in-house team of clinicians, to provide organizations with immediate visibility into team performance. This gave care management leaders an unprecedented tool for coaching, compliance monitoring, and understanding the dynamics of their patient conversations.
Now, with the launch of Insight 2.0, we are taking this to a new level of AI novelty and clinical excellency. We are moving beyond pre-defined templates to a model of deep, collaborative alignment with each organization’s unique standards. This article delves into the sophisticated methodology behind Insight 2.0, explaining how we partner with clients on a journey of inter-rater reliability (IRR) and organizational refinement to build a truly aligned AI.
The Goal: Beyond Accuracy to Deep Alignment
The promise of Artificial Intelligence (AI) in healthcare is not merely to automate tasks, but to amplify human expertise and bring scale to the nuanced, often unquantifiable, aspects of care. While AI can readily parse structured data, its application to the delicate interplay of clinical conversations—a mix of regulatory compliance, clinical knowledge, and profound empathy—presents a far greater challenge. The goal is not just to build an accurate AI, but to create a system that is deeply aligned with an organization’s specific standards of clinical excellence.
This is a challenge we at Laguna Health have embraced with Insight 2.0, our conversational care intelligence platform. The process goes far beyond a simple “human-in-the-loop” model. It is a collaborative journey of inter-rater reliability (IRR) and organizational refinement, one that elevates a client’s own quality standards while simultaneously training a highly aligned AI. This article delves into this sophisticated methodology, explaining how we move from a client’s unique standards to a fully scaled, transparent, and efficient quality assessment engine.
The Foundation: Starting with Clinical and Organizational Expertise
Crucially, our process never begins with a generic, “out-of-the-box” model. The starting point is always the client’s own expertise, embodied in their quality evaluation forms and operational guidelines. We recognize that many organizations have already undertaken their own rigorous inter-rater reliability journeys. They have worked with institutes of clinical excellence and professionalism, debated criteria, compared interpretations across reviewers, and refined their definitions of quality through iterative cycles. When they hand us their form and accompanying guidelines, they are handing us a well-thought-out framework—one that has been forged through careful deliberation about what constitutes excellence in their specific context. This provides the AI with a powerful head start.
This initial step involves converting their form—whether from Excel, PDF, or another system—into a structured digital prototype. But this is where the Laguna Health difference truly begins. Our team is not just composed of engineers; we have nurses and psychologists on staff who are experts in this domain. They perform the initial “teaching” of the AI prototype, grounding it in clinical reality from the very first iteration. They understand that assessing a clinical conversation is fundamentally different from other domains. This is not about “closing a deal” in a marketing call center or “escalating appropriately” for support agents. This is healthcare. These are delicate, abstract skills—motivational interviewing, therapeutic alliance, trauma-informed communication, empathy that builds trust. Our clinical team knows these concepts intimately, both from academic training and from years of practice. This foundational training, performed by clinical experts who speak the language of healthcare, ensures the AI is primed with a deep, domain-specific understanding from day one, getting it remarkably close to the client’s standards before their own team even sees it.
The Evaluation Phase: A Collaborative Inter-Rater Reliability Journey
What follows is the “Evaluation” phase, which is best understood as a formal inter-rater reliability process between two parties: the AI and the organization’s Subject Matter Expert (SME). Importantly, the SME continues their existing audit workflow. They are not being asked to do something new or burdensome. They review calls, listen to recordings, read transcripts, and assess performance across their evaluation form—exactly as they have always done. The only difference is that now the AI pre-populates the evaluation form with its initial assessments. As the SME reviews the call and makes corrections, they are not just “labeling data” in the traditional machine learning sense; they are acting as one of two raters in a continuous reliability study. Their work generates training data as a natural byproduct of their normal responsibilities.
We systematically perform a gap analysis on the discrepancies between these two raters. This analysis is critical because it surfaces two distinct types of gaps, each with different implications:
Consistent Gaps occur when the AI consistently misunderstands a concept, and the SMEs consistently correct it the same way. For example, the AI might consistently score calls as “proficient” in empathy when the care manager uses phrases like “I understand” or “That must be difficult,” but the SMEs consistently downgrade these scores to “emerging” because they recognize that true empathy in this organization’s framework requires not just acknowledgment but also reflective listening and validation of the patient’s emotional experience. This is a clear signal for the AI to learn from the human experts. It is a classic case of domain adaptation [3], where the model is refined to better capture a specific clinical nuance that matters deeply to this organization.
Differential Gaps are more profound and more revealing. These occur when different SMEs correct the AI in different ways, or when they disagree with each other’s assessments. For instance, one SME might score a call as “proficient” in “logical call flow” because all required topics were covered, while another SME scores the same call as “emerging” because the topics were covered in a sequence that felt disjointed to the patient. This reveals an ambiguity not in the AI, but in the organization’s own definition of the criteria. It suggests that the human reviewers themselves are not perfectly aligned—a common challenge in healthcare quality assessment [1] [2]. The criteria may be well-intentioned, but they are not yet crisp enough to ensure consistent interpretation across different reviewers.
A Journey of Organizational Refinement
When differential gaps are identified, we surface these insights back to the organization. This often sparks a crucial dialogue, transforming the AI implementation into a broader quality improvement journey. It prompts teams to ask: “What do we really mean by ’demonstrates empathy’ in this context? What are the specific behavioral markers that distinguish ’emerging’ from ’proficient’? When we say ’logical call flow,’ are we prioritizing completeness or coherence? Are we measuring adherence to a script, or are we measuring the patient’s subjective experience of being heard and understood?”
This is not a superficial exercise. It is a deep dive into the organization’s values and priorities. We work collaboratively with the organization to refine the criteria, often suggesting specific language or examples that can clarify ambiguous concepts. For instance, we might propose that “proficient empathy” be defined as including at least two of the following: reflective listening (“It sounds like you’re feeling overwhelmed”), validation of emotions (“It’s completely understandable that you’d feel that way”), and an offer of support (“Let’s figure this out together”). By making these definitions more concrete and behavioral, we reduce ambiguity and increase inter-rater reliability—not just between the AI and the SME, but among the SMEs themselves.
We have found that clients deeply appreciate this collaborative process. What begins as an AI implementation becomes a journey of organizational self-discovery and refinement. It helps them clarify the very definitions of quality and excellence they are measuring. By iterating together—reviewing gap analysis reports, discussing edge cases, refining criteria, and then testing the refined criteria in the next audit cycle—we help them forge a crisper, more robust set of standards. This journey elevates their clinical quality standards and drives transparency and alignment, ensuring everyone—from frontline clinicians to supervisors to executive leadership—shares a common, operationalized understanding of what success looks like. For many organizations, this process is as valuable as the AI itself, because it brings clarity and consistency to their quality assurance program in ways they had not achieved before.
The Outcome: Scalable Excellence with Transparency and Efficiency
The result of this intensive, collaborative process is twofold. First, the organization emerges with a more clearly defined and consistently applied set of quality standards. The criteria are no longer vague aspirations but concrete, behavioral definitions that can be reliably assessed. Second, the AI becomes deeply aligned with these newly crisp criteria, having learned not just from generic clinical data but from the specific judgments and values of this organization’s own experts.
When we finally launch the full Laguna Insight dashboard, the AI is no longer just a prototype. It is a high-reliability scoring engine, calibrated to the unique DNA of the organization. It can then analyze 100% of conversations—not just a sample of 2% or 5%—providing scores that are not only accurate but also deeply meaningful, because they are based on a shared, well-understood definition of excellence. This comprehensive coverage is transformative. It allows organizations to achieve quality oversight on the most abstract, delicate, and nuanced skills (like empathy, motivational interviewing, and trauma-informed communication) as well as clear-cut regulatory requirements (like HIPAA disclosures and informed consent), with a level of efficiency, transparency, and consistency that was previously unattainable.
The impact extends beyond operational efficiency. With 100% of calls scored on crisp, well-defined criteria, organizations gain unprecedented visibility into their clinical quality. They can identify top performers and understand what makes them excellent. They can spot clinicians who are struggling and provide targeted coaching. They can track trends over time and measure the impact of training interventions. And they can do all of this with confidence, knowing that the scores are not arbitrary or inconsistent, but are grounded in a rigorous, collaboratively refined definition of what excellence means in their context.
Conclusion
Building effective AI for healthcare requires more than just powerful algorithms and vast datasets. It demands a partnership, a shared commitment to a process of discovery and refinement. The human-in-the-loop is not a temporary measure but the engine of a continuous quality journey. By treating the AI and the SME as partners in an inter-rater reliability process, we do more than just build a better AI. We help organizations become better versions of themselves, with clearer standards, more aligned teams, and the ability to scale their unique vision of clinical excellence.
References
[1] Hartling, L., Hamm, M., Milne, A., Vandermeer, B., Santaguida, P. L., & Ansari, M. (2012). Validity and inter-rater reliability testing of quality assessment instruments. Agency for Healthcare Research and Quality (US).
[2] Chow, R., Bruera, E., Temel, J. S., Krishnan, M., Im, J., & Lock, M. (2020). Inter-rater reliability in performance status assessment among healthcare professionals: an updated systematic review and meta-analysis. Supportive Care in Cancer, 28(7), 3047–3057.
[3] Laparra, E., Mascio, A., Velupillai, S., & Miller, T. (2021). A review of recent work in transfer learning and domain adaptation for natural language processing of electronic health records. Yearbook of medical informatics, 30(01), 282-291.
[4] Lyell, D., Wang, Y., Coiera, E., & Magrabi, F. (2023). More than algorithms: an analysis of safety events involving ML-enabled medical devices reported to the FDA. Journal of the American Medical Informatics Association, 30(7), 1227-1234.