How Laguna Health Cut Clinical AI Inference Costs by 85% with NVIDIA Nemotron

The Problem with Healthcare AI at Scale

Care management is a labor-intensive, conversation-driven discipline. Nurses and care managers at large health plans spend most of their working hours on the phone — checking in on members with chronic conditions, coordinating specialist referrals, reconciling medications after hospital stays. At a large payer, thousands of care managers handle hundreds of thousands of calls every year.

Every one of those calls generates work that extends well beyond the call itself. There is documentation — a clinical record of what was discussed, what was agreed to, and what happens next, which drives care continuity and feeds compliance auditing. There is quality monitoring — supervisors need to understand whether care managers are communicating effectively, whether members are engaging, whether safety flags are surfacing across the team. And there is the operational intelligence question: across thousands of calls and hundreds of care managers, what is actually happening, and where does it need to change? Most of this work is still done manually, inconsistently, or not at all.

Laguna Health builds AI systems for this problem. Our products — Companion, which assists care managers before, during, and after calls; LISA, a voice agent that handles outbound scheduling; and Insight, a supervisor analytics platform — run across payer organizations at scale. AI is not a feature in our stack. It is the stack.

That means the model decisions we make are not just engineering decisions. Latency affects care manager throughput. Cost per call affects whether an AI-assisted program can survive on a health plan's budget. And the accuracy bar for clinical documentation — what a care manager signs off on and enters into the system — is one that no amount of throughput optimization can substitute for.

This is the context in which we started working with NVIDIA on Nemotron.

How This Started: A Suggestion From NVIDIA

For some time, we had been running our most demanding AI tasks — particularly post-call clinical documentation — on frontier models via API. The quality was there. The cost and latency were not ideal, especially at the volumes we operate. We had been exploring small language model distillation as a path to purpose-built, efficient models for specific tasks, using other open-source models as candidates.

When we began collaborating with NVIDIA through the NVIDIA Inception program for startups and applied AI teams, they suggested Nemotron as a distillation candidate — specifically their Nemotron 3 Nano 30B model, deployed via Amazon Bedrock. Nemotron 3 is an open model family utilizing a hybrid Mamba-Transformer MoE architecture for fast, long-context reasoning in multi-agent systems. Nemotron's architecture, open datasets, and training methodologies made it well-suited as a base for knowledge distillation for complex, domain-specific tasks.

Before investing in distillation work, we needed to know where we stood. So we ran a rigorous out-of-box evaluation: the same task, the same production dataset, the same prompt — just a different model.

Deploying on Amazon Bedrock: Infrastructure for a Regulated Environment

For healthcare AI companies, infrastructure is not a neutral choice. PHI boundaries, audit requirements, BAAs, and customer-specific data residency constraints mean that where a model runs — and how data flows to and from it — is a contractual and regulatory question, not just an engineering one. Self-hosting models is one path, but it introduces operational overhead and complicates the auditability story for enterprise health plan customers.

Amazon Bedrock gave us a managed inference path that fit within our existing compliance architecture. Nemotron 3 Nano is available on Bedrock in our existing region, which keeps inference within our established AWS environment. Data does not leave the environment for model inference, the API fits our existing service layer, and the deployment requires no new infrastructure.

For both evaluations and for the Laguna Lens production deployment described below, all inference ran through Bedrock's Converse API with no external model calls or data egress outside the controlled environment.

Use Case 1: Establishing a Distillation Baseline for Clinical Documentation

Our post-call summary is the most demanding of our AI tasks. It takes a full care management call — transcript, member context, clinical history — and produces a structured clinical record with multiple sections: the disposition of the call, a narrative summary, the member's care goals and progress, agreed self-management actions, clinician follow-up commitments, and outstanding items. The output is reviewed and signed off by the care manager before entering the care management system.

The quality requirements are strict. The structure must be complete and consistent on every call. Factual errors — a misrepresented medication, an invented appointment, a wrong follow-up attribution — have direct care continuity implications. And the output needs to be comprehensive: care management documentation is a clinical record, not a verbal summary.

We ran Nemotron 3 Nano on 25 recent production calls using the live production prompt, with no modification — measuring structural completeness, semantic alignment with our commercial LLM baseline, latency, and sentence-level accuracy through an LLM-as-a-judge evaluation with manual review of every flagged sentence.

The headline results were stronger than we expected. Nemotron 3 Nano produced complete, well-formed outputs on every call. Semantic alignment with our production reference averaged above 0.91 across all sections. Most significantly: zero hallucinations across 343 evaluated sentences. In a clinical documentation context, where fabricated content carries real downstream risk, that number matters more than aggregate error rate.

Latency was roughly half that of our frontier model baseline, with meaningfully tighter variance across calls. Cost reduction was approximately 85% versus a frontier model reference.

The gap relative to production was real, but specific and quantifiable. Sentence-level judging surfaced two concentrated failure modes. In one section, 37% of sentences showed overinterpretation — stating a member's uncertain or aspirational language as a settled fact or commitment. In another section, 32% of sentences contained contact-identity errors, most often documenting that the member was reached directly when a family member, caregiver, or facility representative had actually answered. The remaining five sections were substantially cleaner, with problem rates between 7% and 18%. Outputs were also about a third shorter than the production baseline, driven mainly by the summary section.

These are not random failures. They are learnable patterns — exactly the kind of targeted behavior that knowledge distillation is designed to address. We now have a quantitative baseline at sentence level, with severity tagged, across real production calls. That is the starting point for the fine-tuning work ahead.

Use Case 2: What Happens When the Baseline Is Better Than You Expected

The post-call summary evaluation was designed to answer one question: is Nemotron a viable distillation candidate for our most demanding task? The answer was yes — and it opened a second question we had not originally planned to ask.

If Nemotron performs this well out of the box on a task this complex, what could we get from it with prompt engineering alone on a task with a lower precision ceiling?

The candidate was Laguna Insight — our supervisor analytics platform, named one of TIME's 200 Best Inventions of 2024. Insight gives health plan leaders a real-time view of quality, productivity, and wellbeing trends across their care management organization. At its core is Laguna Lens: AI-extracted conversational signals that run automatically on every call transcript — communication quality indicators like reflective listening and conversational alliance, safety flags, and care manager wellbeing signals including empathic engagement and moral distress.

To make this concrete: on a typical call, a care manager checks in with a member managing congestive heart failure and type 2 diabetes. The member mentions a recent ER visit, some difficulty affording a new medication, and uncertainty about a specialist referral. The care manager listens, asks follow-up questions, and commits to following up with the specialist's office before the next call. Laguna Lens processes the transcript of that conversation and extracts, among other signals, whether the care manager demonstrated reflective listening during the medication discussion, whether any safety or financial barrier flags were present, whether the member showed signs of disengagement, and a composite wellbeing score for the care manager based on the emotional tenor of the interaction. None of these are binary checkboxes — they are graduated, reasoned extractions from natural language. Getting them right requires a model that understands clinical context, not just keyword patterns.

We had previously run Laguna Lens on a commercial LLM. As with the post-call summary work, we were exploring open-source alternatives — Nemotron among them — to reduce cost and dependency on closed models. Getting consistent, well-calibrated output required iteration on every model we tried, commercial or open: working through the tendency to overstate positive signals, resolving ambiguity in edge cases, and establishing reliable output structure across the diversity of calls we process. We went through the same process with Nemotron: multiple prompt engineering iterations, refining the framing and evaluation criteria until its outputs were consistent and clinically grounded.

Nemotron 3 Nano is now running in production on Laguna Lens across customer environments.

The case for deployment was straightforward: OOB quality that met the bar for this task, latency and cost that worked at scale, and an infrastructure path through Bedrock that required no new architecture. The prompt engineering route meant we could move without a training cycle.

What These Two Stories Add Up To

Taken together, these two use cases point to something more useful than either one alone: a way to think about SLM strategy in a regulated environment without having to guess.

Neither conclusion was obvious going in. The suggestion from NVIDIA to evaluate first — before committing to a distillation path — is what surfaced both.

Evaluations conducted March 2026 on Amazon Bedrock in our current region. Post-call summary evaluation: 25 production calls, real-world care management dataset, production prompt, commercial LLM baseline. Laguna Lens: Nemotron 3 Nano 30B, prompt-engineered, running in production across customer environments.

‍