The Documentation Challenge: Why Static AI Is Not Enough
AI-powered clinical documentation can reduce documentation burden by more than sixty percent, freeing care managers to focus on what they do best: caring for people. But generating a note is only the beginning. The harder question is whether that note reflects the precise standards, preferences, and clinical judgment of the organization it serves.
Every organization brings its own documentation culture: how a clinical narrative should flow, what level of detail is appropriate, when to include contextual information from prior calls, and when brevity is more valuable than comprehensiveness. These expectations are rarely written down in full. They live in the collective expertise of clinicians, social workers, behavioral health specialists, and other specialized members of care teams, expressed in the small, deliberate edits they make every day to the notes that AI produces.
The initial calibration of an AI documentation system captures a snapshot of an organization’s standards at a single moment in time. But clinical programs evolve. New regulatory requirements emerge, team composition shifts, and supervisors refine their definitions of quality through months of practice. A system that was well-aligned at launch can gradually drift out of step with the organization’s living standards.
This paper describes a methodology for addressing that challenge: a continuous, automated system that monitors documentation quality, learns from user edits, proposes refined AI instructions, rigorously evaluates those proposals, and presents them for human review before any change reaches production. It is, in essence, a system that learns how to write better clinical notes by studying how clinicians improve the notes it has already written.
A system that learns how to write better clinical notes by studying how clinicians improve the notes it has already written.
Why Individual-Level Learning Fails in Regulated Healthcare
Some AI documentation products offer a seemingly intuitive approach: let each user “teach” the model what they like. A clinician edits a note, the system remembers, and next time it writes more like that individual prefers. In consumer or low-stakes settings, this can work. In the highly regulated world of care management within payer organizations, it is fundamentally the wrong approach.
The reason is organizational, not technical. Clinical documentation is governed by regulatory requirements, accreditation standards, and the clinical judgment of quality leadership. When a care manager edits an AI-generated note, that edit may reflect a correction aligned with organizational standards—or a personal preference, a time-saving shortcut that loses clinical detail, or even a misunderstanding of the requirements. Individual edits are noisy. They contain signal, but they also contain variance and occasionally errors.
This is the challenge that clinical leaders already face without AI. Achieving standardization across a team of dozens or hundreds of care managers is one of the hardest problems in care management operations. Supervisors invest significant effort in calibration, coaching, and audit reviews to align their teams around a shared definition of excellent documentation. Allowing each clinician to independently “train” an AI to match their own style would undermine that effort entirely—producing as many documentation standards as there are clinicians.
Our approach is deliberately different. The system learns at the team and organizational level, not the individual level. It aggregates edit patterns across all clinicians within a defined unit, filters for consistency, and surfaces insights to clinical leaders—the supervisors, quality managers, and program directors responsible for documentation standards. The question is not “What does this clinician prefer?” but “What does this organization’s collective behavior tell us about its standards, and where is the AI falling short?”
The question the system asks is not “What does this clinician prefer?” but rather “What does this organization’s collective behavior tell us about its standards, and where is the AI falling short of them?”
This distinction has profound implications for the taxonomy of gaps the system identifies—a taxonomy that mirrors the framework we described in our work on evaluation alignment with Insight 2.0.
Continuous Monitoring: Making AI Quality Visible
The foundation of any improvement process is measurement. Traditional quality assurance for clinical documentation is episodic: a supervisor reviews a sample of notes and provides feedback. Our approach replaces this with continuous, automated monitoring of every AI-generated note across every team and organizational unit.
Each day, the system calculates two core metrics for each configuration in production:
- Edit distance — a normalized measure of how much a clinician changed the AI-generated note before finalizing it. Zero means no changes; higher values mean more rewriting.
- Untouched rate — the percentage of notes that clinicians accepted without any modification.
Together, these metrics provide a clear, quantitative signal of how well the AI’s output matches the organization’s expectations. They are compared against configurable quality thresholds defined collaboratively with the organization — for example, average edit distance below six percent and at least forty percent of notes accepted without edits.
Based on these thresholds, the system classifies each unit into one of several states: performing well, still collecting data, or needing optimization — which triggers the next phase of the process. This classification is updated daily and surfaced through automated dashboards and notifications, creating a living, organization-wide view of AI documentation quality that ensures no drift goes undetected.
Learning from Clinician Edits: The Intelligence in Human Corrections
When the monitoring system identifies a configuration that needs optimization, it initiates a process that treats clinician edits not as corrections to be applied, but as a curriculum to be studied.
The system retrieves a representative sample of recent conversations, including three artifacts for each: the original transcript, the AI-generated note, and the final version as edited by the clinician. It performs a section-by-section comparison, analyzing each component independently — clinical narrative, member goals, self-management plan, follow-up agreements, and preparation notes.
For each section, the system performs sentence-level alignment, matching each AI-generated sentence to its closest human-edited counterpart. This reveals which sentences were kept intact, which were modified, which were deleted, and which new sentences the clinician added. The sample is split into a training set and a held-out test set — a separation essential for distinguishing genuine improvement from overfitting.
From the training set, the system generates a comprehensive edit analysis report, organized by section and including both edited and unedited examples. Unedited examples are particularly important: they represent notes the clinician deemed satisfactory, defining the target for length, style, and level of detail. Edited examples reveal systematic gaps — content that clinicians consistently add signals what the AI should include; content they consistently remove signals what it should avoid. The report also calculates word count and sentence count baselines for each section, ensuring the AI’s output matches the organization’s expectations for density and length.
Clinician edits are not corrections to be applied, but a curriculum to be studied.
The Taxonomy of Gaps: Consistent Patterns and Human Variation
Not all discrepancies between AI output and human edits carry the same meaning. A critical innovation in our methodology is the classification of gaps into two fundamentally different categories:
- Consistent gaps emerge when the AI produces output that clinicians across the team systematically correct in the same direction. For example, the AI might consistently generate self-management plans in the third person (“The member will contact their PCP”), while clinicians consistently rewrite these in a collaborative tone (“Agreed to contact PCP by end of week”). Or the AI might restate the call purpose in the clinical narrative, while most clinicians delete it because the disposition section already captures that information. These are clear, learnable patterns. The team has spoken with one voice, and the AI should listen.
- Differential gaps emerge when different clinicians edit the same type of AI output in different ways. One care manager might expand the follow-up section with detailed context from prior conversations; another trims it to a single sentence. One adds emotional tone (“Member expressed frustration about...”); another removes subjective language in favor of behavioral observations (“Member stated that...”). These are not errors in the AI — they are variations in human judgment, reflecting different interpretations of the same organizational standard.
Differential gaps are more complex and more valuable. When the system surfaces them, it holds up a mirror to the organization’s practices and asks: Which of these approaches do you actually want? This is a question many organizations have never had the data to ask at this level of specificity.
The system presents both types of gaps to clinical leaders with full context: anonymized examples showing the AI’s output alongside the divergent corrections. Leaders can then adopt the majority pattern, choose a specific approach as the new standard, or refine their guidelines to reduce ambiguity. The AI’s instructions are updated to reflect leadership’s explicit decision — not an algorithm’s inference from noisy individual edits.
Which of these approaches do you actually want? This is a question for clinical leadership, not for the AI team.
This is the fundamental difference between a system that passively absorbs individual preferences and one that actively facilitates organizational alignment. The former optimizes for individual satisfaction at the cost of standardization. The latter optimizes for institutional excellence while giving leadership the transparency and control they need to govern their documentation standards with confidence.
Automated Prompt Refinement: AI That Learns to Write Better Instructions
The edit analysis report becomes the input for the most novel phase of the process: automated generation of improved AI instructions.
A separate, highly capable AI model — distinct from the production note-generation system — receives the current instructions alongside the full edit analysis report. Its task is to act as an expert in clinical documentation optimization: analyzing edit patterns, identifying systematic discrepancies between current instructions and the organization’s demonstrated preferences, and producing revised instructions designed to close those gaps.
This meta-level AI prioritizes patterns that appear across multiple examples rather than idiosyncratic corrections from a single reviewer. It treats unedited examples as the primary model for length and style, and interprets deletions as signals of content to avoid. It generates revised instructions for each section along with anonymized examples of the desired output, synthesized from training set patterns to reduce ambiguity.
Crucially, the output is a draft — a proposed revision, not a deployed change. The system treats prompt refinement with the same rigor a pharmaceutical company applies to formulation changes: the proposal must be tested before it is considered approved and updated in the product that reaches patients.
Rigorous Evaluation: Testing Before Deployment
It is important to understand what this optimization process is—and is not—designed to address. Clinical AI documentation systems must be factually accurate; they must not fabricate information. But factual accuracy is a baseline requirement, not the frontier of the problem. Modern large language models, when properly constrained and grounded in a transcript, rarely hallucinate in structured documentation tasks. Accuracy is maintained as a safety guardrail, but it is not where the meaningful work of alignment happens.
A clinical summary can be entirely accurate and still be a poor summary. It might be too verbose, burying critical information. Too terse, missing contextual richness. It might follow a structure that differs from the organization’s preferred flow, or use language that feels detached when the organization values warmth. It might include correct but irrelevant information, or omit what the protocol requires.
This is the domain of standards, delight, and efficiency. The question is not whether the AI got it right or wrong, but whether the note aligns with the organization’s protocols, meets the team’s stylistic expectations, and saves the clinician time. A note that a clinician accepts without changes is not just an accurate note—it is a note that felt right: the right level of detail, the right tone, the right structure, the right emphasis. That feeling of rightness is what we are optimizing for.
A note that a clinician accepts without changes is not just an accurate note — it is a note that felt right: the right level of detail, the right tone, the right structure, the right emphasis.
The proposed instructions are evaluated using the held-out test set. For each conversation, the AI generates a new note using the proposed instructions, compared against both the original AI output and the human-edited version (gold standard). Word-level analysis measures whether the new output is closer to the human-edited version; section-level scoring evaluates each component independently.
Beyond quantitative metrics, the evaluation includes an AI-as-judge assessment — a separate, expert-level AI model that scores each test output on four dimensions:
- Completeness — Does the output capture all information that clinicians deemed important enough to include? This is not about capturing every transcript detail, but matching the organization’s judgment of what constitutes relevant, protocol-aligned documentation.
- Accuracy — The essential safety guardrail. Does the output remain factually grounded in the transcript? While not the primary optimization target, it ensures that improvements in style never come at the cost of fabricated content.
- Structure — Does the note follow the organization’s preferred documentation flow? The expected sequence of topics, logical progression from disposition through narrative to follow-up, and appropriate allocation of detail across sections.
- Style quality — Often the most consequential dimension for clinician satisfaction. Does the output match the professional tone, formality, person-centered language, and conventions the team has established through its collective editing behavior? A note that reads like a generic medical transcript has not achieved alignment, even if it is complete and accurate.
Each dimension is scored on a five-point scale with behavioral anchors and written reasoning. Results are aggregated into an evaluation report with per-sample breakdowns, dimension averages, and a side-by-side comparison viewer for human inspection.
Human Review: The Final Gate
The evaluation report and the side-by-side sample viewer are the deliverables of the automated process. They are not a deployment decision. They are a recommendation, presented to the clinical AI team for review.
This is deliberate. The system automates the labor-intensive work of analyzing edit patterns, generating hypotheses, testing them rigorously, and presenting the evidence clearly. But the decision to deploy a change to a system that generates clinical documentation — documentation that informs care decisions and enters the medical record — remains with human experts.
The review is designed for efficiency. Key metrics (judge scores, dimension averages, word count alignment) are surfaced at the top level for quick assessment. The sample viewer provides deep inspection when needed, with full transparency into how the proposed instructions would change the AI’s output. A diff view of the instructions themselves highlights precisely what changed and why.
If favorable, the revised instructions are promoted to production and the monitoring cycle begins anew. If concerns arise, the AI team adjusts and triggers another optimization cycle.
The Virtuous Cycle: From Monitoring to Mastery
The power of this methodology lies not in any single component, but in the closed loop that connects monitoring, learning, proposal, evaluation, and deployment. Each cycle produces measurable improvement: edit distance decreases, untouched rate increases, and the AI moves closer to the organization’s evolving standards.
When we plot these metrics as a time series from the moment an organization goes live, a characteristic pattern emerges. Edit distance starts at a baseline; the untouched rate reflects how well the initial calibration matched actual practices. As the first optimization cycle completes, we see a step-change: edit distance drops, untouched rate climbs. The system stabilizes, collects more data, and the next cycle produces another step. The result is a staircase pattern — each release producing a measurable upward shift in acceptance, with stable plateaus between releases as the system gathers fresh evidence.
This time-series view serves multiple audiences. For the AI team, an objective record of whether each cycle delivered improvement. For clinical leaders, visual proof that the system is learning. For executives, a compelling story: a plot that trends consistently up and to the right, driven not by abstract model retraining but by a transparent process anchored in the organization’s own clinicians.
The compounding effect is significant. Notes that require fewer corrections mean less editing time and more direct patient care. More examples of accepted output mean more precise understanding of preferences. Smaller detected gaps mean more targeted optimization cycles — refining nuances rather than addressing broad misalignments.
This also addresses the trust gap in healthcare AI deployment. By making the entire process transparent — what the system learned, how it proposes to change, how those changes performed — and providing full human control over deployment, organizations get the evidence they need to trust that the system is improving in the right direction [1] [2].
A plot that trends consistently up and to the right — an AI system that gets better at its job with every passing month, through a transparent process anchored in the expertise of the organization’s own clinicians.
Implications for the Field
The approach described in this paper has broader implications for how the healthcare industry thinks about AI alignment and quality.
First, it demonstrates that clinician behavior is a rich source of alignment data that does not require any additional labeling effort or workflow disruption. Every edit a clinician makes to an AI-generated note is an implicit annotation—a signal about what the organization values, what it considers sufficient, and what it considers missing or excessive. Treating this behavioral data as a first-class input to the AI improvement process transforms routine documentation work into continuous alignment feedback.
Second, it shows that automated improvement does not mean unsupervised improvement. The system automates the analysis, hypothesis generation, testing, and reporting—but it preserves human authority over deployment decisions. This is not a limitation; it is a feature. In a domain where errors can have clinical consequences, the combination of automated rigor and human judgment produces better outcomes than either alone.
Third, it challenges the assumption that AI systems must be periodically “retrained” by a central engineering team to stay current. Instead, this methodology enables continuous, organization-specific evolution of AI behavior, driven by the daily work of the clinicians who use it. The result is not a one-size-fits-all model that degrades over time, but a family of configurations that grow more aligned with each organization’s standards with every passing week.
Finally, it provides a concrete framework for the kind of post-deployment monitoring and improvement that regulators and clinical leaders increasingly recognize as essential. As the healthcare AI ecosystem matures, the ability to demonstrate continuous quality improvement—backed by quantitative evidence and rigorous evaluation methodology—will become a differentiating capability for organizations that deploy AI in clinical workflows [3] [4].
Conclusion
Clinical documentation is not a solved problem the moment an AI system generates its first note. It is a continuous journey of alignment—between the AI’s capabilities and the organization’s evolving standards, between automation and human expertise, between efficiency and clinical rigor. The methodology described here transforms that journey from an episodic, manual process into a systematic, evidence-driven cycle of improvement.
By monitoring quality continuously, learning from clinician edits at scale, proposing improvements through rigorous automated analysis, evaluating those proposals against held-out clinical data, and preserving human authority over deployment, we create a system that does not merely produce clinical notes—it gets better at producing them, every day, for every organization it serves. The result is not just better AI. It is a partnership between human clinical expertise and machine intelligence that grows stronger with use, producing documentation that clinicians trust, organizations can rely on, and patients ultimately benefit from.
References
[1] Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., ... & Dennison, D. (2015). Hidden technical debt in machine learning systems. Advances in Neural Information Processing Systems, 28.
[2] Wu, E., Wu, K., Daneshjou, R., Ouyang, D., Ho, D. E., & Zou, J. (2021). How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals. Nature Medicine, 27(4), 582-584.
[3] Feng, J., Phillips, R. V., Malenica, I., Bishara, A., Hubbard, A. E., Celi, L. A., & Pirracchio, R. (2022). Clinical artificial intelligence quality improvement: towards continual monitoring and updating of AI algorithms in healthcare. npj Digital Medicine, 5(1), 66.
[4] Vokinger, K. N., Feuerriegel, S., & Kesselheim, A. S. (2021). Continual learning in medical devices: FDA’s action plan and beyond. The Lancet Digital Health, 3(6), e337-e338.