Blog / How to Measure Automated Medical Coding Accuracy: Precision, Recall, Routing
How to Measure Automated Medical Coding Accuracy: Precision, Recall, Routing
Code-level precision and recall, notes fully right, micro and macro averages, a fair test set, and why a coder that completes only the notes it is sure of can be held to a higher standard.


The short version
- A clinical note usually needs several codes, so "accuracy" has to be measured at more than one level.
- Code-level precision: of the codes the system assigned, how many were correct. Code-level recall: of the correct codes, how many it found.
- Notes fully right: the share of notes where every code matched the audited answer. It is the strictest score, and the one a coder feels.
- A fair test uses unseen, double-audited notes from several specialties, on the code set the regulator requires for your claims.
- Confidence routing: the system completes only the notes it is sure of and sends the rest to a human coder. Read the accuracy on those notes next to the share it completes.
To measure automated medical coding accuracy, score it at three levels: code-level precision (how many assigned codes are correct), code-level recall (how many correct codes were found) and notes fully right (every code on the note correct). Then check coverage: which notes the system completes on its own, and which go to a human coder.
What does "accuracy" mean for an automated medical coder?
How closely the system's codes match those a careful human coder would assign to the same note. One note rarely has one code.
A home-care visit note may carry a principal diagnosis, several secondary conditions, a history code and a status code. An emergency note may need an injury code, an external cause code and a place-of-occurrence code. In machine-learning terms this is a multi-label task: every note has a set of correct codes, and the system proposes a set of its own.
Comparing two sets can go wrong in three different ways. The system can add a code the note does not support, leave out a code it does support, or pick a neighbouring code at the wrong level of detail. A single "accuracy" figure mixes these together, so researchers report several measures side by side. A recent study, Code Like Humans (Motzfeldt, Edin et al., EMNLP Findings 2025), reports micro and macro F1, the exact match ratio and recall for each part of its system.
How do you calculate code-level precision and recall?
Count codes. Precision divides correct codes by all codes the system assigned. Recall divides correct codes by all codes the note should have.
Google's machine learning course (Google for Developers) gives precision as TP / (TP + FP) and recall as TP / (TP + FN). For coding:
- True positive (TP): a code the system assigned that is also in the audited answer.
- False positive (FP): a code the system assigned that is not in the audited answer, for example a diagnosis the note does not support, or, in outpatient and home-care coding, an uncertain "rule-out" diagnosis, which the ICD-10-CM outpatient guidelines say not to code.
- False negative (FN): a code in the audited answer that the system did not assign, for example a missed secondary condition.
A worked example makes it concrete. Suppose a note truly needs four codes: A, B, C and D. The system proposes A, B, C and X. Three codes match. X is wrong, and D is missing. Precision is 3 of 4 proposed codes. Recall is 3 of 4 true codes. And the note is not fully right, because one code must still be removed and one added.

Precision and recall pull against each other. A system that proposes many codes finds more of the true ones, so recall rises, but it also adds unsupported codes, so precision falls. Reporting both stops one number being improved by quietly giving up the other.
They also match how auditors think. In the Department of Health Abu Dhabi JAWDA home-care audit logic, an unsupported diagnosis (a precision error) costs a claim more points than a missing secondary diagnosis (a recall error). Both cost points.
What is "notes fully right", and why is it so much harder?
A note is fully right only when the system's set of codes equals the audited set exactly. One extra code, one missing code or one wrong character, and the whole note fails.
Researchers call this the exact match ratio. It is the measure closest to a coder's working day: a fully right note can be accepted as it is; every other note needs correction. Errors compound, so a note with eight codes has eight chances to fail.
The research shows how hard it is on real hospital charts. MDACE (Cheng et al., ACL 2023) is a public set of inpatient and professional-fee charts annotated with the evidence for each code; its inpatient part has 302 charts with 3,934 evidence spans. When the Code Like Humans study scored systems on MDACE, almost no chart had every code exactly right, for any system tested (Motzfeldt, Edin et al., Table 1, 2025).
General-purpose language models used on their own do not close the gap. A Mount Sinai team tested them on more than 27,000 unique diagnosis and procedure codes from 12 months of routine care. The study, "Large Language Models Are Poor Medical Coders" (Soroush et al., NEJM AI, April 2024), found that "an unacceptably large number of errors remained" (Mount Sinai press release, 2024).
Many misses are near misses. A verification study found that codes at the wrong level of the right family explain a substantial portion of language-model failures, and added a verification step to catch them (Yuan et al., EMNLP 2025 Industry Track). A preprint tested on single medical terms, not full notes, reported that retrieving candidate codes first and then ranking them helps (Kwan, arXiv preprint, 2024). The lesson for measurement is simple: a code that is "nearly right" still counts as one wrong code and one missed code, so it lowers precision and recall and fails the note.
| Measure | The question it answers | How it is counted | What it can hide |
|---|---|---|---|
| Code-level precision | Are the codes it assigns correct? | Correct codes / all codes assigned | Missed codes: a system can stay precise by assigning very few |
| Code-level recall | Does it find the codes the note needs? | Correct codes / all codes in the audited answer | Extra codes: a system can find more by assigning too many |
| Notes fully right (exact match) | How many notes can be accepted untouched? | Notes with an exactly matching code set / all notes | How close the other notes were; one wrong character fails a note |
| Coverage (with routing) | How many notes does it complete on its own? | Notes completed / all notes | Accuracy, unless it is reported next to the score on those notes |
| Micro average | How good is it on the codes you see most? | Pool every code from every note, then divide | Weakness on rare codes |
| Macro average | Is it good on rare codes too? | Score each code separately, then average | How often each code really occurs |
Micro or macro: which average should you trust?
Ask for both. A micro average pools every code from every note, so common codes dominate. A macro average scores each code separately, so a rare code counts as much as a common one.
In a replication study on the MIMIC-III and MIMIC-IV hospital datasets, Edin et al. (SIGIR 2023) found that the macro F1 score in earlier work had been calculated sub-optimally, and that correcting it doubled the figure. It also found models underperforming from weak configurations and poorly sampled train-test splits, and all models struggling with rare codes.
So how a number is counted, and on what data, can change it as much as the model. Ask: micro or macro, and on what notes?
How should a fair test set be built?
One the system has never seen, audited carefully, broad enough for the notes you code, and scored on the code list the regulator (DoH or DHA) requires for your claims.
- Never used in training. Otherwise the score measures memory, not coding.
- Audited, ideally twice. The answer is only as good as its authors; the verification study above released outpatient notes double-annotated by experts (Yuan et al., 2025).
- Spread across specialties. Obstetrics, newborns, injuries, oncology and psychiatry each have their own coding rules; one specialty's score says little about another.
- The right code set version. A code that is valid in one ICD-10-CM release may not exist in another. In the UAE, Abu Dhabi uses the ICD-10-CM FY2021 January update. Dubai uses the 2021 set until date of service 31 December 2026 and ICD-10-CM 2024 from 1 January 2027 (DHA circular GC 06-2026). A test should say which list it scored against, and a switch of list is a reason to measure again.
- Dated and described. Say when it ran, how many notes it used and how they were chosen.
What is confidence routing?
Confidence routing means the system judges, note by note, whether it is sure of its codes. It completes the notes it is sure of and sends all other notes to a human coder.
In machine learning it is called selective classification, or a "reject option": a classifier may decline to answer when it is unsure (Geifman and El-Yaniv, NeurIPS 2017). The user sets a target error rate, and the classifier rejects as many inputs as it must to meet it. The trade-off has two names: coverage, the share of inputs the system answers, and risk, the error rate on those answers.
The same approach has been studied for medical text. Vazhentsev et al. (2025) describe a diagnosis system that is allowed to abstain from a decision, based on modelling its predictive uncertainty, and include multi-label ICD-10 code prediction on MIMIC-IV among their tasks. A related line of work, conformal risk control, gives a way to bound the expected value of a chosen loss, for example the false negative rate, which in coding means missed codes (Angelopoulos et al., ICLR 2024).

Book a demo of the medical coder WhatsApp us
Why does confidence routing raise accuracy on the notes a system completes?
Because the hard notes are not forced through. A system that may hand a note to a person no longer has to guess on unclear documentation or rare codes, and the notes it keeps are the ones it handles well.
So a system with routing is judged on two numbers together: the accuracy on the notes it completes, and the share of notes it completes. Either alone can mislead. High accuracy on a tiny share saves little work; a large share at lower accuracy pushes errors onto the auditor.
Routing does not hide errors as long as it is reported honestly: the test set still contains every note, and the notes sent to a human still count. When you read an accuracy claim, check that the share of notes completed is stated next to it, and that the other notes go to a person rather than being silently dropped.
What do Xrero's numbers look like on this method?
We measured Xrero's coder on these levels. This page shows only results above 80%.
About these numbers. Measured on 21 November 2026 on 320 practice notes across 8 specialty groups, written and double-audited by our own team and never used in training. This is our internal test, not an independent certification.
| Which notes | Measure | Result |
|---|---|---|
| All 320 notes | Code-level precision: codes assigned that were correct | 93.1% |
| All 320 notes | Code-level recall: correct codes that were found | 90.6% |
| Notes the coder was sure of (most of the notes in that test) | Code-level precision | 97.0% |
| Notes the coder was sure of | Notes with every code right | 86.2% |
| Psychiatry, substance use and neurology notes | Notes fully right | 87.5% |
In that test the coder completed on its own only the notes it was sure of; all other notes went to a human coder. In the product, a human coder reviews and approves every case: the coder suggests, the human decides. Coders see only their own cases, and every final decision is recorded in an audit trail.
The scores are against the ICD-10-CM FY2021 January release, 72,621 codes: the same list the Abu Dhabi Department of Health publishes for routine reporting, with 0 differences when we compared them. The coder uses this FY2021 list today, not the ICD-10-CM 2024 list that Dubai requires for dates of service from 1 January 2027.
How it works. Rules read the note, including sections, negation, and history and family-history handling, and propose candidate codes from the ICD-10-CM index and tabular list. A language model that runs on our own servers picks the final codes from that list. A confidence check then decides which notes it may complete and which go to the coder. Patient notes are processed on a server in Dubai, UAE; no outside AI service receives patient text.
The coder covers home care first, and now all specialty groups. See the medical coding software page for the details.
What should you ask any coding vendor about accuracy?
Ask for the method, not just the number. These questions fit any system, ours included:
- Is the figure code-level precision, code-level recall, or notes fully right? Micro or macro?
- How many notes were tested, from which specialties, and were any of them used in training?
- Who wrote the audited answers, and were they checked by a second coder?
- Which ICD-10-CM release was the test scored against, and is it the one the regulator requires for your claims?
- Does the system complete some notes on its own? If so, what share, and what is the accuracy on those notes?
- Where do the other notes go, and who approves the final codes?
- Is it an internal test or an independent one, and where is patient text processed?
To see the coder on your kind of notes, call +971 54 468 2468 or WhatsApp +971 55 776 9320.
Book a demo of the medical coder WhatsApp us
Frequently asked questions
What is the difference between precision and recall in medical coding?
Precision is the share of the codes the system assigned that are correct. Recall is the share of the correct codes that the system found. A coder that adds extra codes loses precision; a coder that leaves codes out loses recall. You need both, because an auditor penalises unsupported codes and missing codes.
What does 'notes fully right' mean?
A note counts as fully right only when every code on it matches the audited answer: nothing wrong, nothing missing. Researchers call this the exact match ratio. It is the strictest score and the closest to what a coder experiences, because one wrong code means the note must be corrected.
Why is one overall accuracy number not enough?
Each note carries several codes, so one number can hide whether errors are extra codes, missing codes or whole notes that need rework. Ask for code-level precision, code-level recall and notes fully right, the size and make-up of the test set, the code set version, and which notes the system completes on its own.
What is confidence routing in automated coding?
Confidence routing means the system checks how sure it is about each note. It completes on its own only the notes it is sure of and sends all other notes to a human coder. Research calls this selective prediction, abstention or a reject option. Always report the share of notes completed next to the accuracy on those notes.
What accuracy has Xrero's medical coder reached?
On all 320 notes, 93.1% of assigned codes were correct and 90.6% of correct codes were found. On the notes it was sure of (most notes; the rest went to a human coder), 97.0% of codes were correct and 86.2% had every code right. Measured 21 November 2026 on 320 practice notes, 8 specialty groups, never used in training; internal test, not an independent certification.
Does an automated coder replace the human coder?
No. In Xrero's coder a human coder reviews and approves every case: the coder suggests, the human decides. Coders see only their own cases, and every final decision is recorded in an audit trail.
About the publisher: Xrero (xrero.com; in Arabic اكسريرو) is a Dubai-based business software company. Xrero is not Xero and is not connected to it.
Page updated 21 November 2026.

