Experiment 17
Our Image Quality Score Was Measuring the Wrong Thing
We tested our own most impressive number and it did not survive. This is what we found, what we are retiring, and what we are doing instead.
Our image quality model sorts ultrasound images of the fetal heart into GREEN (good enough to diagnose from), YELLOW (needs adjustment) and RED (unusable). It reported 99.89% accuracy, and that was the strongest-looking number our research had produced.
It was not measuring image quality. It was recognising the artificial blurring and contrast changes we had applied ourselves when creating its training examples. We are withdrawing the figure from all use.
We wrote down this exact test, and committed to this exact consequence, before we ran it. That commitment is published on our research page and timestamped in our public code repository.
To teach a model what good and bad images look like, you need examples of each, labelled by someone who knows. We did not have access to a clinician, so we generated the bad examples ourselves. We took clear expert images, and:
- GREEN was the original image, untouched.
- YELLOW was the same image rotated slightly, blurred a little, and faded.
- RED was the same image rotated hard, blurred heavily, given artificial noise, and partly covered up.
A model scoring 99.89% on this might have learned to judge diagnostic quality. Or it might have learned something far easier: whether we had run a blur filter over the picture. Both explanations predict the same excellent score, and completely different behaviour in a clinic.
The decisive test needed no clinician at all. If the quality task can be solved just by measuring blur and brightness, then a deliberately simple method should match our sophisticated one.
So we measured nine basic properties of each image: how sharp it is, how noisy, how bright, how much contrast it has, and similar. None of these have any concept of a heart, a chamber, or a diagnosis. Then we asked a basic statistical method, far simpler than a neural network, to sort the images using only those nine numbers.
We also ran our quality model over 775 real ultrasound images that we had never artificially altered, and separately took real images and degraded them step by step to see whether the model's scores tracked how bad they were getting.
Results
| Method | Accuracy | Notes |
|---|---|---|
| Guessing the most common answer | 40.00% | Baseline |
| Logistic regression on 9 simple measurements | 99.05% | No understanding of anatomy |
| Random forest on the same 9 measurements | 99.52% | No understanding of anatomy |
| Our EfficientNetV2-S quality model | 97.62% | Previously reported |
A basic statistical method using nine numbers scored higher than our purpose-built model. That settles it. Sorting these images into GREEN, YELLOW and RED never required understanding an ultrasound. It only required noticing that we had blurred some of them.
Across 775 real ultrasound images, including freehand scans recorded by inexperienced operators that certainly contain unusable views, the model returned a RED rating zero times.
Meanwhile, on our own artificially damaged images, it identified RED with perfect confidence every single time. The warning that is supposed to tell an operator "do not save this image" has never fired on a real picture.
We took real images and blurred them progressively. The model noticed, moving them out of GREEN. But it never moved them to RED, not even at blur levels heavier than those in its own RED training examples.
| Blur applied | Rated GREEN | Rated RED |
|---|---|---|
| 1 (none) | 90.0% | 0.0% |
| 3 | 88.0% | 0.0% |
| 5 | 84.0% | 0.0% |
| 7 | 70.5% | 0.0% |
| 11 | 51.0% | 0.0% |
| 15 | 30.5% | 0.0% |
| 21 (severe) | 7.5% | 0.0% |
The reason is that our RED examples were made by applying five different kinds of damage all at once. The model learned that specific combination as a fingerprint. A real image that is simply blurry does not match the fingerprint, so the warning never triggers. Separately, reducing an image's contrast by only 15% was enough to push nearly every image out of GREEN, which tells us the decision is resting on overall brightness rather than on anatomy.
What is withdrawn
The 99.89% accuracy figure, and every claim resting on it. It may only be described from now on as accuracy at spotting our own artificial image edits, a task a much simpler method performs slightly better.
What still stands
This does not affect the part of our system that finds the heart in the image, which was measured separately and differently. Nor does it mean a quality guidance tool is a bad idea. The design is sound. What failed is how we created the training labels, and a model trained on real expert judgements could work perfectly well. This one was simply never tested against the thing it claims to measure.
What changes now
Getting expert clinicians to rate real images moves from important to essential. It is now the only route we have to a quality signal we can trust, and we cannot build our way around it. Generating fresh artificial labels with different settings would only reproduce this same failure. If you are a sonographer or physician willing to rate a set of anonymised research images, that is the single most useful thing anyone could offer us right now: info@theodoresheartinitiative.org.
Why we ran this at all
Nobody asked us to check. The number looked good, and we could have kept quoting it. We tested it because a figure that impressive, produced by labels we generated ourselves, deserved suspicion rather than celebration. Finding this now costs us a headline. Finding it after a clinical partner relied on it would have cost considerably more, and it would not have been ours to lose.