How We Validate Synthetic Patient Data Against Real US Statistics

§ · What "calibrated" means

A generator can produce values that are the right type — a number for age, a category for insurance — without producing the right distribution. Calibration is the step that fixes the distribution to a real one. For each field, we take a published reference — the share of adults who are obese, the split of household income brackets, the mean systolic blood pressure — and draw values so the generated marginal matches that target.

When a field is conditioned on others, the per-stratum rates are rescaled so the implied overall marginal still hits the published figure. Diabetes prevalence rises with age, for instance, but the age-weighted average across the whole population is held to the national rate. Calibration is explicit and auditable — every rate traces to a source — and it is deterministic: the same seed reproduces the same distribution every run. It is not a model learned from real records, so there is nothing to memorize or leak.

§ · The benchmark library

Validation is only as good as the numbers you validate against. The reference values live in a version-controlled benchmark file, each row carrying its source dataset, year, population scope, and a confidence grade. Seventy-plus figures were adversarially reconciled against primary federal tables. A few honest scope notes come with the territory: some federal figures are all-ages while the generator is adults 18+, and the Census classifies areas urban/rural only, with no "suburban" tier — those are flagged rather than scored as misses.

§ · The check: 20,000 records against the references

Plot each generated marginal against its federal benchmark and the picture is one line. A point on the dashed diagonal is a perfect match; the seventeen metrics — spanning 6% (Asian race) to 90% (has a primary-care provider) — all sit on it.

The full table, with the source behind each figure:

The numeric means line up just as closely:

§ · Beyond the average: distribution shape

Matching a mean is a low bar — two very different distributions can share one. The stronger test is whether the whole shape lines up. The Kolmogorov–Smirnov distance (KS-D) is the largest gap between the generated and reference cumulative distributions, from 0 (identical) to 1. We measure it against NHANES micro-data, survey-weighted, at population scale — one million records, where the synthetic-side sampling noise collapses and the number is the true distance.

Three of the four body measures sit in the "excellent" tier (KS-D below 0.03); weight (0.035) and the vitals and labs are "good." HbA1c and systolic blood pressure carry a small known residual — they run a touch high — but neither crosses a tier boundary, and both are documented rather than smoothed away. The joint structure holds at the same scale: height and weight correlate at r = 0.51, BMI and waist at r = 0.86, both on their clinical targets.

§ · The verdict, and where it drifts

Of the seventeen marginals, fourteen land within two percentage points of the federal figure and every one is within 3.2. Five of the six numeric means are within one unit of the reference, and the full distributions — not just their averages — match NHANES to a KS-D under 0.04 for every body measure. Body measures, blood pressure, HbA1c, obesity, smoking, and the sex and ethnicity splits are effectively exact.

Two gaps are worth naming rather than hiding. Any-prescription use runs 3.2 points high (68.0% vs 64.8%), and mean sleep is about half an hour short of the NHANES workday figure (7.0 h vs 7.6 h). Neither is a scope artifact; both are real targets the generator can tighten. A handful of other rows carry a scope note — the federal source is all-ages while the generator is adults 18+ — where a small offset is expected, not an error.

§ · What calibration does not do

Matching marginals is necessary, not sufficient. A dataset can hit every one of these targets and still be full of impossible people if the fields do not move together correctly — which is the separate job of the joint structure. And calibration only reproduces the relationships it is built to model; a specific real-world correlation nobody encoded will not appear. For a conclusion that hinges on one particular association, validate it against real data first — the line is drawn in what synthetic data is.

§ · Reproduce it

Every figure here is a few lines. Generate a sample, read it into pandas, and check any marginal against its published reference:

import pandas as pd

df = pd.read_json("people.jsonl", lines=True)   # 20,000 rows, seed 42

# categorical marginals (%)
df["diabetes_status"].isin(["diagnosed_t2dm", "type1"]).mean() * 100   # ~12.2
(df["bmi"] >= 30).mean() * 100                                        # ~42.5  (NHANES 42.4)
(df["smoking_status"] == "current").mean() * 100                       # ~11.4  (NHIS 11.6)

# numeric means
df[["bmi", "systolic_bp_avg", "a1c_value"]].mean().round(1)            # 29.5 / 122.0 / 5.8

Why joint distributions matter →What drives HbA1c →NHANES-calibrated data →Person Profile generator →