Checking models against the experiment and the data
Class 9 · Model diagnostics
Gavin Simpson
Aarhus University
Wednesday, 4 November 2026
Class 9 schedule
Time
Work
12:30–13:40
Recall, checking tools and shared count example
13:40–13:55
Break
13:55–14:55
Four group cases
14:55–15:10
Break
15:10–16:00
Change, recheck, project check and report
Opening recall: animals share pens
Last class, several animals shared each pen, and a diet was assigned to a whole pen. Discuss with your seated group:
What must the model say about animals sharing a pen?
What output would make you doubt that fitting worked?
What pattern would suggest that the model misses something?
Three questions for any fitted model
Question
Evidence to inspect
Does it respect the experiment?
Who received each diet; shared conditions
Did the calculation work reliably?
Warnings, extreme estimates or very large standard errors
Does it describe the data?
Patterns left unexplained or differences from simulated data
A standard error describes uncertainty in an estimate under the fitted model.
What the model leaves unexplained
For daily gain, a residual is observed gain minus fitted gain, in g/day.
In this simulated example, the residuals form a horizontal cloud around zero with a similar spread across fitted values. Look for patterns against starting weight and by pen as well.
Normal QQ plot for gain residuals
A normal QQ plot checks whether the residuals have the bell-shaped distribution the model assumes.
Each point pairs an ordered residual with the corresponding quantile of a standard normal distribution. The dashed line shows the pattern expected for normal residuals at their observed scale.
Points close to the line are consistent with normality. A bend or a systematic departure at either end suggests a different distribution shape.
Count checks compare data with simulations
Simulate counts.performance uses DHARMa to simulate many datasets from the fitted model, keeping each pen’s diet and room fixed.
Locate the observed count. For each pen, find the fractions of simulated counts below and at or below its observed count. The randomised quantile residual is a uniform draw between those fractions, or their common value if they agree.
Compare quantiles. The QQ plot shows theoretical quantiles of a uniform distribution on 0 to 1 on the x-axis and sample quantiles of the residuals on the y-axis. Points should fall roughly along the diagonal if the residuals follow that reference distribution.
One event count per pen
Researchers assigned diets to 64 pens in four rooms. Each room had eight pens per diet. Researchers recorded one count of treatment events per pen over the same follow-up time.
Open the guided count worksheet. Set Control as the reference diet and R1 as the reference room.
# A tibble: 2 × 4
room pen diet events
<fct> <chr> <fct> <dbl>
1 R1 R1-P01 Control 1
2 R1 R1-P02 Control 1
Poisson model for event counts
We want to compare expected event counts under High fibre and Control within the same rooms. In events ~ diet + room, diet gives that comparison and room lets average counts differ across rooms.
The Poisson model assumes that the variance of counts equals their mean at each diet and room combination.
# Overdispersion test
dispersion ratio = 3.975
Pearson's Chi-Squared = 234.526
p-value = < 0.001
The dispersion ratio is about 4.0, well above one. The observed pen counts vary more than the Poisson model allows after accounting for diet and room.
Researchers followed each pen for the same time and assigned diets to pens separately. What else could explain the extra variation in counts? Name two possible causes before changing the model.
Poisson QQ plot for the pen counts
Compare the sample residual quantiles with the theoretical uniform quantiles. Where do the points depart from the diagonal?
The seed argument makes the simulation reproducible.
The negative binomial model allows more variation in counts among pens on the same diet in the same room. method = "ML" fits the model by maximum likelihood.
# Overdispersion test (using simulated residuals)
dispersion ratio = 0.868
p-value = 0.78
In the check with 500 simulated datasets, the dispersion ratio is now close to one.
Negative binomial QQ plot for the same counts
Compare the departures from the diagonal with the Poisson plot. Which part of the pattern improved? Which possible cause would still require information about the pens?
Simulate. Use each fitted model to generate 500 new sets of pen counts, keeping each pen’s diet and room fixed.
Compare. For each possible event count, compare how many pens had that count in the study with how many had it in the simulations.
Look for a mismatch. Can the model recreate how often pens have zero events? What about unusually high counts?
check_model() calls this a posterior predictive check. Here we simulate at the fitted estimates. A Bayesian posterior predictive check draws parameters from their posterior distribution before simulating new counts.
Which model recreates the zero counts?
Green points show the observed numbers of pens. Blue points and bars show the simulated medians and intervals. Both panels show the observed range of 0 to 18 events per pen.
Poisson
Ignoring unknown labels:
• size : ""
• alpha : ""
Negative binomial
Ignoring unknown labels:
• size : ""
• alpha : ""
Which pens does the diet comparison represent?
newdata = "balanced" gives one prediction row for each diet in each of the four study rooms. That gives every room the same weight. Averaging over the observed pens gives the same room weights here because every room has 16 pens.
comparison = "lnratioavg" averages the predicted counts for each diet, divides the High fibre mean by the Control mean and takes the log. transform = "exp" returns the estimate and interval to the count-ratio scale. One means equal expected counts.
The estimate of about 0.65 means 35% fewer expected events under High fibre. The approximate 95% interval stays below one under the Poisson assumption.
The estimate is about 0.67, but the wider interval now includes one. Once the model allows extra variation among pen counts, the data are compatible with a reduction, equal expected counts or a small increase.
What changed when the model allowed extra variation?
For these pen counts, record one concern about the starting model and one reassuring feature of the changed model.
Name another possible cause of the Poisson pattern and the pen records you would need to investigate it. Explain how the wider interval changes what you can say about the two diets.
Break 13:40–13:55. Continue with the workshop deck.
