Measuring Whether Anything Moved
The five numbers that show whether a rota change worked, why one of them should get worse, and how to read a result that goes the wrong way.
One site, before and after relaxing four constraints over a year
Five measures improved, one got worse by design and one got worse in a way that shows the others are real. Recorded lateness rose because the arrangement people had been using to avoid it was removed. This is one employer's own data over a year, not a benchmark.
A rota change either moved something measurable or it did not, and the measurement has to be decided before the change rather than found afterwards. Five numbers cover almost every case.
The test in “Measuring Whether Anything Moved” needs an unchanged baseline and a trustworthy record of what happened after the change. Teams exploring a transparent how to monitor employees without being intrusive process for how to monitor employees without being intrusive can strengthen that comparison while preserving corrections, exceptions and the decision owner’s explanation.
The discipline matters because rota changes generate strong impressions in all directions, and impressions are formed in the first fortnight, which is the one period that proves nothing.
Additional context for “Measuring Whether Anything Moved” is available from HSE shift-work guidance. Use external guidance to prompt a specific validation check rather than replace local review.
The five numbers
Unfilled shifts. Overtime or agency hours. Individuals working beyond what the pattern intends. Distribution of unsocial shifts across the team. And whatever the service itself measures.
- Unfilled shifts.
- Overtime or agency hours.
- Individuals working beyond what the pattern intends.
- Distribution of unsocial shifts across the team.
- Whatever the service itself measures.
Four are about the rota and one is about the work, and the fifth is the one that stops an efficient rota being declared a success while the service gets worse.
Where each comes from
All four rota numbers come from the rota system or the spreadsheet, and all four can be produced for a past period, which means the before can be reconstructed even if nobody thought to collect it.
The service number comes from somewhere else entirely and usually requires asking. Ask early; it is the measurement most likely to be unavailable at short notice.
The number that should get worse
Any real change has a cost, and if none of the numbers moved the wrong way the measurement is probably incomplete.
A change that frees capacity usually spends something: supervisor time, a room, a budget line, a small loss of continuity. Naming that in advance and watching it is what distinguishes a trade from a claim.
The number that gets worse and is good news
Occasionally a measure deteriorates because a workaround has been removed. Recorded lateness rises when people stop arranging to avoid a rounding cliff; recorded absence rises when a reporting route becomes easier.
Those are the most informative results in the whole exercise, and they are routinely read as failures by anybody looking at a single line on a dashboard.
Reading a null result
Nothing moved. That happens, and it means one of three things: the constraint was not binding, the change was too small, or something else absorbed the effect.
All three are worth knowing and all three are better than not having measured. Record it as tested with no effect, and go to the next constraint on the list.
Reading a bad result
The change made things worse. Reverse it, say so plainly, and record what happened.
An organisation that reverses one change publicly can attempt ten more. One that defends a bad change because reversing it would look like failure will not get another trial agreed for years.
How long to wait
Two full cycles before drawing any conclusion, and a year before treating a result as settled. Seasonal effects are large in most operations and will produce both false positives and false negatives inside a quarter.
Where a decision has to be made sooner, say that the evidence is partial and name the date when it will not be.
Who should read the numbers
Somebody other than the person who proposed the change, at least once. Not as an audit: as a second reading of the same page.
The proposer is the worst-placed person to notice that the improvement started a month before the change did, which is the most common way these results turn out to be about something else.
Publishing the result
To the people affected, in the same five numbers, whichever way it went. A short note, once.
This is the step that makes the next change possible. People cooperate with changes when the last one was measured and reported honestly, and they do not when results only appear if they are good.
The measure that is not a number
One question, asked of the people on the shift two months in: is this better or worse than before, and in what way?
It belongs beside the numbers rather than instead of them. Where the numbers improve and the answer is worse, something has been moved onto people in a way the measurement did not capture, and that is exactly what the question is for.
What to record on the register
Constraint, what was changed, the date, the five numbers before and after, and the decision. One row.
After two years that table is a record of what this particular organisation's rota responds to, which is more useful than any general advice about scheduling, including everything in this collection.