A model id that silently starts pointing at different weights turns every number you published against it into a claim about something else.
ConceptWhat it is
Model versioning is pinning the exact model behind a result and keeping that pin meaningful over time. For self-hosted weights it is a checkpoint identity. For a hosted API it is a model id — and the difference between those two cases is the whole subject.
A checkpoint you hold cannot change underneath you. A hosted id can. Providers ship updates behind aliases, deprecate dated versions, and adjust defaults for parameters you did not set. None of that produces an error. The call succeeds, the output looks reasonable, and every published measurement taken against that id is now describing a model that is no longer there.
How it worksThe mechanics
The rule is to pin the most specific identifier the provider exposes — a dated or numbered version rather than a floating alias — and to record it on every result alongside the parameters that shape output. Temperature, max tokens and system prompt belong in the pin, because a default that moves changes behaviour exactly as much as new weights would.
Then the pin has to be watched, because pinning does not stop a provider retiring a version. A dated id has an end of life, and the migration is a re-measurement rather than a find-and-replace: the held-out set is re-run against the new version and the results are compared before the id changes anywhere else. Published numbers name their model version, so a reader can see what was measured rather than assuming it is current.
At a glanceSee it
Pinning a dated version and its parameters is what makes a published number mean something later. Deprecation then makes migration a re-measurement, not a rename.
When to use itWhere it fits
- Any time a number is published against a model, which on this site is every kit.
- Before a comparison between two runs, since an unpinned id makes the comparison unfalsifiable.
- When a provider offers both a floating alias and a dated version — take the dated one.
- Alongside a deprecation watch, because a pin that has been retired fails at the least convenient moment.
When NOT to use itLimits & anti-patterns
- As a reason to stay on an old version indefinitely; pinning is for attribution, not for avoiding upgrades.
- Pinning without recording parameters, which leaves half the behaviour unpinned and looks complete.
- In exploration, where a floating alias is genuinely the right default.
- As a substitute for re-running evals after a migration — a new version is a new measurement.
Trade-offsAdvantages & costs
Advantages
- Keeps a published result meaningful by naming exactly what produced it.
- Makes an upgrade a decision with evidence rather than something that happens to you.
- Turns provider-side change from an invisible event into a scheduled one.
- Costs nothing at runtime; it is a string and a discipline.
Trade-offs & costs
- Dated versions get retired, so a pin is a commitment to migrate rather than a permanent answer.
- Not every provider exposes a stable dated id for every model.
- Parameters are easy to forget and carry as much behaviour as the weights do.
- Pinning across several providers multiplies the deprecation calendar you have to watch.
ExampleIn the real world
A kit publishes an eval result against a model id and the numbers hold on every re-run for weeks. The id was a floating alias. When the provider moved it to newer weights, the re-run produced different outputs on roughly a fifth of the set — not worse, but different, and the published comparison against the baseline had quietly become a comparison between two different models. Nothing errored at any point, and the only signal was that a number moved without any change on our side.
ToolsHow to implement it
- A dated or numbered provider idthe most specific identifier available, never the floating alias.
- The parameter set stored with ittemperature, max tokens and system prompt, pinned together.
- A deprecation calendarso an end-of-life date is a planned migration rather than an incident.
- The model version field on every published recordwhat makes a stale number visible as stale.
Cost & effortWhat it takes
No runtime cost. The ongoing cost is the migration cadence a pin commits you to, and the re-measurement each migration requires — which is real spend on any kit that pays per call. That re-measurement is the point rather than the overhead: a version change that has not been measured is an unpublished change in behaviour.