New Model Migration Rehearsal
After a new model launches, replay de-identified production tasks to identify quality regressions, cost changes, and the conditions for a safe rollout.
When a new model launches, the biggest risk for an application owner is being persuaded by leaderboard results and switching traffic immediately. They first export de-identified historical requests from their logs, retaining the original prompts, retrieval snippets, tool responses, and the outcomes users actually chose. The product reruns the same task set on the old and new models, letting the team set separate thresholds for formatting, factuality, refusals, latency, and cost per call.
The results page does not hide problems behind a single average score. It groups similar regressions into categories, such as missing fields in a quote, incorrect dates, or requests the old model handled but the new one suddenly refuses. Each category is reduced to the shortest reproducible case: the old output and production outcome on the left, the new output and its triggering context on the right. From there, the owner can revise a prompt, add an evaluation sample, or keep this request type routed to the old model for now.
Tasks that pass the thresholds move into a rollout card: which low-risk requests to cover first, which errors should halt the rollout, and what cost increase requires a rollback. Every rehearsal retains the model version, parameters, and data snapshot, so the next model release can be checked directly to see whether the issues actually disappeared.
The first version accepts JSONL task replays and manually labeled expected outcomes, focusing on text generation and structured output. It does not take over production routing or ask teams to upload raw customer data; the owner still confirms the release decision.
Why now
DeepSeek V4 Flash 0731 published evaluation results on July 31. Higher leaderboard scores may prompt application teams to consider switching models, but they cannot establish whether real business performance has regressed. S1 On August 8, the Hacker News post ranked third, with 421 points and 251 comments, making migration decisions more likely to be an immediate team concern. S2
Target user
Application owners or evaluation engineers with LLM features already in production. After a new model release, they must balance performance claims, cost pressure, and release deadlines. What they lack is not another generic leaderboard, but a concrete view of how historical traffic changes on the new model. It is especially suited to small and mid-sized product teams that have historical requests but no dedicated evaluation team.
Minimal entry point
Start with a stable JSONL import schema for prompts, retrieval snippets, tool responses, and expected outcomes. Use configurable HTTP adapters for model calls, retaining model names, parameters, and response metadata. Validate structured outputs deterministically with JSON Schema first. Assess factual, refusal, and formatting issues through rules plus human labels. Initially cluster regressions with text embeddings and hierarchical clustering, then let users confirm the categories. The first release does not connect to production routing; it exports only rollout cards and reproducible cases.
Punching above its weight
Acquire early users through model-release posts, evaluation communities, and LLM engineering groups. Publish a reusable migration-replay template with each major model update. Open-source local JSONL de-identification and sampling commands to lower the barrier to a trial run. Real examples of grouped regressions will reach application owners more effectively than generic model-scoring content.
Competitors & gaps
- BraintrustGoogle
- Braintrust already covers evaluation datasets, experiment snapshots, scorers, and production-log feedback loops. Teams can compare configurations side by side and add failure traces to regression sets. It also supports integrating evaluations into continuous integration, making it well suited to teams with mature evaluation systems. S3 Its public workflow is more oriented toward building general-purpose evaluation infrastructure. A migration owner still has to interpret score changes and identify which requests should not yet switch. The opportunity here is to automatically group similar regressions and distill them into reproducible cases. Results should also culminate in a rollout card that specifies eligible requests, stop conditions, and rollback thresholds. That makes the product about making a migration decision, not simply adding more evaluation capability.
- LangSmithGoogle
- LangSmith already provides datasets, experiment comparisons, and per-item result views. It can flag regressions and improvements relative to a baseline experiment, and supports JSON and YAML output diffs. Users can also filter results by latency, status, and feedback metrics. S4 These capabilities work well for investigating individual runs and metric changes. In a migration, however, an owner still needs to synthesize shared failures from many rows before deciding which traffic can roll out first. This product could group regressions by business meaning rather than merely rank them by scoring columns. Each issue type would be compressed into the shortest viable case and linked to prompt changes or a decision to defer migration. Rollout conditions and snapshots from previous model versions should remain in the same workflow.
How it makes money
Charge a monthly fee per team workspace, with a fixed replay allowance included. Bill overages by model-call volume, and reserve historical snapshots and audit exports for higher-tier plans.
The case against
Historical requests may contain customer information, internal retrieval content, and tool responses; even a small gap in de-identification rules can block procurement. Replays may also fail to reproduce production conditions because dynamic retrieval results and external tools may have changed. Generative tasks rarely have one correct answer, so automated scoring can mistake stylistic differences for regressions. If clustering combines different root causes, the shortest case may point teams toward the wrong fix. Running both models also increases call costs and wait times. If the results still require extensive manual, item-by-item review, teams may stick with existing scripts and spreadsheets.