AI Model Outage Drills
Teams rehearse a model outage with sanitized real tasks to identify replaceable open models, quality gaps, and the cost of switching.
Product teams that depend on a single model or cloud service often do not discover which workflows cannot be moved until pricing rises, rate limits hit, or the service suddenly becomes unavailable. They start by importing a set of sanitized real tasks, current outputs, acceptance criteria, and latency requirements. For example, a customer-support summary must retain the order number, a code review must identify a specified risk, or document extraction must not miss table fields.
The product replays these tasks against candidate open-weight models, different quantization configurations, and the current model. Instead of producing a generic leaderboard, it shows task by task which workloads can switch immediately, which need only prompt adjustments, and which still fall short on quality. Each failure retains the input, output, and failed acceptance criterion, so engineers can determine whether the problem is factual accuracy, formatting, speed, or tool use.
The team can then run an outage drill: temporarily bar the current model or cloud service and rerun the real workflow. The interface identifies where critical workflows break, which tasks can be automatically degraded, and which must be handed to people. For each task type, teams can also set a fallback model, a minimum acceptable quality level, and the incremental cost after switching.
The first version runs only on sanitized historical tasks in an isolated environment and does not migrate production traffic for the team. It delivers a task-level replacement path, turning “we have a backup model” into a tested plan that can be executed on the day an outage occurs.
Why now
On July 23, policy debate over restricting access to Chinese open-weight AI entered the public lobbying agenda for startups. S2 When observed on July 24, the related Hacker News post had 693 points, 626 comments, and rank 2; teams dependent on a single model are more likely to be asked whether their fallback plan actually works. S1
Target user
The core user is a product team that has connected customer support, code review, or document processing to a single model. The trigger is typically a vendor price increase, rate limit, regional restriction, or policy risk entering planning discussions. Engineering leaders need to answer which tasks can switch, not compare abstract leaderboards. Security and procurement teams also need to see sanitization boundaries, added costs, and human handoff points before they can approve a fallback plan.
Minimal entry point
The first release can use Promptfoo as its evaluation execution layer. It already supports multi-model providers, generic HTTP interfaces, custom scripts, and assertions, covering both cloud models and local endpoints. S3 The product imports sanitized tasks, acceptance criteria, and latency ceilings, then compiles them into test cases. Its results layer retains every input, raw output, assertion failure, and duration. The initial outage drill disables the primary model through a routing switch and replays only fixed workflows in an isolated environment. It will not automatically revise prompts or shift production traffic; the priority is task-level replacement findings, blocking steps, and human handoff points.
Punching above its weight
First publish a reusable “model outage drill checklist” for engineering leaders using open-weight models. Then offer a local CLI that converts existing Promptfoo configurations into task-level replacement reports, using its developer community to reach early testers. Create sanitized example repositories for customer-support summaries, code review, and document extraction. When policy or service changes occur, quickly update drill templates for the affected models and invite teams to submit anonymized failure cases.
Competitors & gaps
- PromptfooGoogle
- Promptfoo can already run the same test cases across multiple models and check content, structure, cost, and latency with assertions. It supports generic HTTP, custom scripts, and local execution, and can retain raw outputs and failure reasons. S3 Teams can therefore assemble model comparisons quickly. Its core unit, however, remains the evaluation configuration and results matrix. Its public documentation does not present replaying an entire business workflow after disabling the current model as a primary workflow, nor does it produce task-level plans for replacement, graceful degradation, and human handoff. The new product must turn evaluation results into an executable outage contingency plan, rather than build another general-purpose evaluator.
- LangSmithGoogle
- LangSmith already offers datasets, experiments, evaluators, and run tracing. Teams can compare models or prompts and inspect each sample’s inputs, outputs, scores, latency, cost, and execution trace. S4 It is well suited to continuously improving an existing LLM application and can build evaluation data from production traces. The gap is that its public documentation centers on experiment comparison, quality analysis, and production monitoring. It does not inherently define outage conditions, backup-model thresholds, or human handoff points for a team. A new product that only scores models side by side would be easy to subsume; its differentiation must lie in fault injection, workflow-break detection, and migration checklists.
How it makes money
Charge a monthly subscription per team project, with tiers based on the number of model endpoints, task-set size, and drill frequency. Isolated deployment, audit exports, and dedicated migration support are priced separately.
The case against
The main risk is that historical tasks may not represent long-tail production inputs, so a successful drill can still fail under live traffic. Overly mechanical acceptance criteria can mistake valid formatting for business usability, while heavy reliance on model-based grading introduces unstable judgments. Tool use, retrieval, and multi-step state make replay environments harder to reproduce, and integration costs can quickly exceed those of a one-off model comparison. Sanitization may also remove context that determines the outcome, distorting conclusions. Without clear quality floors and an accountable workflow owner, the report will remain a risk display rather than become a switching plan.