Testing AI Behavior Before Production with Deployment Simulations
Benchmarks, adversarial prompts, and safety evaluations can reveal important weaknesses before a language model is released. Still, they cannot recreate every situation the model will encounter once real people begin using it.
Users provide incomplete instructions, unexpected combinations of context, and conversations that develop over many turns. AI agents add another layer of complexity through tool calls, retrieved documents, permissions, and changing external systems. A model that performs well in a controlled test may behave differently inside a working application.
Deployment simulation gives teams a preview of that behavior. It places a candidate model in conditions that resemble production, records what happens, and evaluates the results before users see them.
How Deployment Simulation Works
A simulation begins with examples of the traffic and context the model is expected to encounter. These might include sanitized production conversations, historical requests, representative user journeys, simulated tool responses, or synthetic users based on real usage patterns.
The candidate processes those inputs as though it were serving the application. Its responses remain inside an isolated evaluation environment, and tool calls are prevented from changing production systems.
A typical pipeline may:
Select a representative sample of earlier interactions.
Remove or transform sensitive information.
Recreate the surrounding conversation and application context.
Run the candidate with the proposed instructions and settings.
Record its responses and tool requests.
Apply automated checks and human review.
Compare predictions with behavior observed after release.
OpenAI’s research on Deployment Simulation describes a process that removes identifying information from eligible production conversations, preserves the earlier conversation, and replaces the next assistant response with one generated by a candidate model. The new responses can then be searched for known problems and audited for unexpected behavior.
This approach has an important limitation. Because the simulation reuses an existing conversation, it cannot show how a user would react to the candidate’s different response.
Why Traditional Evaluations Miss Production Behavior
Evaluation sets are usually built around risks teams already understand. Security tests might target prompt injection, data leakage, prohibited content, or deception. Product evaluations may focus on accuracy, instruction following, and tool selection.
These tests provide deliberate coverage of serious risks, including failures too rare to appear in sampled traffic. Their design can also make them feel different from ordinary use. Handwritten prompts may sound artificial, and familiar benchmark formats may signal to a model that it is being evaluated.
Deployment simulation draws from prompts and contexts that resemble expected traffic. In OpenAI’s experiments, this produced better forecasts than challenging-prompt baselines for important changes in undesired behavior. A retrospective audit also indicated that the method could have surfaced a previously identified behavior called “calculator hacking” before release.
The results still depend on how closely the sample and test environment match production. Users may interact differently with a more capable model, and tool simulations may fail to reproduce changing external systems. OpenAI also notes that its experiments should not be expected to measure behaviors occurring less often than about once in 200,000 messages. Targeted evaluations and red-team exercises remain necessary for rare, severe failures.
Simulating Agentic Workflows
Agent simulations require more than replaying prompts. The environment must recreate the tools, permissions, resources, and state that influence the agent’s decisions.
A coding agent may need a temporary repository, command-line environment, test suite, and package manager. A procurement agent may need mock vendor records, approval rules, and purchasing APIs. These tools must behave realistically without giving the candidate access to production.
Teams should examine the full sequence of actions. Did the agent choose the correct tool? Were its arguments valid and authorized? Did it recover from failures, preserve the user’s goal, and stop at the right time? Would any proposed action have caused an unacceptable side effect?
Anthropic’s guidance on agent evaluations recommends evaluating outcomes and intermediate behavior across realistic, multi-step tasks.
Privacy and Isolation
Production conversations may contain personal information, credentials, confidential documents, or internal business data. Sanitization should happen before that information reaches the candidate, though automated removal cannot guarantee that every sensitive detail is gone.
Simulated tools should use mock services or disposable environments when practical. Any connection to a live system needs strict access controls, request limits, and narrowly scoped credentials. Network rules should also prevent the candidate from contacting unauthorized destinations.
Detailed logs should connect each input with the model response, proposed tool calls, evaluator results, model version, prompt version, and test configuration. That record makes failures easier to understand and reproduce.
Production Takeaways
Deployment simulation adds realistic evidence to the release process:
Use representative traffic while protecting sensitive data.
Recreate the surrounding application and tool context.
Isolate responses, tool calls, and other side effects.
Evaluate complete workflows as well as final answers.
Continue using targeted tests and adversarial evaluations.
Compare predictions with post-release monitoring.
No simulation can reproduce every condition a model will face. It can, however, show how a candidate behaves across a realistic sample before users are exposed to the results. That evidence gives teams a stronger basis for release decisions, safeguards, and controlled rollouts.