Agent API Drillground
Before an API is released to agents, simulated customers independently complete real tasks in a sandbox to expose unsafe call sequences, semantic misunderstandings, and permission gaps.
Before opening an API to autonomous agents, teams often rely on Swagger documentation and fixed scripts that can show an endpoint works, but not whether an agent can safely complete a real task. Developers upload an OpenAPI description, provide isolated test accounts, and define a few realistic goals, such as “create a trial for a customer and send an invitation” or “cancel a subscription opened by mistake.”
The product launches simulated agents with different behavioral strategies. They read the documentation, choose their own call sequence, and complete the goals in a sandbox. It records the instructions each agent read, request parameters, permission scope, and recovery actions after failures. If an agent bypasses a required confirmation, requests excessive permissions, or charges repeatedly after an error, the report stops at that call trace and includes the smallest reproducible request.
Teams can turn failed traces from the report into CI tests in one click, then add missing field descriptions, error-code semantics, or permission limits. The first release runs only in sandboxes and never touches real customer data or production payment flows. It tests how agents, as a new kind of client, will explore an API.
Why now
As observed on September 15, Elva ranked 14th in Product Hunt’s new-product feed and explicitly calls agents the new consumers of APIs. S1 Teams preparing to open their interfaces will therefore encounter a gap sooner: an API may work, yet still offer no proof that an agent can complete a real task in a safe sequence.
Target user
Platform teams preparing to hand internal or public APIs to agents, especially before first exposing write operations, billing endpoints, or account permissions. At this stage, standard integration tests have passed, but the team still does not know how agents will interpret documentation or handle failure. Security, platform, and API owners need a shared, reproducible trace.
Minimal entry point
The entry point accepts an OpenAPI file, a sandbox URL, and restricted test credentials. It first converts endpoints into model-callable tools, then runs natural-language tasks through an agent loop. Every request passes through a policy proxy that limits domains, permission scope, and the number of side effects. The execution layer can reuse Schemathesis to generate malformed parameters and stateful call paths, while Docker isolates each task run. S3 Reports initially provide call traces, policy violations, and minimal reproducible requests. Regression output is limited to YAML scenarios and generic test scripts; it does not yet support every CI platform.
Punching above its weight
Early users cluster among API teams preparing to expose MCP or agent tools. Release an open-source CLI scanner that lets developers check permission descriptions, error semantics, and side-effecting endpoints for free. Its results generate shareable failure traces that bring security owners into review. Then add GitHub checks, making every API change a repeatable acquisition channel.
Competitors & gaps
- ElvaGoogle
- Elva can already discover APIs from code and generate permission-controlled MCP services. Its live agent playground lets agents choose tools on their own and shows the call process. The platform also records tool failures, latency, and agent feedback, covering post-launch governance. S2 That substantially overlaps with this product’s task-focused drills. A remaining opening is to test the same task across agents with different behavioral strategies. The focus is not proving that one agent succeeds, but finding privilege escalation, repeated side effects, and unsafe error recovery. Its public capabilities do not emphasize turning failure traces into standalone regression tests. This opening would narrow quickly if Elva adds a policy matrix and security assertions.
- SchemathesisGoogle
- Schemathesis reads OpenAPI or GraphQL descriptions to generate both valid and malformed requests. It adapts tests based on responses and supports stateful, multi-step call sequences. Its reports can produce JUnit output and reproducible requests, making it well suited to CI integration. S3 It excels at finding service errors, specification deviations, and state-machine flaws. Its test objectives primarily come from structural constraints and predefined checks. Teams must still encode business goals, confirmation steps, and least-privilege requirements themselves. Explaining why an agent misunderstood documentation or chose a dangerous sequence is not its central purpose. It can serve as the underlying fuzzing engine, with differentiation in semantic policies and safety judgments.
How it makes money
Charge per team subscription, with plans differentiated by the number of API projects, concurrent drills, and monthly task runs. Bill overages per drill, and reserve high-risk permission policies and private deployment for an enterprise tier.
The case against
Reliable acceptance criteria are difficult to derive automatically from OpenAPI. A successful API response does not mean the customer received an invitation or that a cancellation followed approval rules. Teams must define preconditions, allowed side effects, and end states for every business goal. If the sandbox differs from production in permissions, asynchronous jobs, or third-party callbacks, reports can create false confidence. Model outputs also vary, so putting them directly into CI can cause intermittent failures. Isolated accounts, data cleanup, and model calls increase the cost of each run. Without stable assertions for business rules, the product becomes expensive randomized API testing.