AI Patch Attack Replay
Before an AI-generated fix is merged, replay the real attack path against both versions to prove the vulnerability is closed without creating a new privilege-escalation route.
When security engineers review AI-generated fixes, the hardest question is not whether tests pass, but whether the original intrusion path has actually been cut off. A compromise involving an AI-assisted autofix showed teams that even with green functional tests, boundaries around authentication, authorization, or third-party integrations can still be broken. Before merging a patch into the main branch, teams need replayable attack evidence—not a code explanation that merely sounds plausible.
Developers attach the vulnerability description, affected endpoints, and reproduction conditions to a security PR. In an isolated environment, the service deploys both the pre-fix and patched versions, then reconstructs the same attack path using the call chain, permission configuration, and test accounts. It runs that path against each version, records where the request gains access, reads data, or invokes a sensitive endpoint, and attaches the behavioral differences back to the PR.
Rather than showing only a risk score, the review page presents a clickable trail: "the old version escalates privileges here; the patched version is denied here." If the patch blocks the old path but opens a new high-privilege call, bypasses logging, or expands token scope, the check flags that new path. Cases that cannot be replayed reliably go to human security approval, so uncertainty is not presented as a confirmed fix.
The first version covers authentication, authorization checks, and third-party ticketing integrations in web applications, using the team’s isolated accounts and test data. It does not scan every repository or replace penetration testing. Its purpose is to leave behavioral evidence of why an attack failed with every AI security patch before it is merged.
Why now
On August 17, Wiz disclosed a vulnerability that had been exploited in the wild: Copilot Autofix was listed as a co-author on the merge commit, while AI review failed to catch script injection. S1 As of August 18, the post ranked fifth on Hacker News, with 306 points and 123 comments, making it easier for security teams to ask whether a patch truly breaks the attack chain. S2
Target user
Primary users are application-security engineers who approve security fixes and senior developers accountable for merging them. The critical moment is after an AI or automated tool submits a patch and before a protected branch is released. Unit tests can show that functionality is not obviously broken, but not whether the original attacker identity can still escalate privileges. For changes to authentication, authorization, or third-party tokens, they need repeatable behavioral evidence before signing off.
Minimal entry point
Launch as a GitHub App that listens for security PRs and check requests. Teams submit roles, seed requests, prerequisite state, and success assertions in a declarative file. GitHub Actions checks out the baseline and patched commits separately and starts isolated environments with Docker Compose. Replay HTTP and browser flows with Playwright; connect server-side services to audit logs or OpenTelemetry traces. Write step-by-step differences back through the Checks API, with redacted request evidence. Start with authentication, object-level authorization, and ticketing APIs rather than trying to generate arbitrary exploits automatically.
Punching above its weight
Acquire users through publicly reproducible security patches. Create open-source replay examples and a corresponding GitHub Action for GitHub Actions injection, IDOR, and OAuth permission expansion. Each example shows the exact steps through which an attack can still succeed after ordinary tests pass. Security consultants and application-security engineers can attach these artifacts directly to client PRs, bringing the first private-repository trials through real review workflows.
Competitors & gaps
- Bright STARGoogle
- Bright STAR can build and start an application in CI, run dynamic scans, report only findings reproducible in a running environment, and rescan patches to verify fixes. On PR triggers, it can also narrow test scope based on code changes and replay CodeQL or SARIF findings. S3 That comes close to post-deployment verification. Its public documentation does not say that it treats a team-submitted existing attack chain as a fixed test case, or that it compares baseline and patched commits step by step. Differences in privilege acquisition, data access, and sensitive calls may still be reduced to a vulnerability status. The opportunity is not another automated remediation agent, but a dependable dual-version evidence format that preserves failed steps, identity context, and audit events so reviewers can see why an attack no longer works.
- StackHawkGoogle
- StackHawk already provides DAST for running applications, authenticated routes, and CI/CD scanning. It supports multi-role BOLA and BFLA testing, custom security scripts, and routing scan results into GitHub PR checks and vulnerability-management workflows. S4 It therefore solves much of the automated testing problem for protected endpoints. Its public documentation does not make a single known attack chain the central object of patch acceptance, nor does it describe deploying baseline and patched versions side by side to show behavioral changes. General-purpose scanning is better suited to finding a set of vulnerabilities; reviewers must still determine whether a given fix addresses the original intrusion path. This product can turn reproduction conditions, roles, token scopes, and success assertions into reviewable artifacts, while also checking whether a patch introduces new high-privilege calls rather than merely confirming that the old alert has disappeared.
How it makes money
Charge a monthly fee per active private repository, with a base replay allowance included. Bill overages by isolated-environment execution usage.
The case against
Teams must run two testable versions at once and supply least-privilege accounts with resettable data. Single sign-on, short-lived tokens, and third-party callbacks can make replays unreliable. Attack scripts may leak secrets or damage test data, so isolated environments also need egress restrictions, managed credentials, and automatic cleanup. Comparing status codes alone can lead to false conclusions, since the vulnerability may still succeed through another route. Adding audit logs and call traces materially increases deployment-adaptation work. As role matrices grow more complex, execution time and compute costs become harder to fit into the PR workflow. If the evidence occasionally labels an unfixed issue as fixed, security teams will quickly stop enforcing the gate.