Threat-model an AI development workflow

Choose a limited scenario

Start with one workflow that people can understand. For example, an agent reads an issue, edits a repository, runs tests, and opens a pull request. Include the systems that make these actions possible.

List the assets that matter: source code, customer information, credentials, release artifacts, and service availability. Identify who owns them. Then identify the people and systems that can read or change each asset.

OWASP recommends modeling the system, identifying threats, choosing responses, and validating the result. Use the method early and update it as the system changes. Threat modeling guidance.

Draw the trust boundaries

For the fictional issue-to-PR workflow, draw these connections:

Issue → agent → repository → test runner → artifact store → deployment

Add the model provider and the secret store. Mark where content comes from a less trusted source. Mark where an identity gains a new capability, such as moving from reading an issue to writing repository files.

The application diagram alone does not show the whole development risk. A production database can be private while a CI job exposes a credential. Include temporary environments and support access when they affect the scenario.

Write a concrete failure path

Avoid entries such as “AI might be unsafe.” Write an actor, an action, an affected asset, and a consequence. Include the conditions needed for the scenario to occur.

These are examples, not a complete threat list. Your data, tools, and operating environment determine the relevant scenarios.

Choose a response with an owner

Prioritize consequences and credible exposure. Avoid presenting a numerical score as precision you do not have. Record the uncertainty and the evidence that could change the priority.

A response can remove the risky capability, reduce its scope, add a control, or accept a defined residual risk. Acceptance needs an authorized owner and a reason. It should not be an agent’s unreviewed conclusion.

Turn the chosen response into work with an observable acceptance condition. “Improve agent security” is difficult to verify. “The test job cannot read the production secret” defines a boundary that can be checked.

Review after material changes

A new connector, model route, environment, or permission can change the threat model. Add these changes to the review triggers. Also use incidents and failed evaluations to update assumptions.

Try the risk-review exercise to change a scenario’s data, authority, audience, and recovery conditions. The result suggests questions. It does not replace a system-specific threat model or authorize the work.