Interactive AI misuse risk model

The following is an interactive model of AI misuse risks described in the research paper "An Example Safety Case for Safeguards Against Misuse." The model leverages evidence from expert surveys, capability evaluations, and safeguards evaluations to estimate the annual expected harms caused by novice actors with the help of a particular AI assistant.

Evidence from expert surveys

Baseline Success Probability
iThe distribution below indicates the probability that a novice 'attempt' to cause a large-scale harm will succeed given varying levels of time spent, where an 'attempt' must involve at least two weeks of earnest effort. This curve assumes conditions prior to the deployment of the AI assistant in question.

Success at 2 months (%):

Success at 6 months (%):

Success at 24 months (%):

Baseline Success Probability

Effort Distribution Parameters
iThe distribution below indicates the probability that a novice 'attempt' to cause a large-scale harm will involve less than or equal to the amount of time indicated. An 'attempt' must involve at least two weeks of earnest effort. We assume this distribution is fixed and is unaffected by the deployment of the AI assistant.

Effort at 2 months (%):

Effort at 6 months (%):

Effort at 24 months (%):

CDF(Effort of attempt)

Expected Annual Attempts

Expected Harm Per Success

Queries Per Month

Baseline expected annual harm (e.g. lives lost):

25,223

Evidence from capability evaluations + experts

Pre-Mitigation success probability
iThe distribution below indicates the probability that a novice 'attempt' to cause a large-scale harm will succeed given varying levels of time spent, where an 'attempt' must involve at least two weeks of earnest effort. This curve assumes conditions where the AI assistant in question is deployed without safeguards.

Success at 2 months (%):

Success at 6 months (%):

Success at 24 months (%):

Pre-Mitigation success probability

Pre-Mitigation expected annual harm (e.g. lives lost):

65,500

Summary of predictions

Baseline expected annual harm (e.g. lives lost):

25,223

Pre-Mitigation expected annual harm (e.g. lives lost):

65,500

Post-Mitigation expected annual harm (e.g. lives lost):

22,950 ✅ (< baseline)

Evidence from safeguards evaluations

Queries executed vs effort
iThe results of a safeguards evaluation. A red team tries to obtain responses from an AI assistant to a representative set of misuse queries. The y-axis shows the average number of queries executed by members of the red team as a function of effort expended.

Drag me!

Bans vs Queries Executed
iDuring the safeguards evaluation, the developer keeps track of the number of times the system 'bans' a member of the red team. The figure below shows the average number of bans a member of the red team receives as a function of the number of harmful queries they successfully execute (receive nearly-complete responses to).

Time lost to bans
iThis plot is estimated by experts. It shows the amount of time a novice misuse actor would likely need to spend to reacquire access to the AI assistant after being banned.

Queries executed vs effort (w/ bans)
iThis plot synthesizes the other three plots. It shows the number of harmful queries members of the red team obtain responses to over time, factoring in the additional time that a real misuse actor would lose due to being repeatedly banned.

Post-mitigation success probability
iThe distribution below shows the probability that a novice attempt to cause a large-scale harm succeeds given the amount of time spent on the attempt. The solid blue line shows the post-mitigation success probability, while the dashed lines show the baseline and pre-mitigation probabilities for comparison.

Baseline Success Probability
Pre-Mitigation Success Probability
Post-Mitigation Success Probability

Post-Mitigation expected annual harm (e.g. lives lost):

22,950

Estimating how quickly developers need to respond to changing deployment conditions

We can use this quantitative model to simulate "what-if scenarios" where users identify a way to reliably jailbreak models. How much time do developers have to correct the deployment?

Unacceptable risk contribution (annual expected harm):

Risk post deployment (w/o jailbreak)
Risk post deployment (w/ jailbreak)

A formal description of the model:

Joshua Clymer, March 8, 2025.

Interactive AI misuse risk model

Evidence from expert surveys

Baseline expected annual harm (e.g. lives lost):

Evidence from capability evaluations + experts

Pre-Mitigation expected annual harm (e.g. lives lost):

Summary of predictions

Baseline expected annual harm (e.g. lives lost):

Pre-Mitigation expected annual harm (e.g. lives lost):

Post-Mitigation expected annual harm (e.g. lives lost):

Evidence from safeguards evaluations

Queries executed vs effortiThe results of a safeguards evaluation. A red team tries to obtain responses from an AI assistant to a representative set of misuse queries. The y-axis shows the average number of queries executed by members of the red team as a function of effort expended.

Time lost to bansiThis plot is estimated by experts. It shows the amount of time a novice misuse actor would likely need to spend to reacquire access to the AI assistant after being banned.

Queries executed vs effort (w/ bans)iThis plot synthesizes the other three plots. It shows the number of harmful queries members of the red team obtain responses to over time, factoring in the additional time that a real misuse actor would lose due to being repeatedly banned.

Post-Mitigation expected annual harm (e.g. lives lost):

Estimating how quickly developers need to respond to changing deployment conditions

A formal description of the model:

Queries executed vs effort
iThe results of a safeguards evaluation. A red team tries to obtain responses from an AI assistant to a representative set of misuse queries. The y-axis shows the average number of queries executed by members of the red team as a function of effort expended.

Time lost to bans
iThis plot is estimated by experts. It shows the amount of time a novice misuse actor would likely need to spend to reacquire access to the AI assistant after being banned.

Queries executed vs effort (w/ bans)
iThis plot synthesizes the other three plots. It shows the number of harmful queries members of the red team obtain responses to over time, factoring in the additional time that a real misuse actor would lose due to being repeatedly banned.