misuse-monitoring

How it is built

Architecture overview

Misuse monitoring is the defender's side of running an AI model, or any online service that strangers can call. It decides, for every request and for every account over time, whether use stays inside the usage policy, and it answers in proportion to the evidence: let it through, steer it, slow it down, send it to a person, warn the owner, or end the account. It runs two loops on different clocks. A real-time loop puts fast classifiers and limits in the request path. An offline loop records everything, aggregates it per account and per cluster of accounts, and finds harms that no single request shows, such as a campaign spread over many harmless-looking conversations. The idea that makes it work is that the account, not the request, is the unit of enforcement: a request-level score is only a signal, and the decision to warn, restrict or ban rests on the pattern of signals, confirmed by a second reviewer (a stronger model, then a person) before anything hard to undo happens.

This page covers public, documented designs (Anthropic, Microsoft, OpenAI, and the open-source tools Osprey and Coop) and only the defending side.

Parts:

• Real-time classifiers — score each prompt and completion against the policy categories — Text classification, Calibration, Transformer block, Byte-pair encoding (BPE) tokenization, Linear and logistic regression, Linear-time regular expression matching, Aho-Corasick multi-pattern matching
• Request limits — per-key and per-organization rate and spend caps that bound the damage of any account — Token bucket, Sliding-window counters
• Event log — one durable, ordered record of requests, scores and actions, with privacy controls on content — Partitioned commit log, Consumer offsets, Data masking, Named entity recognition, Text normalization, data-deidentification (no page yet), k-anonymity (no page yet), differential-privacy (no page yet)
• Rules and state engine — turns streams of signals into per-account state and actions — Windowed stream aggregation, Complex event processing, Reputation scoring, Bayesian rate estimation
• Account and cluster analysis — links accounts, finds coordinated groups and near-identical prompts — embedding-inference (no page yet), HNSW, Density-based clustering (DBSCAN), Near-duplicate detection with MinHash LSH, Union-find, Entity resolution, Device fingerprinting, Isolation forest, graph-community-detection (no page yet), identity-resolution (no page yet)
• Review — model first pass, summaries per account, human queues with escalation — LLM-as-judge, Top-k selection, Escalation policies
• Enforcement and audit — warnings, restrictions, termination, appeals, and a log that shows what was decided and why — Tamper-evident audit log

 client ─► gateway ─► per-key limits ─► input classifiers ─► model ─► output classifiers ─► client
 (key, org,           (token bucket,     (block / steer /      │       (block / log)
  end-user id)         spend cap)         log)                 │
                          │                   │                │              │
                          ▼                   ▼                ▼              ▼
                  ┌──────────── event log (partitioned commit log) ─────────────┐
                  │ request id, account, org, scores, actions, content (limited) │
                  └──────┬──────────────────────────────────────┬───────────────┘
                         ▼ seconds                              ▼ hours to days
               rules + state engine                      offline analysis
               (windows, labels, reputation)             (summaries per account,
                         │                                clusters across accounts)
                         ▼                                        │
               review queue ◄──── model first pass ◄──────────────┘
                         │
                         ▼
               decision: warn · restrict · ban ─► account state ─► gateway (read per request)
                         │
                         ▼
               audit log + appeals + feedback to policy, classifiers and training

How the parts fit together

Policy and taxonomy

Everything starts from a written policy. Anthropic describes a usage policy with specific guidance for areas such as child safety and cybersecurity, and external experts who stress-test it (source: Building safeguards for Claude); Microsoft's counterpart is a code of conduct with severity levels. The contract with the rest of the system is the taxonomy: its categories and severities are the classifiers' label set, the model reviewer's rubric and the human reviewer's checklist. If the three differ, a "flag" means something different at each stage and queues fill with disagreements.

Real-time classifiers

Text classification is the workhorse, and the system climbs the ladder that block describes. Cheap matchers (Linear-time regular expression matching, Aho-Corasick multi-pattern matching) run after Text normalization and catch exact known strings; small classifiers cover broad categories; larger models handle the ambiguous middle. Documented systems use all of these:

• Anthropic runs prompted or fine-tuned Claude models as classifiers in real time, several at once, and says the system must handle trillions of tokens while limiting compute overhead and over-enforcement (source: Building safeguards for Claude).
• Constitutional classifiers are input and output classifiers trained on synthetic data generated from a natural-language constitution of permitted and restricted content. The updated design is reported at an absolute 0.38 % increase in production-traffic refusals and 23.7 % inference overhead (source: Sharma et al. 2025). The first prototype was reported to refuse too many harmless queries and cost too much compute (source: Anthropic post).
• Llama Guard is an open, instruction-tuned LLM that classifies prompts and responses against an adjustable taxonomy (source: Inan et al. 2023).
• OpenAI's Moderation API returns a flag, per-category booleans and scores from 0 to 1 for text and images, and warns that scores may shift when the model is upgraded, so thresholds may need recalibration (source: OpenAI Moderation guide).

That last warning is why Calibration belongs here: actions hang on thresholds, and a drifting score silently moves the false-positive rate. Log the score with the model version, calibrate on a labelled holdout whenever the classifier changes, and keep the action a separate decision from the score.

Inside the request a classifier may block, log silently, or steer. Anthropic describes prompt interventions: when a classifier flags likely spam or malware generation, extra instructions are added to the system prompt, and in narrow cases the model is stopped from responding (source: Building safeguards for Claude). Steering costs little and does not punish an account for a borderline request.

Request limits

Token bucket and Sliding-window counters bound what one key can do per unit time. They detect nothing, but they fix the blast radius while detection catches up and are the cheapest enforcement lever. Anthropic's API docs say limits exist to prevent abuse while minimizing impact on common usage, are set per organization by usage tier, may start lower for new organizations and rise as history builds, use the token-bucket algorithm, and that sharp usage increases can trigger separate acceleration limits (source: Anthropic rate limits). Tier and history are Reputation scoring in disguise; limits keyed on the API key alone stop nothing when one actor holds many keys, which the account-analysis part addresses.

Event log

A Partitioned commit log is the spine. Each request writes an event keyed by account, so per-account order holds, and independent consumers read the same stream at their own pace with their own Consumer offsets: rules engine, cluster builder, summarizer, dashboard, audit sink. A new detector is a new consumer, not a change to the request path. The log is also where privacy is enforced:

• Microsoft stores prompts and completions for human review in a store separated per customer resource and geography. Reviewers read only data already flagged or part of a flagged pattern, via request-id queries, secure workstations and just-in-time manager approval. Qualifying customers can apply for modified abuse monitoring, with no content storage and no human review, at the stated cost that detection may be less accurate (source: Microsoft docs).
• Summaries carry the same access controls as the text they summarize (source: Anthropic, hierarchical summarization). Data masking and Named entity recognition strip personal data from anything analysts see or aggregates use.
• Clio's research runs require clusters to exceed minimum numbers of accounts and conversations and audit summaries for private data; its safety runs skip those thresholds so violating accounts can be actioned, with access limited to a small authorized group (source: Tamkin et al. 2024).

Customers help attribution. OpenAI asks developers to send a stable, hashed per-end-user identifier (safety_identifier), which helps it monitor for abuse, give actionable feedback and act on one user instead of the whole organization (source: OpenAI safety best practices).

The privacy blocks decide what leaves the restricted store in which form. data-deidentification (no page yet) replaces account, email and device identifiers with stable pseudonyms and generalises exact values (a timestamp to the hour, an address to a region) in the copies that analysts and research jobs read, while the keyed mapping back to the real account stays with the small group allowed to action it. k-anonymity (no page yet) is the rule behind the cluster thresholds above: a cluster, a dashboard cell or an exported table is shown only if it covers at least a minimum number of accounts, and smaller groups are suppressed or merged, so an aggregate cannot point at one person. differential-privacy (no page yet) applies where counts leave the safety group altogether, such as published usage statistics or shared trend figures: calibrated noise is added to each count and every release spends from a privacy budget, so repeated queries cannot be combined to undo the protection. None of the three is applied on the enforcement path, where the account must be identifiable.

Rules and state engine

Per-request scores become decisions only when accumulated. Windowed stream aggregation counts flags per account per window by category and severity; Complex event processing matches sequences over time; Reputation scoring and Bayesian rate estimation turn small counts into a score that one flagged request on a new account does not trip, but a steady rate on an old one does. Microsoft's abuse-pattern component fits this: it weighs frequency and severity of detected harmful content and the intentionality of behavior, such as recurring attempts to defeat safeguards (source: Microsoft, Abuse monitoring).

Osprey makes the pattern concrete. Events stream in; human-written rules extract named features and evaluate each event; labels on entities (user id, email, IP address) persist across events as memory; matched rules fire effects such as adding a label, banning or returning a verdict; every result records the matched rules and fired effects (source: Osprey docs). A rule engine that cannot say why it acted cannot be reviewed or appealed.

Account and cluster analysis

The offline part finds what rules written in advance miss.

• Per account: Anthropic's hierarchical summarization condenses each interaction into a short structured summary with a harm tier, then aggregates a user's summaries into a usage report citing representative interactions for verification. Reviewers rated 96 % of interaction summaries accurate enough for harm determination and 98 % complete enough; usage-level summaries are harder to validate (source: Anthropic, hierarchical summarization). This is LLM-as-judge used as a data reducer, making logs small enough to read.
• Across accounts: Clio embeds a description of each conversation, clusters with k-means, titles and summarizes clusters and builds a hierarchy. In safety use it found clusters that were individually benign but coordinated across many accounts, such as bulk keyword generation, role-play with identical prompt structure and access resale, and accounts were removed. Anthropic states it takes no automated enforcement based solely on Clio clusters (source: Tamkin et al. 2024). The matching blocks are embedding-inference (no page yet: a pretrained embedding model served as is, with tokenizing, batching, pooling and caching of vectors, and no training), nearest-neighbour search (HNSW), density clustering (Density-based clustering (DBSCAN)) and Near-duplicate detection with MinHash LSH for template-built prompts. The public sources describe only the k-means clustering; HNSW, DBSCAN and MinHash are design options.
• Linking accounts: Entity resolution, Union-find and Device fingerprinting merge accounts that share payment instruments, devices or organizations, so action lands on the actor, not one key. Community detection over the account graph (graph-community-detection, no page yet) finds rings.
• Unknown unknowns: Isolation forest ranks accounts whose volume or mix is odd; it says where to look, not what is wrong. Anthropic also compared Clio cluster concern with classifier flag rates to find false positives and negatives (source: Tamkin et al. 2024).
• Threat intelligence: analysts study severe cases and feed findings back into detection. Anthropic reports banned accounts, a classifier tailored after a case, and indicators shared with authorities and partners (source: August 2025 report).

Review

A ban is costly to get wrong, so a flag gets a second look. Microsoft reviews flagged samples first by automated means including LLMs, and brings in humans when automated review misses confidence thresholds or is unavailable (source: Microsoft, Abuse monitoring). Anthropic's harm tiers triage human review. The queue is a priority problem: Top-k selection picks the next cases by severity and age, and Escalation policies route an unowned high-severity case up through levels on timers. Coop packages the human side: a review console, queues, routing and automatic enforcement rules, and an analytics view of automated versus manual actions (source: Coop README).

Known illegal imagery takes a separate, exact path: Anthropic notes hash matching against CSAM databases for uploaded images on first-party products, and Meta's Hasher-Matcher-Actioner implements hash, match, act over banks of known content.

Enforcement and audit

The decision is written to an account state that the gateway reads on every request, so a restriction applies on the next call. Documented actions form a ladder: warnings and termination for severe patterns (Anthropic); an email notice, a chance to explain and remediate except for severe or recurring abuse, then suspension or termination (Microsoft). A Tamper-evident audit log records each decision with evidence references so it can be reviewed, appealed and shown unedited.

Data and control flow

One request (real-time loop):

1. The gateway authenticates the key, reads account state and per-key limits; a rejected request writes an event and stops.
2. Input classifiers score the prompt: pass, steer, block with a policy message, or pass and log. This stage must cost a small fraction of the model's own cost.
3. The model generates; output classifiers score the completion with the same outcomes.
4. An event is appended to the log: account, organization, optional end-user id, classifier versions, scores, action, content reference. Time goes to the model first, then classifier inference; the append is off the critical path.

One flagged account (detection to decision):

1. The state engine sees an account cross a rule or reputation threshold, or the offline job places it in a suspicious cluster, or an analyst reports an indicator.
2. A model reviewer reads a bounded sample of its flagged traffic or its usage summary and returns a verdict citing interactions.
3. Confident and low severity: warn or restrict. Uncertain or severe: a case enters the human queue.
4. A reviewer confirms or overturns, picks an action and writes the reason; the gateway applies the new account state on the next request.
5. The labelled outcome returns as training and calibration data.

One campaign (offline loop):

1. A scheduled job embeds conversation summaries, clusters them and ranks clusters by concern and number of distinct accounts.
2. An analyst opens a cluster, checks linked-account signals and decides whether it is coordinated.
3. Accounts go through the same review and enforcement path; indicators go to the rules engine.

Scaling

In the order the limits bite:

1. Classifier cost on the request path. Levers: cheap matchers first and escalate only the uncertain tail, smaller classifiers, running them in parallel with the model.
2. Log volume and retention. Levers: scores and summaries kept long, content briefly; partition by account so consumers scale out.
3. Offline analysis cost. Levers: summarize only accounts above a cheap pre-filter, small models first, summarize in a hierarchy.
4. Reviewer capacity. Human review does not scale with traffic. Levers: model first pass, severity ordering, one decision per cluster.

Failure modes

Real implementations

The alternative: a pure request-level filter, with no account state, is simpler and stores little, but it cannot see slow, split or coordinated misuse and punishes a user for one borderline request. The designs above filter per request, decide per account and confirm before acting.

Sources

• Anthropic, Building safeguards for Claude; Monitoring computer use via hierarchical summarization; Detecting and countering misuse, August 2025; March 2025 report.
• A. Tamkin et al., Clio: Privacy-Preserving Insights into Real-World AI Use, 2024. paper
• M. Sharma et al., Constitutional Classifiers, 2025. paper, post
• H. Inan et al., Llama Guard, 2023. paper
• Anthropic API docs, Rate limits.
• Microsoft Learn, data privacy for Azure OpenAI.
• OpenAI API docs, safety best practices.
• ROOST Osprey data flow and docs; Coop and Hasher-Matcher-Actioner READMEs.