The post is the vocabulary.

Overview

The shape end-to-end:

              main agent
                  │
                  │ proposes action
                  ▼
              supervisor  (step in the loop)
                  │
                  │ fan out to specialists in parallel
                  ▼
        specialist 1, 2, ... N
        (regex · SQL · AST · classifier · narrow LLM)
                  │
                  ▼
        aggregation: union, not vote
                  │
       ┌──────┬───┴────┬──────┐
       ▼      ▼        ▼      ▼
       ok   nudge     flag   block
       │      │        │       │
   execute correction execute  refuse;
           + replan   + record main agent
           (feeds     for      must replan
           back to    human    from scratch
           main       review
           agent)
                  │
                  ▼
        replay artifact (signed, re-runnable)
                  │
                  ▼
            feedback log
                  │
                  ▼
        accumulation per deployment

Four moving parts. A taxonomy that names every known failure mode of the main agent. A specialist per entry in the taxonomy, narrow by construction. A decision layer that fans actions out and aggregates verdicts by union. A feedback log that turns every flag and every nudge into training signal for that specific deployment. Almost everything in this post is variations on those four parts.

What it actually is

A main agent is the thing that does the work. It plans, calls tools, edits state, opens PRs, runs commands. It runs between human checkpoints. The interesting main agents in production today are coding agents, support agents, research agents, browsing agents, sales agents, ops agents. They share one property. They do real work without a human watching every action.

A supervisor agent is a separate process that sits inside the main agent’s cycle. It observes what the main agent is about to do. It decides whether that action is acceptable. Then it acts on that decision. The action might be ok (let it proceed), nudge (refuse this exact action but send a correction back so the main agent can replan), flag (allow but record for human review), or block (refuse outright and require the main agent to replan from scratch).

That’s the whole thing. Watch, decide, act, record. Inside the loop, every cycle.

Five properties follow from this definition. They’re what separate a supervisor from the four nearest things people confuse it with.

Separation with placement. A supervisor is a separate process. Its own weights, its own memory, its own prompts. But it sits in the main agent’s loop, not outside it. Every proposed action passes through it before execution. The independence is in state. The placement is in the cycle. If the supervisor lives inside the main agent’s reasoning, it isn’t a supervisor. It’s a self-critic, and self-critics fail in correlated ways with the agent they critique.

Deployment-time. Evals are good. Evals aren’t supervisors. Evals tell you how the agent did against a golden set last Tuesday. A supervisor tells you what is happening to a customer’s account right now. Most broken behaviour shows up only in deployment, against the real distribution, with real noise.

Memory across sessions. A guardrail evaluates one request at a time. A supervisor accumulates. It remembers that this particular main agent tried this particular trick three times this month. It remembers which nudges were heeded and which were ignored. Without accumulation, the supervisor is reset every session, and you’ve built a stateless check, not a supervisor.

A taxonomy of failure modes. The supervisor isn’t watching for “anything wrong”. It’s watching for a named, published list of ways this class of main agent is known to fail. Each named failure mode becomes a unit of decomposition. The taxonomy is the foundation of the whole system, and I’ll come back to it.

Authority across a graded set of verdicts. A supervisor is not a dashboard. It’s a process that can let an action through, return a correction, flag for human review, refuse outright, or roll a state back. The graded authority is what separates a supervisor from the observability stack. Observability tells you. A supervisor decides, and the decision feeds back into the main agent’s next plan.

If a system you’re looking at doesn’t have all five of those, it might be a useful tool. It isn’t a supervisor agent.

Why this layer doesn’t exist yet

The agent field for the last two years has been about capability. Can the agent do the task. Can it use tools. Can it maintain context. Can it plan and replan. Can it spawn sub-agents. The frontier question has been “what can this thing do”.

Supervision is a different problem and has moved much slower. A few reasons for the gap.

Supervision is boring. The exciting paper writes itself if your main agent does something new. The supervisor for that main agent doesn’t write a paper, it writes a log line. The incentives haven’t lined up.

Supervision is invisible when it works. A well-supervised agent looks like an agent that did the right thing. The supervisor’s contribution is the bad outcome that didn’t happen, or the nudge that quietly redirected the agent back onto the original task. That’s hard to demo and hard to sell.

The people building agents have mostly been using them in friendly conditions. Their own machine, their own codebase, their own tolerance for weirdness. The failure modes that need supervision show up at scale, under adversarial use, under drift across months. Those conditions haven’t had time to hurt most teams yet. They will.

The fourth reason is the interesting one. The obvious first version of a supervisor agent is broken, and the field hasn’t collectively concluded that yet. People are still trying to make it work.

Why a foundation LLM doesn’t fit

The default move when someone needs a supervisor is to wire a frontier foundation model in as a reviewer. The prompt looks like this:

You are a senior reviewer for an autonomous agent.
Before the agent's action is committed, review it.
Flag anything unsafe, incorrect, off-policy, or harmful.
Be thorough. Be careful. Use your judgment.

This pattern is in every production-agent tutorial right now. It doesn’t work. The reason isn’t that the model isn’t smart enough. The reason is structural, and it splits into three.

The first is correlated failure. If your main agent is Claude and your supervisor is also Claude with a different system prompt, you haven’t added an independent check. You’ve added a correlated one. They share the same training distribution, the same RLHF priors, the same aesthetic preferences. They get charmed by the same kinds of confident output. They miss the same kinds of subtle errors.

The obvious objection is to use a different model family. Claude watches GPT. GPT watches Claude. The failure modes decorrelate, the intuition goes, because the supervisor and the main agent are now different animals.

It doesn’t really work. They decorrelate a little. Not enough.

The frontier models all trained on essentially the same internet. Same code, same papers, same Stack Overflow, same Wikipedia, same GitHub. Their RLHF labellers come from overlapping populations and rate against similar conceptions of “what a good response looks like”. Their architectures are the same architecture class (transformer, autoregressive, next-token prediction). The benchmark suites they optimise against are largely shared, which means the blind spots those benchmarks fail to surface are shared too.

The fundamental failure modes (hallucination under uncertainty, charm by confident output, sycophancy, susceptibility to prompt injection) are properties of the architecture class, not the specific model. Two different frontier LLMs reviewing each other are two slightly different lenses on the same world model. They’re not two independent components. They’re two correlated components with slightly different fingerprints.

This is a basic reliability point. Redundancy doesn’t help if redundant components fail for the same reason. A second engine that fails for the same reason as the first one is not real redundancy. It is duplication. Two transformer-class language models reviewing each other are two engines from the same factory running on the same fuel. Cross-family review is better than same-family review. It is not low-correlation enough to ship downstream guarantees against.

You can argue a model catches its own mistakes some of the time, and a different-family model catches more. True. The question is variance and floor. A supervisor’s job is to be reliable, not averaged-good. Reducing correlation a notch by swapping the model doesn’t get you there.

The right answer isn’t a smarter foundation model. It isn’t a different-family foundation model. It is to stop putting a generalist in this slot.

The shape that works

A supervisor agent should be a decomposition. Not one model with a big prompt. Many small specialists, each watching for exactly one failure mode, each refusing to answer questions outside its domain. The decomposition is the architecture. Everything else falls out of it.

The taxonomy comes first. Before you write a single line of supervisor code, you publish a list of the failure modes for the class of main agent you’re supervising. Not a wish list. A specific, named, exhaustive-as-you-can-make-it catalogue of the ways this kind of agent is known to fail. Each entry has at minimum a short name, a one-line description, an example drawn from a real incident, a severity, a default verdict (nudge for recoverable failures, block for unrecoverable ones), a reference to an existing risk-classification system if one applies (CWE for security, HIPAA for healthcare, SOC2 for ops, MITRE ATT&CK for adversarial behaviour), and a detection-method label like the following:

• deterministic

• small-classifier

• narrow-llm

• hybrid

The taxonomy is published, not internal. Publishing forces precision and forces the field to converge on names. Without shared names, every team builds bespoke supervisors that don’t compose. Shared names are an API.

Each entry in the taxonomy gets exactly one specialist. A specialist is the smallest unit of supervision. It takes one input (a proposed action plus context) and emits one output, a verdict and a confidence, and where the verdict is nudge, a structured correction the main agent can consume. The contract:

// supervisor/specialist.ts
type Verdict = "ok" | "nudge" | "flag" | "block" | "refuse";

interface SpecialistResult {
  verdict:     Verdict;
  confidence:  number;
  reason?:     string;    // what the specialist saw
  correction?: string;    // populated only when verdict is "nudge"
}

interface Specialist {
  name:   string;             // matches a taxonomy entry id
  domain: ActionPredicate;    // declared at construction
  evaluate(
    action:  ProposedAction,
    context: Context,
  ): SpecialistResult;
}

A specialist is allowed to refuse. Refusal isn’t failure. A specialist asked about something outside its taxonomy entry should return refuse with high confidence, and the decision layer treats refusal as “not the right specialist for this”. This is how you keep specialists narrow. A specialist that answers questions outside its domain isn’t narrow anymore, and once it stops being narrow, the properties you bought by decomposing go away.

The decision layer fans out to every specialist in parallel, with a hard per-specialist timeout. Each specialist returns a verdict or refuses. The decision layer aggregates by union, not vote. Any block blocks. Any nudge becomes a correction sent back to the main agent. Any flag flags. Refusals are ignored. Default is ok if no one objected.

Voting is the obvious alternative and it’s wrong. A specialist is responsible for exactly one failure mode. Inside its domain, it is the authority for that failure mode. Asking other specialists to vote on whether it was right is asking experts in unrelated fields to weigh in on something outside their domain. The first specialist to confidently identify its failure mode wins.

Most specialists should not be LLMs. This is the counterintuitive part. For most named failure modes, the right specialist is a deterministic check. A regex over the proposed action. An AST walk over a diff. A SQL query against the system state. A comparison against a hard-coded budget. These checks aren’t glamorous. They’re correct. The set of inputs a deterministic check returns block or nudge for is exactly the set you said it should. No temperature, no jailbreak, no clever phrasing that gets past it. You can’t prompt-inject a regex. You can’t smooth-talk an AST diff. And critically, a deterministic check is structurally different from the main agent in a way no LLM-on-LLM check can be. Its correlation with the main agent’s failure modes is zero by construction.

When the failure mode is too semantic for a deterministic check, the right specialist is a small classifier or a narrow LLM, fine-tuned on that one failure mode and asked only that one question. Not a foundation model with a general prompt. A small model that has seen ten thousand examples of “this exact failure” and “this exact non-failure” and learned to discriminate between them.

In production, the supervisor is something like fifty specialists. Maybe thirty deterministic checks, fifteen small classifiers, five narrow LLM judges. Most are cheap. All are independent. Each one is responsible for one row of the taxonomy. None of them is asked to “review the action”. Each is asked exactly one question.

Specialists run in-loop, synchronously, on every proposed action. The supervisor is a step in the main agent’s cycle. The latency budget is real, and the answer is parallelism and tight per-specialist timeouts. Cheap deterministic specialists fit in any budget. Expensive narrow LLM judges only fit if the budget is wide enough, or if they’re reserved for severity-tier actions where the cost is worth paying.

There’s a second tier of specialists that run after execution, not before. Post-action checks that verify the action’s effects on the world (the database row that was actually written, the file that was actually committed, the message that was actually sent). Those can also flag, record, or initiate a revert. The pre-action specialists shape what the main agent does. The post-action specialists shape what gets undone.

Specialists accumulate per deployment. Each specialist owns a feedback log. Every flag and every nudge is recorded with the proposed action, the context, the correction (if any), whether the main agent heeded it, and (when a human eventually reviews) a label indicating whether the verdict was correct. Over time, the log is the training signal. False positives go into the negative set, new patterns go into the positive set, the specialist gets sharper for this deployment. Per-deployment memory lives here, in the fine-tuning data for the small models and the expanded rule sets for the deterministic ones. Not in a giant prompt.

The taxonomy itself accumulates too. When a human notices a failure mode no specialist caught, the team publishes a new taxonomy entry, builds a specialist for it, and adds it to the supervisor. The supervisor grows. The main agent stays the same.

Three specialists

Three sketches across three domains, to make the shape concrete.

Support agent. Duplicate refund check.

// supervisor/specialists/duplicate_refund_same_order.ts
const duplicateRefund: Specialist = {
  name:   "duplicate_refund_same_order",
  domain: action => action.type === "refund",
  evaluate(action, ctx) {
    const count = ctx.db.queryOne(
      `SELECT COUNT(*) AS n FROM refunds
       WHERE order_id = $1
         AND status   = 'completed'
         AND issued_at > now() - $2 * interval '1 day'`,
      [action.orderId, ctx.refundWindowDays],
    ).n;
    return count >= 1
      ? { verdict: "block", confidence: 1.0, reason: "prior_refund_in_window" }
      : { verdict: "ok",    confidence: 1.0 };
  },
};

No model. A SQL query and a comparison. Correct, fast, cheap, impossible to inject. The verdict is block, not nudge. You can’t issue half a duplicate refund. There’s no correction string that makes this action acceptable. The main agent must replan from scratch.

Research agent. Fabricated-citation check.

// supervisor/specialists/citation_resolves_and_quote_appears.ts
const citationCheck: Specialist = {
  name:   "citation_resolves_and_quote_appears",
  domain: action => action.type === "cite",
  async evaluate(action, ctx) {
    const fetched = await httpGet(action.url, { timeoutMs: 2000 });

    if (fetched.status !== 200) {
      return {
        verdict:    "nudge",
        confidence: 0.9,
        reason:     "source URL did not resolve",
        correction: `The citation URL ${action.url} returned HTTP ${fetched.status}. `
                  + `Find a different source for this claim, or remove the citation.`,
      };
    }

    if (fetched.body.includes(action.quotedText)) {
      return { verdict: "ok", confidence: 1.0 };
    }

    const paraphrase = await narrowParaphraseJudge.run({
      candidate: action.quotedText,
      source:    fetched.body,
      model:     "paraphrase-judge-small-v2",  // single-purpose fine-tune
    });

    return paraphrase === "yes"
      ? { verdict: "ok", confidence: 0.8 }
      : {
          verdict:    "nudge",
          confidence: 0.85,
          reason:     "quoted text not present in source",
          correction: `The quote you cited does not appear in the source at ${action.url}. `
                    + `Either update the quote to match the source verbatim, or replace `
                    + `this citation with one that supports your claim.`,
        };
  },
};

Mostly deterministic. A small fine-tuned model handles only the paraphrase decision because verbatim string match is too strict. The model is trained for paraphrase detection, nothing else. The verdicts here are nudge, not block. A failed citation isn’t catastrophic. The main agent should be told what’s wrong and which fix is acceptable, then replan. The correction string is structured enough that the main agent can act on it without a second human round-trip.

Browsing agent. Out-of-domain click check.

// supervisor/specialists/click_outside_allowed_domains.ts
const outOfDomainClick: Specialist = {
  name:   "click_outside_allowed_domains",
  domain: action => action.type === "click",
  evaluate(action, ctx) {
    const d = parseRegistrableDomain(action.targetUrl);
    return ctx.allowedDomains.includes(d)
      ? { verdict: "ok",    confidence: 1.0 }
      : {
          verdict:    "block",
          confidence: 1.0,
          reason:     "domain_not_in_allowlist",
        };
  },
};

Three lines of real logic. Arithmetic-equivalent. Catches a real failure mode that has caused real harm in real deployments. Block, not nudge. The allowlist is a hard boundary; the agent has no business outside it. The main agent gets the block and has to replan a path that stays inside the allowed surface.

You could write twenty of these in an afternoon for a given main agent. That’s what the supervisor is. Twenty of these. Then forty. Then a hundred. Each one narrow, cheap, named, with the right default verdict for the failure mode it watches.

Hard parts

The architecture above is the easy part to describe. The hard parts are these.

Coverage. How do you know your taxonomy is complete? You don’t. You know it’s incomplete the moment the main agent fails in a way no specialist caught. Coverage is a moving target, and the only way to close the gap is to instrument every supervisor decision and every human override, then look at where the supervisor missed. The log is the input to the next taxonomy revision. Without that loop, the supervisor stops accumulating, and a static supervisor against a learning main agent gets worse over time.

Latency. The supervisor is in the cycle, so every action pays its budget. A one-second supervisor in front of a sub-second main agent is unacceptable for interactive use. The answer is parallelism, tight per-specialist timeouts, and a clear policy on timeout. The right default is timeout = refuse, treated as no-vote, action proceeds. The wrong default is fail-closed on timeout, which turns one slow specialist into a denial-of-service against the main agent. The latency budget is configuration, not a global constant. Different actions get different supervisor budgets; a wire transfer can afford 400ms, a tool-call to retrieve a search result cannot.

Authority. The supervisor can record, flag, nudge, block, or revert. Which authority does it have, for which specialist, on which action? This is a configuration problem, not a code problem, but it’s the configuration that decides whether the supervisor is useful or theatrical. A supervisor that can only record is observability. A supervisor that can flag is a queue for human review. A supervisor that can nudge is a coach. A supervisor that can block is operational. A supervisor that can revert is autonomous. Different deployments need different levels, and the same deployment may need different levels per specialist. The duplicate-refund check might be allowed to block. The citation check might only be allowed to nudge. The browser-domain check might be allowed to block, and the budget check might be allowed to revert.

Nudge compliance. A nudge only works if the main agent heeds it. The supervisor has to detect when the main agent ignores a nudge and re-proposes the same action (or a near-equivalent variant). The mechanic is per-session memory keyed on the specialist that issued the nudge. After N repeats inside a session, the verdict escalates from nudge to block. N is a configuration knob. The default I’d choose is two: a nudge, a second nudge with a sharper correction, then block. Without this escalation, nudges become a corrective lullaby that the main agent learns to ignore.

Replay. A good supervisor produces a portable, deterministic record of every decision. Given the same input, the supervisor must produce the same verdict every time it replays against historical actions. That’s what makes the supervisor itself auditable. The replay artifact has to include the proposed action, the context the specialists saw, every specialist’s verdict, the correction strings for any nudges, the aggregation, and the final decision. It has to be signed, time-stamped, and re-runnable. This is what lets a regulator, a customer, or a future engineer ask “what did the supervisor see at 03:14:09, and why did it allow this action” and get a real answer.

The supervisor for the supervisor. Obvious next question. The answer isn’t fully satisfying. Deterministic specialists supervise themselves by construction; same input, same verdict. Small classifiers are validated against held-out sets with continuous precision/recall monitoring. Narrow LLM judges are sampled and human-reviewed in batch. There’s no perfect answer. The argument is that the supervisor’s surface is small, structured, and narrow enough that the validation problem is tractable in a way the main agent’s isn’t. A foundation-LLM-as-supervisor can’t be validated this way. A union of deterministic checks, small classifiers, and narrow judges can. The validation problem moves from intractable to merely difficult, and merely difficult is the kind of problem we can ship against.

FAQ

Isn’t this just a rules engine with extra steps?
Partially. The deterministic specialists are a rules engine. The small classifiers and narrow LLM judges aren’t, the nudge mechanic isn’t, and the feedback loop that retrains specialists per deployment isn’t. A static rules engine doesn’t accumulate. The supervisor does. Calling it “rules engine plus” undersells the accumulation loop, which decides whether the supervisor gets sharper or stays flat across months.

What if I use a different model family for the supervisor, Claude watching GPT or GPT watching Claude?
The decorrelation is weaker than it looks. Frontier LLMs trained on largely the same internet, the same architecture class, and the same RLHF aesthetic from overlapping rater populations. They share blind spots in correlated ways. Cross-family review is better than same-family review, but it’s still two correlated components, not two independent ones. And the structural problems (variance under broad prompts, no accumulation) don’t go away when you swap the model. The slot itself is wrong for any foundation LLM, regardless of which lab trained it.

Why is nudge a separate verdict instead of just “block with a message”?
Because the contract is different. block says “this action is unacceptable, replan from scratch, the supervisor has no opinion on what you do next”. nudge says “this exact action is unacceptable, but here is a structured correction the supervisor expects you to incorporate into your next plan”. The main agent treats them differently. A block resets the planning context. A nudge feeds back into it. Conflating the two costs you the most productive verdict in the system.

Why union aggregation and not weighted voting?
A specialist is responsible for exactly one failure mode. Inside its domain, it’s the authority. Outside, it refuses. Voting introduces noise from specialists that don’t know the failure mode. Weighted voting just hides the noise under a coefficient. Union is the only aggregation rule that lets every specialist remain the sole authority in its own narrow lane, which is the property the decomposition was supposed to deliver.

What about a giant LLM with retrieval over the failure history?
Three problems. The retrieval prompt grows with deployment age and stops fitting in cache. The retrieval system itself becomes a critical, hard-to-validate component. And the underlying model is still a generalist failing in correlated ways with the main agent. You moved the accumulation problem out of the prompt but you didn’t fix correlated failure or variance. A giant LLM with retrieval is a better foundation-LLM supervisor than one without retrieval. It still isn’t a supervisor.

Don’t evals, guardrails, or a critic already cover this?
Evals run offline, against a fixed dataset, before deployment. Guardrails run inline against the prompt, one request at a time. A critic runs inside the main agent’s own reasoning loop. None of them are a separate, deployment-time, stateful, taxonomy-driven, authoritative process sitting in the main agent’s cycle. Each fills a different slot. The supervisor slot is empty.

Who builds the taxonomy if I’m starting fresh?
You do. You start with the failures you’ve actually seen in your own deployments. You add entries as new failure modes show up. Over time, the cross-team common entries get pulled into a shared taxonomy and the long tail stays per-deployment. The same way CVE and CWE evolved.

Most of the agent infrastructure I see being built right now is doubling down on capability and pretending supervision will figure itself out. It won’t. The supervisor for the autonomous main agent has to be designed, not gestured at, and it has to live inside the cycle, not next to it.

I’m not announcing a roadmap. I’m writing down the object so the conversation can get past “we have a supervisor, we use GPT to watch Claude Code” or whichever cross-family combination is currently in the tutorial. When the field shares the words, it’ll be possible to compare actual supervisors and notice which ones are real. The post is a definition. The work is the specialists, one row of the taxonomy at a time.

The hard part of the next phase of agent deployment isn’t making them smarter. It’s making every autonomous act leave enough structure behind that another process can say yes, here is a better way, here is what to log, or no.

I built it last week because I couldn’t answer either question for my own codebase.

Overview

The pipeline end-to-end:

  git log
    │ null-byte format parse
    ▼
  tagged commits  ←  Co-Authored-By email  →  nine-row model table
    │
    ├──►  hotspots (AI % per directory)
    ├──►  velocity (AI commit size ÷ human commit size)
    │
    ▼
  (model × language) pair  ←  SecLens benchmark  →  blind spots
    │
    ▼
  Risk Score  =  0.4 × AI-coverage  +  0.6 × (1 − language-weighted recall)

Detection

Most developers already produce the core signal. If you use Claude Code, Cursor, Copilot, Codex, Gemini, Devin, or Windsurf, those tools auto-append a Co-Authored-By: line to your commit message whenever the assistant writes or rewrites code. The ground truth for “AI wrote this commit” is already in git log. The scanner reads it.

The whole detection table fits in nine rows:

// src/intelligence/models.ts
const AI_EMAILS: Record<string, ModelFamily> = {
  "noreply@anthropic.com":                { tool: "claude-code", provider: "anthropic", family: "claude" },
  "claude@anthropic.com":                 { tool: "claude-code", provider: "anthropic", family: "claude" },
  "copilot@github.com":                   { tool: "copilot",     provider: "openai",    family: "gpt" },
  "cursor-ai@users.noreply.github.com":   { tool: "cursor",      provider: "cursor",    family: "unknown" },
  "cursor@cursor.sh":                     { tool: "cursor",      provider: "cursor",    family: "unknown" },
  "codeium@codeium.com":                  { tool: "windsurf",    provider: "codeium",   family: "unknown" },
  "devin-ai-integration[bot]@users.noreply.github.com": { tool: "devin", provider: "cognition", family: "unknown" },
  "codex@openai.com":                     { tool: "codex",       provider: "openai",    family: "gpt" },
  "gemini@google.com":                    { tool: "gemini",      provider: "google",    family: "gemini" },
};

Nine email addresses, nine tools. No classifier, no LLM call, no inference. For every commit in your repo, the scanner reads the Co-Authored-By: trailers in the body, looks them up in this table, and tags the commit with the model that wrote it. You can audit the method with git log --grep 'Co-Authored-By' on any repo. Everything else in the tool is variations on “group the tagged commits by X and count.”

Model names inside trailers aren’t standardised, so they get normalised on the way in:

// "Claude Opus 4.6 (1M context)" → "claude-opus-4-6"
// "Claude Sonnet 4.6"            → "claude-sonnet-4-6"
export function extractModelName(coAuthorName: string): string | null {
  if (!coAuthorName.trim()) return null;
  let name = coAuthorName.trim();
  name = name.replace(/\s*\(.*?\)\s*/g, "").trim();   // strip "(1M context)" etc.
  if (!name) return null;
  return name
    .toLowerCase()
    .replace(/[\s.]+/g, "-")
    .replace(/-+/g, "-")
    .replace(/^-|-$/g, "");
}

Parsing git log

I thought this part would be easy. It wasn’t. Commit messages can contain any character: newlines, tabs, quotes, emoji, adversarial trailers, even the output of git log itself. Split on newlines or commas and some commit message somewhere will eat you.

Git log has had a solution forever. Use --format with placeholder bytes that can’t appear in normal text.

// src/scanner/git-log.ts
const RECORD_SEP = "\x1E";   // ASCII record separator (1960s)
const FIELD_SEP  = "\x00";   // null byte

// %x00 between fields, %x1E between records. Git expands these to real bytes.
const format = "%H%x00%aN%x00%aE%x00%aI%x00%s%x00%b%x1E";

const raw = execFileSync("git", [
  "log", "--all",
  "-n", String(maxCommits),
  `--format=${format}`,
  "--numstat",
], { cwd: repoPath, maxBuffer: 100 * 1024 * 1024, encoding: "utf-8" });

\x00 and \x1E are the original record separators from ASCII. They exist to split records unambiguously. They almost never appear inside commit messages, because you can’t type them on a keyboard. Parsing becomes raw.split("\x1E").map(r => r.split("\x00")). No regex acrobatics, no shell-quote hell. --numstat gets you line-count stats per file on the same command, same parser, a few extra lines.

Hotspots

Once every commit has a detection tag, the question shifts from “how much” to “where”. A 61%-AI repo with AI work spread evenly is different from a 61%-AI repo where one directory is 100% AI and the rest is all human. The second one is a risk surface.

The hotspot computation walks every analysed commit, groups line additions by top-level directory, and keeps anything above 30% AI:

// src/scanner/insights.ts
function computeHotspots(analyzed: AnalyzedCommit[]): AiHotspot[] {
  const dirs = new Map<string, { ai: number; human: number }>();

  for (const { commit, detection } of analyzed) {
    const isAi = detection !== null;
    for (const file of commit.filesChanged) {
      const dir = getDirectory(file.path);
      const entry = dirs.get(dir) ?? { ai: 0, human: 0 };
      if (isAi) entry.ai    += file.additions;
      else      entry.human += file.additions;
      dirs.set(dir, entry);
    }
  }

  const hotspots: AiHotspot[] = [];
  for (const [directory, { ai, human }] of dirs) {
    const total = ai + human;
    if (total < 20) continue;               // skip trivial dirs
    hotspots.push({ directory, aiLines: ai, totalLines: total, aiPercentage: ai / total });
  }

  return hotspots
    .filter(h => h.aiPercentage > 0.3)
    .sort((a, b) => b.aiPercentage - a.aiPercentage)
    .slice(0, 5);
}

On my own backend repo this surfaced apps/analytics, apps/intelligence, and apps/realtime at 100% AI. I had noticed the 53% top-line. I had not noticed that three entire directories were pure Claude.

Risk scoring

I went back and forth on how to score “risk” and landed on a weighted sum of two factors, with the blind-spot term carrying more weight:

// src/scoring/index.ts
// Risk Score = AI Coverage (40%) + Language-Weighted Blind Spot Severity (60%)
const confirmedCommits = aiCommits - heuristicCommits;
const weightedAI = confirmedCommits + heuristicCommits * 0.6;
const aiCoverage = totalCommits > 0 ? Math.min(weightedAI / totalCommits, 1) : 0;

const blindSpotSeverity = 1 - languageWeightedRecall;

const raw   = aiCoverage * 0.4 + blindSpotSeverity * 0.6;
const score = Math.round(raw * 100);

const grade =
  score >= 75 ? "F" :
  score >= 60 ? "D" :
  score >= 45 ? "C" :
  score >= 25 ? "B" : "A";

Heuristic commits get weighted at 0.6. Trailer-based detection is ground truth. A commit either has the trailer or it doesn’t. The heuristic detector (mass-add diff shape plus AST-level structural tells that tree-sitter picks up from AI-generated code) is noisier, so its contributions are discounted in the coverage factor. I trust it less than the trailer.

Blind-spot severity uses language-weighted recall, not generic category scores. Recall means: when you run a model against OWASP-seeded vulnerable code, what fraction does it catch? A Python-heavy repo with Claude Opus 4.6 (63% Python recall on SecLens) scores differently from a JavaScript-heavy repo with the same model (31% JavaScript recall). The severity weighting follows the actual language mix of your code.

Blind-spot severity uses language-weighted recall, not generic category scores. Recall means: when you run a model against OWASP-seeded vulnerable code, what fraction does it catch? A Python-heavy repo with Claude Opus 4.6 (63% Python recall on SecLens) scores differently from a JavaScript-heavy repo with the same model (31% JavaScript recall). The severity weighting follows the actual language mix of your code.

SecLens is the benchmark feeding those recall numbers. 12 models × 8 OWASP Top 10 categories × 10 languages × 35 scoring dimensions, running known-bad code through each model and counting what they catch. A slice of the recall table:

| Model              | Python | JavaScript | Java  | Go    | Overall |
|--------------------|--------|------------|-------|-------|---------|
| Claude Opus 4.6    | 62.5%  | 31.2%      | 27.8% | 55.6% | 39.0%   |
| Claude Sonnet 4.6  | 70.8%  | 62.5%      | 61.1% | 85.2% | 42.1%   |
| Claude Haiku 4.5   | 70.8%  | 68.8%      | 77.8% | 85.2% | 37.8%   |
| GPT-5.4            |  8.3%  |  0.0%      |  5.6% | 14.8% | 31.1%   |
| Gemini 3.1 Pro     | 83.3%  | 75.0%      | 77.8% | 70.4% | 45.8%   |

Two things surprised me on first read. Gemini 3.1 Pro beats the Claude family overall. And GPT-5.4 has near-zero recall on three of these four languages, which is not where I would have placed it going in. The risk from AI blind spots is heavily language-dependent, and it rarely matches the intuitive model leaderboard.

Results

Results

I ran it on overwatch-backend, a Django-ish service, 74 commits over six weeks:

The 3.0x commit-size ratio is the number that stuck with me. AI commits average three times the size of human commits. Three times more lines per commit for me or a reviewer to read. The real question is how much of that I reviewed, and that scales in the opposite direction from my attention budget.

The top-line AI percentage is a vibe. The review-delegation estimate (AI-authorship percentage combined with commit-size ratio) is the accountability question. You can ship 100% AI code if you reviewed every hunk. You can ship 30% AI code and be worse off if those hunks were merged unread.

The name

I posted this tool to r/ClaudeAI on Tuesday. I called it vibe-check. That was a joke. Zero upvotes. A couple of people came after me. One called it a pattern of dishonesty, then asked me a question I keep coming back to: “what would the non-hedged version of this post look like to you?” I answered honestly. They came back softer. This post is part of that answer.

The real name is AI authorship. I renamed the npm package:

npx @mattersec/ai-authorship scan

MIT, runs locally on your .git, no telemetry. Repo: https://github.com/mattersec-labs/ai-authorship

Limitations

Limitations I know about:

• Trailer-based detection is ground truth only if trailers aren’t stripped. git commit --amend with a manual rewrite removes them. Developers who want to hide AI authorship can. The heuristic is the attempt to catch that case but it’s noisy, which is why it’s weighted at 0.6.

• Stylistic tells per model are tuned on Claude. Detection is strongest on Claude-heavy repos. Other models are supported but noisier. I have been staring at Claude output for a few hundred hours and it shows.

• Newest models (released in the last month or two) don’t have full SecLens coverage yet. If your scan lands on one, you’ll see an unknown model fallback in the blind-spot block.

• The 3.0x commit-size ratio is a proxy, not a direct measurement of unreviewed code. I want to correlate against review traces (who opened which PR, who squashed what, who LGTM’d without comment), but that needs GitHub API data I haven’t integrated yet.

FAQ

Can developers strip the trailers to hide AI authorship?
Yes. git commit --amend with a manual rewrite removes them, and git filter-repo does it at scale. The heuristic detector is the attempt to catch the rewrite case, but it’s noisier than trailer matching. Heuristic commits are discounted to 0.6× in the coverage factor for that reason. If you want to hide AI authorship, you can. The tool is built on the assumption that most people don’t bother.

How is this different from git log | grep Claude | wc -l?
Not that different for the top-line number. Three things the scanner adds: (1) mapping trailer emails to the right model/provider via the nine-row table, (2) per-directory hotspot computation, so you can see where the AI code is concentrated instead of only how much, and (3) cross-referencing the detected (model × language) pair against SecLens to surface blind spots specific to your repo’s language mix. If all you want is the top-line, git log --grep is fine.

Why is blind-spot severity weighted higher than AI coverage (60/40)?
The thing that damages you is not that AI wrote your code. It is what the model writing your code fails to write safely. A repo 90% written by a model with 90% OWASP recall is safer than a repo 50% written by a model with 20% recall. Coverage tells you how much of a problem could exist. Severity tells you how bad the problem is if it does. The formula prioritises the second.

My AI tool isn’t in the nine-row table. What happens?
The commit falls through trailer-detection into the heuristic pipeline, which flags on diff shape and AST tells. That catches some of it and misses some of it. If the tool emits a stable Co-Authored-By: email, PR a new row. The table is the only thing that needs updating.

Does it work on rewritten history (rebase, squash merge)?
Partly. It reads whatever is in git log at scan time. If the rebase or squash preserved the trailers on the final commit, they get counted. If the squash dropped them, the heuristic may flag the commit as Likely AI based on diff shape, or it may miss. Rewritten history is the known soft spot.

Month three of building again, alone. I keep noticing versions of this same problem. Shipping security tools for the last thirteen years had a familiar shape: see something nobody else had seen, plan, staff, build the instrument for the view, then use it months later. This weekend I wrote the question, an agent wrote the instrument, and I used it the same day. The tool that measures Claude’s authorship was, funnily enough, built with Claude Code.

I’m not announcing a cadence. I am shipping instruments for things I suspect we should be looking at and haven’t. AI authorship is the first one. There will be others. Some will be wrong. The repo is MIT, the scanner runs locally, and I’m reading the comments.

The next twelve years are going to look nothing like the last twelve. I’d rather write while I figure out why than after.