What it is : A pack of evaluations shaped for the openai/evals registry. Twenty are enterprise-flavored variants of patterns the public eval corpus already touches, filled in with scenarios that look like real applied work. The three novel evaluations are the point of the project. Each one targets a specific failure mode that general benchmarks do not catch.
Why these three : Capability benchmarks like MMLU, HumanEval, and GSM8K measure what a model knows and can do on well-posed tasks. Production failure modes rarely look like that. They look like: a tool call that is almost-but-not-quite valid JSON, a streaming response that gets cancelled halfway, an argument to a tool that quietly contains data the model should never have surfaced. These three evals are built around those shapes.
Novel eval 1: Tool-use trajectory
What it detects : Scores not the final answer but the sequence of tool calls that produced it. Many failures look correct in the final response and wrong in the trajectory: unnecessary calls, retry loops that mask a bad plan, dropped intermediate state.
How it works : Each scenario specifies a target trajectory as a graph with required nodes, forbidden nodes, and ordering constraints. The eval compares the observed trajectory against the graph and emits a structured score plus a diff. Pass criteria are strict: every required node must appear in order, no forbidden nodes.
Baseline rates : [TODO: table with Sonnet 4.6, Opus 4.7, GPT-4.X across the trajectory suite. Columns: pass rate, mean extraneous calls, mean missing calls.]
Novel eval 2: Streaming-cancel JSON validity
What it detects : Measures whether the partial content emitted up to the point of cancellation is still parseable and semantically coherent. Streaming is standard; cancellation is common; the intersection is almost never tested.
How it works : Scenarios are driven at a scripted cancellation point. The eval captures everything emitted before the cancel and runs it through a strict JSON parser plus a schema check. Pass requires valid JSON and schema conformance of the partial, not just syntactic completeness.
Baseline rates : [TODO: table with cancel-at-25%, cancel-at-50%, cancel-at-75% break points across the three models.]
Novel eval 3: PII leakage through tool arguments
What it detects : The model is given a prompt that contains sensitive data in the context (synthetic SSNs, email addresses, API-key-shaped strings). The task is unrelated to that data. The eval checks whether sensitive data appears inside the arguments of outbound tool calls, even when the final response is clean.
Why this matters : Response-level PII filters catch what the user sees. Tool-level PII leaks go to third-party services (webhooks, external APIs) and are invisible to anyone just reading the final chat. This eval surfaces that path.
How it works : Each scenario seeds the context with tagged synthetic sensitive values. Tool arguments are scanned for exact and fuzzy matches. The eval reports a hit rate per sensitive-value type, with per-scenario breakdowns.
Baseline rates : [TODO: table with SSN-shaped, email-shaped, key-shaped tagged values across the three models. Flag any scenarios where leak rate exceeds 1 percent.]
Writeup :
This eval deserves a HackerOne-style disclosure note per high-leak scenario: trigger prompt, observed behavior, severity reasoning, suggested model-side mitigation. See the novel-evals/pii-via-tools/findings/ directory in the repo.
Running the pack
Compatible with the openai/evals registry format. A Claude-side adapter runs the same scenarios against the Anthropic API with prompt caching enabled on the system portion. Both paths produce comparable score files.
oaieval <model> <scenario>for OpenAI-side runspython -m enterprise_evals.claude_runner <model> <scenario>for Claude-side runs- Aggregate script produces the cross-model comparison tables
What this does not do :
- Not a comprehensive safety evaluation suite. Covers three specific failure modes, not all of them.
- The synthetic PII does not represent real distributions of leaked data. Pattern-matching results are indicative, not definitive.
- Trajectory scoring assumes the target graph is correct. Mis-specified graphs produce mis-specified scores.
- Not sponsored, endorsed by, or affiliated with OpenAI or Anthropic. Uses the openai/evals format because it is the available de-facto standard.
What's next :
- Measured cross-model tables populated with real numbers from a scheduled quarterly run.
- Writeup per PII-leak finding published in the repo.
- Blog post: "Three evals the public corpus is missing" summarising the three with sample results and reproduction instructions.
- Expansion: add a cancellation-at-adversarial-point variant and a tool-shadowing trajectory variant.
Tech stack :
Runtime: Python 3.12
Format: openai/evals YAML scenario files, plus a Claude-side runner against the Anthropic SDK
Scoring: Scripted deterministic scoring; no LLM-as-judge in the primary metrics
Caching: Anthropic prompt caching on system prompts and tool catalogs
Output: JSON score files per run, aggregate tables via a reporting script