Introducing PatentBench:
How We Built the First Patent Prosecution AI Benchmark

The Benchmark Desert in Patent AI

Software engineering has SWE-bench. Medical AI has MedQA. Legal reasoning has LegalBench. But patent prosecution -- the multi-billion-dollar practice of responding to USPTO Office Actions -- has had zero standardized benchmarks. Until now.

Every patent AI vendor claims "95% accuracy" or "attorney-quality output." None of them can prove it. There has been no shared evaluation framework, no common test set, and no way for practitioners to compare tools on equal footing. Vendors grade their own homework, and attorneys have no way to verify the claims.

We built PatentBench to fix this. It is the first open-source, reproducible benchmark specifically designed for evaluating AI systems on patent prosecution tasks. It measures what actually matters: Can the AI correctly parse an Office Action? Can it construct a legally sound 35 U.S.C. §103 argument? Can it draft claim amendments without introducing new matter? And critically -- does it hallucinate citations?

Four Capability Dimensions

Patent prosecution is not a single task. It requires distinct capabilities that must be evaluated independently. PatentBench measures AI systems across four dimensions:

Each dimension is scored independently, producing a four-axis capability profile for every AI system tested. A tool might excel at parsing but fail at amendments -- PatentBench reveals these asymmetries instead of hiding them behind a single aggregate score.

Five Difficulty Tiers

Not all patent prosecution tasks are equally difficult. A deadline calculation is fundamentally different from constructing a novel non-obviousness argument. PatentBench defines five difficulty tiers calibrated against real practitioner roles:

The tiered design serves two purposes. First, it provides meaningful baselines: any production system should achieve 100% on Tier 1 tasks. Second, it reveals where AI systems plateau. Most current models handle Tiers 1-2 well but degrade significantly at Tier 4+, where legal judgment and strategic reasoning dominate.

Four-Layer Evaluation Architecture

A benchmark is only as good as its evaluation methodology. PatentBench uses a four-layer evaluation architecture where each layer catches failures the previous layer cannot:

The layered approach means deterministic checks catch obvious failures before expensive LLM evaluation runs. Layer 2 handles the bulk of quality assessment at scale. Layer 3 provides relative ranking. Layer 4 provides absolute calibration against the standard that ultimately matters: would a practicing attorney accept this output?

Anti-Hallucination Methodology

Hallucination in patent prosecution is not merely inconvenient -- it is an ethical violation. Citing a fabricated MPEP section in a USPTO filing is sanctionable conduct. PatentBench treats hallucination as a first-class failure mode with dedicated testing methodology.

Any Critical-severity hallucination is an automatic benchmark failure for that test case, regardless of how good the rest of the output is. This reflects real-world stakes: one fabricated MPEP citation in a filing can trigger Rule 11 sanctions.

Real Data: 604 Cases Across 9 Technology Centers

PatentBench is built on real patent prosecution data, not synthetic examples. We pulled 604 benchmark cases from 98 real patent applications using the USPTO Open Data Portal (ODP) API, spanning all major technology areas.

The dataset covers the full spectrum of patent prosecution scenarios:

Every case includes the original Office Action, the cited prior art references, the application-as-filed claims, the specification, and -- where available -- the applicant's actual response. This provides both the input for AI evaluation and a human baseline for comparison.

PatentBench-Mini: 300 Initial Test Cases

The full PatentBench dataset is large. For rapid iteration and initial evaluation, we release PatentBench-Mini -- a curated subset of 300 test cases that balances coverage across technology centers, difficulty tiers, and rejection types.

PatentBench-Mini is designed for fast runs. A full evaluation takes approximately 2-4 hours on a single machine with API access to the model under test. The mini subset uses stratified sampling to ensure proportional representation:

• Proportional coverage across all 9 Technology Centers
• Balanced distribution across 5 difficulty tiers
• At least 20 poison-pill test cases for hallucination detection
• Mix of 101, 102, 103, and 112 rejection types
• Both single-reference and multi-reference 103 rejections

How to Contribute

PatentBench is open source and community-driven. We need contributions from patent practitioners, AI researchers, and developers to make it the definitive standard for patent AI evaluation.

Get Started

PatentBench is available as a Python package and as a raw dataset. Install it and run your first evaluation in under five minutes:

# Install PatentBench
pip install patentbench

# Run PatentBench-Mini evaluation
patentbench evaluate --model your-model --suite mini

# Run full benchmark
patentbench evaluate --model your-model --suite full

# View results
patentbench report --output results.html

Frequently Asked Questions