PatentBench

PatentBench is the first open-source, reproducible benchmark specifically designed for evaluating AI systems on patent prosecution tasks. It measures four capability dimensions across 7,200+ test cases derived from 604 real USPTO Office Actions spanning 9 technology centers and 5 technical domains. Unlike vendor self-reported accuracy claims, PatentBench provides a shared evaluation framework with expert-validated rubrics so practitioners can compare AI tools on equal footing.

Patsnap’s PatentBench evaluates AI performance on patent novelty search (finding prior art references). ABIGAIL’s PatentBench evaluates AI performance on patent prosecution tasks: parsing Office Actions, constructing legal arguments (35 USC 101, 102, 103, 112), drafting claim amendments, and detecting hallucinations. They measure different things. Patsnap’s benchmark asks "can the AI find relevant prior art?" ABIGAIL’s benchmark asks "can the AI draft a filing-ready Office Action response that a practicing attorney would accept?"

PatentBench evaluates four capability dimensions: (1) OA Parsing, can the AI correctly extract rejection types, cited references, claim mappings, and examiner reasoning from raw Office Action text? (2) Legal Argument Construction, can the AI construct a sound legal argument to overcome a specific rejection ground? (3) Claim Amendment Drafting, can the AI propose amendments that address the rejection without introducing new matter under 35 USC 132? (4) Hallucination Detection, does the AI fabricate MPEP sections, case citations, prior art references, or specification passages?

PatentBench includes 7,200+ individual test cases derived from 604 real Office Action cases across 9 USPTO Technology Centers and 5 technical domains (software/electrical, mechanical, chemical, biotechnology, and business methods). Test cases are organized into five difficulty tiers calibrated against real practitioner roles, from deterministic tasks (deadline calculations, fee lookups) to strategic reasoning (prosecution strategy selection, appeal vs. amendment decisions).

PatentBench uses a four-layer evaluation architecture: Layer 1 (Deterministic), automated binary pass/fail checks for objectively verifiable outputs like deadline dates and fee amounts. Layer 2 (LLM-as-Judge), a separate evaluation LLM scores outputs against rubrics for completeness, legal accuracy, and argument quality. Layer 3 (Comparative), head-to-head ranking of multiple systems on the same test case using Elo-style ratings. Layer 4 (Human Expert), USPTO-registered patent attorneys with 5+ years experience evaluate whether the output would be acceptable to file as-is, acceptable with minor edits, or requires substantial rework.

Every citation in every AI-generated output is verified against source documents. MPEP section numbers are checked against the current MPEP. Prior art column/line citations are verified against actual patent documents. Specification paragraph references are matched against the application as filed. Hallucinations are classified by severity: Critical (fabricated legal authority like fake MPEP sections or invented case law), High (non-existent prior art citations), Medium (correct source but wrong location), Low (minor paraphrasing inaccuracies). Any Critical-severity hallucination is an automatic benchmark failure for that test case, regardless of overall output quality.

Yes. PatentBench is open for submissions. AI tool vendors, academic researchers, and law firm innovation teams can participate by running the benchmark test suite against their system and submitting results. The methodology, test cases, and evaluation rubrics are published openly. Participation details and the submission process are available on the PatentBench GitHub repository and in the PatentBench White Paper.

Yes. The benchmark methodology, evaluation rubrics, scoring framework, and test case structure are open source and available on GitHub. The goal is to create a shared standard that the entire patent AI community can use and contribute to, similar to how SWE-bench serves the software engineering AI community and MedQA serves medical AI. Academic researchers who contribute to PatentBench evaluation or methodology may be offered co-authorship on published papers.

The PatentBench leaderboard ranks AI patent prosecution systems by their aggregate scores across all four capability dimensions and five difficulty tiers. It provides transparent, side-by-side comparison of tools on identical test cases using identical rubrics. The leaderboard is live at abigail.app/patentbench and is updated as new submissions are evaluated.

Every patent AI vendor claims "95% accuracy" or "attorney-quality output" but none can prove it because there has been no shared evaluation framework. Vendors grade their own homework. PatentBench exists to give practitioners a way to verify claims, compare tools objectively, and hold vendors (including ABIGAIL) accountable to measurable standards. The benchmark was created by Roger Hahn, a USPTO Registered Patent Attorney (Reg. No. 46,376) with 25+ years of prosecution experience, to bring the same rigor to patent AI evaluation that benchmarks like SWE-bench brought to software engineering AI.