How accurate is pdf2takeoff? Every benchmark, published
pdf2takeoff is accurate at measuring from PDFs exported from CAD (99.8% on our ScaledDrawingBench) and much weaker on degraded scans (37.3%) and on counting doors, windows and rooms without an AI model. We test it on four benchmarks, three external, and publish every number with its method, including where it falls short.
Every benchmark at a glance
| Benchmark | What it tests | pdf2takeoff | For comparison |
|---|---|---|---|
| AECV-Bench | Counting doors, windows and rooms on 120 floor plans (five-field exact match) | 0.295engine alone, no AI model0.585with the opt-in AI count on GPT-6.1 Sol | 0.572same model, the benchmark's own prompt0.543best published model |
| AEC-Geometric-Bench | Finding walls on 15 construction sheets (wall pixel F1) | 0.519no AI model, laptop CPU | 0.287best general AI model0.931a production pipeline built for these sheets |
| ScaledDrawingBench (ours) | Scale, length, area, count and revision questions on 200 sheets | 99.8%PDFs exported from CAD37.3%degraded scans0 of 92unanswerable questions answered anyway | No outside scores yet |
| ResPlan-FP | Wall geometry on the 1,000-plan test split (wall F1) | 0.635engine alone, no model | 0.968model fine-tuned for this dataset0.811Gemini 3.1 Pro, 200-plan subset |
How to check these numbers yourself
- Get the code. Clone github.com/ajflow/pdf2takeoff and run npm install (Node 22.13 or later).
- Rebuild ScaledDrawingBench. Run npm run bench:scaled -- generate to rebuild the 200 sheets from a fixed seed.
- Score the engine. Run npm run bench:scaled -- reference --set bench/scaled-drawing-bench/v0.1 to answer and score every question.
- Run the external benchmarks. Fetch each external dataset from its publisher and follow its method page, which names the harness and settings.
AECV-Bench: can it count doors, windows and rooms?
With no vision model, the engine scores 0.295 five-field exact match: bedrooms are right 71% of the time, but doors and windows only 11 to 12%. With the opt-in AI count on GPT-6.1 Sol it scores 0.585, against 0.572 for the same model on the benchmark's own prompt and 0.543 for the best published model. Most of that gain is the model, not pdf2takeoff. pdf2takeoff adds checkability: each count arrives as items on the drawing for a person to confirm, and when repeated runs disagree, the count is right only 25% of the time, against 75% when they agree.
AEC-Geometric-Bench: can it find walls?
On 15 construction sheets the engine finds walls with a pixel F1 of 0.519, about 1.8 times the best general AI model (0.287), with no clicks. A production pipeline built for these sheets scores 0.931, and our object score is poor (F1 0.031): the door finder over-fires on real drawings, and fixtures are not attempted.
ScaledDrawingBench: can it measure from a scaled drawing?
No public benchmark tested measuring from a scaled drawing, so we built one and released it openly under CC BY 4.0: 200 synthetic sheets, half CAD-exported PDFs and half degraded scans, with about 2,000 questions. The engine scores 99.8% on the CAD half and 37.3% on the scans, where blur hurts most. On 92 questions with no valid answer it answered none: without a readable scale it abstains. It is synthetic and we wrote it, so it is not independent proof.
ResPlan-FP: how does it compare with trained models?
On the 1,000-plan test split of clean residential plans, the engine scores a wall F1 of 0.635. A model fine-tuned for this dataset scores 0.968, and Gemini 3.1 Pro 0.811 on a 200-plan subset, where we score 0.647. That is a published gap, not a lead.
Data and licensing
We publish numbers, predictions and harnesses, never images we have no right to share. AECV-Bench has no licence and most of its plans derive from CubiCasa5K (CC BY-NC 4.0), so we run it locally and publish scores and predicted counts only. The harnesses and our predictions are in the repository: see the evals guide and the full ScaledDrawingBench results.
FAQ
How accurate is pdf2takeoff?
On PDFs exported from CAD, very: 99.8% on our ScaledDrawingBench. On degraded scans, 37.3%. Counting doors, windows and rooms is harder: 0.295 exact match for the engine alone, and 0.585 with an opt-in AI model.
Can AI do a takeoff on its own?
Not reliably. The best published model gets 54.3% of floor plan counts exactly right on AECV-Bench. pdf2takeoff uses AI to propose items that a person confirms.
Is the AI count score down to pdf2takeoff or the model?
Mostly the model. GPT-6.1 Sol scores 0.572 on the benchmark’s own prompt and 0.585 through pdf2takeoff, and that gap is within the margin of error.
Why publish the weak results?
Because estimators need to know where to check, and a page that only shows wins cannot tell you when to trust a number.
See how it does on your drawings
Benchmarks are averages: trust the engine on CAD-exported PDFs, and check it on scans and counts.
Related guides
- Takeoff from photos and scans: images, scanned PDFs, phone photos and screenshots.
- MCP server for construction takeoff: let Claude, Cursor or Codex run the takeoff for you to review.
- Set the scale on a PDF drawing: presets, Auto-read and the check that tells you whether to trust it.
- pdf2takeoff home: free construction takeoff software that runs in your browser.