Measured on every IPC benchmark problem
We benchmarked our planner against 14 other planners on all 6,384 problems from 11 International Planning Competition tracks, classical, numeric and temporal, under identical limits.
Problems solved
Problems each planner solved with a plan the VAL validator accepts, within 60 seconds and 4 GB of memory on one core. A dash means the planner does not support that kind of planning.
| Planner | Classical | Numeric | Temporal | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1998–20182,742 | 2023140 | 2023400 | 2026320 | 2002436 | 2004976 | 2006450 | 2008390 | 2011240 | 2014200 | 201890 | |
| Autoplan | 2,450 | 109 | 383 | 270 | 338 | 715 | 308 | 348 | 218 | 181 | 86 |
| LPG-td1.4 | 1,297 | 20 | 152 | 61 | 324 | 445 | 192 | 264 | 126 | 119 | 63 |
| Count-Downward42de017 | 1,914 | 32 | 223 | 234 | – | – | – | – | – | – | – |
| LAMA24.06.1 | 2,209 | 56 | – | – | – | – | – | – | – | – | – |
| OPTICgcc8-2019-01-08 | 824 | 0 | 72 | 43 | 141 | 361 | 141 | 201 | 89 | 26 | 39 |
| POPFpopf2-11jun2011 | 751 | 0 | 56 | 30 | 132 | 343 | 141 | 213 | 110 | 41 | 36 |
| Metric-FF2.1 | 1,383 | 18 | 129 | 124 | – | – | – | – | – | – | – |
| ENHSP20-0.21.0 | 1,041 | 14 | 195 | 147 | – | – | – | – | – | – | – |
| Panino-LC67c8c08 | 1,316 | 17 | – | – | – | – | – | – | – | – | – |
| Tyr0.0.15 | 1,111 | 9 | 109 | 66 | – | – | – | – | – | – | – |
| TFD0.4 | 0 | 0 | – | – | 157 | 97 | 140 | 323 | 140 | 83 | 44 |
| Pattya5001cc | 197 | 0 | 279 | 100 | – | – | – | – | – | – | – |
| Panino67c8c08 | – | – | 290 | 217 | – | – | – | – | – | – | – |
| UPMurphiabdd3fc | – | – | 10 | 11 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| DiNof95f07d | – | – | 5 | 2 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
The highlighted number in each column is the most problems any single planner solved there.
Domain by domain
Each domain against the planner that solved the most of it. The grey bar is that planner, the blue bar is Autoplan.
The best plan found within 60 seconds
Every planner improves its plan for the full minute where it can, and we compare the best plan each one finds. Quality is the problem's own metric, such as total cost or makespan, or the number of steps where the problem states none.
Each dot is one domain: Autoplan's plan cost divided by that of the planner that solves the most of the domain, taken over the problems both solve and averaged geometrically. Left of the line, our plans are cheaper.
Wall-clock time to the first plan
How long each planner takes to produce its first valid plan, from start-up to the plan file, on the problems both solve. Left of the line, Autoplan is faster.
How we measured
Every problem from eleven International Planning Competition tracks, 1998 to 2026: classical, numeric and temporal, 6,384 in all, as published.
One planner per container, pinned to its own performance core of an Intel Core i7-12700F (12th generation, up to 4.9 GHz), with 4 GB of memory, no swap and 60 seconds of wall-clock time. Every planner ran under exactly the same limits on the same machine.
A problem counts as solved only when the VAL validator accepts the plan. Plan quality is VAL's value of the problem's metric. VAL rejects one domain outright (data-network 2018) and misjudges one constraint (tpp-metric-time-constraints 2006) for every planner; both are counted as published.
Each planner in its authors' satisficing configuration, given the time limit where it takes one. Where a planner writes several plans as it improves, its best valid plan counts for quality and its first for speed. Autoplan's coverage and speed come from its default run, its plan quality from a run with --anytime.
Who built each one, and how it works: see the planners.
Version 0.3.606, November 2026.