How the reproducibility score works

What the score measures

The score answers one question: if you follow what the project publishes, how much of the robot can you build? It looks at the published design files, parts list, wiring, control code, build instructions and hardware license, then at whether anyone has built it. It does not rate how good the robot is, how much it costs or how popular it is.

The same scoring function runs everywhere a score appears: robot pages, the atlas, the public API, the README badge, the command line tool and the GitHub Action. Scores are not set by hand.

The seven criteria

Each criterion has a weight, and the weights add up to 100. A criterion that is met earns its full weight, one that is partly met earns half, and one that is missing earns nothing. The score is the sum, rounded.

Quality tiers

Within each criterion the engine records a quality tier: how much of what a builder needs the evidence actually holds. A criterion is met once it reaches the tier marked below, partly met at any lower tier above zero, and missing at tier zero. No criterion earns credit without evidence the engine can point to: a file at the commit it read, an entry inside an archive, a release asset or a linked page.

A parts-list feature counts only when it covers most rows, and a list of fewer than five rows stops at tier 2. A vendor link is a link to where the part can be bought, or a supplier column; a manufacturer column is not. A total counts only for a table that covers the robot, not one subsystem (the consumables, one electronics order, a stand or test jig, the power supply), and the robot's own table decides whenever one exists; a circuit board's parts list alone stays at the lowest tier. A computed total needs every row's quantity; without them only a total the table states counts.

Which sources count

Each robot lists its sources with a role. Design evidence (CAD, parts list, wiring, build instructions) counts only from its hardware, parts-list, documentation and build-guide sources. Control code also counts from its own software sources. A simulation package, a software repository's docs and a related project (an upstream design it remixes) never count as the robot's hardware. A manual or guide file counts when the project's docs link it as the build guide, or the listing declares it as one; a video counts when its link names the robot and its build or assembly. Files stored with Git LFS count only when GitHub serves them.

The cap, and how reviewed builds lift it

Published files can only show that a robot is buildable on paper. Until at least one person has built it, the score stops at 89, so the top band of 90 and above always means someone reproduced the robot.

A build counts once its report has been reviewed: by the robot's maintainer when the page is claimed, otherwise by a Motari admin. Pending reports count for nothing, and nobody can approve their own report. One to four reviewed builds half meet the "Known reproductions" criterion and lift the cap; five or more meet it in full. Nothing else can raise that criterion.

Reviewed overrides

The engine reads what it can reach at a pinned commit. Some projects keep their CAD on Onshape, their parts list in a spreadsheet or their build guide on a docs site, which the engine records but does not open. A reviewer may then correct one criterion, under strict rules:

• Every override cites a public https evidence URL that the reviewer opened and checked, plus a note saying what it shows and the date of the review.
• The robot page marks the criterion "Reviewed" and shows the evidence link, the note, the date and what the engine found on its own, side by side.
• Overrides go both ways. They also lower a criterion the engine credited in error, for example a software license read as the hardware license.
• An override must change the engine's verdict, there is at most one per criterion, and the tier it gives must match its state. Known reproductions can never be overridden.
• Where no concrete evidence exists, the engine's verdict stands, even when that lowers the score.

The license gate

The license criterion reads the hardware license only: the terms on the design files. A software repository's license does not count when the hardware lives elsewhere, and neither does a documentation license.

• An OSHW-compatible license that covers the hardware (CERN-OHL, CC BY, CC BY-SA, MIT, Apache, BSD, GPL) meets it.
• Recognized terms that are not OSHW-compatible, or that cover only part of the hardware, partly meet it.
• Non-commercial (NC) or no-derivatives (ND) terms, or no license at all, earn nothing.

The same reading decides whether a robot can ever be offered as a commercial kit. Non-commercial, no-derivatives and unknown hardware licenses never qualify.

Reproducible by anyone

Each robot page names the engine version and the commit it read, for example "Engine v3.1.1 at abc1234". The engine is deterministic: the same sources at the same commits give the same result, byte for byte. Motari keeps a snapshot of each run, and evidence links point at the files as they were at that commit, so a later change upstream never silently changes a score. Scores move when the snapshot is rerun on new commits or a reviewer adds evidence.

Known engine gaps

These are the places where the engine is honest but limited. Each is a reason a score may be lower than the project deserves, and a reason for a reviewed override.

• Onshape and Fusion documents, Google Drive folders, Google Sheets, docs sites, wikis, Thingiverse, Printables and Instructables are recorded as linked but not opened.
• ZIP archives are listed and credited by their contents. Other archives (tar.gz, 7z, rar) are not opened.
• Diagrams are recognized by their file names; a diagram inside a PDF named otherwise is missed.
• PDFs are credited as documents, but parts tables inside them are not read.
• A parts list written as prose or bullets, not a table, is not recognized as a BOM.
• Files over the engine's read limits are skipped, and the robot page says so.
• A linked external guide is credited by what the link says, at the lowest tier, without the engine opening it.
• Repository metadata such as the description is read when the engine runs, not at the pinned commit.

Anti-gaming rules

• A file named like evidence (wiring.md, an empty firmware folder, a "Build" heading) earns at most the lowest tier unless its type or content supports more.
• Empty and stub files earn nothing; models of bought parts and circuit-board 3D exports do not count as the robot's CAD.
• A parts-list feature counts only when it covers most rows, and a list of fewer than five rows cannot pass for a full BOM.
• A photo of wiring is not a diagram, a component's manual is not a build guide, and a demo video is not a build video.
• An embedded image counts as a wiring diagram by its own file name, by being an SVG or PDF drawing under a wiring heading, or when its caption or the sentence introducing it calls it a diagram; a JPEG counts by name only when named a diagram or schematic, and a hookup for flashing a bootloader is not the robot's wiring.
• CAD coverage counts parts, not files: one mount exported as STEP and IGES is one part.
• One mount or spare part in STEP next to a printed robot is not its CAD; neither is an OpenSCAD script that only imports a mesh, nor a Git LFS file GitHub no longer serves.
• The license criterion reads the hardware license, so a permissive software license cannot stand in for it.
• Overrides need public evidence and are shown as reviewed; build reports need review and cannot be self-approved.
• No score reaches 90 without a reviewed build.

Why scores changed

Until 2026-09-30 the scores in the atlas were set by hand and often ran above what the published sources support. They are now computed from the engine's snapshot plus reviewed overrides only, under the stricter engine 3.0 rules above, after an independent audit found the first computed scores still generous; engine 3.1 then fixed what a second audit found the stricter rules missed or misread (diagrams embedded as HTML images, stand and power-supply tables, totals without quantities). Scores dropped where a project's hardware is not published in a form anyone can check, for example a robot that ships meshes but no editable CAD, a parts list without quantities, or a software license standing in for a hardware one. Others rose where the engine found evidence the hand-set score had missed.

Maintainers can raise a score the direct way: publish the missing piece in the repository, or ask for a correction with a link to where it already lives.