Written from the specification alone

A specification that only describes can be wrong without anything noticing. The only honest test of whether a document is sufficient is whether somebody built the thing from it and the bytes matched. That test has been run, and the site reports what it returned, including what it returned badly.

Measured when this page was generated: the corpus is 114 documents, and the reference encoder produces canonical bytes for 114 of them. That is the size of the thing 11 independent implementations agree about — stated as a number, because the word robust would be doing work the number does not support.

Twelve implementations of spec/unimsg-v0.umsg, each written from the specification alone by a different author with no sight of the others. They are kept because their independence is the asset: when they disagree, one of them has found something, and when they agree the specification said what it meant.

They are not merged and should not be. Merging them would produce one implementation with twelve authors, which tests nothing.

They were written in order, and the order shows. Go was first and had the most found in it; .NET was last, written from docs/implementing-unimsg.umsg, and arrived with guards none of the earlier six had on the day they were written. That is the whole argument for writing the findings down: a defect found once becomes a paragraph, and a paragraph prevents it. Where there was no paragraph, the seventh implementation made the same class of mistake as the first.

The eighth is the C one, for microcontrollers, and it is the first to be measured somewhere other than a developer's machine: it has run on an ESP32-H2, where it parses a document nested 400 deep on a 3 KB task stack and moves the heap by zero bytes across fifty round trips. It is also the first to arrive with the guide's guards already in place and to find two things the guide had never said — that a headerless document may open with a comment, and that the specification's errors table is a check rather than an illustration. Both are now in the guide.

The ninth, in Swift, returned five findings and every one of them is a place the guide was WRONG rather than silent — including a trap it reported as closed that was still open, which the ninth then met exactly where the guide promised it would not. It also brought a method: it found a cubic formatter cost by plotting time against depth and reading the slope, where every earlier formatter defect here was found by something timing out. Applying that method the same afternoon found the same cost in this repository's JavaScript.

The tenth, in Python, is the second attempt in that language and shares no code with the first, which is archived and fails 18 of 20 documents. It is dependency-free where the archived one needed cbor2, writing its own CBOR codec. It returned SEVEN findings, and two of them CORRECT advice the guide was giving rather than filling a silence: the remedy for quoting keys refuses valid sigil payloads, and the curve recipe reports a false failure against any correct formatter, because the canonical text of a deep chain is quadratic by construction. It also found that the concatenation defect the guide files under formatting is reachable in an ENCODER, and contributed a fourth formatter structure that removes the exponential rather than mitigating it. All seven are now in the guide.

The eleventh, in C++, is the first whose LANGUAGE forced a question the guide had never thought to answer. The eight before it all had Unicode general categories in the standard library; C++ has isalpha and a process locale, which would make the same document lex differently on two machines, so the word-rune ranges had to be vendored. That is now in the guide, together with the thing this implementation got wrong on the way: it took the generated table from the C implementation rather than regenerating it, and a generated table is still a source read.

It is also the first entry here whose most useful finding came out of the REVIEW rather than out of the writing, and the finding is against the guide's own advice. Told to memoise the formatter, it keyed the memo on the address of the value node — correct for every node a document holds, and wrong for the nodes a formatter INVENTS. The degrading !cbor/tag spelling is built, emitted and dropped, so the next invented node lands on the freed address and reads the previous tag's text: a map with two distinct unknown tags formatted both as the first. Eleven bytes of input, a formatter changing the value, and nothing in its suite could see it — two hundred thousand fuzzed decodes and every single-byte mutation of a valid message, none of which format, against a corpus in which no document carries an unknown tag. A degrading spelling is by definition the path the corpus does not exercise.

The twelfth, in Julia, was handed a brief that told it what to read and deliberately did not tell it what to look for, and what it came back with is a SECOND cubic in the formatter. carry-the-indent-as-a-number is written as a rule about indentation and is not one: it is a rule about any quantity of length O(depth) that a tree walk rebuilds at every node, and a unimsg formatter has two of them. The other is the PATH — which exists because the guide and the specification both say comments and authored column order are kept beside the value, KEYED BY PATH, and a path written as a string grows with the depth. Following every word of the guide still produced a formatter that spent two seconds on a document it now renders in twenty-six milliseconds.

It also brought the instrument. Every earlier measurement here was wall time, which the tenth implementation showed needs a denominator and which this one found still needs a variance rule — one reading per depth reported a sixteen-fold spike on a formatter that was correct. BYTES ALLOCATED per output character is deterministic, needs no repetition, and stated the whole defect in one column: 277, 292, 380, 479, 580, 680, straight in the depth. That is the cubic signature with none of the noise, and it is the cheaper half of the recipe.

Status

Four measurements per implementation, and they are not the same measurement. tools/compare encodes every document in examples/, corpus/ and profiles/ and checks the bytes against the reference in syntax/; it catches divergence in the large. tools/vectors runs the conformance table carried by the specification itself — the vectors it must encode and, since 2026-09-02, the rejects it must refuse — and catches divergence in the small. tools/order formats one fixture and checks that the order its blocks and its columns were written in survives. tools/fmtcompare formats every document compare encodes and checks the TEXT against the reference's, which is the check the other three cannot be: the specification promises one canonical rendering, and until this existed nothing read the rendering.

THE FORMAT COLUMN IS WHERE THE FLEET STANDS FURTHEST FROM THE SPECIFICATION, and it was invisible for a month. Nine formatters produced nine texts from one document — key columns padded or not, a blank line after the header or not, comments carried or dropped — and every check was green, because order compares order and forgives exactly those things on purpose. Part of that is the implementations and part of it is the specification, whose formatting section does not say what the reference does about key alignment. Both halves are owed before the freeze; see docs/50.

EVERY ROW BELOW SAYS WHEN AND WHERE IT WAS MEASURED, because the rows are written by a run and not by a hand: sh implementations/check.sh -record merges the run into results.umsg and rewrites the table from it, and a row from a shell that could not find a toolchain never overwrites a row from one that could. That is how a Swift row measured under WSL sits beside rows measured on Windows, each saying which. The stale /20 rows this table once held could not happen that way, because a number typed here by hand fails go test ./tools/checkreport.

THE FIRST RUN IN WHICH THE MAINTAINED IMPLEMENTATIONS DID NOT ALL AGREE was 2026-09-01: four accepted all 88 documents and five refused one, examples/entry-pluto.umsg, which wrote a table whose first data row began on the same line as its header with no separator between them. The split was recorded here as an open reading, with the five as the odd ones out. IT WAS THE OTHER WAY ROUND. The specification had said since 2026-08-23 that x [ | a b | 1 2 ] is malformed; the five were following it and the reference, together with four others, was not. Nothing had pinned the sentence, so nothing noticed. On 2026-09-02 the sentence became a reject vector — the specification now carries a rejects table beside its vectors — the reference and the four were corrected, and the document was rewritten in block form. What a disagreement between implementations means is that somebody has found something, and this time the finding was that a rule the reference did not follow is a rule nothing enforced.

THESE FIGURES WERE STALE FOR A FORTNIGHT and the way they went stale is the argument for not writing them by hand. This paragraph said 36 documents and 28 vectors, measured 2026-08-15; the corpus had since grown to 85 and the vector table to 32, and nothing failed because nothing checks a paragraph. It was found when a number copied out of here reached a public website. The three tools print their own denominators on every run — see not-written-by-hand-for-long below.

The third check was added on 2026-08-22, after four implementations — the reference among them — were found to be scrambling the column order of every document they formatted while passing both byte checks perfectly. Neither byte check could see it: the authored order deliberately never reaches the bytes, which is the whole point of the rule and the reason it needed a check of its own. Every maintained implementation that ran keeps 5/5 orderings. See F25 in docs/16-format-weaknesses.umsg.

Not written by hand for long

EVERY NUMBER IN THE TABLE ABOVE IS MEASURED, and none of it is measured by this document. Re-measure before trusting it:

sh implementations/check.sh

It prints a row per implementation, and each of the three tools prints its own denominator, so a corpus that has grown since the last edit shows up as a larger number rather than as a silent disagreement with the prose.

A MISSING TOOLCHAIN IS REPORTED AND NEVER SKIPPED, which is the behaviour to keep — but read the report carefully, because NOT RUN means only that the harness could not find the command in the shell it was given. Swift is the standing example: a swiftly-managed toolchain lives under ~/.local/share/swiftly/bin, which is on PATH for a login shell and not otherwise, so the same machine reports it missing from one shell and passing from another.

THE FIGURES COME OUT OF A RUN. check.sh -record hands its own transcript to tools/checkreport, which writes results.umsg — one row per implementation, with the machine and the date — and rewrites the Documents, Vectors and Measured cells of the table above from it, leaving the Implementation and Notes cells as their author wrote them. go run ./tools/checkreport -check fails when the table disagrees with results.umsg or when README.md is stale against this document, and go test ./tools/checkreport runs that check. So the numbers cannot go stale silently any more. The prose around them still can, and this paragraph is prose.

Decoding is total, and only one of them was checked

errors.decoding-is-total-and-side-effect-free is a conformance requirement, and the recommended way to verify a decoder — re-encode and compare — cannot see it. It verifies that what was decoded round-trips, and says nothing about whether decoding terminates. So an audit of the Kotlin implementation checked it there, found and fixed six defects, and the other four were left on the strength of a re-encode check that was never going to report this.

They were then checked, against thirty adversarial inputs: declared lengths near the top of the word, arrays and maps claiming two billion items, nesting thousands deep, ill-formed UTF-8, duplicate keys, indefinite lengths, non-minimal encodings and truncation. All four failed, in four different ways, and the failures were not near-misses: 9a7fffffff is five bytes and asked Rust's allocator for 137 GB, which it answered by killing the process. Go accepted duplicate keys and ill-formed UTF-8 outright, both of which the specification names.

All eight now refuse all thirty. The guards are the same three in every language — a depth cap shared by the decoder, the text parser, the encoder and the formatter, a declared count checked against the bytes that remain before anything is allocated, and a set rather than a scan for duplicate keys — because the defects were the same three, found once each in a different implementation.

Worth stating plainly, since it is the second time this directory has learned it: eight implementations agreeing proves nothing about a property none of them was tested for.

The one that was not an attack

Every other entry in the table above is a malformed input. This one is a document an author could write.

Format a value nested forty deep — a [ [ [ ... ] ] ], or the same with maps. It is well inside every implementation's depth cap, it is valid, it round-trips, and three of them never finished formatting it — Go, JavaScript and .NET. Measured at 1.9x per level: depth 32 took 683 ms and depth 40 did not return.

Nested maps and nested sequences are separate paths and had to be tested separately. Go and JavaScript were exponential in both; .NET only in maps, its sequence path having got the structure right. Two of the three were declared fixed on the strength of the sequence test alone before the map test was written, which is the reason both are named here.

The cause is the same in all three and follows from what canonical formatting is. The formatter renders the inline form, measures it against the line budget, and falls back to the block form when it does not fit — which, written directly, renders every child twice at every level, and two renders per level is 2^depth.

It survived every check this repository has. The vectors are single values; the corpus is shallow; 36 agree says nothing about a document nobody wrote. Two of the four had already been through an adversarial suite of thirty hostile inputs and passed all thirty, because every one of those inputs is malformed and this one is not.

The fix differs by language — memoise the render keyed by node AND indent, or decline the inline attempt when the value is provably too deep to fit on one line — and both are written up in docs/implementing-unimsg.umsg. One implementation's sequence path had it right by structure, rendering once and reusing, and the map path in the same file by the same author rendered twice and hung.

The lesson is narrower than test more and worth stating exactly: a corpus tests the documents someone thought to write. Cost, unlike correctness, is invisible to a comparison that only checks bytes.

There was a second cost underneath the first, and it took a ninth implementation to see it. Removing the exponential from the JavaScript formatter by memoising the rendered text left a CUBIC cost in place and made it worse: every level prefixes its indentation onto a child string that already carries its own, so the characters created across a render grow with the cube of the depth, and a cache turns that from transient garbage into retained heap. Decoding a 1 021-byte document nested 1 020 deep asked for 3.8 GB and was killed.

Nothing failed while that was true. It agreed on all 36 documents and passed all 28 vectors on the morning it was killing itself on a kilobyte of input. The ninth implementation found the same class of defect in its own formatter by timing depth 100, 200 and 400 and reading the slope — 1 382 ms to 63 ms once fixed — and that method, not another corpus document, is what was missing here.

Why they are not allowed to read each other

Each of these was written from spec/unimsg-v0.umsg and docs/implementing-unimsg.umsg alone, with no sight of any other implementation's source. That rule is load-bearing rather than ceremonial, and the table above is the reason: eight implementations agree on 36 documents is evidence only if the eight are independent readings. Two of them once carried the same half-precision mantissa mask in the same place with the same constant, and that was a problem precisely because nobody knew they were related.

The rule has not stopped earning. The two most recent entries in the findings table below both come from the eighth implementation, written this month — one refusing a document the other seven accept, one answering a prescribed input with a message that named no fix. What independence tests has changed: it used to find defects in the format, and now it mostly finds defects in the guide. That is still worth having while the guide is what onboards the next one.

The cost is real too. Three implementations shipped an exponential formatter that any of them would have avoided by reading a fourth, and good structure — an iterative walker, a render-once formatter, a conformance runner that finds the repository itself — has had to be arrived at separately each time.

So the rule stays and the benefit is carried across a different way. The implementations are reviewed periodically, and whatever structure turned out to work is written into the guide's patterns-worth-copying section in prose, with the defect it prevents beside it. A pattern described is not a reading copied: the next implementation gets ten authors' worth of structure and still reads the specification with its own eyes. The eleventh is the first case where that review earned its keep in the other direction: the pattern it was handed was RIGHT and incomplete, and following it produced a formatter that changed the value. Every implementation owes that review one thing in return — a section in its README saying what the guide did not prepare it for, which is the only measure of whether the guide is working.

Running them

Toolchains needed: cargo, go, dart, node, python3, a C compiler, a C++20 compiler, a .NET SDK, a Swift toolchain, and a JDK with kotlinc for the Kotlin implementation, and julia. Kotlin is a zip from JetBrains rather than a package, so check.sh reuses an already built jar and reports NOT RUN when there is neither. Julia needs one extra step and it is not a toolchain check: its CLI is loaded as a package so that the compiled code is cached, and check.sh precompiles it once before measuring rather than paying three seconds in each of the 264 processes these three tools start. The C++ row prefers cmake, which is how the implementation packages itself, and falls back to compiling its six sources directly: requiring cmake made a present compiler look like an absent implementation once, which is the failure this whole section exists to prevent.

No machine here has had all eleven at once, and that is expected rather than a problem: check.sh reports what it could not run and the rows it did run are still measurements. Swift lives under WSL on the development machine while the rest are on Windows, so a complete picture is two runs, and the two runs disagree about which rows say NOT RUN rather than about any implementation.

Being on PATH is never taken as evidence that a toolchain works, because three separate incidents here began exactly that way. A kept jar is checked against its sources: one older than the code it was built from reports on code nobody is running, and it reports a pass, so check.sh says NOT RUN and names the file that is newer. A dotnet on PATH does not imply an SDK behind it, so build.sh tests dotnet --list-sdks rather than the host. And a dart on PATH does not imply a dart that runs — under WSL the Windows Flutter is reachable through /mnt/c with CRLF launchers that cannot execute — so the script asks it for --version first. Each of those three, left unchecked, reported a correct implementation as broken.

./implementations/check.sh

A missing toolchain, or a Dart package tree that will not resolve, is reported as NOT RUN — never compared. An implementation that was not run must not look like one that passed, and the first version of this script broke that rule twice: it reported an unresolved Dart as 20 differ, because Dart prints its resolution failure to stdout and exits 0, so the harness dutifully compared an error message against CBOR.

Some CLIs write raw CBOR and some write hex. That difference used to be bridged by a generated shell script, which cost an hour once and then cost rather more: tools/compare runs its command directly rather than through a shell, so on Windows a wrapper script is not an executable command, and three of the five implementations reported every document as an error while being entirely correct. The bridging is gone — tools/compare -raw declares the difference and the tool converts — and the lesson is the one above, arrived at twice: when the harness breaks, its output blames the implementation.

What is archived, and why

rust-second-attempt — a later Rust implementation by the same author, written to see whether experience improved the result. On the current corpus it scores worse: it refuses a comment that follows a value on the same line, which the specification explicitly permits, and whose own notes record as *three accepted a trailing comment and one refused*. It is that one, and it predates the ruling.

It is kept rather than deleted because it is not worse in design. It separates lexing into its own module and carries a dedicated big-integer module, where the maintained one folds both into the parser, and its CLI emits hex directly rather than raw bytes — the better choice for a tool meant to be diffed. If the Rust implementation is ever restructured, those three decisions are the ones to take from here.

dart-before-web-fixes — superseded. It carries two defects found by compiling for the web: an integer literal above 2^53 that dart2js refuses to compile, and ByteData.getUint64, which is unsupported there and throws at run time on any float. The maintained dart/ has both fixed, and a smoke test that runs the compiled output rather than only building it.

Python, and what an unrun implementation looks like

python/ was the earliest implementation written and was overlooked when this directory was organised — it sat in examples/ while everything else moved. Running it was the first time anyone had.

It fails 18 of 20 documents, and the cause is not staleness. Fourteen failures are multi-line strings, which is not a construct added later: a string may span lines and always could, and nearly every document here relies on it. Three more are trailing comments, the same construct the archived Rust refuses. One is a partial timestamp such as 2026-08. One document encodes differently.

It is archived rather than repaired. A rewrite from the current specification is a smaller job than retrofitting multi-line strings into a lexer that never had them, and the specification is considerably better now than on the morning this was written.

It is kept and reported rather than quietly deleted, because an implementation nobody runs is indistinguishable from one that works, and this one had been in that state since the day it was written — which is the whole argument for check.sh reporting every implementation on every run, including the archived ones.

What they have found

Independent implementations have located every serious defect in this format. Not one came from re-reading the reference implementation.

The reserved-literal and closing-bracket rows were found while organising this directory: the reference implementation was lenient where the grammar is strict, and four independent readers had all concluded otherwise.

The last two are F22, and differ in kind. Every earlier row is one implementation disagreeing with another; these are all of them agreeing, wrongly, because the specification said cells are self-delimiting and never said an identifier is not. A defect present in five independent implementations is a defect in the document they were written from.

The last row is the same defect seen from the other side. The grammar's value production excludes annotation — it appears only in pair — so an implementation written from the EBNF refuses [ hz 62.5 ], | km 38.2 and !ledger/line usd 5.00, all three of which are written in this corpus and all three of which the reference accepts. Annotation belongs to the value position, not to the pair. Say that, and F22 follows immediately: a table cell is the one value position with no terminator, which is why the identifier rule cannot be applied there and why the sixth implementation restricts cell annotations to bare words.

The final row is not a disagreement between implementations but a gap the re-encode check cannot see. decoding-is-total is a conformance requirement, and the recommended way to verify a decoder — re-encode and compare — verifies only that what was decoded round-trips. It says nothing about whether decoding terminates, so a document of four thousand open brackets, or a five-byte header declaring two billion array items, crashes rather than reports.

The row after it is what happened when that was checked in the others rather than left as a suggestion: all four failed, and the section above records how.

The last two rows are a different lesson and the more uncomfortable one. Two implementations widened half-precision floats to single because both carried a mantissa mask four bits too wide — not the same kind of mistake, the same mistake, in the same place, with the same constant. And four of the six were running their self-tests against private copies of the specification that had gone stale, one of them 428 lines against the specification's 567, missing rulings this directory records itself as having made. A conformance test read out of a stale copy passes while testing nothing.

Independence is this directory's whole argument, and both rows are evidence against it: implementations that fail identically were not written independently, and implementations tested against their own frozen copy of the specification are not being tested against the specification at all. Both are now fixed — the mask in both, and every self-test repointed at spec/unimsg-v0.umsg — but the fix is not the point. The point is that the property this directory exists to provide has to be checked too, and nothing here was checking it.

Why the defects are the point

Every serious defect in this format was found by an independent implementation reading the specification differently, and not one came from re-reading the reference. A reserved literal accepted as a bare map key was found by four implementations refusing a document the reference accepted — four readers against the author, and the readers were right. The site tells that story with the specification's correction attached, because it is the best available evidence that the document is the authority and the code is not.

Where each part of this page comes from