Nobody Wrote It Down

Here is a fact that surprises people. StuffIt — the archive format that essentially every Macintosh file passed through between 1988 and 2005 — has no specification. Not a lost one. Not a paywalled one. Aladdin Systems never published one, and neither did anyone who bought them afterwards.

There are millions of .sit files in the world. There is no document that says what one is.

That is the everyday condition of writing software for old computers, and it is much stranger than “the format is old”. Old is fine. Old is documented. What you actually face is a landscape where the thing you need either was never written down, or was written down by someone who is now unreachable, or was written down perfectly by a company whose lawyers would now prefer you didn’t look.

The three kinds of nothing

Honest absence

Nothing was ever published. You know exactly where you stand, and every number you end up with is one you earned.

Clean, in a way

Confident reconstruction

A document that looks like a specification — offsets, field names, tables — and cites nothing. Written in good faith. Circulated. Built on.

Worse, because it reads as authority

Perfect, and forbidden

The real answers exist and are published — under a licence for research, teaching and personal study. We are writing a commercial application, so for us that door is closed. Not locked. Closed, politely.

The one that actually stings

Three ways for the documentation to not be there. Only the first is straightforward.

The first kind is honest absence. Aladdin published nothing; Cyclos published nothing about Compact Pro; MindVision published nothing about Installer VISE. This is clean, in a way. You know where you stand. You are going to have to work it out from the artefacts, and every number you end up with will be a number you earned.

The second kind is worse: the confident reconstruction. Search for any dead format and you will find documents describing it — in prose, in tables, with offsets and field names, looking exactly like a specification. Read the small print and you find they cite nothing. Somebody read an implementation, wrote down what they concluded, and published it in good faith. Now it circulates, gets quoted, gets built on.

We hit a lovely example. A document we held described two bytes in the Compact Pro header as “reserved (ignored)”. When we measured four real archives, those bytes came back 04f7, fbd0, 4b78, 0054 — different in every single one.

Archive	Bytes [0..1]	Bytes [2..3]
`WriteNow_3.cpt`	`01 01`	`04 f7`
`Deep Thoughts Quotes.cpt`	`01 01`	`fb d0`
`Gateway.cpt`	`01 01`	`4b 78`
`Misc Quotes.cpt`	`01 01`	`00 54`

Two bytes hold steady across all four. The two beside them — the documented “reserved (ignored)” field — differ in every archive.

Header bytes read from four real Compact Pro archives.

Not reserved. Not ignored. The unverifiable citation wasn’t merely uncheckable; it was carrying a fact that was flatly wrong, and anyone trusting it would have inherited the error and passed it on.

The third kind is the one that actually stings. Sometimes the perfect document exists. Apple released the entire Lisa operating system source through the Computer History Museum, and it is glorious — the real thing, the actual answers.

It is released under an academic licence: non-commercial, research, teaching and personal study only.

If you are writing a commercial application, that door is closed. Not locked — closed, politely, with a note on it. You may read it as a hobbyist on a Sunday afternoon. You may not read it on Monday if Monday is work. And here is the part that catches people: a copy sitting on your own hard drive is not permission. Neither is a third-party document that quotes it. Contamination travels.

So you can’t just look it up. Now what?

You measure.

This turns out to be more fun than it sounds, and occasionally more rigorous than reading a spec would have been.

The trick that makes it work is known plaintext. If you have a file, and you have the compressed version of that same file, you have an enormously strong constraint on what the compressor did in between. And there is a delightfully direct way to get that pairing: run the original software.

1
Author the input

Runs of every length. A shared long prefix. A single byte changed in the middle. Empty files. Incompressible noise.
2
Compress it on the real thing

StuffIt Lite 3.6, on a 1994 Macintosh, in an emulator. No reimplementation to be wrong about.
3
Measure what came out

You know the input exactly, so every byte of the archive is a constraint rather than a guess.
4
Name the behaviour

Window size, pass structure, header fields — each one pinned by an input built to expose it.

Change one property of the input and go round again. Fifty times.

Chosen‑plaintext, the retrocomputing version: you cannot read the specification, so you commission the evidence.

So we booted a 1994 Macintosh in an emulator, installed StuffIt Lite 3.6 — found, pleasingly, already sitting on a magazine cover disc we owned — and fed it fifty files we had written ourselves. Runs of every length. Two files sharing a long identical prefix. Two files differing by a single byte in the middle. Empty files. Incompressible noise.

Then we read what came out.

Fifty authored pairs settled in one afternoon what a fortnight of staring at archives could not: the codec is whole-file two-pass, its window reaches 16,384 bytes, and the mysterious leading byte turned out to be min(⌊log₂ n⌋, 14) — exact on all sixteen test cases, and it then correctly predicted the value in a real 1990s archive it had never seen.

authored specimen, at its measured length the specimen that exercises the cap DART™ 1.5.3 ReadMe.txt — real 1990s archive, predicted not fitted

Specimens whose input length was measured; several share a length, or sit close enough on a log axis to overlap. One byte either side of 4,096 moves the output a whole step — and the 3,593‑byte file from the 1990s lands exactly where a rule built entirely from authored files said it would.

That is a nice feeling. You didn’t find the answer. You caught it.

The trap that nearly got us

Before the emulator, we tried to answer the same question from the archives we already had. Three of them shared a 1,028-byte identical prefix, and their compressed streams diverged after only 8 bits — which looks like proof that the compressor builds a model of the whole file before emitting anything.

We wrote that up. It was wrong.

What we saw

1,028 bytes of identical prefix

What was actually in it

1,024 bytes — the same byte, repeated 4

A compressor with a simple fixed table diverges on input like that too. The evidence could not tell the two explanations apart — so it was not evidence for either.

The same 1,028 bytes, looked at twice.

1,024 of those 1,028 bytes were the same byte repeated. A compressor with a simple fixed table produces exactly the same divergence on input like that. The evidence was real, the reasoning was sound, and the conclusion was unsupported — because the input we happened to have could not tell the two explanations apart.

This is the actual difficulty of the work, and it is not technical. It is resisting a conclusion that looks earned. The nice thing about authored inputs is that you can build the discriminating case on purpose, instead of hoping the 1990s left you one.

The two-day lesson

We characterised a StuffIt codec from scratch across two days. Careful work, statistically sound, genuinely novel measurements.

It was documented. In prose. On a public wiki. Which we had already used — for a different format — thirty-six hours earlier.

The search we had recorded as “no specification exists” had been a search for the container format. Nobody had searched for the codec. Two adjacent questions, one answered, one assumed.

The rule we wrote down afterwards is embarrassingly obvious and we will never break it again: a prior-art search is per algorithm, not per project.

The dependency you cannot fix

For years the app leaned on a small, well-written MIT-licensed C library for archive decompression. Good code, generously licensed, freely given. No complaints.

We deleted it anyway, and the reasons are instructive.

It was ten C files parsing untrusted input — precisely the shape of thing that turns a malformed download into somebody else’s afternoon. It was a second language in a codebase that is otherwise entirely Swift. And buried inside it was a randomisation table noted as “identical to the bzip2 table” — a BSD-licensed table inside an MIT-licensed library, an attribution the upstream project had not discharged and which we therefore inherited simply by vendoring it. We could not fix that from the outside. The only way to discharge it was to stop shipping the code.

None of this is a criticism of the library or its author. It is a fact about vendoring: you take on everything, including the parts nobody upstream noticed.

Deleting it cost us, immediately and honestly. Compact Pro extraction: gone. StuffIt compressed extraction: gone. An entire archive format we could previously list became unopenable, because the piece that read its catalogue lived in the C we’d removed. We wrote that down as a deliberate regression rather than a bug, and every failure now says exactly what it can’t do and why — “StuffIt arithmetic-coded decompression is not available in this build (the archive is intact)” — rather than quietly producing an empty folder.

Which brings me to the one principle that matters most here.

Wrong is worse than missing

A preservation tool that can’t open your file is annoying.

A preservation tool that opens your file and gives you subtly corrupted contents is a catastrophe, because you won’t know. You’ll file it away. The original will get deleted eventually. And the damage surfaces in ten years when nobody can reconstruct what went wrong.

So the rule throughout is: digest match or refuse. Before deleting the old decompressor, we captured what it produced — 82 files, 82 SHA-256 digests. Any replacement has to reproduce those bytes exactly. Not “look right”. Not “open in a text editor plausibly”. Byte-identical, or the code refuses and says so.

Compact Pro came back that way: reimplemented in Swift from a published prose description, 29 of 29 digests matching. StuffIt’s compressed methods have not come back yet, and we’ve written down precisely why rather than shipping something that mostly works.

The bit I didn’t expect

You would think the frustrating part of this work is the missing documentation.

It isn’t. The frustrating part is that the documentation that exists is often behind the software. Apple’s own developer books lag Apple’s own shipping practice — we found a resource type in use in 1988 and first documented in 1991, and a flag bit that Apple’s documentation says is “reserved; set to 0” which Apple’s own System software cheerfully sets.

Documentation tells you when a fact became publishable. Only measurement tells you when it became true.

Which is, when you think about it, a fairly good reason to keep the old machines running.

This is drawn from work on Disk Phoenix, which reads, repairs and rescues classic Macintosh disk images. Every number above is measured. The emulator really did run StuffIt Lite 3.6, and the two bytes really are different in all four archives.