renest 0.1.15

Concepts

Known by bytes, not by name.

Renest identifies every file in a nest by the sha256 fingerprint of its contents, not by its filename. On the way back, each file is fingerprinted again and compared with what was recorded, so a truncated, corrupted or swapped file is caught and named.

A filename is a label somebody chose

model-v2.safetensors tells you what someone decided to call a file. It tells you nothing about what the file is. The name lives outside the bytes; it can be kept while the bytes change, and changed while the bytes stay. Any system that identifies files by name is really recording claims, and claims drift.

A sha256 fingerprint of a file's contents is different in kind: it is derived from the bytes themselves. Same bytes, same fingerprint; one byte different, a different fingerprint. A filename is a label someone gave the file. The fingerprint is the file's own identity.

A name can stay the same while the bytes change in three common ways

  • Re-quantised or re-exported under the same name. Two checkpoints called the same thing, produced by different conversion runs, with different bytes — and different behaviour.
  • Quietly replaced upstream. An author fixes their weights and re-uploads over the old file. Everyone who downloads "the same model" after that day has a different model, and nothing in the name says so.
  • Half a download. A connection drops partway through and leaves a file with the right name and truncated contents. By name it's present; by bytes it's incomplete.

Every one of these passes a check that only asks "does a file with this name exist?"

Renest stores each file under the sha256 of its contents

Every model file, source archive, lock file and workflow in a nest is recorded in the manifest with the sha256 of its contents and its size, and stored under that fingerprint — blobs/sha256/<first two characters>/<sha256> — not under its name. The name still exists: the manifest records where each file belongs and what it's called on the way back out. But the name is metadata about placement. Identity is the fingerprint. The full layout is in the format spec.

Addressing by content turns missing and altered files into detectable facts

"What's missing" stops being a guess. A rebuild doesn't check that something named model-v2.safetensors showed up; it fingerprints what arrived and compares it with what was recorded. A truncated download, a corrupted transfer, a swapped file — all collapse into the same detectable fact: these bytes are not those bytes. A file that doesn't match is downloaded again, and if it still doesn't match, the rebuild stops and names the file.

"What changed" stops being an argument. Same fingerprint, same file — no matter what it's called or where it came from. Different fingerprint, different file — even if every label insists otherwise.

The same bytes are stored once. When identity is content, deduplication falls out of the addressing: a file already in the bucket under its fingerprint is not uploaded again, and a file that appears at two paths in one nest is downloaded once. What that does and does not cover is spelled out in Deduplication is whole-file.

The fingerprint is sha256, and it stays sha256

There are newer fingerprint algorithms. We use sha256 and intend to keep using it, for a reason that outranks novelty: it's the fingerprint the ecosystem already speaks. Hugging Face and Civitai publish sha256 for the files they host, so a file already on your disk, or already documented upstream, can be recognised as the same file by comparing one value. A private fingerprint would work technically; it would also make every fingerprint already published in the ecosystem useless to you.

One scope note: sha256 verifies the payload — the bytes of your files. It says nothing about whether two machines will compute identical outputs from those bytes; that depends on hardware and software, not on storage. What a rebuild checks beyond the bytes is described in the three gates.

These docs describe renest 0.1.15, the latest release.

Open format

The docs describe a format you own.

Everything here is written against the open nest format. Every nest ships a plain restore.sh that brings back its files and dependencies with zero Renest code.