Concepts

Whole files, whole fingerprints.

Renest deduplicates complete files: identical bytes, identified by one sha256 fingerprint, are stored once. A file with even one byte changed counts as a new file and is stored and transferred in full.

Renest stores identical files once

Every model file, source archive, lock file and workflow in a nest is stored under the sha256 of its complete contents (see Files are known by their bytes). Identical bytes therefore land on the same object:

• In your own bucket, a file that is already there under its fingerprint, at the expected size, is not uploaded again.
• On the hosted drive, files you have stored before are not uploaded again. For a larger file the drive already holds but you haven't stored, your tool can prove it has the same bytes by answering a spot check on byte ranges of your local copy; if the check doesn't pass, the file is uploaded in full.
• On the way back, a file that appears at several paths in one nest is downloaded once and copied, and a rebuild that is run again on the same machine does not re-download files it already verified.

For the way AIGC work accumulates, this covers a lot: the multi-gigabyte base checkpoints are the bulk of the bytes, and they are exactly the files that repeat unchanged from nest to nest. Seal ten variations of a workflow into the same bucket and the base model is stored once, not ten times.

Changing one byte of a file makes it a new file

The unit of "same" is the whole file. Change one byte inside a 6 GB model and, as far as deduplication is concerned, you have a brand-new 6 GB file — stored in full, transferred in full. There is no diffing against the previous version, no splitting files into chunks and reusing the unchanged ones.

This bites in a specific case: a file that is semantically the same but not byte-identical. Re-save a checkpoint with different embedded metadata and the tensors inside may be untouched while the file's fingerprint — computed over header and all — changes, so both versions are stored. Worth keeping in proportion: the common cases still deduplicate. The same published file downloaded by everyone is byte-identical everywhere, and renaming a file changes nothing about its bytes. The gap is the narrower set of files that were re-written, not merely re-named.

We'd rather you learn this trade-off from this page than from a transfer that's bigger than you expected. It is a known limitation and a deliberate one.

The whole-file sha256 is the identity layer, so chunk-level deduplication is not built

Chunk-level or tensor-level deduplication would genuinely save space for re-written files. The reason it isn't here is that the obvious implementation breaks something we won't break.

In Renest, the whole-file sha256 is not just a storage key. It is the identity layer: it's what the manifest records, what a rebuild verifies against, what the escape-hatch script checks with ordinary command-line tools, and what lines up with the fingerprints Hugging Face and Civitai already publish. A scheme that replaced that fingerprint with per-chunk fingerprints or a different algorithm would optimise storage by giving up verification with ordinary tools and ecosystem compatibility. Storage economics are our problem; verifiability is yours.

Any finer-grained deduplication would have to sit beneath the whole-file fingerprint

If finer-grained deduplication is ever added, it has to be a second key underneath, never a different fingerprint on top. The whole-file sha256 would stay exactly what it is: the identity every manifest records and every rebuild verifies. A finer-grained key (for instance, one computed over tensor payloads independently of a mutable file header) would live inside storage only, with manifests, verification and the escape hatch unchanged.

No date is attached to that, and this page is not a promise that it's coming. It states the constraint any future version has to satisfy: the fingerprint you verify with is not negotiable, and savings have to be found beneath it, not instead of it. Until then, deduplication stays whole-file.