All posts

The copy says it’s done. Why verify?

A finished progress bar means the data was sent, not that the drive holds an exact copy. What a checksum is, and how Lightferry reads each copy back to check it.

When an ingest hits 100%, all the progress bar knows is that the data was read from the card and handed to the drive. Whether the drive ended up holding an exact copy is something it can’t tell you. This post starts from the evidence: whether copies really go wrong silently, what a checksum is, why Lightferry uses XXH64 by default, and exactly how it checks your copies.

Do copies really go wrong silently?

Yes, and there is large-scale data to show it. These two widely cited studies measured data-center equipment that has redundancy and monitoring:

StudyScaleFinding
CERN, “Data integrity” (2007, internal report)3,000 servers writing and reading back test data for about 5 weeks500 errors on 100 machines. Of 33,700 files checked (about 8.7 TB), 22 no longer matched the checksums recorded on tape: about one bad file in 1,500
NetApp et al., FAST ’08 paper1.53 million drives over 41 monthsMore than 400,000 checksum mismatches. 0.86% of nearline and 0.065% of enterprise drives returned changed data without reporting any error

The CERN report puts it well: “the different protection mechanisms don’t work 100%”—not every error in the data flow is correctly handled and reported. It also tested 10,000 compressed files and found that “with a likelihood of 99.8% a SINGLE bit error makes the whole file unreadable.” H.264, HEVC and ProRes are compressed too. A damaged clip may not refuse to open, but corrupted blocks and dropped frames can’t be delivered either.

A set is messier than a data center. Every copy passes through a card reader, a cable, a port, an enclosure and the drive itself, and any of them can fail without the system reporting an error:

  • A flaky reader or cable misreads a few bytes now and then.
  • An enclosure or dock with unstable power writes bad data.
  • A drive is unplugged or loses power mid-copy, leaving a half-written file.
  • An ageing card returns different data from the same spot on two reads.

A checksum is a file’s fingerprint

Verifying means computing a fingerprint for each file, called a checksum or hash. The same content always gives the same fingerprint; flip a single bit and the fingerprint is completely different. Here is what we measured on 1 GiB of test data:

DataXXH64 fingerprint
1 GiB of test data0a08cc4c8b94db2b
The same data with one bit flipped5a21bb20086ea713

Comparing fingerprints tells you whether two files are identical without comparing them byte by byte. An XXH64 fingerprint has 2⁶⁴ possible values—about 1.8×10¹⁹—so the chance that a random copy error leaves it unchanged is negligible. It isn’t a cryptographic hash and isn’t meant to stop deliberate tampering; catching accidental errors is exactly what the industry uses it for.

Why XXH64 instead of MD5?

Speed. MD5 dates from 1991 and was designed to cryptographic requirements, which makes it slow. XXH64 exists only to catch data errors and is more than an order of magnitude faster. We measured both on a Mac with the same library versions Lightferry ships:

Single-thread checksum speed (longer is faster)

  • XXH12827.6 GB/s
  • XXH64Lightferry default21.8 GB/s
  • MD50.76 GB/s
Apple M5 Max, single thread, 1 GiB in memory, fed in 8 MiB blocks; xxhash-rust 0.8.21 and md-5 0.10.6, the versions Lightferry uses; median of three runs (measured 2026-11-17). xxHash’s own benchmark on an Intel i7-9700K gives XXH64 19.4 GB/s and MD5 0.6 GB/s, the same order of magnitude.

0.76 GB/s may not sound slow, but today’s cards are faster. Here is MD5’s speed next to the rated read speeds of a few popular cards:

Rated card read speeds; the dashed line is MD5 on one thread

  • CFexpress 4.0 Type BLexar DIAMOND3.7 GB/s
  • CFexpress 4.0 Type ASony CEA-G1.8 GB/s
  • CFexpress 2.0 Type BSanDisk Extreme PRO1.7 GB/s
  • SD UHS-II V90SanDisk Extreme PRO0.3 GB/s
Card speeds are the makers’ “up to” sequential read figures (1 MB = 10⁶ bytes); real ingest speeds are also limited by the reader and interface: Lexar Professional DIAMOND CFexpress 4.0 Type B, Sony CEA-G (CFexpress 4.0 models), SanDisk Extreme PRO CFexpress Type B, SanDisk Extreme PRO SD UHS-II. XXH64’s 21.8 GB/s is off the chart.

For any card to the right of the line, verifying with MD5 alone would hold the copy back. XXH64 is far faster than any card, so verification costs almost nothing—which is why Lightferry always computes it. If a recipient needs another format, turn on Extra Checksums in Settings to compute MD5 or XXH128 at the same time. Netflix’s production data management guidance, for example, says every copy of original camera files should be checksum-verified with xxHash64, xxHash128 or MD5, and recommends the ASC MHL format.

How Lightferry verifies

Lightferry reads the card only once while copying. Each block is added to the fingerprint and then written to every destination at the same time, so a two-drive backup doesn’t mean reading the card twice. Once the files are written, the default, read-back verification, reads every copy back from the drive in full, fingerprints it again, and compares the result with the fingerprint taken from the card.

  1. 01Read the card once, fingerprinting as it goes

    A001Sony · 128 GB

    Fingerprint · XXH649f3a·71c2

  2. 02Write to both drives at the same time

    SSD 2TBWork drive

    HDD 4TBBackup drive

  3. 03Read each copy back from disk and fingerprint again

    Read back from disk

    9f3a·71c2Match

    Read back from disk

    9f3a·71c2Match

Read the card once, write to two drives at the same time, then read each copy back and compare (sample data).

When fingerprints don’t match, Lightferry doesn’t fail the file right away. It reads it again, bypassing the cache. If the second read matches, the file passes, with a prompt to check your reader and cable. If it still doesn’t match, the file is marked as failed. Bad copies are never deleted: once you confirm, they’re moved as-is into a quarantine folder and copied again from the card.

Three verification modes

Read-back verification proves that the copy on the drive matches what was read from the card during the copy. But if the reader misread the card at that moment, the bad data is written to every drive unchanged, and read-back still “matches”. Full verification reads the card again after copying, which catches this case too.

ModeWhat it reads backGood for
OffCopies only, nothing read backNot recommended for original footage
Read-back (default)Every copy, in full, from the driveEveryday ingest
FullEvery copy, plus the card once moreImportant shoots, and before formatting

Full verification reads the card one more time, so it takes longer. You can also copy with read-back verification and click Re-Read Card afterwards to add that step.

A record for whoever comes next: MHL and ASC MHL

Verification is only useful if the next person can check it too. Lightferry writes MHL files to every destination and an ASC MHL record inside the footage folders. ASC MHL is an open specification from the American Society of Cinematographers’ technology council, released as v1.0 in 2022, supporting xxh64, xxh128, md5 and more. Because a new manifest is created for every copy, it “creates a chain of custody” from set to post. Files that failed verification are never recorded as verified.

In short

  • Silent corruption is documented; a finished progress bar doesn’t mean a correct copy.
  • Comparing fingerprints is how you confirm a copy, and XXH64 is fast enough not to slow any card.
  • Read-back verification has to read from the disk; full verification checks the card itself as well.
  • MHL records let the next person verify again.

When it’s safe to format the card is a separate question, covered in another post.

Sources

Say hello