The copy says it’s done. Why verify?
A finished progress bar means the data was sent, not that the drive holds an exact copy. What a checksum is, and how Lightferry reads each copy back to check it.
When an ingest hits 100%, all the progress bar knows is that the data was read from the card and handed to the drive. Whether the drive ended up holding an exact copy is something it can’t tell you. This post starts from the evidence: whether copies really go wrong silently, what a checksum is, why Lightferry uses XXH64 by default, and exactly how it checks your copies.
Do copies really go wrong silently?
Yes, and there is large-scale data to show it. These two widely cited studies measured data-center equipment that has redundancy and monitoring:
| Study | Scale | Finding |
|---|---|---|
| CERN, “Data integrity” (2007, internal report) | 3,000 servers writing and reading back test data for about 5 weeks | 500 errors on 100 machines. Of 33,700 files checked (about 8.7 TB), 22 no longer matched the checksums recorded on tape: about one bad file in 1,500 |
| NetApp et al., FAST ’08 paper | 1.53 million drives over 41 months | More than 400,000 checksum mismatches. 0.86% of nearline and 0.065% of enterprise drives returned changed data without reporting any error |
The CERN report puts it well: “the different protection mechanisms don’t work 100%”—not every error in the data flow is correctly handled and reported. It also tested 10,000 compressed files and found that “with a likelihood of 99.8% a SINGLE bit error makes the whole file unreadable.” H.264, HEVC and ProRes are compressed too. A damaged clip may not refuse to open, but corrupted blocks and dropped frames can’t be delivered either.
A set is messier than a data center. Every copy passes through a card reader, a cable, a port, an enclosure and the drive itself, and any of them can fail without the system reporting an error:
- A flaky reader or cable misreads a few bytes now and then.
- An enclosure or dock with unstable power writes bad data.
- A drive is unplugged or loses power mid-copy, leaving a half-written file.
- An ageing card returns different data from the same spot on two reads.
A checksum is a file’s fingerprint
Verifying means computing a fingerprint for each file, called a checksum or hash. The same content always gives the same fingerprint; flip a single bit and the fingerprint is completely different. Here is what we measured on 1 GiB of test data:
| Data | XXH64 fingerprint |
|---|---|
| 1 GiB of test data | 0a08cc4c8b94db2b |
| The same data with one bit flipped | 5a21bb20086ea713 |
Comparing fingerprints tells you whether two files are identical without comparing them byte by byte. An XXH64 fingerprint has 2⁶⁴ possible values—about 1.8×10¹⁹—so the chance that a random copy error leaves it unchanged is negligible. It isn’t a cryptographic hash and isn’t meant to stop deliberate tampering; catching accidental errors is exactly what the industry uses it for.
Why XXH64 instead of MD5?
Speed. MD5 dates from 1991 and was designed to cryptographic requirements, which makes it slow. XXH64 exists only to catch data errors and is more than an order of magnitude faster. We measured both on a Mac with the same library versions Lightferry ships:
Single-thread checksum speed (longer is faster)
0.76 GB/s may not sound slow, but today’s cards are faster. Here is MD5’s speed next to the rated read speeds of a few popular cards:
Rated card read speeds; the dashed line is MD5 on one thread
For any card to the right of the line, verifying with MD5 alone would hold the copy back. XXH64 is far faster than any card, so verification costs almost nothing—which is why Lightferry always computes it. If a recipient needs another format, turn on Extra Checksums in Settings to compute MD5 or XXH128 at the same time. Netflix’s production data management guidance, for example, says every copy of original camera files should be checksum-verified with xxHash64, xxHash128 or MD5, and recommends the ASC MHL format.
How Lightferry verifies
Lightferry reads the card only once while copying. Each block is added to the fingerprint and then written to every destination at the same time, so a two-drive backup doesn’t mean reading the card twice. Once the files are written, the default, read-back verification, reads every copy back from the drive in full, fingerprints it again, and compares the result with the fingerprint taken from the card.
01Read the card once, fingerprinting as it goes
A001Sony · 128 GB
02Write to both drives at the same time
SSD 2TBWork drive
HDD 4TBBackup drive
03Read each copy back from disk and fingerprint again
Read back from disk
Read back from disk
When fingerprints don’t match, Lightferry doesn’t fail the file right away. It reads it again, bypassing the cache. If the second read matches, the file passes, with a prompt to check your reader and cable. If it still doesn’t match, the file is marked as failed. Bad copies are never deleted: once you confirm, they’re moved as-is into a quarantine folder and copied again from the card.
Three verification modes
Read-back verification proves that the copy on the drive matches what was read from the card during the copy. But if the reader misread the card at that moment, the bad data is written to every drive unchanged, and read-back still “matches”. Full verification reads the card again after copying, which catches this case too.
| Mode | What it reads back | Good for |
|---|---|---|
| Off | Copies only, nothing read back | Not recommended for original footage |
| Read-back (default) | Every copy, in full, from the drive | Everyday ingest |
| Full | Every copy, plus the card once more | Important shoots, and before formatting |
Full verification reads the card one more time, so it takes longer. You can also copy with read-back verification and click Re-Read Card afterwards to add that step.
A record for whoever comes next: MHL and ASC MHL
Verification is only useful if the next person can check it too. Lightferry writes MHL files to every destination and an ASC MHL record inside the footage folders. ASC MHL is an open specification from the American Society of Cinematographers’ technology council, released as v1.0 in 2022, supporting xxh64, xxh128, md5 and more. Because a new manifest is created for every copy, it “creates a chain of custody” from set to post. Files that failed verification are never recorded as verified.
In short
- Silent corruption is documented; a finished progress bar doesn’t mean a correct copy.
- Comparing fingerprints is how you confirm a copy, and XXH64 is fast enough not to slow any card.
- Read-back verification has to read from the disk; full verification checks the card itself as well.
- MHL records let the next person verify again.
When it’s safe to format the card is a separate question, covered in another post.
Sources
- Bernd Panzer-Steindel, “Data integrity”, CERN/IT, Draft 1.3, April 2007
- Bairavasundaram et al., “An Analysis of Data Corruption in the Storage Stack”, USENIX FAST ’08
- xxHash README and benchmarks
- Lexar Professional DIAMOND CFexpress 4.0 Type B product page
- Sony CEA-G/CEA-M CFexpress Type A write speed performance
- SanDisk Extreme PRO CFexpress Type B data sheet
- SanDisk Extreme PRO SD UHS-II V90 data sheet
- Apple File System Programming Guide: Performance Tips
- Netflix: Production Assets: Data Management
- ASC Media Hash List (ASC MHL) Specification v1.0