Tuesday, September 17, 2024

Re: checksums to detect/correct bit-rot

Thank you to everyone who replied -- you've pointed out some useful
design approaches and software.

To summarize:
* OpenBSD doesn't (currently) support any filesystems with embedded
checksums
* mtree (8) can generate a database of files, modify-times, and checksums
(and both the man page and my testing verify that it safely handles
whitespace, non-printing, and/or non-ASCII characters in filesnames)
* There are various programs which can compare a directory tree with
a separate checksum database, including
mtree (8)
bitrot (ports)
yabitrot (ports)
restic (ports)
* /lumia/ (https://lumidify.org/doc/lumia/lumia-current.html) can do
similar things using a *distributed* checksum database (each directory
contains a checksum of the files in that directory)
* I also ran across /snapraid/ (https://www.snapraid.it/), which does
some sort of raid 1 setup (I'm not sure of the details)
* I haven't yet found any program to compare checksum databases which
(a) automatically screens out files which "legitimately" differ between
the databases because the files were written/created in-between
the different backups, and/or
(b) handles "majority voting" among 3 or more checksum databases
to automatically identify which backup(s) is/are likely corrupted

Thanks again to everyone, ciao,
--
-- "Jonathan Thornburg [remove -color to reply]" <dr.j.thornburg@gmail-pink.com>
on the west coast of Canada
"The 'S' in 'IoT' stands for 'Security'."
-- commenter on /Ars Technica/, 2024-05-16

No comments:

Post a Comment