HexOS is a home-user front end for TrueNAS, and in October it added tools for pools with damaged files: Repair, Check pool, and Reset error count. This post is about what those messages mean, what HexOS can and can't do for you, and what to do before you delete anything. The same advice applies to plain TrueNAS or any ZFS pool where zpool status -v lists permanent errors.
First, a correction: Repair isn't the danger
When we read the October 6 release notes, "a Repair action that removes them" sounded like a one-click delete of your files. It isn't. HexOS's own documentation says Repair removes only damaged files from the TrueNAS app catalog, a list TrueNAS downloads again on its own. It re-reads the list on the server just before removing anything, and if any of your own files is on the list, Repair isn't offered at all. That part is designed carefully.
The risky step is the one HexOS leaves to you. For your own files, its instructions are: restore each file from a backup, or, if you don't need it, open the folder from your computer and delete it. That's fine when you really don't need it. It's a problem when "I don't have a backup" turns into "I guess it's gone."
What "can't be read" actually means
ZFS stores a checksum for every block it writes and checks it on every read. If a block doesn't match and the pool has a good second copy (a mirror or RAIDZ), ZFS fixes it quietly. If the pool keeps only one copy, or every copy is bad, there's nothing to fix it from. HexOS's page says this directly, and adds that the cause can be a failing drive, a loose cable, or a memory fault, even on healthy drives.
OpenZFS's ZFS-8000-8A page describes what happens next. Programs that try to read the corrupted data get an I/O error, and that data may be permanently lost. The key word is data. The error is on a block. A large file, such as a video, a disk image, a photo library database, or an archive, is made of many blocks, and the others usually still read fine. A normal copy in Finder, Explorer, cp, or rsync tends to stop at the first error and leave you nothing. That's why the file looks completely dead when most of it isn't.
If ZFS lists damaged files and you have no backup, the file isn't all-or-nothing. Copy what still reads to a different disk before you delete anything, and stop running checks on a drive that's getting worse.
What to do, in order
- Write down the list. Screenshot the HexOS message, or save
zpool status -von TrueNAS. That's your record of which files are affected and whether any snapshot (a name with @) is involved. - Check for a clean copy first. A backup, another computer, a phone, cloud storage. HexOS warns that a backup made after the damage may be missing the file, so look for one from before.
- Copy the readable parts to a different disk. Use a tool that keeps going past read errors. GNU
ddrescue, run from a Linux or Mac machine against the file on the share, copies everything it can and leaves the unreadable stretch empty, for exampleddrescue damaged.mov /Volumes/Other/damaged.mov damaged.map. Write the output to a different disk, never back onto the same pool. A video with a gap may skip for a second. A photo may be partly gray. That's often better than nothing. - Find the cause before you clear anything. HexOS's drive errors page separates a bad cable (one drive with errors, drive itself reports no problems), a bad controller (several drives on the same controller), and memory faults. Its deep memory test restarts the server and takes it offline for hours, so do it after your copy, not before.
- Only then delete, and run Check pool once. After a clean check, Reset error count is fine. Before that, leave the counts where you can see them.
What not to do
- Don't delete the only copy because the copy failed. A failed normal copy means one block didn't read, not that the file is empty.
- Don't run Check pool over and over on a one-drive pool. Each one reads every file on a drive that may be failing. If the list grows each time, the drive is telling you something.
- Don't delete snapshots to make the warning go away. HexOS says to ask in its community first. Old snapshots may hold earlier versions of other files you still want.
- Don't reset the error count to "fix" it. It only changes the number on the screen.
- Don't add a mirror drive and expect it to rebuild the damaged blocks. HexOS recommends a second drive so ZFS can fix future damage on its own, which is good advice. A new mirror copies the pool as it is now, bad blocks included. It can't bring back data that had no good copy.
When it's a lab case
A few damaged files on a healthy pool, with a cable or memory cause found, is not a lab case. Copy what reads, restore or delete, and move on. It becomes lab work when the pool has one drive and that drive is failing, when the damaged list keeps growing, when the damage is in metadata so whole folders or datasets won't open (OpenZFS's page covers that case too), or when the pool stops importing.
What we do for NAS and ZFS cases is image every drive first, sector by sector, and work only from the copies. We don't run scrubs, imports, or repairs on your originals. There's no ZFS job in our public case log yet, so we're not going to point you at a past result. And we'll be plain about the limit: a block that ZFS says has no good copy anywhere is usually gone for us too. What a lab can often save is everything around it, plus everything else on a drive that's failing, before it gets worse. We don't publish a success rate, and we won't promise anything before an evaluation.
Desert Data Recovery is a Phoenix-area lab, founded in 2013 and now a division of ioLogik LLC. Arizona drop-off is on the locations page. Everywhere else is mail-in. If you're shipping drives, pack them like this and label which bay each one came from. Call (602) 686-2622, Mon–Fri 9:00am–5:00pm.
HexOS files can't be read FAQ
Will HexOS's Repair button delete my files?
Can the damaged part of the file be recovered?
Does Reset error count fix anything?
zpool clear. Clearing the counts before you know the cause just hides the evidence.I deleted the file but the warning is still there. Why?
The damaged list gets longer every time I run Check pool. What now?
A file ZFS can't fully read is usually still mostly there. Copy what reads to another disk before you delete it, and stop checking a failing drive over and over. If the pool has one drive and it's getting worse, start with a free evaluation.
Request free evaluation →Free evaluation · No data, no fee · Talk directly with a technician.