Several Drives Dropped at Once. Suspect the Controller, Not the Disks.

Written and maintained by the Desert Data Recovery lab team. · Updated October 22, 2026

We already wrote about RAID 5 with two drives down, about a mirror whose first failure sat for weeks, and about rebuild versus recovery. We wrote about a ZFS scrub and Replace, about zpool import -F, and about Force Online after the RAID battery dies. Synology's button is Repair. QNAP's is Recover. A Storage Spaces Direct pool that went read-only is that page. This page is the screen those jobs are not: several members drop together, and the part they share is still in the box.

Same minute, same path

Read the timestamps before you read the disk count. A member that failed in March and a member that failed this morning are a staggered pair. The RAID 5 page and the mirror page are about that gap. The data on the survivor is often still coherent because the first disk left the set earlier, and the second did not take the first disk's blocks with it. That is a media story, told slowly.

A path story is the other shape. Five bays on one backplane go missing at 01:43. Every disk behind one mini-SAS cable shows write errors in the same minute. An expansion shelf disappears and the head unit's own disks stay up. A ZFS vdev reports UNAVAIL on the disks that share an HBA, and ONLINE on the disks that sit on the other card. The disks have one thing in common, and it is not a manufacturing week. It is the wire, the expander, the controller, the enclosure, or the power rail in front of them.

The parts that do that job:

• The HBA or RAID controller. The card that the disks, or the expander, plug into. A reset, a firmware fault, or a card that stops completing commands looks like every disk behind it failing.
• The SAS expander or the backplane. One expander feeds a row of bays. When it drops, the row drops. The disks at the other end of the chassis stay where they were.
• The cable. One internal or external cable is the whole path for every disk downstream of it. A half-seated connector is a multi-disk failure.
• The enclosure. A NAS expansion unit, a JBOD shelf, or the drive cage's own board. The head unit can be healthy while the shelf is not.
• Power. A supply, or one rail of a supply, that feeds a row. Those disks spin down together. The disks on the other rail do not.
• Firmware on that shared part. A controller build that mishandles one disk's timeout can spread the timeout to the other disks on the card. That is a property of the card, not of the platters.

This list is the order to suspect. It is not a diagnosis of your chassis. A screenshot of six red bays does not name which line in the list you are on. The bay map does. Which disks share a port, and which disks stayed up, is the whole test.

Same-model wear is real, and it is a different clock. Drives from one family can fail in a cluster over a service life. That is the note on our Synology RAID 5 (DDR-2026-0003), where members of the same 3 TB generation failed and the array was never rebuilt in the NAS. Those failures are not four bays emptying in the same minute while yesterday's SMART was clean. If the dates are months apart, use the RAID 5 page. If the dates are the same minute, stay here.

What a public RAIDZ3 outage actually showed

On October 8, 2026, InfiniCLOUD posted an incident report for their zeze node (the zeze1 pool). The incident date they give is October 1, 2026 (JST), and as of that report it was ongoing. This is their outage, not a Desert Data Recovery case. The report separates log facts from inference, and it says the investigation was still open. What follows is their account, not a lab finding about your NAS.

The pool was 11 disks in ZFS RAIDZ3. RAIDZ3 is triple parity. OpenZFS's description of raidz is that a group sustains as many disk failures as it has parity disks — one, two, or three — without losing data. InfiniCLOUD's pool could lose three disks. Six of the eleven sat on one card, so a fault in that card could take out more disks than the parity covered.

Their timeline, in their words: at 01:43 on October 1 a disk on the control card stopped responding to commands. At 01:45 one disk failed and was disconnected, and the pool kept running on the other ten. Around that failure, other disks on the same HBA also became unresponsive, and some data was already corrupted past what RAIDZ3 could rebuild. They state the rule they hit: RAIDZ3 cannot recover a data set when four or more locations in it are corrupted at once. As of October 1, 61 files were unreadable, including system files, and the service was shut down at 07:06. That was still inside a pool that had only fully lost one disk.

They then say the part people skip. All ten remaining disks were reported DEGRADED. They write that this does not mean all ten failed. When ZFS reads data it cannot recover, it cannot tell which disk holds the bad data, so it records the error against every disk. Past a threshold, every disk was classified DEGRADED. OpenZFS documents the same rule on the zpoolconcepts page: if a block cannot be reconstructed, checksum errors are reported for all disks the block was stored on. Their example is a raidz2 with three damaged disks. InfiniCLOUD's was a RAIDZ3. The mechanism they are describing is that one. A status line of DEGRADED on every disk is not a parts list of dead drives.

The next day the shared card kept costing them disks. Their summary says three more disks on that same control card became unusable, one after another. Counting the first disk, four had failed. The timeline is the same fact in smaller steps: on October 2 two further disks stopped responding, the pool went SUSPENDED (they define that as stopped reading and writing), and at 19:21 a fourth disk became unavailable and the pool was UNAVAIL (inaccessible because too few disks remained). RAIDZ3's budget is three. Four took the pool out.

What they say they know from the logs, as of October 8:

• The first fault was one disk. It answered commands with an internal-failure response.
• Just before that, the control card logged processing delays. Other disks on that card missed commands or were not recognized.
• Six of the eleven disks were on this one card.
• The logs recorded no communication errors on the cables between the card and the disks.
• The card was on an older firmware build the manufacturer had already updated.
• The disks that became unusable on October 2 were all on that card.

What they mark as inference: one disk's internal failure set off instability on the other disks on that card, because a card that sees a disk stop answering aborts commands and resets the disk, and they believe those recovery steps disturbed the other disks on the card. They name two contributing conditions. Six disks on one card is more than RAIDZ3 can lose. And they were investigating whether a flaw in the old firmware's error recovery helped the fault spread. They say, in the same section, that the cause of the three October 2 disks was still uncertain — the card or the disks — and that they were prioritizing the card because it was the part those disks shared.

They do not call it a backplane failure. They explicitly record no cable errors. The shared part in their report is the HBA. A backplane, an expander, a cable, or a power rail can paint this screen on someone else's chassis. Their log named the card. Yours has to name whatever your disks actually share.

The day before the report, on October 7, they posted a firmware-maintenance notice. It says their investigation of failures they date to August 19 and October 1 found that, on this type of HBA, one drive failure can trigger several drives to fail together. The work they scheduled is a firmware update, to replace a bug in the old card's error recovery, finished on their side by the end of November 2026, with the service down during each window. That page does not publish an August 19 disk-by-disk timeline, and it does not name the card vendor. The October 8 report is the careful one: investigation ongoing, October 2 still uncertain. Read the notice as their maintenance plan. Do not read it as a firmware string you should flash tonight.

Their own prevention item is the design point. They said they would rearrange connections so one control card, or one cable, cannot hit more disks than RAIDZ3 can lose, and they said a single pool on a single machine is how this class of fault becomes a total loss. That is their plan for their cloud. It is also why a resilver on the surviving disks, through the card that just dropped six of them, is the wrong next click on a pool you only have once.

What not to do while the path is still dropping drives

Do not replace the red disks one by one. Pulling the first UNAVAIL member, inserting a spare, and letting ZFS resilver — or letting a hardware controller rebuild — assumes the other members are healthy and the path to them is stable. You do not have that. The spare activation is the same write. It reads every remaining disk and writes a new one. If the card is still aborting commands, the resilver is more traffic of the kind InfiniCLOUD believes spread their first disk failure to the rest of the card. That belief is their inference about their firmware. The instruction does not depend on it. A rebuild through a path that is still dropping drives faults more members, and it writes the ones it can still see.

Do not clear, force, or re-create the pool. zpool clear clears the device errors. OpenZFS's clear page says that if the pool was suspended, clear brings it back online provided the devices can be accessed. The disks are still behind the same card. zpool online returns a member to the vdev. Force Online, Bring Online, and Mark as Good on a RAID card put a dropped member back into a set that has moved on, or into a set whose other members are still missing. zpool import -F and -FX rewind transaction groups. They do not reconnect a cable. zpool destroy, zpool labelclear, and zpool create write new labels. A controller "create virtual disk," "clear configuration," or initialize does the same job in a different menu. Unraid's Format and New Config are the Unraid page. The old pool configuration is the map. Replacing it so the chassis looks empty is how the map gets thrown out.

Do not leave a rebuild running. If a resilver, a parity rebuild, or a consistency check has already started, it is reading the survivors through the suspect path and writing the result. Power the chassis down. Letting it "finish overnight" is how a path fault becomes a set of half-written members. The rebuild page is the general argument. This page adds the path: the rebuild is only a maintenance job when one disk failed, the rest are healthy, and the bus they sit on is not the thing that failed.

Do not flash the card, the backplane, or the disks to see if the lights go green. InfiniCLOUD's October 7 notice is maintenance they planned for their own servers, with the service taken down on purpose. Their October 8 report says that on October 1, while the incident was open, the data center worked on the control card and updated its firmware, and that the cause of the disks lost on October 2 was still unknown the following week. We are not saying that update caused the later failures. They did not say that. A firmware update is still a write to the only controller, or to the only disks, and it does not put back a block the pool has already failed to read. If a vendor later confirms a firmware issue, that work happens after the members are imaged, on a card you can swap, not as a test of the patient set.

Do not keep power-cycling. Each power-on retrains the links and runs the card's recovery again. If drives are still disappearing across those cycles, the cycles are the fault getting another turn. Shut it down and leave it down. Photograph the screen first if you can do that without clicking Repair, Recover, Clear, or Import.

Do not carry the disks to another machine loose. Bay, expander port, and serial are the diagnosis. A PC that sees a former member will offer Initialize or Format. A second NAS will offer to create a pool. Those are the writes on Initialize Disk and on the Synology and QNAP pages. Label the slot before a drive comes out. A drive goes back only to the slot it left, and only if you are not done with it.

The buttons on the other pages are still the wrong buttons. Synology Repair and QNAP Recover write. Storage Spaces Direct's read-write flag writes. Force Online after a power cut can commit or drop controller cache — if the battery is also dead, read that page before this one, because cache is a separate loss. None of those clicks answers "which part do these bays share?"

What to do right now

1. Stop the write that is already running. If a resilver, a rebuild, a repair, or a spare activation is in progress, power the chassis down. If members are still vanishing while it sits powered on, power it down anyway.
2. Save the status before anyone clears it. zpool status -v is a read. So is a photograph of Storage Manager, DSM, QTS, or the RAID BIOS. Keep the disk names, the states (UNAVAIL, FAULTED, DEGRADED, missing, failed), the READ / WRITE / CKSUM columns if they are on screen, and the time. A controller event log is worth a photo when you can open it without accepting a repair.
3. Map the path on paper. Which bays share a backplane, an expander, a cable, a controller port, or a power supply. Write the serial next to the bay. A row that failed together is the row to mark. Disks that stayed online tell you where the shared part stops.
4. Write down the layout if you know it: raidz1, raidz2, or raidz3; mirror; RAID 5, 6, or 10; SHR; hardware RAID level; the controller model. The parity budget is how many disks the layout can lose. It is not how many disks one card is allowed to take with it. If the volume was BitLocker-encrypted, bring the recovery key. We cannot brute-force BitLocker. See Windows Device Encryption / BitLocker.

If a disk has to be shipped, pack it like any other failed drive and keep the bay label on it. If the only record of which disk sat behind which port is the chassis, send the chassis, or send the controller with the disks. Do not open a drive.

How a lab handles it

We do not resilver your pool on the HBA that dropped it. We do not zpool clear, zpool replace, zpool online, or zpool import -F on the patient disks. We do not Force Online, rebuild, or create a new virtual disk on the original controller. We do not flash that controller to test a theory.

Each member is imaged write-blocked, on a path that is not the one that lost the set, including a disk the UI had already marked failed or UNAVAIL. The pool or the array is read from those images. A wrong guess about which member is stale costs a clone, not the original. Imaging uses the lab tools already named on this site, including the ACE Lab PC-3000 and DeepSpar Disk Imager. Those tools image the drives. They are not a second NAS, and they are not a second HBA running your pool.

The public case log does not have a simultaneous multi-disk HBA or backplane dropout. We will not invent one, and we did not work the InfiniCLOUD pool. The closest published server job is 21607, an HPE SmartArray with two arrays. The RAID 5 had no significant drive failures; the controller dropped offline. Every drive was imaged, and nothing was rebuilt on that controller. The RAID 6 on the same server is the other pattern: members had fallen out months apart. A Synology RAID 5 (DDR-2026-0003) is two failed members of one model family, imaged and reconstructed off the NAS. Use those for what "image first" looks like. Do not use them as a twin of a row of bays that failed in the same minute. We don't advertise a success rate. The free evaluation is how you find out whether the disks are dead or the path in front of them is.

If the dates are staggered, start with two drives down or the mirror that ignored the first failure. If the prompt is Replace during a scrub, start with don't race the replace. If the pool will not import, start with zpool import -F. If the button is Repair or Recover, start with the Synology or QNAP page. For the work itself, see RAID data recovery.