RAID 5 failed during a rebuild? What to do next

One disk in your RAID 5 array failed, you replaced it, the rebuild started, and partway through a second disk dropped out. Now the server or NAS reports the array as failed, offline or “inactive”, and the shares have gone. This is one of the most common ways RAID 5 loses data, and it is also one where the wrong next move can turn a recoverable situation into a very difficult one. Here is what has happened and what to do.

Why RAID 5 cannot survive two failed disks

RAID 5 spreads data and parity across all the disks in the set. Parity is extra information that lets the controller work out the contents of any one missing disk from the others. That is the whole point of RAID 5: it tolerates exactly one failed disk. While one disk is missing the array is “degraded” and still works, but there is no spare redundancy left. If a second disk fails, or even just cannot read some of its sectors, the controller no longer has enough information to rebuild those stripes, and it takes the array offline.

Why the second failure so often happens during a rebuild

It can feel like terrible luck, but there are good reasons a second disk tends to give up at this exact moment:

• A rebuild reads every sector of every remaining disk. In normal use much of the data is rarely read, so weak sectors can go unnoticed for months. The rebuild finds them all at once.
• The disks are usually the same age. Drives bought together, from the same batch, and run for the same hours tend to wear out at a similar time.
• Large disks take a long time to rebuild. With multi-terabyte drives a rebuild can run for many hours or days under heavy load, which is exactly when a tired disk is most likely to fail.
• The first failure may not have been the first problem. Sometimes a disk dropped out of the array weeks earlier without anyone noticing the alert, so the “first” failure was really the second.

What not to do

Most of the damage in these cases is done after the second failure, while trying to get the array back:

• Do not re-create or re-initialise the array. Creating a new array over the same disks, even with the same settings, can overwrite the metadata and in some cases start writing fresh parity over your data.
• Do not force a failed disk back online unless you know exactly which disk failed when and what has been written since. Forcing in the wrong disk can bring back stale data and corrupt the file system.
• Do not start another rebuild or swap in more disks to “see if it works”. Each attempt puts more strain on disks that are already failing.
• Do not change the disk order or move disks between bays without labelling them first.
• Do not run file system repair tools (fsck, CHKDSK) on a volume that has been forced online. Repairs on a half-assembled array write changes based on wrong data.

Safe steps to take now

1. Stop. Shut down the server or NAS cleanly if you can, and leave it off.
2. Write down what happened: which disk failed first, when the replacement went in, how far the rebuild got, and any error messages or log entries. Screenshots of the controller or NAS storage page are very useful.
3. Label every disk with its bay or port number before removing anything, including the disk you took out first. Keep that original failed disk; it may still hold data that helps the recovery.
4. Do not reuse the disks in another system, and do not let anyone “try something” on them until a plan is agreed.
5. Check your backups. If you have a recent, tested backup held separately from the array, restoring from it may be quicker than recovery.

How the data is recovered

A lab does not try to repair the array in place. Each disk is assessed and, where needed, physically repaired, then imaged sector by sector so the original disks are not stressed further. The array is rebuilt virtually from those images: the disk order, stripe size, parity layout and the point at which each disk dropped out are worked out from the data itself. Weak sectors on the second failed disk can often be read with specialist tools, and where they cannot, parity from the other disks may fill in the gaps. The file system is then read from the virtual array and the files copied to new media.

The outcome depends mainly on the condition of the disks and on what has been done since the failure, which is why stopping early matters so much. You can find more detail on our RAID 5 data recovery page, and on server data recovery if the array is in a rack or tower server.

Preventing it next time

• Act on alerts as soon as a disk drops out, and make sure alert emails actually reach someone.
• For large disks, consider RAID 6 or RAID 10, which can cope with more than one failed disk in many cases.
• Remember RAID is not a backup. Keep a separate copy of important data, ideally one held off-site.

Array failed during a rebuild?

Power it down and call Data Retrieval on 020 7064 4996, or request a quote online. We offer a free assessment with no obligation, so you will know where you stand before any work is done.