RAID rebuild failed: do not restart it. Diagnose first.
When a RAID array rebuild fails, the instinct is to "try again". That is almost always the wrong move. The reasons behind the failure need to be diagnosed: restarting blindly only adds stress to the most fragile disks and accelerates further failures.
Three typical causes.
- URE on another disk: during the rebuild, the controller hits an unreadable sector (Unrecoverable Read Error) on one of the surviving disks. On large arrays with high-capacity disks, the URE Bit Error Rate declared by the vendor (e.g. 1 in 10^14 or 10^15) means that while rebuilding TBs of data there is a non-negligible probability of hitting one.
- A second disk at its limit: a disk that was still "ok" under normal load gives out under the additional stress of the rebuild. Typical with disks from the same production batch.
- A controller throwing errors: in some cases the rebuild fails not because of the disks but because of the controller itself (buggy firmware, intermittent backplane, an exhausted cache battery that drops write-back).
Diagnosis before the next attempt.
- Reading the controller log: identifying the exact sector where the rebuild failed.
- SMART check on the surviving disks: which one is marginal?
- An informed decision: retry the rebuild (replacing the most marginal disk first), or recover from clones if the disks are too fragile.
The questions we get most often.
Can I restart the rebuild if I replace the most suspect disk first?
Sometimes yes, if the diagnosis shows that the second disk is the weak link and all the others are genuinely healthy. But not without a diagnosis: replacing disks by trial and error slows the recovery down and introduces new risks.
On large arrays (10+ TB per disk), can a rebuild fail "by design"?
Yes, this is the well-known problem of RAID 5 with high-capacity disks. URE Bit Error Rate × the volume of data re-read during the rebuild gives a non-negligible probability of hitting one. That is why, on large arrays, RAID 6 (double parity) is almost always more rational than RAID 5.