Doh! Raid10 in the dark ...

[Adrian Caspersz]

I have weekly scheduled backups and have successfully restored, I am in
a good place - now need to grow back some hair.

Lesson learned for a cheapskate Dell R620 home lab environment.

Do not set up a Raid 10 system (stripe + mirror) and forget to keep an
eye on monitoring the raid [1] for errors

- OR specifically,

forget to keep an *eye on the monitoring platform* for functioning
itself [2]

I had two drives grow bad sectors, so when I came to pull and replace
one of them, found the other bad disk was mirroring in the same stripe
group.

Hmmm...An uncorrectable array, that I could have avoided had I known and
replaced the first drive earlier[3].

So downtime[4], check all disks and restore all VMs from backup :(

Ho hmmmm... :)


1 - Especially when using cheapo £9 dubious 10 year old 900GB SAS drives
from eBay.

2 - Out of the box, Proxmox configures mail alerts via SMTP -
unfortunately that outgoing is blocked courtesy of my ISP/Spamhaus, so a
GMail workaround now implemented.

3 - Hmmm, time to look at configuring a "hot spare" hard drive.

4 - Yeah, not critical though. I'm lucky I don't do much infrastructure
work like this for a job!

[Andy Burns]

 forget to keep an eye on the monitoring platform for functioning
itself [2]

Don't PERC cards have a fault LED on the chassis

[Adrian Caspersz]

  forget to keep an eye on the monitoring platform for functioning
itself [2]

Don't PERC cards have a fault LED on the chassis?

Yeah, probably for failed drives where the whole raid would have gone
into degraded state.

Instead both the drives that were mirrored were screaming SMART pre-fail
messages that I could not hear.

Otherwise the RAID & drives were working fine silently correcting bad
reads until I pulled out one of the drives, then PERC announced the bad
blocks closely followed by a Proxmox VM backup hang.

[Jeff Gaines]

Do not set up a Raid 10 system (stripe + mirror) and forget to keep an eye
on monitoring the raid [1] for errors

My NAS has 4 x 2 TB SSD in RAID 10. In fact I only have 1.2 TB of data and
wonder if I should use them as 4 x individual drives - is that JBOD?