Glusterfs

Not sure what happened with GlusterFS tonight. Around 19:07 EST, the AWS alarm sounded. After rebooting the GlusterFS servers and the web servers, nothing was bringing the sites back online. There was a terrible lag when accessing the GlusterFS file systems and there weren't any glaring errors in the log files. In an effort to get the sites loading again, I began making an archive all of the files from the GlusterFS server mount point and was planning to copy them onto the main web server (web1).

This morning started off okay, until I checked my email and found an alarm ringing about one of our file servers being offline. Luckily, we have a replicated setup, so no customers were affected, but it is a little worrisome not having the safety net of two replicated servers. The server didn't technically crash, rather, it was a problem with the EC2 instance being unavailable likely due to some unforeseen problem in the Virginia datacenter. Regardless, the first step I took was to get the server accessible via SSH.