Cookies help us deliver our services. By using our services, you agree to our use of cookies. Learn more

Server crashes, data loss and a one day downtime...

Hello there, EpicLPer here!

You might saw that our website was down for almost a whole day yesterday and we're sorry for that. Our server hoster had 3 vNode crashes over the last 2 weeks where the last one completely destroyed our server.

The MySQL database was damaged to a state where it wasn't able for us to recover it anymore, even accessing it just threw a ton of error messages. Most of the files, groups and permissions were broken or missing, the web server didn't even bother to start up correctly anymore other than displaying a crude error message when accessing the site. We were able to boot up the server to a state where we were able to retrieve the most important files still, but that was it. We tried to repair it, that however turned out to be impossible...

We had to reinstall our server completely from scratch. Luckily we had backups from the day before (we generate backups at 2 AM each day) so it wasn't as bad as I first thought it would be.

To prevent such a thing from happening again I'm currently thinking about starting a donation pool or something similar (maybe Patreon?) so we can get a second server for redundancy, right now I (EpicLPer) am paying everything out of my own pocket. But that's something I'll have to talk about with my team.

Anyways, sorry again for the downtime yesterday, I really hope that something like this won't happen again.

Also a huge thanks to Nuke928 for setting up the new server and getting it up and running in less than 2 hours, awesome!

Best regards, Stefan Kern (EpicLPer)