You may have already signed up with us…again…before we even post this announcement. If that’s the case, welcome back, and thank you for your support. We are really excited to officially announce that we are back in business.
We worked really hard the past two months to rethink, redesign, and rebuild the whole infrastructure from the ground up because who wants another complete crash? Yea, about that…We feel like we need to explain what happened and how we are completely avoiding that exact same situation from happening again.
Not saying that the previous infrastructure was not built right, it was, but we did make small mistakes designing our storage system. Although we did have storage redundancy in mind and designed it to avoid crashes like the one we had. We tested it multiple of times by purposely simulating outages, unplugging live equipment, disconnecting random cables and it was working great. The two major mistakes we did not think of were:
- We signed up every one of our customers under one giant storage pool
- Well..you already know this by now but we only had internal backups and not active backups for our customers.
What happened was since we had all of our customers under one storage pool, anything that happened to that storage pool impacted everyone, which is what happened to us…and you. Our redundancy system of choice (cough…GlusterFS…cough) confused customer data with old data that is no longer needed so it took it upon itself to start deleting everything. We went to sleep at night with everything working great, to waking up early in the morning with hundreds of alert emails, angry customer tickets and the whole system was down. By the time we made our announcement about it, everyone’s data was gone! Not fun 🙁
So how are we avoiding the same situation from happening again?
Well, we thought, and thought hard about this. We had 3 options:
- Give up, sell everything and never look back…never!
- Re-brand and start over…no!
- Own it, explain what happened to our customers and start fresh
As you can already tell, we chose option 3. We are a small group of people that are very passionate about technology and love working in this industry. We are all Computer Engineers Graduates and our goal is to see Hyper Expert thrive and succeed but most importantly, see our customers happy with our service.
The damage is done, mistakes happened, we all learned from it and have moved on. We designed our storage system differently. First, we are now only using GlusterFS for non-critical system services that even if it fails, it won’t impact our customers. Second, we no longer have one giant storage pool but have designed it to be a bunch of smaller redundant pools that each houses a very small amount of customers. Third, which is the most important, we are now including nightly onsite AND offsite backups for EVERYONE with all of our plans so you can rest assured no matter what happens, we will always be able to recover.
We worked and will continue working really hard to rebuild your trust and to offer the best service we can possibly offer. We want to make sure every single one of our customers is satisfied with our service. We will be introducing very competitive Web Hosting plans soon. Oh, and while we were at it, we also redesigned our website. Stay tuned for even more awesomeness from Hyper Expert. We are only just beginning 🙂