Around 1530HRS EST today (March 20th, 2021) we were alerted by our monitoring service that the MYSQL service on Server1 has gone down. We immediately begun to investigate and it showed that the MYSQL database was unable to be rebooted. Upon, further investigation it was determined the hard drive containing the MYSQL databases had failed and is corrupted.
Given the inability to recover data from the disk we will be restoring backups that were created this morning at 0100HRS - consisting of all data.
We are currently setting up a brand new webhosting server and once completed will begin the Disaster Recovery restore from our 0100HRS backup. We are terribly sorry for the inconvenience and apologize in advance for the extended period of downtime. However, our main focal point right now is to get customers back online.
Provided below is the timeline of our updates. We ask at this time if possible to refrain from creating tickets in relation to updates as this will slow down support for the remainder of our customers not effected today. As we have updates we will provide them below:
2021 - 03 - 20 1530HRS --- Monitoring Alert regarding MYSQL Service DOWN.
2021 - 03 - 20 1818HRS --- Confirmed Hard Drive Failure
2021 - 03 - 20 2000HRS --- New Server Racking / Setup.
2021 - 03 - 20 2250HRS --- We are now setting up the OS and other softwares. We will be using a new hostname and IP's for this server - new Dedicated IPs will be sent to all affected.
2021 - 03 - 21 0030HRS --- We've installed CloudLinux, Imunify360, Softaculous, Litespeed, Attracta, CloudFlare & Mailchannels.
2021 - 03 - 21 0100HRS --- MYSQL, PHP have all been setup.
2021 - 03 - 21 0200HRS --- We've begun to restore accounts and can see websites coming ONLINE! Please read below to adjust any settings.
2021 - 03 - 21 0400HRS --- Accounts are continuing to migrate successfully. Accounts bigger in size will take longer to migrate. We appreciate everyone's patience thus far in the process!
2021 - 03 - 21 1227HRS --- 76% of the Accounts have been migrated over. Since, more website are starting to come online we are seeing an increase in support tickets for minor fixes. We want to assure you we are working around the clock to get answers to you ASAP.
2021 - 03 - 21 1326HRS --- 91% of the accounts have been restored. We're also making good progress on support tickets.
2021 - 03 - 21 1433HRS --- All the accounts have now been migrated. We are now working on assigning respective Dedicated IP's to customers who had them prior.. and working on clearing support tickets.
NOTE: The server will be a little slow while we continue to run updates... there is also our real-time virus scanner running to ensure all files being restored are malware FREE. This will also cause sluggish performance, however once completed things will be back to normal.
2021 - 03 - 21 0400HRS --- All customers have been provided with their new Dedicated IPs and we consider this event CLOSED.
We now consider the migration completed and will continue to monitor the server. NO further updates will be posted here unless required. There may be periods when services restart as we fine tune during monitoring. Once again we would like to extend our apologies to everyone and all those effected. We truly appreciate everyone's positive response to our disaster recovery approach.
POST MIGRATION COMMON ISSUES & FIXES
Please read the below new DNS servers.. you can still continue to use the old nameservers.
Please access Cloudflare -> DNS and change your IP from 22.214.171.124 to 126.96.36.199 for every record with the old IP. Otherwise, if you use a Dedicated IP please check your account and switch to your Dedicated IP.
If you are getting the above 500 ERROR it means you are missing a PHP option. Please check your error_log and ensure all required php values are enabled with the right PHP version in cPanel under PHP Selector.
Nonetheless, if you are experiencing any issues please reach out to us via a support ticket and we will gladly assist you out.
Saturday, March 20, 2021