Global Update(MySQL, Issues, Etc.)

[Corey]

Update, Please Read Entire Post

Summary(For those with short attention spans).
1.) MySQL is messed up, we are moving it back to local.
2.) We are replacing our secondary 250GB drives with SCSI drives solely for MySQL.
3.) We are taking this time to format the servers and set them up efficiently and correctly.
4.) I ordered another server solely for backups for this occasion.
5.) I will update the bottom of this post with the progress of our actions.
/End Summary
Our original goal was to move all databases from the free server to a single remote server to handle all MySQL processes. MySQL is the #1 resource hog on each server. By doing this we would eliminate the number one cause of lockups and stability issues. The implementation looked easy, the cost was only $250\mo, easy decision. Wrong. We started having issues from the start. We knocked them out slowly one by one. One issue took a whole week, going back between our Datacenter and cPanel. We were finally were told that it's a cPanel bug and may be fixed in the next 3 months. Obviously not a good answer, we investigated and came up with a work around.

This brings us to our very last and most recent issue, PHPMyAdmin. Since we are a free hosting company and on occasion stress software to it's breaking point with our highly unusual usage, it makes it very hard for us to get support from the developers of software. Specifically MySQL and PHPMyAdmin. We were told many different things, tried A LOT of different things, and in the end decided to open a paid support ticket with our Data Center to see if they could figure it out. This brings us to the past two days. The data center had a collapse of communication between their technicians and downgraded MySQL on our database server to 4.1(We were at 5.0). This caused huge issues, on top of that the tech did not make a backup of the databases before doing so.

I exchanged some words with one of the managers yesterday who assured me he would take care of it and everything would be fixed. At this point I just wanted the databases back online, the PHPMyAdmin issue could wait. Well, another catastrophic miscommunication occurred. They made some stupid decisions and delayed the process of fixing things for hours. I got a call from Bryon last night while I was out to dinner for a friend's birthday(I have a life sometimes) informing me of the ticket updates. I then spent 45 minutes on the phone outside going over everything with a tech. They finally got the database server back up this morning. There are of course issues with some databases because of the downgrading and upgrading. We have about 15 backups from the past week that are floating around, we will get everyone databases back up and running when we resolve this.

Where we are at now? Without a definitive answer for PHPMyAdmin in sight I have made the decision to move MySQL back locally, but replace the secondary 250GB HDDs we have with 73GB 15K SCSI drives at an extra cost and have MySQL run solely off these drives easing the burden on the I\O of the main drives. I have also decided that it would be best to format each server one by one and start fresh with all updated software. We've had so many updates, patches, fixes, etc(Apache 1.3->Apache2.2;ASP.NET Support, MySQL 4.1->5.0) over the past few months that these servers are not running anywhere close to efficient. By doing this each server will be 100% optimized and should be good to go for a long time to come.

Implementation. I have just purchased a server solely for backups. We are going to do this one server at a time by migrating all accounts to the backup server, formatting the current server, optimizing and securing then moving the accounts back. I'm hoping we can finish all this in the next two days for all servers. This is obviously not what I really want to do as it is going to cause downtime. But I feel that the overall improvement that we will get out of this will be worth it

So, please be patient, I am working as hard as I can on this. For those of you who have been with us a while, you know we will work as hard as possible until we solve these issues. We have a great team of staff members, and lots of drinks with caffeine, we will be around to answer questions until we are done. I will also try to keep everyone updated on our progress as much as possible. Thank you for sticking with us, we'll be back to 100% soon.

-Corey
You will not lose your data\accounts, this does not affect paid members, please excuse typos.

Progress:
Waiting for backup server to come online, ETA 2 hours.
Backup server is online, recompiling Apache, setting up MySQL, and updating cPanel to prepare for the account moves.
Server setup complete, copying accounts from Absolut over to backup server.
Final backup of whole hard drive before format
Format and reinstall in progress.
Copying Absolut backup accounts back to Absolut.
Absolut back online.
Stoli in the process of being backed up to the backup server.
All accounts were successfully copied to the backup server.
Accounts are being copied from the back up server back to Stoli.
Stoli has completed the copy back. Please post all move issues in the server move forum.
Starting Cossacks transfer to backup tonight 1/24/08
Cossacks completely transferred to backup.
Cossacks to be formatted in the next 2 hours.

All server moves are complete!

Any remaining issues from the ABSOLUT, STOLI OR COSSACKS transfers should be posted here:
http://forums.x10hosting.com/server-move-issues/
Read the sticky threads before posting!

Upgrade Optimization Order:
Absolut Done
Stoli Done
Cossacks Done

F.A.Q.
Q: Will this move everything back to localhost?
A: Yes, all config files and such will have to be changed back to localhost

Q: Will I lose anything?
A: You should not lose anything, we have multiple backups of everything. Worst case you might lose a few hours of updates from MySQL while your account was being copied between servers.

Q: How will I know when it is done?
A: I will update this post as we complete stages.

Q: Does this affect paid services?
A: No, these move and updates are only going on for free servers.

[Corey]

Do not post support requests here, I will not answer them. Legitimate questions are okay to ask.

[MicrotechXP]

Also, please do not post topics about this, we will tell you when this issue is fixed!

[Nikolasr]

Good corey thanks for this

[Smith6612]

Thanks for the update Corey. I already backed up all of my data this morning when MySQL was up and running long enough for the download to finish, along with all of my data so I should be set if something were to happen with my site. Well, this is an unexpected bump in the road, but it should improve the service a lot once this is done.

[Momiji]

Thank you very much Corey.

You guys are trying your hardest and people can't seem to grasp that.

[animerth]

If your moving MySQL back on to the servers what will be the performance cost even with the optimization

Thanks for all your work

[Smith6612]

There shouldn't be much of a performance issue with the servers. Despite SQL making the servers get busy, just as long as there is enough memory, bandwidth and CPU, which there usually is, this should be fine. The site when I first got it here on x10 before the SQL move had very good page generation times and were very consistent. On the current setup which is about to be trashed and moved back to what Corey said, page generation times were all over the place.

[online87]

Corey & the guys,

#1 - Seriously, thank you VERY MUCH for the detailed and honest update alert. It's nice to be told bout stuff.

#2 - I notice... From 250Gb HDD to a 75Gb HDD (Raptor by any chance?)... that's a pretty big space sacrifice! Although the good thing about SCSI is the ability to link a fairly large number of drives to one controller with no performance hit. Is that your eventual intention?

#3 - I'd like to thank you guys for providing a great free service, but unfortunately it is getting to the point where sometimes I just think "For ****'s sake they're down again.... Jesus..." - as you have to admit you have had an unacceptable amount of down time (even for the average free service), BUT...

#4 - It's really great how dedicated you are to providing the fastest, most up-to-date, 'cool' free hosting service out there. I've tried others and they lag months behind with updates, and don't really care about their free customers. Although it often pisses me off, it's nice to be a customer of such dedicated and professional personnel.

Keep it up, I hope it all goes as smoothly as possible, and sorry if I'm cheeky but let's hope this really DOES solve your problems, not just for my sake but for yours too.

[coolv1994]

Well good luck with that and thanks for the update it sound like a load of fun.