August 2010 crash: Difference between revisions
| Line 125: | Line 125: | ||
In this case our RAID supplier was updating the firm ware and while he did that he crashed it. We managed to get most of it back but one of the DB didn't handle not having a harddisk anymore very well and that one had to be restored from backup. Which had ended about 20-30 minutes before the crash happened. | In this case our RAID supplier was updating the firm ware and while he did that he crashed it. We managed to get most of it back but one of the DB didn't handle not having a harddisk anymore very well and that one had to be restored from backup. Which had ended about 20-30 minutes before the crash happened. | ||
}} | |||
{{forum_message| | |||
| from = [[HT-Tjecken]] | |||
| to = [[User:Pupske|Pupske]] | |||
| msgid = 14298242.135 | |||
| prevmsgid = 14298242.48 | |||
| datetime = 25.08.2010 at 9:54 | |||
| keywords = (Downtime) | |||
| text = Pupske wrote: | |||
''Anne, it's too bad it happened, but we can't turn back time. What I'm really interested in is what is Hattrick going to do to prevent such thing from happening again? | |||
----- | |||
Not updating firmware ever again. | |||
We will naturally see what we can done to prevent this, but right now I have no good answer for you. We're still not over this situation, the site is up again and most funcionality is back - but we still got a lot do with our disk performance before we can call this over. And then we can start looking if there are more things we can do to avoid situations like this. That said, as someone pointed out earlier on - one can never totally avoid the risk of disk crashes, downtimes etc. But is should naturally be kept to the very minimum. | |||
}} | }} | ||
Revision as of 10:31, 25 August 2010
On monday 23 August 2010 Hattrick went down arround 15:20u HT-time
Official announcement
| Down Time | 24.08.2010 |
| As you have noticed, the site was down since 15:20 HT time yesterday. And here is what happened.
We were performing routine maintenance and upgrades to our servers, our hard disk supplier was on site to perform an upgrade to the disk's firmware on our database storage solution. Unfortunately, a bug in this firmware caused all our disks to crash, which meant that we had to restore most of our databases from backup. Because of the enormous amount of data needed to be restored, it took all night. Thankfully, the last backup ended at around 15:00 HT time yesterday, so just before the site went down. This means that only the data for, give or take, 30 minutes was lost. How does that impact your team? Read more about it in system info. These have been circumstances beyond our control, and we are very sorry for the inconvenience this may have caused some of you. | |
Downtime messages
| Hattrick announces | 23.08.2010 |
| During routine maintenance today, our disk cabinet supplier introduced an upgrade to firmware which resulted in our database being erased. This means we need to go back to a backup which was made a few hours earlier on Monday. Because of the enormous amount of data needed to be restored, this will take at least all night to get done. We will then need to sort through the inconsistencies in the data and let the heart of Hattrick catch up for the down time. We are hopeful to be back online by tomorrow afternoon. At this point we can't guarantee that we will be back before Cup games start, but it is our goal and we will keep you updated throughout Tuesday. | |
| Hattrick announces | 24.08.2010 (9:50) |
| Update 09:50: We have now restored all data from a backup taken around 15.00 yesterday (which means not that much data has been lost). We're checking for inconsistencies in the restored data now, and estimate to be back online around noon. | |
| Hattrick announces | 24.08.2010 (11:30) |
| Sorry to have to take the site down again, but the new disks are too slow as it seems. We're investigating, together with our hard disk supplier. | |
| Hattrick announces | 24.08.2010 (13:00) |
| During routine maintenance yesterday, our disk cabinet supplier introduced an upgrade to firmware which caused all our disks to crash. Because of this we had to restore our databases from a backup, taken about half an hour before the disk crash. This morning we've checked for inconsistencies in the restored data, and the site is ready to launch. Unfortunately, the disk performance is too slow to open the site. We need to wait until our hard disk controller has finished checking the disks for errors, and this will take some hours more. We hope to be back around 16:00 HT-time. We regret any inconvenience this may cause you. | |
| Hattrick announces | 24.08.2010 (14:25) |
| 14:25 HT-time: During routine maintenance yesterday, our disk cabinet supplier introduced an upgrade to firmware which caused all our disks to crash. Because of this we had to restore our databases from a backup, taken about half an hour before the disk crash. This morning we've checked for inconsistencies in the restored data, and the site is ready to launch. Unfortunately, the disk performance is too slow to open the site. We need to wait until our hard disk controller has finished checking the disks for errors, and this will take some hours more. We hope to be back around 17:00 HT-time. We regret any inconvenience this may cause you. | |
Arround 17:00 the site started to act very odd, and even brought good old Hammo back. Not a good sign.
| Hattrick announces | 24.08.2010 (16:45) |
| 16:45 We will have to keep Hattrick offline for at least a few more hours, in order to increase the disk performance to open the site. Unfortunately, matches scheduled during this downtime will be played. Transfer deadlines will however be extended. We humbly apologize for all the inconveniences this may have caused you. Rest assured that we are working really hard to remedy the situation as quickly as possible. | |
By 19:15 the site was up again. By 7:30 the next day everything (including the forums) worked again.
Forum comments
| From: | HT-Anne | (14298242.5) as reply to (14298242.4) | |
| To: Everyone | 24.08.2010 at 19:09 | ||
| So we brought the site back up a little before noon and then we noticed our disks being excruciatingly slow. We had to run a disk check, which meant taking the site back down. The disk check is taking a very long time, and thus we opted to migrate the Data to another set of disks to be able to be up and running as fast as possible. We were able to abort the disk check to finish the migration. And we are now up and running.
All the problems stem from a buggy firmware update, made by our HD cabinet vendor, which crashed our Database. The Youth section of the site is currently unavailable, we are working to get it back up as soon as possible. | |||
| From: | HT-Johan | (14298243.5) as reply to (14298243.4) | |
| To: Everyone | 25.08.2010 at 8:49 | ||
| The forums are now back up, as you can see. We tried to bring them back online with the rest of the site yesterday evening, but as we had to move the game engine to our backup disk cabinet (which is a bit slower and has a great deal smaller storage capacity) the forums just were too slow to function. (This also why youth could not open either).
We now have them on another temporary disk for the forums and we will have to see how well that works. It is not inconceivable that we will need to take down the forums again when the site gets busier later today. The main disk storage - the one that caused the problems in the first place - is still being prepared to take on the load of the whole system. We can do this over here I realize now :) (14298242.1) | |||
| From: | HT-Anne | (14298242.54) as reply to (14298242.25) | |
| To: Gambit_ | 25.08.2010 at 9:05 | ||
Gambit_ wrote:
Youth teams are still down to allow the youth engine to catch up. They should be back shortly. What do you mean shortly? I am waiting for opening YA since yesterday evening in order to send lineup for my friendly match which starts today at 8:00 :( I'll update the information over there. We don't have an estimate on when the Youth Academies will be back up yet. They are now running on the same disks as the forums and we need to monitor performance to be able to give you an estimate. Right now the Youth engine is trying to catch up for the down time. As soon as I know more I'll of course inform you. | |||
| From: | HT-Anne | (14298242.76) as reply to (14298242.48) | |
| To: Pupske | 25.08.2010 at 9:22 | ||
| Pupske wrote:
Anne, it's too bad it happened, but we can't turn back time. What I'm really interested in is what is Hattrick going to do to prevent such thing from happening again? From the details of what happened, I dont think there is very much that we could have done to prevent it from happening. In this case our RAID supplier was updating the firm ware and while he did that he crashed it. We managed to get most of it back but one of the DB didn't handle not having a harddisk anymore very well and that one had to be restored from backup. Which had ended about 20-30 minutes before the crash happened. | |||
| From: | HT-Tjecken | (14298242.135) as reply to (14298242.48) | |
| To: Pupske | 25.08.2010 at 9:54 | ||
| Pupske wrote:
Anne, it's too bad it happened, but we can't turn back time. What I'm really interested in is what is Hattrick going to do to prevent such thing from happening again? Not updating firmware ever again. We will naturally see what we can done to prevent this, but right now I have no good answer for you. We're still not over this situation, the site is up again and most funcionality is back - but we still got a lot do with our disk performance before we can call this over. And then we can start looking if there are more things we can do to avoid situations like this. That said, as someone pointed out earlier on - one can never totally avoid the risk of disk crashes, downtimes etc. But is should naturally be kept to the very minimum. | |||
| From: | HT-Anne | (14298242.283) as reply to (14298242.271) | |
| To: Schnuff | 25.08.2010 at 11:29 | ||
| We were back before noon yesterday. That is about when the data recovery ended. But then disk checks made the site unbarebly slow. So we had to take it down again. The information regarding that was available on the down web.
While reading you, I feel like you think we want to cause our users harm. Or don't care that some countries had their cup matches starting without them having anytime to check orders. I can assure that nothing is furthest from the truth. We tried all we could to bring back the site before 18:00 HTTime, which is whent he first cup matches started. We even migrated data to another set of disk, but we then were only able to make it back for 19:17. It's unfortunate, but at one point, we were told another 24h. And I can tell you, that I then had tears in my eyes. Because I knew just what that meant for users. | |||
| From: | HT-Anne | (14298242.325) as reply to (14298242.314) | |
| To: Hiddink14 | 25.08.2010 at 12:05 | ||
| -(Long quote form Hiddink14)-
Transfers were extended by 24h when we first brought the site back up before noon yesterday. When we pulled the plug again. We couldnt roll back that extension. So we added an extension on top of that. So transfers between 15h Monday and 15h Tuesday were extended by 24h the first time, and by 5h the next time. But transfers between 15h Tuesday and 19h on Tuesday were extended by only 5h. Is this clearer ? | |||
