chronicles down

I, Brian · 10-12-2008, 03:03 PM

As you've probably noticed, chronicles is currently down so I'll update on this thread:

- on Friday evening I got two emails stating that the forums were showing database errors

- on Saturday I ran a repair on the forum database, but by that time it was too late - the database had already suffered a serious crash, and all 1.2 million posts were wiped from the database.

- I tried running an upgrade, contacted vbulletin support, and contacted the server hosting company. The jist was that there was no way of recovering the missing data from the database itself, and that the hosting company had failed to implement the back ups that were supposed to be running on the server.

Here's the fix I'm implementing:

- there's a backup from August from when I added the new features to the site (such as the blog) and I'm going to get that back online. However, it does means anything since then is lost - including blog entries, uploaded photos, special groups, and of course discussion topics

- I'm rebuilding chronicles on the old chronicles-network.com domain. This is part of a wider expansion for the site I had planned, and I realised after moving to "sffchronicles" that chronicles would always be the chronicles network - with the new social features it is increasingly a social network, and I had already begun redeveloped on a new expanded look. The latter will have to be postponed while I just ensure the forums go back online.

- there's a huge amount of data to move and no easy way to do it. The attachments alone are something like 350MB, ther's another 150MB from the old Gallery, and the forum database with all the search records removed is still a good 1GB.

- It's going to take most of the day to get the site rebuilt - good use of a Sunday, heh - but it remains my priority. There will probably be some bugs, but I'll try to get most ironed out very quickly

- When chronicles is rebuilt, it's probably going to have a few rough edges, and I aprpeciate a lot of people are going to feel pretty bad about the data loss - new members having to re-register, and nearly two months of posts, threads, and discussions missing, including the chrons meet-up. I can't help that I'm afraid - but I can get as much rescued as possible.

- I have made sure the datacenter now has offsite backups made of the server on a daily basis. In the future there should be little if any chance of a crash of this scale being repeated - redundancy for data backup is now definitely in place.

In the meantime, I'll keep this thread updated, and any questions, feel free to ask.

Talysia · 10-12-2008, 03:18 PM

Thanks for all your hard work in trying to get Chronicles back online for us. Hope it all goes well for you, with no more problems - it sounds like a big task.

Overread · 10-12-2008, 03:25 PM

just chiming in to say the same - defiantly not going to be easy work, but is defiantly appreciated

Hope things go well

And I told you all the slurp spiders were planning something big!

sloweye · 10-12-2008, 03:29 PM

It sounds like a hell of a task, good luck and i hope its up and running again soon.

the ace · 10-12-2008, 07:28 PM

Best of luck. Brian. We're rudderless without it.

manephelien · 10-12-2008, 07:30 PM

I do hope you'll get it sorted soon.

Hoopy · 10-12-2008, 07:58 PM

Appreciative of all your hard work, Brian, definitely. Hope all goes smoothly. Don't wear yourself out!

I, Brian · 10-12-2008, 08:00 PM

The database is only a few minutes away from uploading to the new domain - if that works, shouldn't be too long to get *something* up.

Been a bit hellish, really - thought the hosting company had backups, I've spent all day literally working on this, and I'm supposed to going on my first ever holiday abroad in two days time so I should be packing instead.

Overread · 10-12-2008, 08:03 PM

ouch!
its the old thing though - the backups are never checked until they are needed - had similar happen at the end of a school term (just after exams and work deadlines thankfully) it was only after the crash that IT found that the backup machine was not backing up - and had not been for some time.

Alexa · 10-12-2008, 08:05 PM

Have a nice vacation, Brian. I hope you won't forget to drop by again at your return.