General overhaul of the backup system.

admin (no pm's please) · General overhaul of the backup system.

More to the point I am hoping to see some speed improvements and an end to the problems caused by backups.

I have come to the conclusion that the real bottleneck on speed is our RAID5 disk systems, they work well most the time as most operations are reads from the disks and RAID5 is fast on reads, but when the backups get written initially to the RAID5 disk before transfer then the slow write speed cripples us, swamping the disk so people struggle to get to the forums

The solution I am working on is a minor evolution to the current system that pipes the backups directly to ftp. This has tested out okay on Ford and will go live for the database backup tonight, and assuming that is ok the system will be applied across the board.

panther_dust

why not just use raid 10 or raid 1+0...

this way you get fast read speed + the main hard drives are backed up by the others automaticaly.

admin (no pm's please)

You can't "just" use RAID10 when you are on RAID5 and RAID is not a backup system anyway.

We will be going RAID10 on the next server, but RAID10 costs more than RAID5 and on systems where much more is read than written RAID5 is at least in principle a good choice.

panther_dust

If not then why not use your 5 with 1.. or 5+1 if that can be done...

RAID 0 (striped disks) distributes data across several disks in a way that gives improved speed and no lost capacity, but all data on all disks will be lost if any one disk fails. Although such an array has no actual redundancy, it is customary to call it RAID 0.

RAID 1 (mirrored settings/disks) duplicates data across every disk in the array, providing full redundancy. Two (or more) disks each store exactly the same data, at the same time, and at all times. Data is not lost as long as one disk survives. Total capacity of the array equals the capacity of the smallest disk in the array. At any given instant, the contents of each disk in the array are identical to that of every other disk in the array.

RAID 5 (striped disks with parity) combines three or more disks in a way that protects data against loss of any one disk; the storage capacity of the array is reduced by one disk.

RAID 6 (striped disks with dual parity) (less common) can recover from the loss of two disks.

RAID 10 (or 1+0) uses both striping and mirroring. "01" or "0+1" is sometimes distinguished from "10" or "1+0": a striped set of mirrored subsets and a mirrored set of striped subsets are both valid, but distinct, configurations.

admin (no pm's please)

As I say you can't just casually change the RAID setup on a running server.

When the new server is commissioned, existing servers will be emptied and options considered.

panther_dust

yeh i supose unless you have an external sata port, its a shame raid is such a b**h otherwise i would use it on my pc it could be so much faster.

admin (no pm's please)

On reflection I'm extending the test to Slarti, the slarti backup has only just completed, which is getting more than a little silly as regards the performance hit

panther_dust

how long did it take?

admin (no pm's please)

About seven hours

It kicks off fine long after the Brits are past bedtime and as most of the Yanks are on their last Cocoa. But as it nears completion people wake up and as the process is "niced" it slows almost to stopping point. But all the time it is still managing to create a write bottleneck on the disks as well as inevitably reducing disk cache hits for anything reading the disks.

panther_dust

OUCH!

it may sound silly but would something like norton 360 help? its backup system literely has a one time backup thing after that each time a file is edited nortons will back it up.. (Just an idea)

but still 7 hours they are either extremely big hard drives or they need more ram in the systems.. 64GIG should do it hehehehe

admin (no pm's please)

There is no shortage of memory or CPU, it does seem quite plain that the bottleneck is the disk. I guess we will find out tomorrow what difference the new system makes.

panther_dust

well if it doesent i suppose all you can realy do is just add more discs thus lestening the read..well untill it comes to backup again... bad idea i suppose lol

query

admin (no pm's please)

I already pointed out that RAID is not a backup system, but it is a lot more than a defense against hardware failure!#

Anyhow two backups were on the new system today, they completed fine and a little but not much quicker.

More to the point there was none of the usual forum performance issues

Though one day does not really confirm the issues have been resolved.

query

admin (no pm's please)

Speed, with a files data striped across multiple drives RAID systems depending on the RAID level can be very fast.

panther_dust

it is an a way a backup as there is one that mirrors the 1st array. that i class as a backup..

if anything goes wrong with the first array then you have it how it was on the seconed

admin (no pm's please)

Let's not play semantics here. Of course it is a copy of data and you can choose to call that a backup if you want. Many people have chosen to view things that way.....

and regretted it!

query

admin (no pm's please)

The database backups continue to complete reliably under the new system. In fact the main culprit came in more than 2.5 hours ahead of its worst completion time

Hopefully with more database pruning that can improve further, another two hours quicker and the backup would be completed at 0815 in the British morning taking will well into the quiet hours.

admin (no pm's please)

Trimmed another 14% off the slarti database size, taking us down to 9.41am as the completion time.

I can see room for a bit more trimming today, but much more would require a tougher purge strategy.

I have half an idea to place inactive forums into a forum of "purgatory" where they would be simply and safely removed from the database. If someone actually does access the forum the files would be pretty much instantly restored.

Databases tables in such a place would be saved say once a week.

admin (no pm's please)

All system weekly backups have completed on the new system. One a mere hour and 20 minutes ahead of last week, one 5 hours ahead and one more than 10 hours ahead of last week

The system where there was only an hour or so gain, is the one with a much smaller backup size and the system where performance is the least issue, so that figure is to be expected.

admin (no pm's please)

Bumping this again. The thread here was overtaken by events as we had to get a new backup server and that has required a fair bit of work improving the general backup infrastructure.

But now is the time to get back to deal with the way backups can effect forums performance, and this means getting back onto the idea of removing unvisited forums from the backup systems by moving them out of the database directory structures, but making is so that when someone accesses the forum they are snapped back in seamlessly.

admin (no pm's please)

In one sense this is mostly done, the system can move a forum to "purgatory" and if it is accessed it snaps back with no perceivable delay on the office systems

I do need to do some work though on calling forums back from purgatory to be purged and last but not least whilst the system works in the office security is a lot tighter on the actual servers and will need tweaking to allow access to the code to return a forum from purgatory.

It all takes a little thought, for example I was thinking I'd need to notify the system that a forum had return from purgatory, but is it really needed? actually probably it is.

admin (no pm's please)

or not.

Trouble with a notification is that we run a client server system. Your forum may be a myfreeforum.org, but might well reside on the myfastforum.org server, so a notification would for a myfreeforum forum would go from the server the forum is on, to the server for the forum domain and then back to the server the forum is on, this both introduces a dependency and does little for the process being seamlessly quick.

Instead the forum now ask themselves, was I flagged as in purgatory? If so then change those bits in my configuration that deal with that to remove the flags and the return from purgatory code.

admin (no pm's please)

The new code is now installed, every night we run a script that processes all the forums, tonights should start placing forums that no one has accessed in 100 days into "purgatory", we will see the effect of this in the next nights backup.

The one flaw in the system is that the backup system is currently geared to only keep backups for 3 months, thus forums in purgatory would be ultimately vulnerable. The way round this will be to create another backup process to save the purgatory file system once a month.

admin (no pm's please)

The first night sees 25% in purgatory at the start of the day.

I will measure again later and see what the "churn" factor is, even a bot visit will take a forum out of purgatory.

25% is a quite solid number though, and one that we should be able to increase.

Nascarfan13

http://snw.myfreeforum.org/index.php

Could this cause my forum to not be able to be accessed? I could get to my forum about 15 minutes ago, now all of a sudden I can't even get to the index page, and then I checked my modem to see if my internet was out and its not, so something is causing my forum to be inaccessible. And I know its not A rouge admin because right before it went down, there were no members on online at all.

LFF

http://lancasteroh.myfreeforum.org/
We are down also. Support is down too. I fought my way in to leave a message that MFF is down. orcikat@aol.com

namecaps

When I try to access my free forum I get this message

phpBB : Critical Error

Could not connect to the database

my forum is

http://aimclan.myfreeforum.org/

my email is spaceman@bigpond.net.au
could you please advise me if you know what problem is
thanks

Nick(NR)

Your forum shows up to me as does nascarfans, LFF's is still showing a generaal error, which is weird, using a slightly different link I can get to the index, but it only shows the info with no styling and is showing me people logged in.

Nascarfan13

Here are a few more:

http://sdracing.myfreeforum.org/index.php

and

http://up2speed.slarti3.myfreeforum.org/index.php

Hopefully they will be back up within 48 hours or so.

LFF

Lancaster Free forum posters are relentless. They will keep trying, and it is still down. I am still getting emails they cant post. (even though I was able to make a post about the problem) I had to reload, backup, reload until I was able to post a reply. Admin panel is down too. Thanks for tanswering. I know it will be fine soon.

http://lancasteroh.myfreeforum.org/

namecaps

"G'day Nick,

Unfortunately, many of our users are experiencing the error (I won't say all, as I've not been in contact with everyone). All are receiving the PHPBB Error; Cannot Reach Database error. We are in different states, and on different ISPs, so nothing is connected except the forum. It is has only been occurring in the last 5 hours, and is consistantly coming up. Sometimes with multiple refreshes, we can access the pages; othertimes, we just keep getting the error. Any help would be appreciated.
thanks

Nick(NR)

F.A.O. Admin: found this:

Spoiler:

Nascarfan, namecaps and LFF, those forums you listed are loading for me, some after a few refreshes and one of them first time. Until admin makes a clear post here I can't say if it's realted to the forum backups or not, but it is clearly a database connection issue, that should ease, when that happens I can't be specific on, just think on the good side, if db connections are limited to around 20, more people than allowed are trying to get on your forum Wink

LFF

http://lancasteroh.myfreeforum.org/

You can view the index page. When you click a topic..sometimes you can get in. Then you can't post. Only about 5 posts have made it through this morning.

Full Tilt Boogie

This worked for me - see if it works for you guys.

I could only get to my site's index page - after repeated attempts and reminders of "Critical Erorr" etc - but got to it eventually.

So, in an attempt to post and let my members know what's going on and why they can't log-on or post (i.e. MyFF server outage), and just as a punt, I right-clicked on the Board Announcements forum on my site - it opened no problem.

So, once I was in to the Announcements forum, I then right-clicked on an existing thread about the last time the servers here hiccoughed - and happily got the same again, it opened no problem - the upshot is that I was able to post a message telling my members what the issue is, and what is being to done to fix the issue; and that (at time of going to press) there is currently no estimated fix time, and just to hang in there!

You can see it Last post here.

Hope that helps guys - and hope we're all up and running again shortly.!

Thanks to Martin and the staff here - a thankless task at times like this I know! Cheers lads.

LFF

http://lancasteroh.myfreeforum.org/

We are back and doing fine. Thank you so much. This support is amazing.

namecaps

The [AIM] Clan free forum is back up and working fine
thanks for your help Nick it is much appreciated

array

Nick(NR)

admin (no pm's please)

Well back on topic I'm raising the aggression of the purgatory system again, currently we are only seeing about a 20% gain in backup size, a reflection I think of the fact that search engines are liable to spider many dead forums pulling them back in from the cold.

Having the forums bounce in and out of the system is no big deal at all, the file system moves files in a flash and a move is "failsafe" at the file system level, the existing reference is not removed until the new one is in place.

namecaps

Well thanks for your answers Nick I think that is what kept everybody calm and thanks to the ADMIN and helpers for getting the forums up and running again
Much Appreciated
Namecaps

admin (no pm's please)

I'm also tweaking other bits and bobs on the servers and planning other moves to help with load, in these recessionary times it is vital that we utilize the servers as much as we can, it is still the case that we are not really overloaded it is the intermittent spikes from internal processes that seem more responsible for problems than the actual load from the forums.