General overhaul of the backup system.

[Please Register and Login to this forum to stop seeing this advertsing.]

[admin (no pm's please)]

More to the point I am hoping to see some speed improvements and an end to the problems caused by backups.

I have come to the conclusion that the real bottleneck on speed is our RAID5 disk systems, they work well most the time as most operations are reads from the disks and RAID5 is fast on reads, but when the backups get written initially to the RAID5 disk before transfer then the slow write speed cripples us, swamping the disk so people struggle to get to the forums

The solution I am working on is a minor evolution to the current system that pipes the backups directly to ftp. This has tested out okay on Ford and will go live for the database backup tonight, and assuming that is ok the system will be applied across the board.
_________________

Family Friendly Shareware | | Web Design/Services | Free Forums

[panther_dust]

why not just use raid 10 or raid 1+0...

this way you get fast read speed + the main hard drives are backed up by the others automaticaly.
_________________
PD, Pantherdust, Benjamin, BMDProductions

[admin (no pm's please)]

You can't "just" use RAID10 when you are on RAID5 and RAID is not a backup system anyway.

We will be going RAID10 on the next server, but RAID10 costs more than RAID5 and on systems where much more is read than written RAID5 is at least in principle a good choice.
_________________

Family Friendly Shareware | | Web Design/Services | Free Forums

[panther_dust]

If not then why not use your 5 with 1.. or 5+1 if that can be done...



RAID 0 (striped disks) distributes data across several disks in a way that gives improved speed and no lost capacity, but all data on all disks will be lost if any one disk fails. Although such an array has no actual redundancy, it is customary to call it RAID 0.

RAID 1 (mirrored settings/disks) duplicates data across every disk in the array, providing full redundancy. Two (or more) disks each store exactly the same data, at the same time, and at all times. Data is not lost as long as one disk survives. Total capacity of the array equals the capacity of the smallest disk in the array. At any given instant, the contents of each disk in the array are identical to that of every other disk in the array.

RAID 5 (striped disks with parity) combines three or more disks in a way that protects data against loss of any one disk; the storage capacity of the array is reduced by one disk.

RAID 6 (striped disks with dual parity) (less common) can recover from the loss of two disks.

RAID 10 (or 1+0) uses both striping and mirroring. "01" or "0+1" is sometimes distinguished from "10" or "1+0": a striped set of mirrored subsets and a mirrored set of striped subsets are both valid, but distinct, configurations.
_________________
PD, Pantherdust, Benjamin, BMDProductions

[admin (no pm's please)]

As I say you can't just casually change the RAID setup on a running server.

When the new server is commissioned, existing servers will be emptied and options considered.
_________________

Family Friendly Shareware | | Web Design/Services | Free Forums

[panther_dust]

yeh i supose unless you have an external sata port, its a shame raid is such a b**h otherwise i would use it on my pc it could be so much faster.
_________________
PD, Pantherdust, Benjamin, BMDProductions

[admin (no pm's please)]

On reflection I'm extending the test to Slarti, the slarti backup has only just completed, which is getting more than a little silly as regards the performance hit
_________________

Family Friendly Shareware | | Web Design/Services | Free Forums

[panther_dust]

how long did it take?
_________________
PD, Pantherdust, Benjamin, BMDProductions

[admin (no pm's please)]

About seven hours   It kicks off fine long after the Brits are past bedtime and as most of the Yanks are on their last Cocoa. But as it nears completion people wake up and as the process is "niced" it slows almost to  stopping point. But all the time it is still managing to create a write bottleneck on the disks as well as inevitably reducing disk cache hits for anything reading the disks.
_________________

Family Friendly Shareware | | Web Design/Services | Free Forums

[panther_dust]

OUCH!

it may sound silly but would something like norton 360 help? its backup system literely has a one time backup thing after that each time a file is edited nortons will back it up.. (Just an idea)

but still 7 hours they are either extremely big hard drives or they need more ram in the systems.. 64GIG should do it hehehehe
_________________
PD, Pantherdust, Benjamin, BMDProductions

[admin (no pm's please)]

There is no shortage of memory or CPU, it does seem quite plain that the bottleneck is the disk. I guess we will find out tomorrow what difference the new system makes.
_________________

Family Friendly Shareware | | Web Design/Services | Free Forums

[panther_dust]

well if it doesent i suppose all you can realy do is just add more discs thus lestening the read..well untill it comes to backup again... bad idea i suppose lol
_________________
PD, Pantherdust, Benjamin, BMDProductions

[query]

[admin (no pm's please)]

I already pointed out that RAID is not a backup system, but it is a lot more than a defense against hardware failure!#

Anyhow two backups were on the new system today, they completed fine and a little but not much quicker.

More to the point there was none of the usual forum performance issues  Though one day does not really confirm the issues have been resolved.
_________________

Family Friendly Shareware | | Web Design/Services | Free Forums

[query]