Archive for myfreeforum.org Before posting please check the "stickies" in the support forums.
Please ask questions in real English and not "txt". You will get a better response.
Please do not ask support questions via PMs.
 


       myfreeforum.org Forum Index -> What's new? Announcements!
myff admin

Some architectural changes

Already partially done for myfunforum.org as a testbed.  

Basically we had a Denial of Service type attack on the forums last week, and I was able in very short order to mitigate the issue by partially doing some changes that have been on the drawing board for way way too long.

The changes will generally add to speed, stability, security and the ability to deal with DOS attacks.
bazfalty

thanks admin, my radio site had a dos attack as well which got took down for a few hours intill my hosts could solve it. good to see the admin on the ball on this one
myff admin

A small observation as I work through this process.

The whole thing requires extra servers in place and there are choices to be made.

In the end the choice made means we need 12 extra servers!

This is one of the great things though about running a system that works with virtual servers.

Whilst undeniably 12 servers is a lot to set up, there is not a massive cost associated with it, and hence we can make the choice that results in both the simplest and most effective system.
myff admin

tick tock

80%-90% of the core configuration done and 4 of our forum domains are running new code when people create forums or deal with the control panel (not the admin panel), no complaints yet, and I have been switching one domain a day.
All domains need switch before the next stage.

In theory this should be all quite simple, an outage of maybe 10 seconds on each forums server and everything should resume as normal, but hopefully a little faster.
myff admin

Just made an abortive attempt to switch "slarti" over.
There are a few more key steps that all need to be done at the same time, so having killed access for a few seconds on a couple of occasions I'm going to work a bit more on the procedure, and do it rather more off peak!
myff admin

Well I have done slarti, and as an experiment opened the gate to see if last weeks DDOS attack was still on the go, it is

But it has let me shake a few issues out, and as a result forums on slarti are working fine, even though it is going to take some considerable time for the gate to get shut again.
myff admin

A few more glitches getting wonko switched to the new system.

But it is actually quite a big change, needing lots of manual edits, so I'm not to bothered by 5-10 mins of outage on a Saturday morning when you consider the benefits involved.
Mis

thefluffybunny.myfastforum.org

The forum was there earlier this morning but for over an hour there's just a message saying "controlproxytest1.zetabbs.com"

Is this connected with the upgrade or a separate issue?
Mis

Forget that message. The forum came back while I was posting!
myff admin

You have actually highlighted an issue that will still be there. thanks.
myff admin

I don't think it is just my imagination, but the forums on the new system seem to load faster to me.

I have not put in benchmarking to test that, as speed was not the primary purpose of the change.#

But the basic fact is there is now a primary server for a forum that will efficiently cache and serve images/js/css files and a back end server that will only be dealing the php scripting.

So people are getting two dollups of CPU speed as two servers are now running their forum, and the "easy" files are now served by a system that is geared to doing that job well.
Myles

That sounds excellent, thanks!
myff admin

seems we had a bit of a blip with IP addresses being logged as the proxy ip addresses.

This had been dealt with in tests, but subtle difference in php versions mucked it up  

As I have said this really is quite a major change.
myff admin

The graphs are the moment show "Wonko" as dealing with 1/3rd of the connections that it would usually deal with, and basically staying level as more and more stuff gets cached by the new proxy server.

So to recap, more speed, more ability to deal with attacks and also more secure as the forum server itself no longer has an IP address accessible to the general internet at all.
SRSeditor

myff admin wrote:
So to recap, more speed, more ability to deal with attacks and also more secure as the forum server itself no longer has an IP address accessible to the general internet at all.


Very clever. I like it
myff admin

One minor flaw I was bothered about with the new system was that the forums are going to want to cache a lot of data, and I chose to go with copying a proxy system that has been working solidly for months but which is a trifle small even when we divide the forums up into 12 proxies.
Thus far in the 14 hours the busy proxy has been running we have eaten about 1/6 of the available space. Which I think means that the system would overflow in a period of days or weeks, but plainly we can easily schedule cache cleaning to prevent it becoming a real issue  
Tomorrow another system will be proxied/cached, not sure which one yet. But hopefully this time a bit more smoothly.
myff admin

Ford was "proxied" this morning and seems to be running okay. So that's 3 down 3 to go.
myff admin

Zaphod now done, again I'm afraid a few glitches until it came right.

There really are a lot of configuration changes involved, and it only takes one mistake for a link in the chain to fail.
SRSeditor

One thing I've had come to light is that not all emails going to users are getting through.

They are probably getting blocked at the receiving end due to the source now looking different much the same as when we moved ip address's?

Some email are also hitting spam folders since as well

I assume it will get better as time passes and systems realise the move is genuine.
myff admin

This sort of thing always happens and is very frustrating for all.

It may well be the fact that I was not running a mail server to receive emails on the proxies, as of course the forums system has no need to receive emails to the forums servers.

I have now started email servers on the proxies.
SRSeditor

myff admin wrote:
I have now started email servers on the proxies.


Brilliant and I've now received an update email from this site, which I hadn't for a while.

Top banana  
myff admin

Jeltz is now done. Once again a small glitch  

That leaves hotblack, hotblack is the big challenge, as it harks back to the days when we were not running quite such a massive set up and hence is not quite such a simple system to change.
But work is under way.
myff admin

Phase one test of the new DNS system completed without any real hitches.

Several phases to go...

But that was so simply done that it has worried me a bit. But then paranoia is a good thing.
myff admin

Have spent the day throwing rocks at the replacement DNS system including some deliberate misconfiguration to break it and track the broken behaviour, which is not useful in the sense that broken is broken and a borked DNS will mess people up, but is useful in testing some defence mechanism that can be put in place and failing that, that sensible errors get generated.

It is also the case that seeing that a system behaves as expected when mistreated does hopefully prove things about how the system does actually work. In principle that is known already as the new system is the old system just on its own dedicated space, but the importance is such that I think a long re-examination is needed.

Still a few things to do before managing a final rollout.

We also had 3 servers delivered today! Servers that are even older than the servers we bought 5 years ago.

They were bought for a pittance off ebay:

http://cgi.ebay.com/ws/eBayISAPI....2222&ssPageName=STRK:MEWNX:IT

and my logic is that add on 200 for a RAID card and a couple of SATA drives and you have a powerful backup system over the 1gbs ethernet port these have.

Also as we are commissioning (well have commissioned it is sitting paid for and empty at the moment) a 42U server rack in a Datacenter in Leicester, the ability to play rough with kit that costs 135 and not 6,500 will be beneficial in of itself.

Next week we have the 2 new big servers delivered, and the initial Leicester set up will have 4 servers and 3 switches, and I want us to be slick in getting it installed, the cabling involved is going to make the back of your TV look neat and tidy!

One trick I have done is to get network cabling in 4 colors, so we can follow a convention:

Green - Seen
Black and Blue - redundant internal network
Red - NAS storage (Red is a good mnemonic as ReadyNAS is a make of NAS we have used)
myff admin

myff admin wrote:
One minor flaw I was bothered about with the new system was that the forums are going to want to cache a lot of data, and I chose to go with copying a proxy system that has been working solidly for months but which is a trifle small even when we divide the forums up into 12 proxies.
Thus far in the 14 hours the busy proxy has been running we have eaten about 1/6 of the available space. Which I think means that the system would overflow in a period of days or weeks, but plainly we can easily schedule cache cleaning to prevent it becoming a real issue  
Tomorrow another system will be proxied/cached, not sure which one yet. But hopefully this time a bit more smoothly.


Looks like at least 5 more days life left in the system, so basically it can cope with close to two weeks of caching without the cache needing clearing

I am also more certain than ever that speed has increased a lot on the forums in the system.
myff admin

One of the ebay server is now working at least in the sense of having an operating system installed and being able to see both its network interfaces. So the idea of a set of 300 NAS backup servers is a step closer.
In some ways this is not a great operation from the time spent point of view, but I always like recycling ideas and it does get us more in touch with server kit.

       myfreeforum.org Forum Index -> What's new? Announcements!
Page 1 of 1
Create your own free forum | Buy a domain to use with your forum