Archive for Before posting please check the "stickies" in the support forums.
Please ask questions in real English and not "txt". You will get a better response.
Please do not ask support questions via PMs.
  Forum Index -> Forum Blogs
myff admin

Faster forums part two

The code does seem just about ready for this, the only question is rolling it out.

Probably it will start Tuesday with the new code to deal with template editing and copying of templates to the template host, combined with a neutered version of the server outage detection system that will not integrate with forums code at all, the office version of the code is seeing the odd few seconds of outage every several hours, and so it trips the system and resets it a minute later. This of course is what it is meant to do, but is it leaping too quickly and how will it behave on a live server?

We need to run for a few days and see what happens.
myff admin

I have started the template copying process on one server to see how it goes, did some last minute recoding as where forums have moved servers they may be old copies of the template lying around and that could have lead to old copies overwriting new edits on the template server

copies are progressing at about one per minute, so this will not be a fast transition.
myff admin

Still under 200 done, and this really does need pacing. Not much point doing all three servers and then finding out there is an issue.
myff admin

Found a small issue just now when I started the integration, and it will be a few more days before the support forums own template is old enough to go live with the system. Add that to the fact that only 523 template from one server are saved as yet and you end up with the conclusion that we may as well be patient. It will get there.
myff admin

Trouble is that at this rate it may take more than a week to get there There are more than 10,000 templates on each server and after about 18 hours of processing we have only copied 624 templates.

In the long term that is not an issue, we can as a low priority process clearly save 400 templates a night, which is well in excess of the number likely to need copying.
myff admin


I'm now running two processes one to copy the system templates. One to copy the user templates.

I'll code something up for the other servers to just copy the user templates, whilst marking system templates as already copied.
myff admin

All system templates were copied.

I am now running the copy user template process on all servers, the code is also there on all servers, so very slowly all forums should be pulled into the system.
myff admin


I guess templates are copying at about 3 a minute. This may be a glacial pace, but when you think about it long term, are 3 templates a minute reaching a point where no one has edited them for 4 days? More like 3 templates an hour I would guess.
myff admin


The length of this process is certainly going to put a halt to phpbb2 work for a bit, maybe that means it is time to get to grips with a phpbb3 biggy like the profile fields.
myff admin


Not a lot else to say really, but at least it is setting aside time to look at other issues.

The phpbb3 profile fields has got to wait, there are no good solutions out there yet that I can see, and it is not time for our own bespoke solutions.

Instead todays work on optimising phpbb2 databases has been very productive, and hopefully will be able to go forum wide soon.
Ask Mr. Religion

Once everything is copied I hope you have some sort of diff copy scheme that will only pick up templates that were 'touched' so you don't have to go through this again. (I think I am understanding what is going on...then again...

myff admin

The system is simple, when anyone even enters the template editor for a template, then the template is marked as needing to be loaded locally and to be copied again. This is a little brutal but how many people seriously edit templates that much?
myff admin


As a matter of fact on this issue, I am running it as a manual process on each server from an ssh terminal. So when something glitched this morning and those terminals dropped out, so did the processes

When I kicked the processes off again, they took only a little while to process templates that had been previously deal with, and to deal with ones that were newly needing copying before they have carried on plodding away through the rest of the templates.

admin (no pm's please) wrote:
This is a little brutal but how many people seriously edit templates that much?

I would say I make changes to templates at least once a month on average. Sometimes more often, sometimes less so.
myff admin

So with the system your template would be offsited 85% of the time of thereabouts.

In practical terms I doubt people with the templates onsite will see a visible speed penalty, You go from using 4 servers to using 3 and the server you are falling back on is more lightly loaded than it was, and since the offsite templates are served from "shared" hosting it is questionable if they are that fast anyway! The reason they will appear fast is because of the greater parallelism involved.

Cool, no problem then
myff admin

7562 and have just had a file transfer glitch at the template server end of things, all of our servers managed to see the issue, report it and move on to the next template, so okay those templates will be missed from the copy process this time round, but things have failed safe.
myff admin

Next time I write something like this I am going to add a couple of lines of code to show how far its got  
We are up to  10,975 templates done, which has to be something like 1/3 but I am not that sure.

I'm waiting to see the main weekly backup sizes, in the hope they are showing signs of reduction.
myff admin

We got some reduction in backup size, though in all honestly it was a little disappointing. But then with:

16,999 template copied as I post, that is only in the region of half the templates and some will have been edited since and hence made their way back into the backup system.

Clearly we have several days to go to get through all this.

though the fact that we served 1,355,110 from the new template server yesterday is rather good for such a halfway stage.
myff admin

20,888 done and Zaphod has actually finished. I have kicked it off again to see how long it takes now.
myff admin


Never did quite find out the time on Zaphod, but it was very fast. It is now doing it nightly and probably taking less than 10 minutes.
myff admin

25088 and Ford has now completed.

Slarti is bound to be slowest.
myff admin

29505 and now about 5000 to go on slarti.
myff admin

Down to under 1000...

I really want to get on to the phpbb3 version of this, but there are only so many things I want to be dealing with at the same time

I guess we have 24 hours or so to go now before we are complete.
myff admin

Just seen another little glitch on this.

There are obviously a few motivations in doing this, trying to help load levels, keeping reliability high and controlling costs. I had been assuming that a large shared hosting system would be good on reliability and cost when it came to serving static files. Clearly cost is good, but reliability is raising questions

To deal with reliability the system has always had a 1 minute check to switch off the template offsiting if it is supsect, but this is kicking in a lot to often for my liking currently. Forum Index -> Forum Blogs
Page 1 of 1
Create your own free forum | Buy a domain to use with your forum