Archive for myfreeforum.org Before posting please check the "stickies" in the support forums.
Please ask questions in real English and not "txt". You will get a better response.
Please do not ask support questions via PMs.
 


       myfreeforum.org Forum Index -> What's new? Announcements!
myff admin

Another server...

We have put things in progress for what should be the very last server to go into our rack in Maidenhead, it will be a 96gb beast of a machine.

There wasn't really meant to be another server in Maidenhead but between the idea that a very large memory machine will be useful for caching, and the realisation that despite all the effort that is going into migrating systems to our Leicester center things are still a little crowded in Maidenhead, the idea of another server makes sense.

A lot of effort right now is tied up in writing routines for smooth migration of services with little of no down time, and one thing we do need to achieve is to totally clear the original Maidenhead servers for a full update. It has been less than two years since the first one went in, but that is a long time in the world of server virtualisation.

Getting it all upgraded, and migrated where possible will allow us to be a little more generous with the memory allocated to forums systems. Not that I think we are running at all slow anywhere, but one or two proxies in particular do show some degradation of performance over time, and that can be dealt with more easily if we are no longer a little tight on memory.

I'm losing track of just how much memory is on the system in total but I think we will move over the 1tb mark with this new server   That's about 25x more than we had two years ago when the decision was made that staying in our American datacenter was getting too painful. That was not the easiest of migrations, but was probably the best business decision we have ever made.
vais4u22

awesome myfastforum gets better everyday

myfastforum is truly the best free forum hosted site out there and allways on its toes helping with support and adding new things,i commend everyone that works at myfastforum.org for doing awesome job. thank you
myff admin

Just reading back to this post.

I think this server is now finally on order.

Gone are the days where I see it is needed and place the order, we have become a cog in a machine with more and more people having a say on these things.

The latest upshot is that a budget has been assigned to buy blade servers which are the next logical step. It's a large budget, but I'm not sure that people in charge of budgets ever truly get the point that you need to buy what is sensible tech wise to buy with obviously an emphasis on cost.

But the tech comes first and won't correspond to a budget as such.

What is technically interesting is that we may be about to depart from our policy over the last couple of years of basically buying the biggest faster RAID10 servers we can afford. Such servers have been good to us, you can throw any manner of things at them, and aside from the pain of RAID10 giving you only half capacity on the disks they are ideal for most purposes.

But with growth comes specialisation. The buzz words HADOOP/HDFS are flying and you don't even need any kind of RAID if you are running a cross server file system with its own redundancy build in.

You start to want to transition from expensive ultra reliable servers, to dead cheap large numbers of little servers which work in a redundant fashion, so hardware failure is simply a normal part of the process.

That is how google work, but of course they have squillions of such servers.

We will find our transition very painful, the new budget undoubtedly will not stretch easily to cover buying both fast high end servers and a sensible number of cheap Hadoop servers, and Hadoop which is all about parallel processing won't be really useful until you can throw at a guess 20 servers into it, and 20 servers is more servers than we actually have at the minute. Of course Hadoop servers will be much cheaper, but all the same it's a big leap.
myff admin

Further addendum, we are having some initial discussions that come down to creating our own datacenter, well I say our own but it would mean teaming up with another operation that has a viable building and a fat pipe.

It seems a trifle fanciful, but then again we are now on 15 big servers, one more on order and plans that involve scores more. A little over two years ago we had 3 servers and each had a fraction of the power of the current ones.

One exciting thing about the new plans is that there is a strong "green" element.
myff admin

So much for the last one being the last Maidenhead server.

We are just finalising an order for the biggest one yet in Maidenhead, this is partially because of expansion, but also unfortunately partly because the last server we received is behaving very lamely and needs a software rebuild at the very least.

It is somewhat notable that a decision to spend about $10,000 or so is now almost casual, and certainly easy for myff as it is no longer our money being spent, we are the Minnow on the back of the Whale  
myff admin

The new one has not arrived yet

But I have just put in a request for two more larger servers.
myff admin

We have the new one, the two more should arrive in the next couple of week and we putting in a request for another as well.

It that come in we will have 20 big servers most of which will have 96gb of ram, the rest with 48gb.

Somewhat annoyingly, I have redesigned how we will rack these servers, and with the new design we won't retire the server that hosted "slarti" forums, so the downtime Tuesday was not needed

On the scale of things though it is all about "growing pains" and the new plan is based on the lessons learned from our rack Leicester, basically lessons that even now with so many staff we are looking for new much bigger offices, we still don't have capacity to properly have specialists in physical server management and that is best left to the datacenters..... which is a truth despite the fact that we have had no actual outages in Leicester under our own management but we have in Maidenhead where the servers are physically managed by the "experts".
myff admin

After a lot of being given the runaround it seems we have serious space issues in the main data centre.
Why that could not have been made clear in the first place is typical sales stupidity  
Upshot is we will claw forward by retiring less useful bits of kit replacing it with high powered servers which as a strategy gains some breathing space.
But then it all gets a bit tricky, we are waiting on the next generation of Dell blade servers as our major investment and path onwards and don't want to be hobbled by short term datacentre aggravation when making these long term strategic decisions. On one level we are between a rock and a hard place, on another level the money liable to be lost because of the problems is sickening but not going to threaten the business.
It it still very frustrating though, as aggravation like this we could live with out.
myff admin

Been a fun day for incompetence  

Datacenter made every possible screw up getting the latest server in place and whilst they did so another server in that rack went kaput, it had been running without issue for months/years and of course they claim its entirely coincidental that we get a problem at exactly the time they are mucking about in the rack  

Meanwhile on the home front an electrician managed to slice through our radiator pipes  

the server going down should and I hope did result in no particular forum problems as it is designed to "failsafe", and at the end of the day the new server was live, and we have even managed to restore home plumbing  
myff admin

With the new server in to help allow us to move stuff around, we have managed to repair the "lame" server.
This is somewhat frustrating in that a patch failed to help, and so we went the whole hog changing RAID5 to RAID10 and installing a known safe copy of our server virtualisaton system. This has demonstrated the hardware and net connections were sound, but failed to totally identify the actual issue.
My suspicion is that the slow write speeds of RAID5 were the problem, but it would have been good to gnaw down to the bones and really know.

We also have another two servers delivered and awaiting installation
myff admin

As of now the worst case server is only hitting 75% capacity on CPU and we have crazy amount of memory spare, one previously cramped server has over 30gb ram spare!

The general aim at this point is to develop redundancy, but that needs many things including a housekeeping exercise on resource that were originally over estimated for safety.

What should really play a role here is the fact our newer servers have 96gb of ram each, that is a vast resource if properly used. For instance we are possibly over hammering our disk capacity by using small memory buffer where we could use much larger buffers freeing the disks for "big data" use.
myff admin

Another server has gone in today, another is awaiting installation, two more are possibly being ordered... I'm attending a Dell event next month on next generation servers....

Possibly the oddest thing is that the current rush of servers is little more than a holding action before upping the ante.
myff admin

Installing a new server today and was at the Dell event yesterday.

That event did not impress me about 12th generation servers, since they were barely mentioned.

But what it did crystallise is something I have been pushing for recently which is to work hard on reducing even further out disk IO which compared to capacity which is vast is already very low.

So why be bothered? Well all that vast disk capacity could be employed on the new "big data" systems that are all the rage these days

But we can't do that whilst thrashing our disks for other things, so what we do in invest in a big (very expensive) storage array, the storage array would be pants at "big data" but would be a massively fast RAID array far fasted than the existing arrays, and Dell make a very big thing about how long term scalable their solution is, e.g. if at some point we run out of space/speed we just buy another system in the range plug things in and things will transparently start to use the new space. Over the years as disks get faster or replace by SSDs you will end up with the most frequently used data on the new systems and old data on older slower systems all basically without any human management of the process.

The only catch is that it is £35,000 to get in the game Which is pretty hair raising. But in effect that will also get us for free as we free up existing storage a 100 node big data array, when looked at like that it is a bargain.
myff admin

two more servers on order, but still this is only temporising. these are beasts of servers but still anything is only as strong as the weakest link, and we need to get a few things sorted.

We're undergoing a painful transition here, one that would be so much easier if we could invest say £250,000 into the kind of infrastructure needed, rather than having a constant skin shedding exercise as we have now  
myff admin

I thought I should update this thread.

We installed 6 more servers in the last couple of weeks, and have another 6 even more powerful ones on order.

Since May there have been a few others aside from these 12  

Getting to the point where I'm losing count.
myff admin

just noted some figures showing a doubling of bandwidth in the last year.

there may be some very notable announcements happening in the next 6 months, all rather ironic in a way as whilst the announcements will be phenomenal they also will correspond with us struggling to keep to the customer service standards that I would like to define us.
myff admin

We have now for the first time exceeded a full rack of servers  

Another 6 have been going online this week and the next order is under discussion.

The last two orders we have simply ordered 6 at a time, and even then it is not getting us very far ahead of the curve.

Staff are harder to get hold of, especially as our new offices are still a few weeks away from being ready to occupy. But rather like the servers it is getting to the point of asking for 6 at a time  
myff admin

Another seven servers went on order this week.

But there are concerns that this is not enough  
Ask Mr. Religion

Wow!

You are blessed to have these sort of high-class worries. Glad to see your business is expanding...and creating jobs.

AMR
myff admin

And looks like another 6 will go on order this week which will bring us up to 53, which is 50 more than 3 years ago!
Well thereabouts anyway as we had 3 in the US and it was just about 3 years ago we broke ground in the UK with the first UK based server.
myff admin

I was wrong, it will be 8 more servers on order this week  
myff admin

It's almost getting silly now posting server orders, I'm not sure the final paperwork has gone through but today in theory we have ordered another 17   so that will have been 25 delivered in Dec/Jan or about 3 a week.
myff admin

looks like another week for this next load to be delivered, but it is kind of notable going through the current servers, a lot of them are ceasing to have identities, a few of us can talk about the "nostromo effect" to identify a pattern seen on a server, but these days that "desk" is occupied by four system admins who were not even in the company when that server was bought.
Today had 6+ people chasing a weird bug, and I had to defend the fact that some people had other priority’s   it's a far cry from the old myff days with 3 servers and less employees.
myff admin

hmm paperwork going a little slow on this new lot, but we seem to be upping the order to 28 servers  
That's even more hair-raising than 17. It will take us to 83 servers in total. e.g. rather more than 2 full racks when you add in the assorted bits and bobs.
Nick(NR)

datacentre soon
myff admin

Still a little ways from that but we are at least at the stage where their coming to see us next week to discuss future business.
Plus of course direct USA capacity is getting more on the agenda, and we now actually have US servers (to be accurate live US servers doing work, we have always had "utility" servers state side), though I'm not counting them in the total as they are hired.
myff admin

The next server order was placed last week.

This was for 30 more servers. I was looking at the dates and thinking its been a relative while.
But then again that is in effect about 10 weeks, so I suppose 3 a week
Not quite sure where these will be going, the forums have got off fairly unscathed but the truth is we have been having insane fun and games with our core datacenter in the last month   To be honest I think if if had effected the forums and I had posted the truth of what's gone on I would not expect the more tech aware of our customers to believe me, I think they would strongly suspect I was lying to defect blame from some of our own chronic incompetence  
myff admin

Just occurred to me that so far this year we have been spending something like £2000 a day on new servers.e.g. if this was to increase just a little more we would be at the million quid a year on new servers mark, it's not surprising the staff numbers are getting to 50+ and after 6 months in new offices we are again looking for a much bigger place.
myff admin

Another 46 servers due to arrive next Friday, which will be a big relief.
myff admin

Seems about 100kgbp more servers than I thought are on order   it's getting hard to keep track of, I guess bandwidth is fast approaching 1TB an hour...
It's quite funny looking at the "Server Appeal" which I leave up because it shows what you can do on a myff forum and because it lets people show appreciation and of course myff is still a seperate business to the mayhem of what is having to be the major focus.
myff admin

Monday we are moving into new offices which are about 4x the size, which is just as well as the current place is beyond rammed.

This is almost a year to the day from moving from "the farm" into what seemed a large office.

Along with the move last week we finalized our next datacenter location, the old Greenham Common "Bunker", and will be putting another 48 servers on order.
There was a kind of discussion ranging between 32 to 64.

48 is a "modest" 30% expansion since the last order in August, I last mentioned staff levels as exceeding 50, guess it is more like 75 now  
myff admin

actually more than 100 servers are on order.

This is really getting to measuring by racks and data centers rather than by servers  

       myfreeforum.org Forum Index -> What's new? Announcements!
Page 1 of 1
Create your own free forum | Buy a domain to use with your forum