Archive for myfreeforum.org Before posting please check the "stickies" in the support forums.
Please ask questions in real English and not "txt". You will get a better response.
Please do not ask support questions via PMs.
 


       myfreeforum.org Forum Index -> What's new? Announcements!
myff admin

Important: Forum spam issues

I expect a few people got a database connection error this morning.

This was a combination of things.

1) The database backup completing late, when the backup completes is always a bit of pressure on the server.
2) The backup being late because it gets slowed by spambots.
3) spambots adding to the load at the time of backup completion.

As usual the issue is that people have let phpbb3 forums run to spam So I have done the rounds and suspended a couple of dozen phpbb3 forums.
myff admin

For the record I am ramping up the anti-spam campaign.

We have I think run a fairly tight ship on this, but the spambots are now much more rampant and we need to up out game in response.
myff admin

Down to a trickle of forums being flagged by the new anti-spam system now.

Quite coincidentally as this purge had already started, we have been contacted and push again by our advertising guys about providing a list of vetted forums of some description.

Increasingly it is the case that that advertisers are fussy about user generated content and don't want to be on forums they deem unsuitable, and what they deem unsuitable is pretty extensive.

Now we have had the suggestion in the past that we mould myff into the premier high quality forum company where advertisers want to be, and where any forums that don't fit the bill are unwelcome!

I have politely declined that idea. Policing peoples freedom of speech by the arbitrary standards of advertisers and showing them the door if they pass through hoops set by companies that themselves don't set any ethical highs in my view, is not a game I want to play.

Nonetheless we cannot be completely pious about this, there are forums on the system that I think have every right to be there, but which a lot of advertisers won't touch with a barge pole, and without advertising we do not exist.

So some kind of vetting that means some forums get better or worse adverts is a commercial fact of life.

So what might get you on the "unvetted" list.

1) It won't be porn! Porn will get you on the kicked out of myff list under our own T&C, as will any other violations of our own terms and conditions.
2) Racist crap, we don't tolerate hate forums, but we do let forums tolerate racists. It's a stance that I think is right for myff. Advertisers certainly don't agree.
3) trash talking, foul language is frequent on teen forums (and some others) and is a very easy thing for advertisers to red flag.
4) "Adult" topics. This is the really frustrating one, we do allow adult topics of conversations, advertisers have been known to freak out at discussions of breast feeding   e.g. they can be totally stupid and unreasonable.

All we can do is try and steer a line, that deals with the commercial realities.
myff admin

We seem to be running at about 150 suspended forums now, mostly forums that were spammed.

One result is that the backups are completing in 7 hours rather than 9, and that is with the spammed forums still being part of the backup.
e.g. it is indicative of the damage spambots do to general system performance
myff admin

As of today I'm also implementing a couple of automated policies.

1) phpbb3 upgrades that have not fully registered on the system, an odd occurrence probably associated with incomplete upgrades will be disabled at the board admin level.
2) phpbb3 forum that are not the main forum type and are receiving posts that look at all dodgy will also end up disabled at the admin level.

Note that a board disable is something an admin can deal with and which only effects the phpbb3 version of the forum. So it is a lot gentler than suspending forums which effects both the phpbb2 and phpbb3 versions.
Also note that this effects only phpbb3 forums that are not set to be the main forums, e.g. only "works in progress" which really should not be seeing bad posts at all.
myff admin

I'd also add that to help us with the advertising dilemma, there is now a big brother side to this, with tiered advertising. It's not something I like, but basically the rule will be that if the language on a forum is out of the gutter, then the discerning advertisers will be filtered out.
As of Thursday when that part goes live, a lot of forums should see better adverts
myff admin

Well hopefully a lot of forums are now on better adverts.

The new system has bought to light quite a lot of spammed phpbb2 forums that had crept under the radar, so a lot more suspensions have taken place.

I'd remind people that from here on there is only so much bad language in proportion to posts before advert quality will be effected on a forum.
myff admin

phpbb2 has code now to force visual confirmation on if it looks like there is a spambot issue.

Plainly spambots have gone hyper in a big way and even more measures are going to be needed.

I can see a lot of "mass user deletes" being needed on some forums
myff admin

The next stage has been applied, and it is quite scary the number of bots it is detecting and blocking.

Obviously with all blocking attempts there is the risk of false positives, but an investigation of quite a lot of the reports has not come up with anything that should not have  been blocked so far.
myff admin

One thing that is very apparent is that the bots are working on lists of forums and there are hundreds of spammers running such scripts continually, regardless of success. 99% of the time they hit against our uncracked visual conformation system and they always fail, but they are always back for more.
myff admin

We now have an IP list of over 1000 spambots  

the good news is that we are now configured so that basically their attack will hang their scripts, this means that they have to take us out of circulation They don't care about failure if failure just takes a second, as their scripts will still complete, but they do care about their scripts slowing and dying.

The fall off in spam rate is massive.
myff admin

I think there might be a negative aspect to this for forum admins.

I have noticed a distinct upsurge in manually entered spam here on the support forum.

Basically denied the honey pot of spamming vulnerable forums automatically, the decent forums could come under more fire
myff admin

Progress continues to be made, with I think only a very small number of people getting unjustly firewalled.

I sincerely believe people will have been seeing their forums run faster and with overall less spammers getting through, both on phpbb2 and phpbb3 forums.

I'm now only getting a trickle of emails alerting me to spammers, it's a bit of a shame that I have only got graphing of the levels of spam attacks in place over the last day or two. So I can't give solid figures on the reduction in the attack levels.

But it does seem clear that a lot of them are quitting attacking us
myff admin

Just to keep people updated, the alerts I'm getting are trailing off to a very low level.

The raw level of attacks is a bit of a different story, it is clearly down perhaps by about 40% or so.

Probably the contradiction is simply that I don't get alerts for the most blatant attacks and so these are probably not being reduced as much as the ones I get alerts for.

I guess the key question is how is this effecting people in practice?
myff admin



That's the graph of phpbb2 registration attempts so fat this month on the busiest server, the data is not "cherry picked" phpbb3 shows similar results, but I'd have to explain away bits of the graph that represent inaccurate data from the start of the month.

The grey line is just a line that shows the data 24 hours earlier. red dots are missing data.
The line is heavily "smoothed" as the raw data is a mass of spikes and lulls.

Bear in mind as well that this is not spammers being graphed but registration attempts, the majority of which are probably still spammers, but we hardly want the graph to drop to zero!
myff admin

Not so far bad enough to change the direction of the "smoothed" graph upwards but a major upsurge again today

I wonder if this is going in a cycle of long term firewall bans expiring, whatever it is it only goes to emphasise the point that this battle is never actually won.

Of course the graph of spam attempts is not to say the spammers are getting through the defenses.
myff admin

Just caught someone here replying to a spammer and hence potentially legitimising their spam.

Always look at the poster before responding, do they have only 1 or two posts, and do those posts or their sig contain spam links? or any links for that matter? If so then treat with suspicion. The post may typically seem helpful, but all they are doing is SEO spam for themselves.
kagome

myff admin wrote:
Not so far bad enough to change the direction of the "smoothed" graph upwards but a major upsurge again today

I wonder if this is going in a cycle of long term firewall bans expiring, whatever it is it only goes to emphasise the point that this battle is never actually won.

Of course the graph of spam attempts is not to say the spammers are getting through the defenses.


I noticed a drop in spammers myself but today they started up again. I was wondering, don't know if I am allowed to post here but it is the perfect place to ask, do you use the Anti-Spam ACP by EXreaction? I found out about it for phpBB here:

http://www.stopforumspam.com/contributions

Since you seem like you track spam so I just was curious.
myff admin

stopforumspam would not accept our submissions after we became their top contributor.
So I think any data they have would by and large extend little what we have and by the nature of the their submission process actually be less reliable.
So no we don't use a system that relies on their data.
myff admin

kagome wrote:

I noticed a drop in spammers myself but today they started up again.


This type of report is not very useful.
1) forum url
2) forum type phpbb2/phpbb3
3) Is this attempts at registration?
4) Are registrations succeeding?
5) What image verification system are you using?
6) What Country are the spammers from?
7) Are they managing to spam post?
8 ) Is it from newly registered accounts or older ones?
9) What are they spamming?
10) What type of numbers are you seeing?

That's just off the top of my head and maybe people can give all that information, but they can certainly give us some. No information does not help anyone.
myff admin

Spam attempts have bounced back a lot in the last few days to perhaps within 30% of the peak figures recorded.

Of course one thing I may have failed to point out is that even the peaks we recorded on the graphs are data from after we made very major inroads in the spam rates, and of course "attempts" are not the same as "success".
myff admin

In the last day or two we have seem peaks in spam registration attempts twice as high as anything so far recorded, but the good news is that they simply are not having any noticeable success rate.

What I do think is plain is that we are not seeing the simple random ebb and flow of uncoordinated activity. But your guess is as good as mine as to what goes on  
myff admin

Been musing over some more stats in the light of the sometimes very high number of attempts still occurring, and I have realised that even the peaks simply have to be only about 10% of what was hitting us before action was taken.

e.g. whilst even a single spammer is a spammer too many we must not mistake the current molehills for the old mountains.
myff admin

Quite a few forums are now getting warning emails on spam, including quite large active forums.

The spammers are plainly now trying their luck anywhere and everywhere, and whilst we are not making their lives easy, some are getting through and admins needs to be on their game.

The main thing is to look out for the new member who tail ends a few threads with something innocuous, but who has links in the post or their sig.

Admins and Mods need to be on the ball and ruthless.
Bravo

myff admin wrote:
The main thing is to look out for the new member who tail ends a few threads with something innocuous, but who has links in the post or their sig.


Yep, there has been loads of those on this forum last few days, quite often actually on topic for the thread but still spammers nonetheless
myff admin

Attack levels markedly down in the last day, no rhyme or reason to that that I can think of.

What is notable though is to look at the level of actual forum registrations, they remain pretty steady unaffected by the level of attack. e.g. the bots don't make it through the net
myff admin



Attacks still down, in fact if you look at the graphs and are very sneaky about time periods shown and smoothing factor, you can produce a graph that shows we are at the lowest levels of attack since we started the system  
myff admin

not surprisingly the graph of the last post has proved as selective as I thought it might, the peaks in attacks are if anything increasing, probably in response to the continual shutdown of spamming accounts that were already active, prompting the spammers to try and create more accounts.
Bravo

They are definitely getting harder to spot now, many are actually being on topic with a thread, on very large and active forums that are used to several new members posting every day I can see a lot will be missed.  However at least it is forcing them to use human spammers which ultimately will cost them much more in time and money, which for a post that can be deleted in a heartbeat can't be worth it in the end.
myff admin

At this point the trend in spamming attempts is resolutely going up but the success rate on the registration front remains almost nothing on the scale of things  
myff admin

Spam attempts are down, but we are seeing a clear backwash of spamming of inactive forums  

This is a shame as whilst we can tolerate a certain amount of spam on forums, we can't let such forums sit there being filled with spam, and our counter measures only serve to filter out some not all spam.
The upshot is that inactive forums are probably on a path to suspension and eventual deletion  
myff admin

Spam continues to flat line at 33% of peak levels.

I have just taken a decision to suspend a forum where the admin is not visiting and the forum is full of end of thread spam that might almost seen like genuine members.

I don't know if that will seem harsh  
myff admin

myff admin wrote:


Attacks still down, in fact if you look at the graphs and are very sneaky about time periods shown and smoothing factor, you can produce a graph that shows we are at the lowest levels of attack since we started the system  


When I posted that attempts started to spike again to some of the worst levels, I could now make exactly the same post again   albeit there is a longer term look to the drop off in spam attempts that may be more hopeful.
myff admin

I have introduced more banning measures this morning, there are still some persistent spammers that can be blocked my such measures.
myff admin

I have taken another very large step, and where it is possible to identify spammers, their posts can now and have been automatically  deleted from forums.

100's of known spammers have been scanned for and their posts deleted. This has effected a very few threads on properly maintained forums where people have chosen to reply to the spammer leaving their post alive. But the net effect is that 100's of abandoned forums have been totally cleaned of spam.

This does not give those forums much  likelihood of going places, but does improve the standing of everyone else's forums in the eyes of search engines.

I thought a lot before going down this road, as we have always made a very big thing about the forum owner being totally responsible for their forums.  Frame it as you will this is a bit of a drift from that policy

However on balance this is a good thing.

The major thing that I'd like thoughts on is that this process needs to be fed new spammers!

The way it has worked is that when a spammer has been identified, all forums have been scanned for that spammer and the forums they have spammed are reported. Those forums (or at least some of them) are then manually checked, and if a forum has been spammed once, the chances are it has been spammed by others, the other spammers are then fed into the system.  Thus an almost every increasing list of spammers get found.

As of now we are fresh out of "seeds" I don't like the idea of people reporting spammer details, what we actually need is links to spam posts we have missed, which means the equally annoying issue of asking people to let spam remain for a while
myff admin

Even more measures are on the go now.

phpbb3 is virtually now on the same anti-spam measures

I do have my eye on one clear milestone, we are almost at 50 million posts.

We have been that ways for a couple of months now!

It would be rather good to pass that point with the view that those posts are 99% spam free.
myff admin

We have cleared now maybe 2 million spammers from the system.

Recently as an experiment I reopened some suspended heavily spammed forums, and the consequence has been another massive rising tide of spam attempts.

I was wondering if that would happen and if some spam "honey pots" would help at all, by letting us idenetify and globally ban spammers via the honey pots?

The experiment needs to be played out a little more, but I think the answer is a resounding no.
myff admin

August and September saw a rising tide of spamming going on, not sure if much more was getting through but the attempts were getting to silly levels again.

To help counter this a few more measures have been put in place which are clearly helping
myff admin

The logs suggest the new measures have put off spammers trying with bots, but bots never did have much success here, so we see the attempts fall massively, but the numbers of spammers succeeding dropping only marginally. So whilst the servers appreciate the efforts, whether forums owners appreciate the work is more debatable.
uncle aj

myff admin wrote:
The logs suggest the new measures have put off spammers trying with bots, but bots never did have much success here, so we see the attempts fall massively, but the numbers of spammers succeeding dropping only marginally. So whilst the servers appreciate the efforts, whether forums owners appreciate the work is more debatable.


I certainly appreciate all the hard work, thanks.  
myff admin

Figures seem to have stabilised at 5% of previous levels.

But as I keep on saying its a bit like coastal defences against tidal erosion. When we stop 95% that was crashing against the walls and finding some success rate on abandoned forums, then it may well be that you get spammers giving up on that method and doing more manual spamming on active forums.

I know on the support forum we are still getting waves of spam

I do have a next step in mind, which would be for the first few posts by a member on forums to be automatically checked and for the member and their posts to be put on "moderation" if suspect.
myff admin

Shortly after that post things start to get out of control again  

But this month there was another rework and on hotblack spam attempts that we peaking at over 5000 an hour are down to 10% of that.

Though as ever I would point out that that is a measure of the crap that is getting through to stage 2 you might say, at which point our servers and the forums are having to deal with the load, but they are almost certainly and clearly not being able to actually spam at that rate or anything like it. So it does not follow that forum owners see a positive effect.

Of course 5000 requests an hour does not sound like a lot of load, but with BOTs that could be hitting in a small part of the hour.

       myfreeforum.org Forum Index -> What's new? Announcements!
Page 1 of 1
Create your own free forum | Buy a domain to use with your forum