Need help thwarting spambots to speed up the forum again

Post by **jacob** » Thu Jan 17, 2013 7:37 am

I think it's because the forum has been extremely active over the past 48 hours (a particular thread comes to mind) which has left cloudflare unable to cache since pages have had to be generated often.
Other than today and yesterday, I rarely get the "busy" signal on the forum anymore.
PS: It's both nice and annoying that the forum/blog seems to have reached a size/popularity that it practically requires a professional webmaster/dedicated server to keep running smoothly.

mrjay · Post by **mrjay** » Thu Jan 17, 2013 7:43 am

since joining a few weeks ago, i've found the site nearly unresponsive between, i dunno, about 9 pm and 11:30 or so central standard time nightly. It's no big deal, i just come later, or earlier...i assumed it was a very large volume of traffic....

Spartan_Warrior · Post by **Spartan_Warrior** » Thu Jan 17, 2013 3:45 pm

Eh, maybe. I feel like the problem has been happening since before that thread started. But it is on-and-off (I'm not having problems this morning), and it also seems like other people are getting through and posting when I can't. Maybe it is a traffic problem. Not that that changes the annoyance level.

But I'll see if it persists.

George the original one · Thu Jan 17, 2013 6:23 pm

Cloudflare is starting to interfere with posting. I've lost one post and nearly lost a second. Basically the site forgot I was logged in, so the post didn't make it into the database.

Spartan_Warrior · Post by **Spartan_Warrior** » Thu Jan 17, 2013 8:26 pm

The latest trick seems to be when I post something new, it's added to the thread, but doesn't show up in the forum index (e.g. I just posted in "that thread" but the most recent reply still says GandK from 18 minutes ago).

My_Brain_Gets_Itchy · Post by **My_Brain_Gets_Itchy** » Fri Jan 18, 2013 3:57 pm

Forum is super fast today..

Spartan_Warrior · Post by **Spartan_Warrior** » Fri Jan 18, 2013 5:14 pm

And yet still glitchy as hell. From what I can tell no one is able to see any of my recent posts in the "Can't Save? Here's Why" thread. (We'll see if anyone notices this one. o_o) My posts show up for me when I'm logged in, vanish when I'm logged out, and the thread post count goes from 179 to 172.
Needless to say this is annoying. On the other hand, perhaps it's just a well-needed sign to give up on that thread...

Spartan_Warrior · Post by **Spartan_Warrior** » Fri Jan 18, 2013 5:41 pm

That's it--I'm out. Can't deal with this anymore. Every fucking time I go to post I get the "Website is unreachable, try again" stuff, hang-ups, posts evidently showing up for no one but me--and I just don't have the patience or the time in the day. Maybe I'll check back in a month or so.
In the meantime, I've saved my journal. If I come back I'll repost it. If I start a new blog or journal elsewhere I'll come back and try to force through a post on my whereabouts, in case anyone's interested.
@Jacob: Please delete my journal thread here for privacy reasons.
It's been fun chatting with you and the forum's been a unique experience all around... even now, in being uniquely bad. Good luck all and thanks for all the fish.
(Let's see how long it takes to get this post through...)

Post by **jacob** » Sat Jan 19, 2013 1:45 am

@Spartan_Warrior - It seems that the spam plugin marked your posts as spam. I've reverted them. If a post is marked as spam, only the poster and the moderators can see it. You should be good now.
Are you sure you want to have the thread deleted? If I do so, I can't revert it, ever.
On a personal level, I feel like cloudflare will throttle my attempts if I reload the forum too often. Maybe that's what's been happening to you? The way cloudflare works (decentralized), the site may appear fast to some but slow to others depending on how responsive their local server is.

dot_com_vet · Post by **dot_com_vet** » Sat Jan 19, 2013 2:47 am

FYI, I had slowness earlier, but it's running better than ever now!

simplex · Post by **simplex** » Wed Jan 30, 2013 12:11 pm

Hi Jacob,
I still have sometimes that the forum seems offline with cloudfare. Have you tried the bad-behaviour plugin?

squashroll · Post by **squashroll** » Wed Jan 30, 2013 8:32 pm

Eh... Read through this thread. Not a computer guy, so I really don't get it.

I've been having A LOT of trouble using this forum. I find it interesting but its EXTREMELY slow. After clicking a link there is typically a 30 second or so wait. I'd really like to learn and participate, but this is quite the deterrent. Any fix on the horizon?

Post by **jacob** » Thu Jan 31, 2013 7:14 pm

The speed goes between "stuck" and "ultrafast". One day it takes 5 minutes to get a page to load and the next day it loads at the blink of an eye, that is, 300 miliseconds

.
I cleaned up the blog to require less CPU processing. My account is no longer under any CPU restrictions. So I think it's a problem of 1) the site having outgrown its current host (shared server) and 2) the forum software not being designed to take this much load (there are a lot of lurkers!).
Some data ...

Blog pageviews per month: 120-150k

Forum pageviews per month: 300k (200k users, 70k spammers, 30k search engines)

Wiki pageviews per month: Dunno
This means either throwing money+time (to get a dedicated server) or time+more time (to figure out a better forum software solution and transfer the existing database) at the problem. I'm not interested in either.
Is anyone interested in taking over the forums? We could do a spin-off... PM me if interested. (Only people people who have written 200+ posts will be considered.)

henrik · Post by **henrik** » Thu Jan 31, 2013 9:43 pm

@jacob If you don't mind losing exposure to search engines, consider restricting access to authenticated users. It should increase authentication load a little, but at the same time decrease traffic from spammers, search robots, lurkers etc. Might be a significant difference for a quick fix.

justjohn · Post by **justjohn** » Sat Feb 02, 2013 9:42 pm

Jacob, do have any stats for amount of data transferred?

Post by **jacob** » Sat Feb 02, 2013 11:08 pm

@justjohn - Cloudflare says 50.5Gb/month.

Post by **jacob** » Thu Feb 07, 2013 2:58 am

I haven't personally experienced any forum issues for a few days now.
Cloudflare says that spam hits have dropped from 70k/month to 50k/month.

dot_com_vet · Post by **dot_com_vet** » Thu Feb 07, 2013 4:38 am

Working great for me here!

Tyler9000 · Post by **Tyler9000** » Thu Feb 07, 2013 5:27 am

Just one data point:
The site works fine for me everywhere but from home. I have ATT Uverse Internet. As an experiment, the site works fine on my phone over 3G, but as soon as I turn on my wifi at home, I get the same cloudfare "ere is offline" error that I do on my home computer.

dragoncar · Post by **dragoncar** » Thu Feb 07, 2013 7:03 am

Tyler9000, sounds like ATT cached a cloudflare page at a bad moment. Hopefully it clear up. It's been working quite well for me in general.
One note, it seems like a lot of readers here are also MMM readers. If things get bad, we could try to get an ERE subforum there. Assuming, that is, that the forum here isn't providing much revenue to Jacob.