Donations to Upgrade Forum Server?

Questions and comments
bigato
Posts: 2798
Joined: Sat Mar 05, 2011 12:43 pm
Location: Brazil

Post by bigato »

I like the way the plan is structured and I think it's going to be fun to try to figure it out how to migrate the data. Also, if option 4 is acceptable, I know that I can take my time to work on the migration. Because option 4 is always there and is very easy.
Then I think that it's a matter of settling on some software and waiting to hear from Spartan. I will do some research about it this weekend.


bigato
Posts: 2798
Joined: Sat Mar 05, 2011 12:43 pm
Location: Brazil

Post by bigato »

I don't want any money.


llorona
Posts: 394
Joined: Sun Sep 23, 2012 11:44 pm
Location: SF Bay Area

Post by llorona »

Zero tech skills here, but I'm willing to make a contribution.


bigato
Posts: 2798
Joined: Sat Mar 05, 2011 12:43 pm
Location: Brazil

Post by bigato »

After a lot of reading yesterday and today, here's my conclusions so far, regarding the choice of new software.
Discourse - full of cool features and a lot of new ideas, but still not ready for production.
bbPress - While the automated updates would seem to be promising, what I've found browsing their forums is that they do far too much new versions in short time and because of that things tends to break often. Someday that could lead to as much or more trouble than if you manually update every time. There's not a big community behind the forum software itself, and also not a lot of users. The company behind Wordpress pays a developer full-time to maintain this plugin, but it is far from having so much users and developers as Wordpress does. The software is a more recent piece and so its codebase is not so mature.
phpbb - I'm sold on this option. Used in one gaming community that has around one million posts *a day*. Millions of active installations. There are a lot of developers contributing to the code. The people behind it are very conservative in implementing new features. New versions are not very frequent (last stable release was 9 months ago). Good security track since version 3 (current). They pay for external security audits. No bells-and-whistles in the core, but you can have the features you want through plugins. Has more tradition than the others: first version released 13 years ago. Because of the long time it is around and of the big number of installations including high-traffic sites, it has already run over all the problems we are facing and much more. And... it has automatic update too. You just click some buttons on the admin interface, and if you upgrade every release without skipping a version, it should be smooth and easy enough. That coupled with the fact that releases are not frequent at all makes it the best option if you don't want to outsource but also don't want to spend too much time. But should you ever change your mind about paying someone for the maintenance, there is a commercial service that provides the hosting with maintenance included. Some of the people behind it are also developers at phpbb community.
While just updating bbPress to current version should be a nice step, I think I've made clear above what my option is. But I could help too if the option is to stay with bbPress. There is a script to convert from 1x to 2x series, as pointed earlier in this discussion. Note that bbPress 1.0.2 is the last standalone version. Today it is just a Wordpress plugin, so you need to have a full Wordpress install, even if you don't use all of it.
My opinion so far is that migrating directly to phpbb 3.0.11 should be the best option. As already mentioned here, there are scripts to import the data that should do most of the job.
One problem that we will face will be regarding that posts where there is a link to other posts. All those links will be broken, unless someone writes a script to change that. If we are even considering not importing new data at all, that's a minor problem in my opinion. But I could try to solve this too.


bigato
Posts: 2798
Joined: Sat Mar 05, 2011 12:43 pm
Location: Brazil

Post by bigato »

Next step: I'm waiting to listen from Spartan if he will do the job of taking care of backups, exports and coordinating our efforts.


bigato
Posts: 2798
Joined: Sat Mar 05, 2011 12:43 pm
Location: Brazil

Post by bigato »

Off course, I'm also wanting to listen what people here think about the choice of phpbb and if Jacob is ok with it.


aussierogue
Posts: 379
Joined: Thu Nov 10, 2011 1:02 pm

Post by aussierogue »

Im up for cash if some is needed..


Spartan_Warrior
Posts: 1662
Joined: Fri Dec 02, 2011 1:24 am

Post by Spartan_Warrior »

Hey guys. Had a busy weekend and wasn't following the thread closely. I'll catch up today on the posts since I last visited and see where we're at.


Spartan_Warrior
Posts: 1662
Joined: Fri Dec 02, 2011 1:24 am

Post by Spartan_Warrior »

Thanks for summarizing, bigato.
So, it sounds like we've decided on upgrading the forum to new version/software as the most efficient first line of attack against the performance issues. I agree. This will also have benefits overall in reducing the forum's vulnerability.
I should be able to do bigato's Option 4 (new forum from scratch with current one available as archive) with no or negligible downtime. From what I recall instantiating a new forum on a server is pretty plug-and-play. As long as Jacob's server allows multiple SQL databases(?), both forums should be able to run simultaneously. At that point, registration, posting, and maybe even logging in on the old (bbPress 1.0.2) forum can be disabled and it can serve as an archive.
Advantages of this method are: Simplicity and ease of implementation; no messy database exporting/translating; current forum lives on in search engines and link references--no risk of dead links, etc.
Disadvantages are: bbPress forum would still exist on the server, so if anything intrinsic to its mere existence/continued access is slowing down the server, the problem may not be solved; people would have to start new accounts and make new journals.
Option 5 is more complicated and is basically what I've been assuming throughout my posts--updating the forum, but then trying to migrate the old data into the new forum. I'm not as confident that this effort will be successful (at least in my hands), and I'm also not aware of a way of doing it that wouldn't require at least a few hours downtime (so no new posts are made in the old forum while exporting that data and importing it to the new one).
If folks would be happy enough with Option 4, I'm confident in my ability to do it. That said, I notice bigato seems very knowledgeable about this stuff (and a little more interested in it than I am) and has also volunteered. I am quite happy to pass the torch to him if he and others don't mind. I only volunteered because no one else was speaking up! I consider myself more of a candidate of last resort...
Also, I second this:
"2. And where are all the techies? They are the dominant readers of this place, yes? Geez does the younger generation not have any LAMP guys? (no smiley face. insert anti-smiley face here.)"
I'm a technical writer with a Philosophy major, for crying out loud! (anti-smiley face)
Bigato, if you do want to take the lead role, I am happy to offer support in terms of how to back things up and export the database, or anything else you're unsure on that I might know.
On the other hand, if I am to be the lead/general contractor, does anyone have a good FTP client they can recommend for uploading files to the server? I seem to recall I used WS_FTP back in the day, but it doesn't seem to be freeware any longer.
Also, after finalizing who is taking the lead, the final decision, and perhaps more salient to the community at large, is exactly which software to use. I am also still in favor of phpBB mostly because of its broad support and my brief experience with it in the past. However, perhaps we should narrow down the candidates and take votes?


jacob
Site Admin
Posts: 12426
Joined: Fri Jun 28, 2013 8:38 pm
Location: USA, Zone 5b, Koppen Dfa, Elev. 620ft, Walkscore 73
Contact:

Post by jacob »

Question: If you simply start a new forum next to the old forum, how would that solve the traffic problem to all the posts in the old forum?
The solution should certainly aim to migrate the posts to the new forum. It's okay if intra-links are lost (I don't think there are that many). Archiving should be the last resort.


Spartan_Warrior
Posts: 1662
Joined: Fri Dec 02, 2011 1:24 am

Post by Spartan_Warrior »

If the issue is spamming of the registration page, disabling the registration page on this (the "old" forum) would theoretically stop the bottlenecking.
The whole solution of upgrading the forums is premised on the idea that there's something about this forum software or a vulnerability in it--not the level of traffic itself--that's causing the performance issue. If it's just the level of traffic, we're back to square one of requiring a server upgrade with more bandwidth, as far as I know.
Migrating the data to the new forum is really just one extra step, though, since there appears to be a bbPress 1.0.2 --> phpBB X.X script. I can try that full migration (option 5) as the goal, as originally intended. But I know I can do Option 4, as a fall-back plan.


jacob
Site Admin
Posts: 12426
Joined: Fri Jun 28, 2013 8:38 pm
Location: USA, Zone 5b, Koppen Dfa, Elev. 620ft, Walkscore 73
Contact:

Post by jacob »

The issue could also be that bbpress is just slow/uncached(?). Each page has a lot of information on it to other posts/threads. If the SQL database is traversed in an inefficient way each time a new page is requested...


bigato
Posts: 2798
Joined: Sat Mar 05, 2011 12:43 pm
Location: Brazil

Post by bigato »

Spartan, who is going to lead is irrelevant to me, as long as the choices are not very absurd. It's just that I prefer not having access to production site and not be responsible for backup and restore. I know how to do it, but I vastly prefer that someone else does that part.
Given enough time, I'm quite confident that I can figure it out how to do everything to migrate the data, including fixing links. Or someone could migrate the data while someone else takes care to translate the internal hyperlinks. Any division of labor if fine to me as long as I don't have to mess with production data directly. That's my only requirement.
Jacob, regarding archiving the old posts, if that is the path chosen, my idea would be to turn them all into static html. That's quite simple to do. We couldn't left the old software online because it could still be vulnerable and would have no maintainers producing bugfixes for it.
Regarding the spamming on the registration page, I would be curious to know if that change that I've sent is able to stop them. Did you try it? That's not a solution, just a hack, but if the problem really is located there, it could make things better until the definitive solution.
Jacob, bbPress last version is reported to work with higher traffic than we have. Keeping the software and just upgrading the version is not a very bad option, even if I prefer phpbb as I explained above. What is not very efficient is trying to outsmart the thousands of hours that users and developers already have put towards the newer versions. A quick hack doesn't require much time, but trying to optimize it the way you are suggesting only would make sense if the person is working in the last version and together with the rest of the developer community.


jacob
Site Admin
Posts: 12426
Joined: Fri Jun 28, 2013 8:38 pm
Location: USA, Zone 5b, Koppen Dfa, Elev. 620ft, Walkscore 73
Contact:

Post by jacob »

I tried blocking the reg page at the server level (using httpaccess). That didn't solve the problem.
I'm in favor of bbphp with migrated posts. Translating the internal hyperlinks, but I don't think it's that important. There can't be that many of them.
I think Spartan gets the keys then. (Do back up the site and have a plan to restore ASAP if something goes wrong. We are talking live data here.)
Bigato, did you get the PM I just sent?


bigato
Posts: 2798
Joined: Sat Mar 05, 2011 12:43 pm
Location: Brazil

Post by bigato »

Ok, so phpbb seems to be majority. After the migration and tests, if the slowness persists, we will probably need some help from the crew at your hosting provider to narrow down the sources of trouble. They will be more than happy to help to lessen the burden on their resources.


JamesR
Posts: 919
Joined: Sun Apr 21, 2013 9:08 pm

Post by JamesR »

I strongly recommend using the SSH interface of the webhost, if one is provided, instead of FTP. Generally, FTP is outdated (unsecure), and usually is disabled these days in favor of SSH (which provides SFTP & SCP too). So SFTP or just SSH directly into the user account and use the command line..
An example of backing up the bbpress database:

I would typically take a peek at the configuration files of bbpress to determine the mysql server address/login/pass, assuming it's in plaintext like in wordpress installations. I think shared webhosts usually keep MySQL on a different server from the regular user account. Then I would run a command like:
mysqldump -h actual-mysql-server.com -u login -p bbpresstablename > bbpress-dump.sql
-p option will cause it to ask for a password

The login/pass won't necessarily be the same as the actual webhost/ssh account.
Once it's been dumped, then it's easy to import that into another database if necessary.
--
I'm somewhat more in favor of upgrading bbPress first and seeing how that goes.. maybe try phpbb later?


anomie
Posts: 439
Joined: Sun Apr 29, 2012 2:13 pm
Location: midwest, usa

Post by anomie »

Hello,
Filezilla is great free FTP / SFTP client:
https://filezilla-project.org/download.php?type=client


bigato
Posts: 2798
Joined: Sat Mar 05, 2011 12:43 pm
Location: Brazil

Post by bigato »


I'm somewhat more in favor of upgrading bbPress first and seeing how that goes.. maybe try phpbb later?

Why?


anomie
Posts: 439
Joined: Sun Apr 29, 2012 2:13 pm
Location: midwest, usa

Post by anomie »

Hi bigato,
I private messaged you with my contact information. Asynchronous and offline contact like email will work best for me.
I will also try to be on IRC, though will not be monitoring it 24/7.
I will private message S_W as well now.


KevinW
Posts: 960
Joined: Mon Aug 02, 2010 4:45 am

Post by KevinW »

I had a busy week and am catching up on this conversation. My $.02 FWIW:
- I agree, with reservation, that options (4) or (5) are the proper way to go. We are caught in the crossfire of two best practices that are contradictory in our case. The first best practice is to run up to date, actively maintained software so that security patches are available and can be applied in a timely manner. However another best practice is to diagnose problems and determine their root cause rather than sweeping them under the rug. I agree vociferously with

Be careful assuming your past experience with something roughly analogous is applicable -- do careful experiments and measure the results. It is very easy to waste a lot of time, and sometimes make things worse by doing what "makes sense" in a complex system with so many moving 'parts'.

It makes me uneasy to abandon the current software without ever determining why it was slow. IMO if we start having similar problems with the new software we really need to diagnose the problem. That will be easier with a software package that is modern and has a sizable community.
- Personally I refuse to maintain anything PHP-related, which is part of why I suggested Discourse. But if someone else is volunteering to be an ongoing sysadmin and is more comfortable with a different package, I will defer to them. Another best practice is that the person(s) doing the work should have the strongest veto power.
- I have periods of intense business (e.g. last week) which could make me a negligent sysadmin so I'm not throwing my hat in that ring at this time. However I am willing to help with the one-time transition. In particular I am willing to help write a migration script.


Post Reply