Large community usage?

Hello @PhoenixDragon,

The entire software is designed to run smoothly on low-budget shared hosting, an environment where even the smallest performance gotcha will horribly backfire. Additionally we pay attention to design features to scale nicely, you won't find a single feature that will suddenly overwhelm your system just because it has to deal with large amounts of data.

Large and very active forums are:
http://zeldauniverse.net/forums/
https://forum.fcbayern.com
https://www.bisaboard.de
http://www.dogforum.de

Quote from PhoenixDragon

I'm not concerned about shared hosting as my current setup requires a dedicated server.

It is not that much about shared hosting, but about designing a software that runs well with even with low amount of resources. The individual resource usage per request is pretty low, which is extremely important for sites with a high concurrent visitor count as it enables you to serve more users with the same hardware.

I would be happy to provide you with more information, if you could tell me a bit about your current setup. Especially the number of average concurrent and median peak PHP requests would be interesting, as it tells an in-depth story of what is actually happening. A high "users online"-count is a nice looking number, but unfortunately does not expose enough details such as time frame (e.g. 1.000 users are counted across the last X minutes) and the behavior (e.g. average of 3 requests per minute).

Quote from PhoenixDragon

Assume I've done the math already and 1,000+ visitors at every given moment of every given second is accurate. With the minimum request being 1,000+ at any given moment.

Thank you for clarifying this, I was simply asking because this term is anything but a fixed metric and especially with vBulletin quite a lot of people tampered with the time period for their user online statistics.

Quote from PhoenixDragon

Assume there is also 56 million post to query which vary from 1 page to 300 page threads with the average being 30+ pages.

This is a non-issue, the number of posts doesn't impact the performance.

Regarding your average number of pages, there is an active German thread on our forums which is at 536 pages at the time of writing, you can try it out yourself. The FC Bayern forum (link above) is another good example as their threads usually average at about 40 pages. We solve the issue with very long running threads by having two separate queries that fetch the posts: The first only selects the ids and the second fetches all the data using the result from the first query. This ensures a constantly low query runtime no matter how large your thread is, I can go into detail if you're interested.

Quote from PhoenixDragon

A simple yes or no will suffice, but the question is, do you feel confident your software could manage such a load without locking up?

In all honesty, I can't even recall ever stumbling across a forum that huge (with the notable exception of gaiaonline.com), so I can only make assumptions at this point. I mean forums with 5 or 10 million posts aren't that uncommon today, but 56 million is a whole different story. The number of posts are a complete non-issue as everything is designed to run with larger amounts of data and there are many forums running with Burning Board having more than a million posts - without ever experiencing a degraded performance. Every relevant query runs against database indices, preventing any deal-breaking full table scans or similar.

Regarding the number of concurrent requests: There are no deadlocks or other issues when it comes to a high request count, so technically speaking it can scale indefinitely without every locking up. We offer both a memcached integration (as a replacement for the file-based cache) and elasticsearch (replacement for MySQL FULLTEXT search, which does anything but scale), to remove any potential bottlenecks that may show up when working with large amounts of data.

In general there are a few key principles we've followed that ensure proper scaling:

• Avoid table locks, InnoDB's row-level locking helps a lot solving this.
• Queries make use of indices to avoid full table scans.
• Caching of complex data that is more or less static but requires some time to compute, for example the forum list is using cached data to show the structure and the last post.
• Denormalization of data when necessary, e.g. the list of the first three users that liked a post are cached for each post in database to avoid running an extra query per post.
• Queries that would return large results are performed using two separated queries (for example posts on a thread page) to prevent MySQL of having to work with too many data. This design allows a fast lookup for the relevant object ids and we can then fetch all additional data by using exactly these ids, making full use of table indices and primary keys.

To sum it all up: Based on our experience with actively visited forums having several million posts we can say that there is absolutely no evidence of deadlocks or slowdowns. zeldauniverse.net's forums contains 5.6 million posts and runs perfectly fine and I'm confident that this is true even with 10 times the data. But as I said earlier, this is only an assumption/educated guess based on our combined experience building a forum software for almost 15 years so far.

Quote from PhoenixDragon

How many queries do your software have and on average the time to complete that query for 1 individual?

It requires 2 separate queries:

• Read the post ids for current page of the thread.
• Read all necessary data (including users via LEFT JOIN) for the post ids previously fetched with the query above.

Both queries use only indices (including foreign keys) to retrieve the data, resulting in a static index lookup by MySQL.

Quote from PhoenixDragon

Both of these are standard without relying on a 3rd party addon?

Memcached is built into the core (the next major introduces Redis as an additional alternative) and elasticsearch is available as a paid plugin offered and maintained by us.

Quote from PhoenixDragon

This is the 2nd time I've been informed that a feature was pending in the next major release. Your next major release is 4.2 is this correct?

That is correct, we've outlined the major changes in this post: Outlook on the future of WoltLab Community Framework 2.2

Quote from PhoenixDragon

Will the features you've previously mentioned (CDN support and Redis) be included in 4.2 or are you speaking of a yet further release?

Redis has already been implemented, but the CDN support is still on our list. There has been no work done towards this task yet, mainly because there are other components that need to be stabilized first. I'm afraid this is everything I can tell you at this point