Big Board Optimization

[Kaelon]

Hi there,

Many months ago, I had tried out CES VaultWiki for my Big Board community (1.5 million posts, 40,000 registered users, about 200-300 concurrent at peak times), but it stalled/died during the NuWiki Import process. So, a couple of questions:

1. Does the NuWiki import process work without coming to a screeching halt on a Big Board configuration? How has the NuWiki import process been improved?

2. How has VaultWiki been optimized for Big Board performance? Do you have any Big Board clients that are using VaultWiki?

Thanks!

[pegasus]

1. The "screeching halt" you refer to was due to a full table scan in one of the early versions of the importer.

The import process is now done incrementally, with multiple pages for each step if there are many items to import. If you're concerned about importing from NuWiki, I would recommend cleaning out your "nuwiki_revision" table.

1 & 2. After your issue, we spent the next few weeks reviewing all of the code in VaultWiki, optimizing queries, and fixing any other table scans.

2. With almost every build, we rewrite some sections of code to make them more efficient - most recently we improved auto-linking performance by between 30-70% (depending on how heavily the feature is used on a given page).

There are options to disable some of the more server intensive features, i.e. auto-linking (creating wiki links without BB-Code) and link caching (saving info about every wiki link in the database).

When we update VaultWiki for vBulletin 4, VaultWiki's tables will follow suit and be converted to innoDB.

We keep our clients' information confidential, so I can't divulge any names or URLs - but I believe our largest client has about 1.7 million posts, 180,000 registered users. I don't recall them ever reporting performance issues.

[Kaelon]

Thanks, pegasus. Can you tell me more about how the NuWiki Impoter works, with regards to its incremental import? I just want to make sure that we won't deal with time-outs, and that we can monitor its progression in semi-real-time on the browser.

[pegasus]

It's similar to the way that vBulletin upgrades occur. Between each step, you have to click the 'Next' button to continue. By contrast, in the original version you had tried about a year ago, it attempted to do everything on one page load.

The following is an outline of each step:

1. Finds any articles that were in "Article"-type forums (only the creator could edit them), and populates the 'protect' table with the compatible protection level. If there are more than 200, imports them in pages of 200 per page (you have to click a 'Next' button between each one). Then, converts the NuWiki forums into VaultWiki-compatible forums.
2. Finds any NuWiki books and creates "cover-page" articles for them. If there are more than 50, imports them in pages of 50 per page.
3. Finds chapters of NuWiki books, and makes them as VaultWiki-compatible (200 per page).
4. Finds NuWiki forum headers, and creates articles in the VaultWiki header format. Removes the old forum header (25 per page). Note: any comments made directly to headers will be lost.
5. Find any articles that were protected, and make the protected status VaultWiki-compatible (200 per page).
6. Use NuWiki's permissions to set (some of) VaultWiki's permissions (200 usergroups per page). So, don't edit any custom permissions until after importing!

[Kaelon]

Thanks, pegasus. At which step does it convert normal "wiki" posts/threads? (That is, non "Article" type wiki posts?) I have around 2000 of those.

Does it, for any reason, scan all of the posts table all at once?

[pegasus]

Normal wiki posts/threads are converted earlier when you set-up your namespaces (if you use the same forums as NuWiki). If you haven't done this for some forums, this occurs when the forum is modified in Step 1. Unless you're using a very old version of MySQL, I don't see how any of the current queries would scan the entire post table.

As I mentioned before, we have not received any performance issue reports from our big board clients since your debacle.

Since you seem hesitant (and rightly so), I recommend waiting for the new build (2.5.6 being released tomorrow), because we have made several small performance improvements over the public release.

[Kaelon]

Normal wiki posts/threads are converted earlier when you set-up your namespaces (if you use the same forums as NuWiki). If you haven't done this for some forums, this occurs when the forum is modified in Step 1. Unless you're using a very old version of MySQL, I don't see how any of the current queries would scan the entire post table.

As I mentioned before, we have not received any performance issue reports from our big board clients since your debacle.

Since you seem hesitant (and rightly so), I recommend waiting for the new build (2.5.6 being released tomorrow), because we have made several small performance improvements over the public release.

Thanks so much for this information, pegasus. A couple more questions:

1. Does VaultWiki continue to support legacy board-wide functionality, like the [template] bbcode? Many of my users rely on this.

2. Do you plan to announce when 2.5.6 has been released?

3. What specific plans do you have for vBulletin 4.x? We are still debating whether or not to participate in the vB4 sale, but wanted to know what you thought, performance wise, about the vBulletin 4 transition.

Thanks again for your thoughts!

[pegasus]

1. VaultWiki supports a template BB-Code. This only works when used inside wiki articles. Please see our doc section on "Introduction to Templates".

2. There will be an announcement in our News forum when 2.5.6 is released. Users with VaultWiki installed will receive a ping in their AdminCP alerting them about the new version.

3. We have quite a bit of new functionality that we are holding out on adding until vBulletin 4. Performance-wise, vBulletin 4 looks to be a big improvement over vBulletin 3, although it's possible only big boards will see a noticeable difference. If you're unfamiliar with the advantages of innoDB, there's a recent blog post about it by one of the devs at vbulletin.com.

By what's been discussed so far, I expect that sites with many concurrent users will see an improvement, and that big boards will be able to use vBulletin's built-in search again (I can only imagine what fulltext search was like on 1 million+ posts).

[Kaelon]

Thanks, pegasus. A few more questions:

1. For our users who use the template BBcode outside of Wiki articles, what will happen to those template tags? NuWiki allowed site-wide template code usage.

2. Re: version 2.5.6, any update on when it will be released? I noticed the recent announcement regarding character sets, and I am excited to see the final version that is released.

3. Let's talk Namespaces. I am sure that I have asked this question before, but many of my users like referring to documents that do not have a namespace (e.g., "Article Name" instead of "NameSpace:Article Name"). Does VaultWiki support the use of no namespace for its default article set? How does it handle Namespace usage, in general? Can you link me to more documentation about this?

4. Also, I am not sure if you know this, but all efforts to purchase updates/renew/etc. on the Members Area is leading to an Internal Error 500 notice on your web server. Does this mean you are not presently selling VaultWiki licenses?

Thanks!

[pegasus]

1. VaultWiki's template BB-Code is not compatible with non-wiki posts. The difference is in the way templates are parsed and stored. In NuWiki, this is done using regular expressions and the post can't be cached. In VaultWiki, a separate parser is used altogether than the BB-Code Parser, and the post can be cached. At this point, it should be relatively simple to have a modification that allows templates in non-wiki areas using a similar method to NuWiki, but by calling the correct parser. Otherwise, non-wiki template usages will resolve as links to the template.

2. Version 2.5.6 was released on Saturday. The announcement you read was the announcement about its release.

3. The default namespace will not be included in your article URLs (you will have 'Article Name' instead of 'NameSpace:Article Name'). Right now the documentation on Namespaces and namespace management is discussed here: Introduction to Namespaces.

4. Thanks for reporting this.

[Kaelon]

Thanks, pegasus. Do you also offer support for all legacy NuWiki bbcodes? Specifically the [h3] in addition to VaultWiki's native [h=3]? Do you have a detailed page for NuWiki users thinking about transitioning over?

[pegasus]

There is NOT support for all legacy codes. However, there is support for the NuWiki H2,H3,etc tags. The most detailed page we currently have is VaultWiki Importer.

Other codes that are unsupported:

[showthreads]
[forumindex]
[bookindex]
[span]
[div]

Of course, since NuWiki is no longer in existence I can't check this against their site to see if these are the correct spellings or if there's anything else.

[Alfa1]

You can check this now. NuHit is back online.

Are there any big boards using Vaultwiki? Or sites with 5.000+ concurrent users?

[InformationNirvana]

You can check this now. NuHit is back online.

I see you are moderating the forums.
But all the products are dead still
Nuhit's Nuwiki seems quite primative to vaultwiki.
I don't know why anyone would want to go with nuwiki, even if it was vB4 compatible.

[pegasus]

Are there any big boards using Vaultwiki? Or sites with 5.000+ concurrent users?

I believe I've mentioned this elsewhere, but yes I seem to remember a number of big boards that have installed VaultWiki. Of course I have to leave it up to them to chime in with respect to how it performs on their particular setups.