Why is the first build of a new release always bug ridden?

[Alfa1]

We know that every release of VW has a large number of bug fixes and improvements that are of much benefit. But a new release almost always has a significant number of bugs to discover and address. A large portion of those are not unique to a specific environment, but encountered by most of us.

Could you please consider new methods to prevent and identify such issues before release?

Is there not a method to stress test software on a number of environments and automatically carry out a set of actions on it? Then collect the bug reports and address those.

And also run a beta live here for a few days and ask all customers by email to test the beta before release. This would require the XenForo demo to be much more prominent accessible.

[pegasus]

The next version is always live here and on the XF demo, often weeks to a month in advance of the release. The upgrade process is tested at least 5 times on various web sites and the install process 3 times before it is released. It is not released while the tests are unsuccessful.

Some upgrade steps are designed to correct damage from bugs that might only exist on certain users' installations. It is often difficult to test those steps exhaustively. The behavior of damaged entries is always undefined.

Most bugs that are encountered in new releases are not new bugs in that specific release. A small number of them are, but even those "new" ones were often "hiding" behind other issues that were fixed in a release.

It is also worth mentioning that we have a rather extensive list of known bugs that we have identified internally over time, but if we always tried to fix all of them in one release, it could be several months between releases (and we already consider the current release schedule to be "too slow"). We try to identify the bugs that are the most problematic and/or would affect the largest number of users, and to address those issues first; in addition, we try not to release at all if the total (public/published) bug count that is listed on our web site is 10 or higher.

Especially with respect to the last paragraph, this means that in a new release, if you look hard enough, you will definitely find a bug.

However, I'm not sure what you're referring to exactly. You seem to be implying numerous issues with 4.0.17, but it has been out for a day, and you have not posted any issues that you encountered on your installation. Rather, you have only posted issues with our demo site, and those issues are not new in this release. Looking at your reports from the past several months, most of the issues you have encountered are either E_NOTICEs (which are usually semantic issues and don't cause problems), small style/phrase tweaks, or import issues. Among these, almost none of the bugs were new in the version you reported them in. Regarding imports, they are not a common occurrence, and errors are often dependent on what data appears in what combinations in a source database, small changes to the config file, and even small changes to forum settings in the target database.

In past releases, there was usually a bug with build 001 related to our downloads system, where our download system could not figure out what file versions it was supposed to put in the ZIP. It usually went away at build 002. There have been numerous factors that contributed to this over the years, but with the release of 4.0.17, we noticed it happening because the timestamps for build 001 would go out of sync whenever a build was finally approved by QA (a process which by the way, does often take a week or more). In previous versions, this sometimes meant that the build 001 that a customer downloaded was not the same build 001 that QA had approved; this usually caused a large number of "erroneous" bug reports during the first 1-2 days of release -- issues such as:

- bugs claimed to be fixed by the release do not appear to be fixed
- fatal errors due to mixed file versions
- database errors after a fresh installation due to mixed file versions in the installer

This was not a bug in VaultWiki itself, but with the part of our web site that zipped-up the files for you. It was zipping up the wrong files.

A lot of bugs are not related to VaultWiki at all, but the reports still come when we release new versions, because users are more likely to install/upgrade a version close to its release date. These reports include: conflicts with other add-ons and server configuration problems (such as in Cloudflare or in php.ini).

[Alfa1]

The next version may always be live here and on the XF demo, but no one knows that. The XF demo is tucked away in a way that its hard to find. This will not result in any beta testing by customers.

If you have an extensive list of bugs for VW4, then are you working towards fixing them all?
IIRC VW4 first version was in 2012 and now we are in 2017. Often when I talk to webmasters in need of a wiki, they mention the bugs as a reason not to buy VW.

Surely when I make VW4 available to my 4 million users, then this will result in a large number of issues. I would really like to prevent that scenario. I hope you can understand where I am coming from.

I have not reported any 4.0.17 bugs, because I have not installed it. I generally wait for build 2 like many of us do. This is why I have taken a look at the XF demo and reported bugs from there.

[pegasus]

Yes, we are working towards fixing them all. That is part of the reason why the changelog for each version is significantly longer than if you just compared it to the number of entries marked fixed on our Support tab.

However, the bugs on our internal list are unlikely to affect most users. Such as

- vbulletin mobile style needs sidebar templates b/c of WIDGET

Others are regressions that no one has reported yet:

- regression from VW3: ability reverse comments order

Others are so minor, I doubt anyone would notice them, but they are bugs nonetheless:

- parentprefixid is not a proper asort key - maybe stick prefixkey at end of sortkey?

If someone ever uses bugs as a reason not to buy VaultWiki, I always try to find out what bug(s) they are referring to. Often it's a bug that was fixed two years ago or more, usually within a few days of when it was reported. There is nothing we can do about it if that is their reason. We fix bugs we find, and we fix bugs our customers report to us. If a customer reports a bug, we often fix it in a manner of hours.

[Alfa1]

I know that you are extremely fast with bug fixing. Either it gets fixed in hours or you are sleeping. Absolutely no complaints there.

In VW3 the experience of big board admins was that there always were bugs encountered and to some it felt like an endless beta loop. They ask if its still the same and I have to answer with yes. The bug tracker speaks for itself.
I hope that we will get to a point where a new release will lead to a few bugs and that is all.

[pegasus]

The bug tracker typically has 10-20 real bugs that customers have reported within a single cycle, with most customers encountering 0 of them. Some issues are duplicates or very minor issues like style tweaks. I cannot agree with the assessment that the bug tracker implies an endless beta loop.

By comparison, XenForo's tracker contains about half the number of bugs as we do within a single cycle. They have had the benefit of 2 more years than us to resolve bugs, as well as a larger customer base to report those bugs.

The only things that are in endless beta from our point of view are importers, which, unfortunately, have been your main experience using VaultWiki 4 so far.

This is not to say that there are no bugs or that occasionally we do not learn about any serious ones. But as a whole, it is far from Beta at this point in time.

====

That said, I think it might be reasonable to add a "beta" flag next to features that are considered to be problematic (such as the importer) in their respective menus to clarify which parts of the wiki are considered to be in a constant beta state, or features which are otherwise new. That way, admins might be warned ahead of time that newer options have less of a track record. We could remove "beta" flags for individual features / options that are not in a constant state of flux after a set number of versions.

Additionally, I think it might be reasonable to add a "beta" flag next to new releases for the first 5-7 days of their release. This flag would be removed automatically after that time passes or if build 002 is posted, whichever occurs first. In this way, customers can choose to avoid new versions during the early-adopter phase, if they wish, without having to fear new versions in general.

====

Since the release of 4.0.17, of the 14 bugs reported (as of the time of this edit): most are related to importers or duplicates of other bugs. In a handful of them, the user is not even using 4.0.17, but some earlier release. 1 of the bugs reported was new in 4.0.17, a fix for a related bug in every other version has caused a conflict for some users. 1 of the bugs is somewhat new in 4.0.17; the error message that represents a bug from 4.0.16 has changed.

[hollosch]

Where can i find the XF demo ?

[pegasus]

Scroll to the top of this page and click the "Try XenForo Demo" in sub-links of the navbar.

[Alfa1]

Can you please put it in the main navigation?

[pegasus]

Another thing I have noticed that the XenForo developers do that gives a sense of "fewer bugs", is that there are different forums for Open Bug Reports vs Resolved Bug Reports. We have them all lumped together in one spot, which I suppose can make it look scarier. If you consider total bug reports as a metric for actual number of bugs of a product, total bug reports at XenForo has been about 4x our total bug reports; adjust for time, more like 3x.

I suppose if we moved closed bug reports to a different spot every time we resolved one, it would seem like there are fewer bugs here too, since the open reports section would almost always have only 5-15 bugs listed.

Regarding putting it in the main navigation, I need to figure out a way to do this that would save horizontal space. Perhaps make the tab work as a drop-down "Wiki Demos" menu that has "vBulletin Wiki" and "XenForo Wiki" as choices there. What do you think of this approach?

[Alfa1]

You could just put this in the main navigation:
XF Wiki
Change 'Wiki' to 'vB Wiki'

This would work well for us as it gives direct access and it does not need much space.

[Alfa1]

I see that I have so far reported 216 bugs for Vaultwiki version 4. (not counting the 111 bugs I reported for the VW4 beta) This is double the amount I reported for VW3 which I was running live with 4 million monthly readers. I am not yet running VW4 live.

A substantial share of those VW4 bugs were discovered in the first build of a new release. So my personal experience is that a new release may need some significant testing or a wait for the second build.
All of these bugs were swiftly addressed by you. I just wish that we will see a significant reduction of the number of issues encountered per release.

[ACL]

Regarding putting it in the main navigation, I need to figure out a way to do this that would save horizontal space. Perhaps make the tab work as a drop-down "Wiki Demos" menu that has "vBulletin Wiki" and "XenForo Wiki" as choices there. What do you think of this approach?

Since the header logo contains both the vbulletin and xenforo logos, could implementing an image map with <map> act as a bit of a shortcut to the two wiki versions? Xenforo logo link to XF VaultWiki, vBulletin logo link to vB VaultWiki, remaining image area link to VaultWiki homepage. Consider this as an idea to use in addition to a text link.