Issue: Download-Error

Not much we can do about this. Our database server is usually undergoing maintenance at 3:12. Please try again at 3:13.

Hi, it's not 3:13 and error since monday:

As it stands, 99 attempts out of 100, I am able to successfully download the ZIP from your account, even under the below conditions. So my recommendation if you ever see this error would be to wait one minute and try again.

I notice that it is theoretically possible for this to occur, although I can only make it happen if I delete all copies of all download packages from the server, forcing it to regenerate when requested (even so, regeneration normally takes 3-5 seconds). It appears to be a bug in PHP where it randomly throttles itself for a few seconds on certain commands especially when repeated in a loop. Ten iterations of the same command can take 0.001s, but then the next iteration of the same exact code will take .1s. If it throttles too many commands in a row like this or worse, the script can time out, which is essentially what you are seeing.

I'm not sure if there is a way to resolve the situation fully. The bug has existed in PHP since at least 2002 and they have never offered a solution or a workaround. In the mean time, I will just see if there is a way to make regeneration more efficient.

It is strange that you would encounter this situation at all, let alone so frequently. It might be possible to cause a race condition during regeneration if you double-click the download button before it grays out. Have you been having problems with your mouse double-clicking unexpectedly? I recently replaced my mouse for doing that.

Ok, i'll try it again ;-)

thx

I have made some changes which should eliminate the race condition mentioned above. I have also made some changes which should improve the overall speed of the download script by:

• changing over 10k in_array calls to isset
• changing a number of ifs to switches
• rewriting the connection to the backend server to test whether the port is already open and skip the connection code in that case which was always taking 1-4 seconds

I noticed that a regeneration of a missing download package dropped from >15 seconds to ~7 seconds, so I am stopping here for now.

In the future, we should be able to make more improvements, such as using the same file hashes for everyone, but something like that could only affect future versions.

So after further investigation, I decided to check the slow query log which was over 8GB because it had failed to rotate for a few months. After deleting the old log file and triggering this problem on purpose, I noticed that right before this error occurs, the slow log records a query that takes > 8 seconds to complete. So actually, when the download works really fast, it's because the results of that query are already stored in query cache.

The query takes over 8 seconds because it basically must scan over 50K rows in order to get the locations of files to put in the download package. (The locations are not known because the server only stores files that were changed since the last version.)

The solution to this was to change the database map for the file locations. The table was replicating the file system behavior of only recording files that were changed for each version. This was over 50K rows. To make the query faster we had to increase the size of the location-map by fully populating it for each version. The result is a table with > 1.5M rows, but the query is much faster now (< 0.5 sec).

Marking this as fixed.

actually follwing error occurs:

string(78) "/home/vw/arsenal/vaultwiki/4/0/19/1/upload/vault/resources/js/append.js.min_js" string(71) "/home/vw/arsenal/vaultwiki/4/0/19/1/upload/vault/resources/js/append.js"

Looks like some Javascript files weren't reading from the minify-cache. Fixed. Please let me know if you have any further issues.

I've been watching this in the logs and some queries were still running slow (although nowhere near as slow as the query discussed earlier). I've added some dummy fields to a few tables today which should hopefully eliminate the file-sorting that has been going on.