Server problem, maybe related with VaultWiki

[Alan_SP]

I'm really not sure is this have any connection with VaultWiki, but I'll ask here, just in case.


My server started to have problems, this is description I asked my server support, but it's very limited, I'm on unmanaged server:
---------
Yesterday around 17:00 hours (European, Berlin time) I noticed that server is overloaded.

Before that I didn't installed anything on server or on my site (vBulletin forum) for days. Also, I didn't noticed anything strange in any log, at least as I can read it.

I started monitoring it and this is what happens:

One Apache webserver process for some reason goes to 100% processor load and completely chokes one processor core. This happens about every 7-8 minutes to one webserver process, choking one core. As I have 4 cores, it takes about half an hour to choke almost completely processor.

I tried to reboot server, it helped, but not for long as it continued to happen every 7-8 minutes to one core.

Then I tried to restart webserver process (httpd), this also helped temporarily as every 7-8 minutes one process would go to 100% and choke one core.

Something not logged in error log hangs one httpd process which in turn blocks one core. And apache couldn't recover out of it.

I also tried to reduce MaxRequestsPerChild hoping that it would solve problems by killing sooner httpd processes. Didn't help anything.

Now I tried to employ Monit (program to monitor different processes on server) to automatically restart Apache, but it didn't worked out as I hoped. It restarted Apache only once.

As I said, I didn't worked on server, site, touched anything. I don't know if it is some kind of attack from outside that is able to crash Apache process, or there is something changed in hardware, but please help me. As time goes server is less and less responsive.

------------------
This is why I'm contacting you:

I received this response from server support at the end:
----------

OK, as a final resort, I have checked the processes via strace. The trouble process seems to be doing some rather strange things, it seems to primarily stay in the /var/www/vhosts/slobodni.net/httpdocs/vault/ directory. The process appears to be doing the same thing over and over again:

"/var/www/vhosts/slobodni.net/httpdocs/vault/functions/bbcode/handle.php"
"/var/www/vhosts/slobodni.net/httpdocs/vault/class/bbcode/autolink.php"
"/var/www/vhosts/slobodni.net/httpdocs/includes/functions_login.php"
"/var/www/vhosts/slobodni.net/httpdocs/vault/class/bbcode/autolink.php"
"/var/www/vhosts/slobodni.net/httpdocs/vault/functions/bbcode.php"
"/var/www/vhosts/slobodni.net/httpdocs/includes/functions_login.php"
"/var/www/vhosts/slobodni.net/httpdocs/vault/functions/bbcode/handle.php"
"/var/www/vhosts/slobodni.net/httpdocs/vault/class/bbcode/autolink.php"
"/var/www/vhosts/slobodni.net/httpdocs/vault/functions/bbcode.php"
"/var/www/vhosts/slobodni.net/httpdocs/includes/functions_login.php"
"/var/www/vhosts/slobodni.net/httpdocs/vault/functions/bbcode/handle.php"

This is about the only thing this process does, whereas the other processes, seem to be people.

Unfortunately, I don't believe there is much else we can help you with here. These are UNmanged servers, server administration is your responsibility.

---------------

Now I see you posted new version of VaultWiki. Is it possible that there's a check every 7-8 minutes or so that calls your site and somehow it messes with my Apache server process driving them to 100% and choking my server? Or something like that? Connected somehow with you?

I'm just asking, I'm tried to turn off VaultWiki, didn't helped, but not sure what really could be wrong, so I thought to ask...

And, just to ask you, if you could help me, you maybe know how to program script to automatically kill runaway process with cron. I found out this site with similar script: http://www.squidoo.com/cron-kill


*/10 * * * * username ps -o sz,etime,pid,comm,args -U username | grep scriptname.php | grep -v grep | perl -ane '($m,$s) = split /:/,$F[1]; $kb=$F[0]; $pid = $F[2]; if(($m>3)&&($kb>10000)) {print "$m :: $s $kb $pid\n"; kill 9, $pid;}'

Of course, my script should check for process overload, using something like:

ps -o sz,pid,comm,args,%cpu -U apache

And if %cpu is over 90, kill that process.

Can you maybe help me with script like this?

[pegasus]

Hi,

I'm afraid I can't be much help with this either. We had a server process that was eating up CPU back in May, but this was due to a faulty SATA connection on the server motherboard that prevented the system from being able to kill the process naturally.

There is not a cron job running every 7-8 minutes at the server level. There is a cron job that runs every so often which sends out subscription emails; this is not necessarily a VaultWiki process but could feasibly call VaultWiki functions. Furthermore, this would not continuously eat up CPU, because it is a PHP process and would die if an error occurred. I doubt it is a cron issue because I see no mentions of the cron folder for the runaway process, as far as you posted it.

The most confusing part for me is the repeating calls to functions_login.php. VaultWiki doesn't call this file or have any plugins that are called by this file. Perhaps you have another mod installed which calls one of the login_ hooks, and which might be erroneously starting VaultWiki.

Also, I know that some users have wanted to extend VaultWiki's auto-link functionality for their own uses. If you have done something like this, it may be using the auto-linker inefficiently, and you may want to review your plugin code. After 3 years, VaultWiki's auto-links on their own are about as efficient as they can be without sacrificing utility or being turned off completely.

[Alan_SP]

Thank you anyway. This is really very strange. Now I logged to my server and it was fine, working as it should. After some time, it started again to make problems. Maybe it's allergic on me?

I mean, it was hours without supervision and worked fine now and I logged in and here it goes again. Totally confusing I must say...

Do you know how to make this cron script I mentioned? For killing runaway processes?

[pegasus]

It sounds like you have a plugin that runs when you login and starts something that forms an infinite loop. If you want, PM me and I'll take a look at your plugins.

[pegasus]

I checked your logs. Every 10 minutes, you have a cron job that posts threads from RSS feeds. It posts what looks like over 100 threads at once, in one PHP process. This process starts every 12 hours (one of those cycles is around 17:00) and recurs every 10 minutes for several hours until it's complete.

Posting this many threads at once is likely to strain your CPU. You should lower the number of threads posted each time, or check each feed in different intervals other than just 12 hours (preferably lower), so it finds fewer new entries each time. Some of your feeds are at lower intervals, but about half are fetched every 12 hours. You also may want to reinvestigate those feeds to make sure its content is relevant, and see if there is another way to filter that content so that you only get the items you really need.

You may also want to upgrade the mod [DBTech] Advanced User Tagging Lite. There have been several new updates; I think one of them fixed a compatibility issue with VaultWiki.

I find it hard to believe that you only have one process for PHP files, and that is the problem process, but the particular file sequence posted would never occur exactly as such. The closest would be is when a multiple-login is detected, a new thread is posted, and that thread needs to be parsed, which is why VaultWiki's BB-Codes are loaded. But I checked the forum, and these threads are making it through fine.

I think the cron job is just locking things up for you. There may be an error occurring which is preventing it from finishing, and it just keeps restarting over and over.

[Alan_SP]

Thank you very much. I just disabled all mods and for about one hour of testing there was no runaway processes. This brings hope, it's a problem with some mod, or two or more mods colliding.

Now I'll test by trying to enable most important mods like core mods (CMS, Blog, skimlinks) and paid ones (VaultWiki and vBSEO). After that, if everything works, I'll try with user tagging lite, this possibly problematic version just to be sure it's what's causing problems.

After that I'll try with new version. Didn't know there was some compatibility issues with this mod and VaultWiki. I had it both for relatively long time without problems.

I think that this testing could take long, but anyway, I must go slow, trying to find what's causing problems. I'll inform you when I found what's causing problems...

[pegasus]

Since it has to do with RSS, I would try testing the mod "RSS items as posts" sooner rather than later. With it disabled you get infinite copies of each thread, because there's no handler for the "post" type now.

[Alan_SP]

I'll test it among first ones. But, I looked in RSS poster, as posting feeds as threads is disabled, it seems to me that there wouldn't be no new RSS posts. It looks like the RSS is deactivated for time being (no option selected, post it like thread and post it like notice aren't selected). But, I'll test it certainly among first mods.

I need to go slowly, giving enough time to mod start misbehaving.

[pegasus]

If no option is selected, or the option isn't recognized, vBulletin posts it as a thread. Regardless if posting as threads are enabled, because it bypasses any checks like that. So I would just test it soon so you don't keep getting 100 threads every 10 minutes that you don't want.

[Alan_SP]

Thank you for information. I just enabled it and setting it again, some information was lost... Will see if it makes impact on server...

[Alan_SP]

I logged in and saw one process that was runaway. There was only one, everything else was fine. I straced it, and you have it in attachment, what was output (only part of it, but this is more or less what's important).

The interesting thing, I couldn't find post that triggered it, it started about 3 minutes before I logged to server.

But, it looks like there's definite connection with DragonByte Tech: Advanced User Tagging (Lite) v2.00.

Also, this process managed to recover by itself. Don't know how, but when I finished straceing it, there was no more this process. Before process like this would go on and on, for hours if not manually killed. I'm confused as I used VaultWiki and DragonByte Tech: Advanced User Tagging (Lite) for quite some time and it worked fine, no problems there. Only few days ago it started to happen (1.7). Very strange I must say.

I'll now install latest version of DragonByte Tech: Advanced User Tagging (Lite) and hope that it will solve problem.

[Alan_SP]

Just to inform you, latest version didn't help. I even tried to disable VaultWiki, there were problems still.

At the moment I think that my servers is unstable and that I need to reinstall it from ground up. But at the moment I'm going to (long) vacation and it will wait. After that I'll again try to solve the problem, first with reinstalling server and then I'll see how everything works.

Minimal configuration (core vBulletin mods), vBSEO and VaultWiki seems stable enough. What's exactly wrong with my server I don't know, but your VaultWiki holds it stable.

[Alan_SP]

I checked your logs. Every 10 minutes, you have a cron job that posts threads from RSS feeds. It posts what looks like over 100 threads at once, in one PHP process. This process starts every 12 hours (one of those cycles is around 17:00) and recurs every 10 minutes for several hours until it's complete.

Posting this many threads at once is likely to strain your CPU. You should lower the number of threads posted each time, or check each feed in different intervals other than just 12 hours (preferably lower), so it finds fewer new entries each time. Some of your feeds are at lower intervals, but about half are fetched every 12 hours. You also may want to reinvestigate those feeds to make sure its content is relevant, and see if there is another way to filter that content so that you only get the items you really need.

Just to inform you, I finally disabled RSS completly, mod for posting in posts too and server is stable for some time. I overlooked this mod and whatever mod I disabled, things happened again.

It seems that there's problem with RSS, you spotted it, but I for some reason overlooked it and thought it works fine.

For now I have it disabled. Later when I return from vacation I'll try to check RSS more frequently and other things, maybe check that cron job, or completely abandon using RSS.

Thank you very much for your help and support.