Suddently get high loads on server with 500+ tasks running

One thing I noticed from this screenshot:

http://i291.photobucket.com/albums/ll28 ... hload1.jpg

is your 96.2%wa. Here is an explanation of the parameters in case you don't know:

us = User CPU time (the time the CPU has spent running user's processes that are not niced)
sy = System CPU time (the time the CPU has spetn running the kernel and its processes
ni = Nice CPU time (the time the CPU has spent running user's processes that have been niced)
wa = iowait ie Waiting for I/O (the amount of time the CPU has been waiting for the I/O to complete)
hi = Hardware IRQ (the amount of time the CPU has been servicing hardware interrupts)
si = Software IRQ (the amount of time the CPU has been servicing software interrupts)
id = Idle CPU

So a high wa to me most likely means you are swapping. To confirm this I also see that your RAM memory usage is maxed out and your machine is wanting to use more memory so since it runs out it instead starts using your hard drives to create virtual memory. That is extremely slow and could cause your load to spike up like it is.

So where I would look is to why your memory is being maxed out. There are some things you can run to help pinpoint it. One thing I always check during times like this is to see what is connecting to your machine. Run:

netstat -npa

Another thing you can do is to make sure you have configured MySQL correctly. Many admins mistakenly configure it completely wrong and due to that they end up allocating way more memory to it then they need to and it eventually eats up all of their RAM and then guess what, you start swapping and load spikes through the roof. I would strongly recommend you run this program called MySQLTuner. Run that program and see if you have configured MySQL correctly. If you see everything as OK then that is good, but if you see any !! showing up then that means you need to resolve that. My first guess would be that for:

Maximum possible memory usage

You might be above your installed RAM. Generally I would make sure that stays under about 50% of your installed RAM. Let me know what MySQLTuner is telling you and if you figure anything out

I think the proper solution here is to simply figure out what is using up your RAM like SpooF said. Please run MySQLTuner and see what it says. Very likely you just have your server configured incorrectly and all of this can be fixed. If you have any similar situation to what SpooF had with the way the MySQL configuration was setup, 8 GB wouldn't even come close to fixing your problem as you would have needed over 235 GB of RAM.

We still do not know for sure that MySQL is your problem, but it does seem to be a very common problem on servers where the admin is not exactly sure what they are doing when they are trying to tweak with my.cnf. We are happy to help guide you in the right direction though