Suddently get high loads on server with 500+ tasks running

  • basketmen
  • Student
  • Student
  • basketmen
  • Posts: 80

Post 3+ Months Ago

Guys i am desperate here, already get this problem for a week sometime suddenly get high load, and still don't find where is the problem yet

normally the top result like this, load only 0.71, only have 200+ tasks
screenshots : ... ormal3.jpg

other screenshots : ... ormal2.jpg

but sometime suddenly get high load (usually 1-3 times a day), load very high 90-200, with 500+ tasks
screenshots : ... hload1.jpg

Daily Process Log, i already repair and optimize all table, like you can see the mysql process is very low (in white line, not in yellow or red line), so its looks like the problem is not in mysql
The red line is maybe apache
screenshots : ... essLog.jpg

Nb1. I use a dedicated server, only host 1 site, here is my server specs
Intel Single Xeon E3110 3.0ghz L2 cache 6MB
Memory 4 GB DDR2 RAM
Harddisk 500 GB Sata 7200rpm
Cent OS 5.4

Nb2. I already check no backup/cronjob or related scheduled active right now

Nb3. I already disabled the feature one by one, Even i already block all search engine spider now using robots.txt disallow for 2 days, no spiders again in my website until the problem found, its ok i just need to found the source problem

Nb4. I don't think my hosting support can help it, and they cost very expensive for that extra managed service ($69/hour), i want to find the source problem by my self, please analyze and give advice to found the source problem
  • Bigwebmaster
  • Site Admin
  • Site Admin
  • User avatar
  • Posts: 9128
  • Loc: Seattle, WA

Post 3+ Months Ago

One thing I noticed from this screenshot: ... hload1.jpg

is your 96.2%wa. Here is an explanation of the parameters in case you don't know:

us = User CPU time (the time the CPU has spent running user's processes that are not niced)
sy = System CPU time (the time the CPU has spetn running the kernel and its processes
ni = Nice CPU time (the time the CPU has spent running user's processes that have been niced)
wa = iowait ie Waiting for I/O (the amount of time the CPU has been waiting for the I/O to complete)
hi = Hardware IRQ (the amount of time the CPU has been servicing hardware interrupts)
si = Software IRQ (the amount of time the CPU has been servicing software interrupts)
id = Idle CPU

So a high wa to me most likely means you are swapping. To confirm this I also see that your RAM memory usage is maxed out and your machine is wanting to use more memory so since it runs out it instead starts using your hard drives to create virtual memory. That is extremely slow and could cause your load to spike up like it is.

So where I would look is to why your memory is being maxed out. There are some things you can run to help pinpoint it. One thing I always check during times like this is to see what is connecting to your machine. Run:

netstat -npa

Another thing you can do is to make sure you have configured MySQL correctly. Many admins mistakenly configure it completely wrong and due to that they end up allocating way more memory to it then they need to and it eventually eats up all of their RAM and then guess what, you start swapping and load spikes through the roof. I would strongly recommend you run this program called MySQLTuner. Run that program and see if you have configured MySQL correctly. If you see everything as OK then that is good, but if you see any !! showing up then that means you need to resolve that. My first guess would be that for:

Maximum possible memory usage

You might be above your installed RAM. Generally I would make sure that stays under about 50% of your installed RAM. Let me know what MySQLTuner is telling you and if you figure anything out :)
  • basketmen
  • Student
  • Student
  • basketmen
  • Posts: 80

Post 3+ Months Ago

what do you think if i add Memory 4 GB DDR2 RAM again? so the total will be 8GB, do you think its enough so not get high load again?
  • SpooF
  • ٩๏̯͡๏۶
  • Bronze Member
  • User avatar
  • Posts: 3423
  • Loc: Richland, WA

Post 3+ Months Ago

I would suggest you find out whats using up all your memeory. It could be something simple as a mysql configuration. No point in spending the money on the ram if it can be fixed with a few simple edits.

I had the same issue a while ago. Follow what Bigwebmaster said, I did and it fixed my problem. My mysql server thought I had 232 gigs of ram installed.

This is the output from MySQLTuner

-------- General Statistics --------------------------------------------------
[--] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 5.0.88-log
[OK] Operating on 64-bit architecture

-------- Storage Engine Statistics -------------------------------------------
[--] Status: -Archive +BDB -Federated +InnoDB -ISAM -NDBCluster
[--] Data in MyISAM tables: 404M (Tables: 61)
[--] Data in InnoDB tables: 575M (Tables: 48)
[!!] BDB is enabled but isn't being used
[!!] Total fragmented tables: 19

-------- Performance Metrics -------------------------------------------------
[--] Up for: 1d 13h 13m 48s (52M q [393.677 qps], 17M conn, TX: 5B, RX: 3B)
[--] Reads / Writes: 3% / 97%
[--] Total buffers: 638.0M global + 300.4M per thread (800 max threads)
[!!] Maximum possible memory usage: 235.3G (23524% of installed RAM)
[OK] Slow queries: 0% (7K/52M)
[OK] Highest usage of available connections: 25% (205/800)
[OK] Key buffer size / total MyISAM indexes: 512.0M/476.2M
[OK] Key buffer hit rate: 99.9% (160M cached / 206K reads)
[OK] Query cache efficiency: 93.4% (5M cached / 5M selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 0% (1K temp sorts / 236K sorts)
[!!] Temporary tables created on disk: 35% (108K on disk / 304K total)
[OK] Thread cache hit rate: 99% (280 created / 17M connections)
[OK] Table cache hit rate: 90% (593 open / 653 opened)
[OK] Open file limit used: 5% (376/6K)
[OK] Table locks acquired immediately: 95% (10M immediate / 11M locks)
[!!] InnoDB data size / buffer pool: 575.6M/8.0M

Notice the bold? lol
  • Bigwebmaster
  • Site Admin
  • Site Admin
  • User avatar
  • Posts: 9128
  • Loc: Seattle, WA

Post 3+ Months Ago

I think the proper solution here is to simply figure out what is using up your RAM like SpooF said. Please run MySQLTuner and see what it says. Very likely you just have your server configured incorrectly and all of this can be fixed. If you have any similar situation to what SpooF had with the way the MySQL configuration was setup, 8 GB wouldn't even come close to fixing your problem as you would have needed over 235 GB of RAM.

We still do not know for sure that MySQL is your problem, but it does seem to be a very common problem on servers where the admin is not exactly sure what they are doing when they are trying to tweak with my.cnf. We are happy to help guide you in the right direction though :)

Post Information

  • Total Posts in this topic: 5 posts
  • Users browsing this forum: No registered users and 8 guests
  • You cannot post new topics in this forum
  • You cannot reply to topics in this forum
  • You cannot edit your posts in this forum
  • You cannot delete your posts in this forum
  • You cannot post attachments in this forum

© 1998-2017. Ozzu® is a registered trademark of Unmelted, LLC.