Server overloading, what do you make of these logs?

  • camperjohn
  • Guru
  • Guru
  • User avatar
  • Posts: 1127
  • Loc: San Diego

Post 3+ Months Ago

My server is STILL overloading after a month of trying to figure it out. Sometime, for no reason, uptime will start to report:

0.12 0.08 0.09
5.12 0.12 0.08
30.2 5.12 0.12
90.0 30.2 5.12
150.1 150.2 149.0
(server is now toast)

In a very short time, it's as if it gets "stuck". Once it's stuck, I shutdown apache, restart it, then everthing runs fine for 13 more hours, then it happens again.

I did however manage to "catch it in the act" get a report of what its doing. I was unable to do this before, because once it's stuck, even getting a report times out too!

lynx --dump localhost/whm-server-status

So, tell me what you make of these apache status reports:

http://www.photofight.com/xx.txt (started to get stuck)
http://www.photofight.com/xy.txt (a little more stuck)
http://www.photofight.com/xz.txt (now it's really stuck)

Notice the jump halfway down, where the milliseconds for a response becomes 1921913212 ?!?!? Search for 9068 and you will see q2growth.com GET /Printing-Supplies/Printing-Supplies.html HTTP/1.1


Shutdown apache and restarted:
http://www.photofight.com/xq.txt (yay its not stuck)


If it was simply too much traffic, then restarting apache would cause it to overload immediatly. So it can't be just a matter of too much traffic. There is something fishy and I can't figure it out. I can't work on ANY more websites until I figure this out too, since I don't have any money coming in while its 'stuck' and it is top priority.

I love the word stuck. Its the best way to describe it.

Ideas? Notice how requests build up, but dont get out! They just stack on top of each other!
  • Anonymous
  • Bot
  • No Avatar
  • Posts: ?
  • Loc: Ozzuland
  • Status: Online

Post 3+ Months Ago

  • Don2007
  • Web Master
  • Web Master
  • Don2007
  • Posts: 4924
  • Loc: NY

Post 3+ Months Ago

The only thing I can suggest it to run Wireshark on the interface. Try it with different filters.
  • camperjohn
  • Guru
  • Guru
  • User avatar
  • Posts: 1127
  • Loc: San Diego

Post 3+ Months Ago

Wikipedia says thats the same as tcpdump just with a graphical interface. Should I try that?

I have it installed and it does show info, but I don't really understand it yet. I will investigate...
  • Don2007
  • Web Master
  • Web Master
  • Don2007
  • Posts: 4924
  • Loc: NY

Post 3+ Months Ago

I haven't used TCP Dump. Try both of them. It can't hurt.
  • joebert
  • Fart Bubbles
  • Genius
  • User avatar
  • Posts: 13503
  • Loc: Florida

Post 3+ Months Ago

I'm not sure what I'm looking at, but the first thing that comes to mind for me is that you have a bunch of requests that are just sitting there and never actually finishing the response, almost like you have something starting requests and only retrieving a single byte at a time as to purposely take as long as possible to complete a request, like some sort of denial of service attack.

It might be the requests for things like Gay dildo masturbation and other provocative terms that has my conspiracy meter going though.
  • SpooF
  • ٩๏̯͡๏۶
  • Bronze Member
  • User avatar
  • Posts: 3422
  • Loc: Richland, WA

Post 3+ Months Ago

You might setup a cron job to run top every 15 minutes and log the out put of the top 15 processes

Code: [ Select ]
*/15 * * * * top -bn1 | head -n 15 >> top.log
  • Don2007
  • Web Master
  • Web Master
  • Don2007
  • Posts: 4924
  • Loc: NY

Post 3+ Months Ago

I thought that dildo thing was one of the hosted sites on his server. Besides, why would the DoS happen every 13 hours?
  • camperjohn
  • Guru
  • Guru
  • User avatar
  • Posts: 1127
  • Loc: San Diego

Post 3+ Months Ago

Don2007 wrote:
I thought that dildo thing was one of the hosted sites on his server. Besides, why would the DoS happen every 13 hours?


The gay and adult keywords are part of my server. I have two websites that have lots of pages and redirect to an adult pay program.

I only have about 7-8,000 visitors a day to the server. It's almost like the requests get stuck and don't complete...

I am at my witts end here...
  • camperjohn
  • Guru
  • Guru
  • User avatar
  • Posts: 1127
  • Loc: San Diego

Post 3+ Months Ago

Here are the lines that I don't understand:

Code: [ Select ]
83-50 9066 0/0/8 W 0.13 108 0 0.0 0.00 0.04 87.250.230.33 photofight.com GET /robots.txt HTTP/1.1
84-50 9067 0/0/2 W 0.00 108 0 0.0 0.00 0.00 87.250.230.33 photofight.com GET /Hp_support.htm HTTP/1.1
85-50 9068 0/0/0 W 0.00 107 1921913212 0.0 0.00 0.00 87.250.230.33 q2growth.com GET /Printing-Supplies/Printing-Supplies.html HTTP/1.1
86-50 9069 0/0/0 W 0.00 104 1921910080 0.0 0.00 0.00 88.131.106.31 girlswithoutgoals.com GET /SeikoPartsMale/Asian-Herpes-Simplex-Virus-Symptom.html
  1. 83-50 9066 0/0/8 W 0.13 108 0 0.0 0.00 0.04 87.250.230.33 photofight.com GET /robots.txt HTTP/1.1
  2. 84-50 9067 0/0/2 W 0.00 108 0 0.0 0.00 0.00 87.250.230.33 photofight.com GET /Hp_support.htm HTTP/1.1
  3. 85-50 9068 0/0/0 W 0.00 107 1921913212 0.0 0.00 0.00 87.250.230.33 q2growth.com GET /Printing-Supplies/Printing-Supplies.html HTTP/1.1
  4. 86-50 9069 0/0/0 W 0.00 104 1921910080 0.0 0.00 0.00 88.131.106.31 girlswithoutgoals.com GET /SeikoPartsMale/Asian-Herpes-Simplex-Virus-Symptom.html


Request #83 and #84 are fine. Normal requests. Then at request #85, the duration/timestamp shows 1921913212 milliseconds (REQ, Milliseconds required to process most recent request
), which translates to 1921913 seconds, or 22 days.

How can Apache say that there is 22 days Seconds since beginning of most recent request?

There are also 0/0/0 connections during this request, then there are 0 Kilobytes transferred this connection, 0 Megabytes transferred this child and 0 Total megabytes transferred this slot.

Srv Child Server number - generation
PID OS process ID
Acc Number of accesses this connection / this child / this slot
M Mode of operation
CPU CPU usage, number of seconds
SS Seconds since beginning of most recent request
Req Milliseconds required to process most recent request
Conn Kilobytes transferred this connection
Child Megabytes transferred this child
Slot Total megabytes transferred this slot
  • Don2007
  • Web Master
  • Web Master
  • Don2007
  • Posts: 4924
  • Loc: NY

Post 3+ Months Ago

83 is a normal request when it comes to time but looking for robots.txt is on the shady side. Secondly, all of those requests originated in Russia. You can contact the network owners to ask them about the entries in the log files.

role: Yandex LLC Network Operations
address: Yandex LLC
address: 1 bld. 21 Samokatnaya St.
address: 111033
address: Moscow
address: Russian Federation
phone: +7 495 739 7000
fax-no: +7 495 739 7070
remarks: trouble: ------------------------------------------------------
remarks: trouble: Points of contact for Yandex LLC Network Operations
remarks: trouble: ------------------------------------------------------
remarks: trouble: Routing and peering issues: noc@yandex.net
remarks: trouble: SPAM issues: abuse@yandex.ru
remarks: trouble: Network security issues: abuse@yandex.ru
remarks: trouble: Mail issues: postmaster@yandex.ru
remarks: trouble: General information: info@yandex.ru

Post Information

  • Total Posts in this topic: 10 posts
  • Users browsing this forum: No registered users and 88 guests
  • You cannot post new topics in this forum
  • You cannot reply to topics in this forum
  • You cannot edit your posts in this forum
  • You cannot delete your posts in this forum
  • You cannot post attachments in this forum
 
 

© 1998-2014. Ozzu® is a registered trademark of Unmelted, LLC.