httpd has failed, please contact the sysadmin.

October 14th, 2009, 12:02 pm

So my server is crashing again, for no reason:

Code: [ Download ] [ Select ]

Main >> Restart Services >> HTTP Server (Apache)

Restarting Apache Web Server
Waiting for httpd to restart..............finished.


httpd has failed, please contact the sysadmin.

2. Main >> Restart Services >> HTTP Server (Apache)
4. Restarting Apache Web Server
5. Waiting for httpd to restart..............finished.
8. httpd has failed, please contact the sysadmin.

I tried to contact the admin, but I realized I am the admin.

Now what?

I have Cpanel, CENTOS, LAMP...what do I do now? I am a programmer, not an admin person.

October 14th, 2009, 12:03 pm

Oh, email and SSH still work...

October 14th, 2009, 12:49 pm

/usr/sbin/httpd status

Looking up localhost
Making HTTP connection to localhost
Alert!: Unable to connect to remote host.

lynx: Can't access startfile http://localhost/server-status
root@srv1 [/var/log]# /usr/sbin/httpd fullstatus

Looking up localhost
Making HTTP connection to localhost
Alert!: Unable to connect to remote host.

lynx: Can't access startfile http://localhost/server-status

[ATNO/TW]

October 14th, 2009, 1:15 pm

Does this help any?
hosting-forum/httpd-has-failed-please-contact-the-sysadmin-t100563.html

or this?
http://www.linuxforums.org/forum/server ... 6-2-a.html

or better this?
http://www.howtoforge.com/forums/showthread.php?p=74169

[Bigwebmaster]

October 14th, 2009, 10:09 pm

Do me a favor and show me what the error log says after you try to start it.

tail -n 100 /usr/local/apache/logs/error_log

Or whatever your location is.

Also in your logs directory for apache (may be the one above) try listing the contents:

ls -la /usr/local/apache/logs/

See if any files are really really big. Sometimes if you have a log file over a certain size (usually 2GB) and if you are on a 32 bit server it can cause problems with Apache operating correctly.

October 14th, 2009, 11:02 pm

Hmmm, now DNS also not working.

Code: [ Download ] [ Select ]

named has failed, please contact the sysadmin (result was "named is not running").
Oct 15 02:00:04 srv1 named[2447]: shutting down: flushing changes
Oct 15 02:00:04 srv1 named[2447]: stopping command channel on 127.0.0.1#953
Oct 15 02:00:04 srv1 named[2447]: no longer listening on 127.0.0.1#53
Oct 15 02:00:04 srv1 named[2447]: no longer listening on 204.15.192.194#53
Oct 15 02:00:04 srv1 named[2447]: no longer listening on 204.15.192.195#53
Oct 15 02:00:04 srv1 named[2447]: no longer listening on 204.15.192.196#53
Oct 15 02:00:04 srv1 named[2447]: no longer listening on 204.15.192.197#53
Oct 15 02:00:04 srv1 named[2447]: no longer listening on 204.15.192.198#53
Oct 15 02:00:04 srv1 named: succeeded
Oct 15 02:00:04 srv1 named[2447]: exiting
Oct 15 02:00:04 srv1 named: failed

2. named has failed, please contact the sysadmin (result was "named is not running").
3. Oct 15 02:00:04 srv1 named[2447]: shutting down: flushing changes
4. Oct 15 02:00:04 srv1 named[2447]: stopping command channel on 127.0.0.1#953
5. Oct 15 02:00:04 srv1 named[2447]: no longer listening on 127.0.0.1#53
6. Oct 15 02:00:04 srv1 named[2447]: no longer listening on 204.15.192.194#53
7. Oct 15 02:00:04 srv1 named[2447]: no longer listening on 204.15.192.195#53
8. Oct 15 02:00:04 srv1 named[2447]: no longer listening on 204.15.192.196#53
9. Oct 15 02:00:04 srv1 named[2447]: no longer listening on 204.15.192.197#53
10. Oct 15 02:00:04 srv1 named[2447]: no longer listening on 204.15.192.198#53
11. Oct 15 02:00:04 srv1 named: succeeded
12. Oct 15 02:00:04 srv1 named[2447]: exiting
13. Oct 15 02:00:04 srv1 named: failed

I will get back to you on what the tail -n thing says. I really need to learn to be a better admin, without being a worse programmer

October 14th, 2009, 11:05 pm

Code: [ Download ] [ Select ]

[srv1.ryd0.com] nameserver failed @ Thu Oct 15 02:04:46 2009. A restart was attempted automagically.
Service Check Method: [check command]

Cmd Service Check Raw Output: named is not running

2. [srv1.ryd0.com] nameserver failed @ Thu Oct 15 02:04:46 2009. A restart was attempted automagically.
3. Service Check Method: [check command]
5. Cmd Service Check Raw Output: named is not running

October 14th, 2009, 11:19 pm

Ok bigwebmaster,

1) When you need some programming/PHP/C++, just ask I owe you one...
2) Here are some log data:
3) What next? I really don't even know where to start..

Bigwebmaster wrote:

Do me a favor and show me what the error log says after you try to start it.

tail -n 100 /usr/local/apache/logs/error_log

Or whatever your location is.

Also in your logs directory for apache (may be the one above) try listing the contents:

ls -la /usr/local/apache/logs/

See if any files are really really big. Sometimes if you have a log file over a certain size (usually 2GB) and if you are on a 32 bit server it can cause problems with Apache operating correctly.

The error log at tail -n 100 /usr/local/apache/logs/error_log says file not exist. Which is almost normal, since they files don't exist, they are created on the fly (using .htaccess)

Code: [ Download ] [ Select ]

[Wed Oct 14 11:53:04 2009] [error] [client 88.131.106.3] File does not exist: /home/girlswit/public_html/AzerbaijanStrippers/Ashley-Allen.html
[Wed Oct 14 11:53:06 2009] [error] [client 66.249.71.83] File does not exist: /home/photof/public_html/Turquoise-Jewelry-Santa-Fe/
[Wed Oct 14 11:53:06 2009] [error] [client 88.131.106.3] File does not exist: /home/girlswit/public_html/RebeccaStrippers/Avril-Lavigne-Dancer.html
[Wed Oct 14 11:53:07 2009] [error] [client 69.109.113.147] File does not exist: /home/photof/public_html/galleries/Hurt_love_when.jpg
[Wed Oct 14 11:53:08 2009] [error] [client 66.249.71.187] File does not exist: /home/wallpape/public_html/1474010.html
[Wed Oct 14 11:53:08 2009] [error] [client 41.212.167.225] File does not exist: /home/photof/public_html/CruiseShipRecruitment.php
[Wed Oct 14 11:53:08 2009] [error] [client 88.131.106.3] File does not exist: /home/girlswit/public_html/DancerVanessaMontagne/Dancer-Kate-Hudson.html
[Wed Oct 14 11:53:09 2009] [error] [client 206.53.157.47] File does not exist: /home/neverbuy/public_html/favicon.ico
[Wed Oct 14 11:53:10 2009] [error] [client 88.131.106.3] File does not exist: /home/girlswit/public_html/AliciaSilverstone/Barbara-Butt.html
[Wed Oct 14 11:53:12 2009] [error] [client 88.131.106.3] File does not exist: /home/girlswit/public_html/CameronDiazDancer/Anna-Malle.html
[Wed Oct 14 11:53:13 2009] [error] [client 85.10.71.89] File does not exist: /home/photof/public_html/TaniaAmazon.htm

1. [Wed Oct 14 11:53:04 2009] [error] [client 88.131.106.3] File does not exist: /home/girlswit/public_html/AzerbaijanStrippers/Ashley-Allen.html
2. [Wed Oct 14 11:53:06 2009] [error] [client 66.249.71.83] File does not exist: /home/photof/public_html/Turquoise-Jewelry-Santa-Fe/
3. [Wed Oct 14 11:53:06 2009] [error] [client 88.131.106.3] File does not exist: /home/girlswit/public_html/RebeccaStrippers/Avril-Lavigne-Dancer.html
4. [Wed Oct 14 11:53:07 2009] [error] [client 69.109.113.147] File does not exist: /home/photof/public_html/galleries/Hurt_love_when.jpg
5. [Wed Oct 14 11:53:08 2009] [error] [client 66.249.71.187] File does not exist: /home/wallpape/public_html/1474010.html
6. [Wed Oct 14 11:53:08 2009] [error] [client 41.212.167.225] File does not exist: /home/photof/public_html/CruiseShipRecruitment.php
7. [Wed Oct 14 11:53:08 2009] [error] [client 88.131.106.3] File does not exist: /home/girlswit/public_html/DancerVanessaMontagne/Dancer-Kate-Hudson.html
8. [Wed Oct 14 11:53:09 2009] [error] [client 206.53.157.47] File does not exist: /home/neverbuy/public_html/favicon.ico
9. [Wed Oct 14 11:53:10 2009] [error] [client 88.131.106.3] File does not exist: /home/girlswit/public_html/AliciaSilverstone/Barbara-Butt.html
10. [Wed Oct 14 11:53:12 2009] [error] [client 88.131.106.3] File does not exist: /home/girlswit/public_html/CameronDiazDancer/Anna-Malle.html
11. [Wed Oct 14 11:53:13 2009] [error] [client 85.10.71.89] File does not exist: /home/photof/public_html/TaniaAmazon.htm

The second command is like so:

ls -la /usr/local/apache/logs/

Code: [ Download ] [ Select ]

total 2230212
drwxr-xr-x  2 root  root     4096 Oct 15 01:09 ./
drwxr-xr-x 14 root  root     4096 Sep 25 2007 ../
-rw-r--r--  1 root  root  133205988 Oct 15 01:10 access_log
-rw-r--r--  1 root  root  2147483647 Oct 14 11:53 error_log
-rw-r--r--  1 root  root      5 Oct 15 01:09 httpd.pid
-rw-r--r--  1 root  root    767131 Oct 15 01:09 ssl_engine_log
-rw-------  1 nobody root      0 Oct 3 2007 ssl_mutex.10070
-rw-------  1 nobody root      0 Sep 13 2006 ssl_mutex.10283
-rw-------  1 nobody root      0 Oct 14 22:11 ssl_mutex.10396
-rw-------  1 nobody root      0 Oct 4 2006 ssl_mutex.10468
-rw-------  1 nobody root      0 Jan 9 2007 ssl_mutex.10544
-rw-------  1 nobody root      0 Oct 14 12:37 ssl_mutex.11777
-rw-------  1 nobody root      0 Oct 4 2006 ssl_mutex.12476
-rw-------  1 nobody root      0 Apr 15 2007 ssl_mutex.12796
-rw-------  1 nobody root      0 Oct 14 18:55 ssl_mutex.13899
-rw-------  1 nobody root      0 Oct 4 2006 ssl_mutex.14246
-rw-------  1 nobody root      0 Feb 19 2006 ssl_mutex.14843
-rw-------  1 nobody root      0 Jan 11 2006 ssl_mutex.15186
-rw-------  1 nobody root      0 Oct 14 11:47 ssl_mutex.15667
-rw-------  1 nobody root      0 Feb 13 2008 ssl_mutex.1588
-rw-------  1 nobody root      0 Oct 14 23:04 ssl_mutex.16094
-rw-------  1 nobody root      0 May 25 2006 ssl_mutex.16477
-rw-------  1 nobody root      0 May 25 2006 ssl_mutex.16572
-rw-------  1 nobody root      0 Mar 3 2006 ssl_mutex.2076
-rw-------  1 nobody root      0 Jun 6 2006 ssl_mutex.2085
-rw-------  1 nobody root      0 Jan 15 2006 ssl_mutex.21560
-rw-------  1 nobody root      0 Feb 19 2007 ssl_mutex.2203
-rw-------  1 nobody root      0 Oct 15 01:09 ssl_mutex.2207
-rw-------  1 nobody root      0 Apr 1 2006 ssl_mutex.2230
-rw-------  1 nobody root      0 Mar 6 2006 ssl_mutex.2237
-rw-------  1 nobody root      0 Mar 5 2006 ssl_mutex.2239
-rw-------  1 nobody root      0 Aug 29 2006 ssl_mutex.2263
-rw-------  1 nobody root      0 Oct 14 13:48 ssl_mutex.23520
-rw-------  1 nobody root      0 Apr 15 2007 ssl_mutex.2358
-rw-------  1 nobody root      0 Jun 6 2007 ssl_mutex.2359
-rw-------  1 nobody root      0 Apr 20 2007 ssl_mutex.2360
-rw-------  1 nobody root      0 Aug 8 04:08 ssl_mutex.23746
-rw-------  1 nobody root      0 Jul 17 2007 ssl_mutex.2420
-rw-------  1 nobody root      0 Oct 14 20:06 ssl_mutex.25179
-rw-------  1 nobody root      0 Oct 14 14:52 ssl_mutex.2542
-rw-------  1 nobody root      0 Oct 14 14:53 ssl_mutex.2591
-rw-------  1 nobody root      0 Dec 2 2008 ssl_mutex.2595
-rw-------  1 nobody root      0 Aug 16 04:14 ssl_mutex.2609
-rw-------  1 nobody root      0 Aug 19 18:57 ssl_mutex.2621
-rw-------  1 nobody root      0 Jan 9 2007 ssl_mutex.28518
-rw-------  1 nobody root      0 Oct 15 00:16 ssl_mutex.29016
-rw-------  1 nobody root      0 Oct 14 14:42 ssl_mutex.29590
-rw-------  1 nobody root      0 Jan 9 2007 ssl_mutex.29781
-rw-------  1 nobody root      0 Sep 13 2006 ssl_mutex.300
-rw-------  1 nobody root      0 Oct 14 14:55 ssl_mutex.3031
-rw-------  1 nobody root      0 Oct 14 20:59 ssl_mutex.31106
-rw-------  1 nobody root      0 Oct 6 2006 ssl_mutex.32228
-rw-------  1 nobody root      0 Oct 14 15:00 ssl_mutex.3865
-rw-------  1 nobody root      0 May 25 2006 ssl_mutex.6208
-rw-------  1 nobody root      0 Oct 6 2006 ssl_mutex.689
-rw-------  1 nobody root      0 Oct 14 16:43 ssl_mutex.7400
-rw-------  1 nobody root      0 Oct 14 18:00 ssl_mutex.8059
-rw-------  1 nobody root      0 Jan 9 2007 ssl_mutex.8236
-rw-------  1 nobody root      0 Oct 14 15:52 ssl_mutex.9051
-rw-------  1 nobody root      0 Oct 14 15:53 ssl_mutex.9091
-rw-------  1 nobody root      0 Oct 4 2006 ssl_mutex.9870
-rw-------  1 nobody root      0 Oct 15 01:09 ssl_scache.dir
-rw-------  1 nobody root      0 Oct 15 01:09 ssl_scache.pag
-rw-r--r--  1 root  nobody   20207 Sep 7 08:34 suexec_log
-rw-r--r--  1 root  root      6 Sep 8 04:08 suexec_log.offset

1. total 2230212
2. drwxr-xr-x  2 root  root     4096 Oct 15 01:09 ./
3. drwxr-xr-x 14 root  root     4096 Sep 25 2007 ../
4. -rw-r--r--  1 root  root  133205988 Oct 15 01:10 access_log
5. -rw-r--r--  1 root  root  2147483647 Oct 14 11:53 error_log
6. -rw-r--r--  1 root  root      5 Oct 15 01:09 httpd.pid
7. -rw-r--r--  1 root  root    767131 Oct 15 01:09 ssl_engine_log
8. -rw-------  1 nobody root      0 Oct 3 2007 ssl_mutex.10070
9. -rw-------  1 nobody root      0 Sep 13 2006 ssl_mutex.10283
10. -rw-------  1 nobody root      0 Oct 14 22:11 ssl_mutex.10396
11. -rw-------  1 nobody root      0 Oct 4 2006 ssl_mutex.10468
12. -rw-------  1 nobody root      0 Jan 9 2007 ssl_mutex.10544
13. -rw-------  1 nobody root      0 Oct 14 12:37 ssl_mutex.11777
14. -rw-------  1 nobody root      0 Oct 4 2006 ssl_mutex.12476
15. -rw-------  1 nobody root      0 Apr 15 2007 ssl_mutex.12796
16. -rw-------  1 nobody root      0 Oct 14 18:55 ssl_mutex.13899
17. -rw-------  1 nobody root      0 Oct 4 2006 ssl_mutex.14246
18. -rw-------  1 nobody root      0 Feb 19 2006 ssl_mutex.14843
19. -rw-------  1 nobody root      0 Jan 11 2006 ssl_mutex.15186
20. -rw-------  1 nobody root      0 Oct 14 11:47 ssl_mutex.15667
21. -rw-------  1 nobody root      0 Feb 13 2008 ssl_mutex.1588
22. -rw-------  1 nobody root      0 Oct 14 23:04 ssl_mutex.16094
23. -rw-------  1 nobody root      0 May 25 2006 ssl_mutex.16477
24. -rw-------  1 nobody root      0 May 25 2006 ssl_mutex.16572
25. -rw-------  1 nobody root      0 Mar 3 2006 ssl_mutex.2076
26. -rw-------  1 nobody root      0 Jun 6 2006 ssl_mutex.2085
27. -rw-------  1 nobody root      0 Jan 15 2006 ssl_mutex.21560
28. -rw-------  1 nobody root      0 Feb 19 2007 ssl_mutex.2203
29. -rw-------  1 nobody root      0 Oct 15 01:09 ssl_mutex.2207
30. -rw-------  1 nobody root      0 Apr 1 2006 ssl_mutex.2230
31. -rw-------  1 nobody root      0 Mar 6 2006 ssl_mutex.2237
32. -rw-------  1 nobody root      0 Mar 5 2006 ssl_mutex.2239
33. -rw-------  1 nobody root      0 Aug 29 2006 ssl_mutex.2263
34. -rw-------  1 nobody root      0 Oct 14 13:48 ssl_mutex.23520
35. -rw-------  1 nobody root      0 Apr 15 2007 ssl_mutex.2358
36. -rw-------  1 nobody root      0 Jun 6 2007 ssl_mutex.2359
37. -rw-------  1 nobody root      0 Apr 20 2007 ssl_mutex.2360
38. -rw-------  1 nobody root      0 Aug 8 04:08 ssl_mutex.23746
39. -rw-------  1 nobody root      0 Jul 17 2007 ssl_mutex.2420
40. -rw-------  1 nobody root      0 Oct 14 20:06 ssl_mutex.25179
41. -rw-------  1 nobody root      0 Oct 14 14:52 ssl_mutex.2542
42. -rw-------  1 nobody root      0 Oct 14 14:53 ssl_mutex.2591
43. -rw-------  1 nobody root      0 Dec 2 2008 ssl_mutex.2595
44. -rw-------  1 nobody root      0 Aug 16 04:14 ssl_mutex.2609
45. -rw-------  1 nobody root      0 Aug 19 18:57 ssl_mutex.2621
46. -rw-------  1 nobody root      0 Jan 9 2007 ssl_mutex.28518
47. -rw-------  1 nobody root      0 Oct 15 00:16 ssl_mutex.29016
48. -rw-------  1 nobody root      0 Oct 14 14:42 ssl_mutex.29590
49. -rw-------  1 nobody root      0 Jan 9 2007 ssl_mutex.29781
50. -rw-------  1 nobody root      0 Sep 13 2006 ssl_mutex.300
51. -rw-------  1 nobody root      0 Oct 14 14:55 ssl_mutex.3031
52. -rw-------  1 nobody root      0 Oct 14 20:59 ssl_mutex.31106
53. -rw-------  1 nobody root      0 Oct 6 2006 ssl_mutex.32228
54. -rw-------  1 nobody root      0 Oct 14 15:00 ssl_mutex.3865
55. -rw-------  1 nobody root      0 May 25 2006 ssl_mutex.6208
56. -rw-------  1 nobody root      0 Oct 6 2006 ssl_mutex.689
57. -rw-------  1 nobody root      0 Oct 14 16:43 ssl_mutex.7400
58. -rw-------  1 nobody root      0 Oct 14 18:00 ssl_mutex.8059
59. -rw-------  1 nobody root      0 Jan 9 2007 ssl_mutex.8236
60. -rw-------  1 nobody root      0 Oct 14 15:52 ssl_mutex.9051
61. -rw-------  1 nobody root      0 Oct 14 15:53 ssl_mutex.9091
62. -rw-------  1 nobody root      0 Oct 4 2006 ssl_mutex.9870
63. -rw-------  1 nobody root      0 Oct 15 01:09 ssl_scache.dir
64. -rw-------  1 nobody root      0 Oct 15 01:09 ssl_scache.pag
65. -rw-r--r--  1 root  nobody   20207 Sep 7 08:34 suexec_log
66. -rw-r--r--  1 root  root      6 Sep 8 04:08 suexec_log.offset

October 14th, 2009, 11:25 pm

More status stuff as I try to find out how to fix:

Code: [ Download ] [ Select ]

root@srv1 [~]# /usr/sbin/httpd status

Looking up localhost
Making HTTP connection to localhost
Sending HTTP request.
HTTP request sent; waiting for response.
Alert!: Unexpected network read error; connection aborted.
Can't Access `http://localhost/server-status'
Alert!: Unable to access document.

lynx: Can't access startfile

1. root@srv1 [~]# /usr/sbin/httpd status
3. Looking up localhost
4. Making HTTP connection to localhost
5. Sending HTTP request.
6. HTTP request sent; waiting for response.
7. Alert!: Unexpected network read error; connection aborted.
8. Can't Access `http://localhost/server-status'
9. Alert!: Unable to access document.
11. lynx: Can't access startfile

[joebert]

October 14th, 2009, 11:29 pm

Quote:

-rw-r--r-- 1 root root 2147483647 Oct 14 11:53 error_log

That looks like one of those 2GB log files he was talking about. I don't know if that's it, but based on what I've read here I would make sure Apache is stopped, move that file to another location, then start apache.

October 14th, 2009, 11:59 pm

joebert wrote:

Quote:

-rw-r--r-- 1 root root 2147483647 Oct 14 11:53 error_log

That looks like one of those 2GB log files he was talking about. I don't know if that's it, but based on what I've read here I would make sure Apache is stopped, move that file to another location, then start apache.

HOLY CRAP THAT WAS IT!!!!!

So what do you need programmed? A gallery creator? Need something debugged? How about a stock market prediction program? I'll write that for you!

Next thing is how to I prevent it from becoming a 2gig file again?

[joebert]

October 15th, 2009, 2:41 am

Is there a time during the day when traffic is slow enough that you can squeeze a graceful Apache restart in ?
http://httpd.apache.org/docs/2.2/logs.html#rotation

Under the section I linked to, there's a section about "piped logs", which are basically a way to direct log entries to an application instead of a file. That application takes care of rotation/etc and Apache never needs to be restarted.

Apache apparently comes with such an application, rotatelogs.

I've never had to mess with any of that so I can't be of much help. My home servers are running Ubuntu and that's always taken care of log rotation for me.

[Bigwebmaster]

October 15th, 2009, 1:24 pm

Glad you were able to figure it out and fix it!

camperjohn wrote:

Next thing is how to I prevent it from becoming a 2gig file again?

Since you are using cPanel, its relatively easy to do. In your WHM goto the section on the left menu labeled:

Service Configuration

Under that area you should see:

Apache Configuration

Click that link and you will see a bunch of options in that area such as Global Configuration, PHP and SuExec Configuration, etc. At the very bottom is one called:

Log Rotation

Click on that and check all the boxes there for access_log, error_log, and the others. Then click save. Now whenever any of those files get larger than 300 MB they will automatically be compressed and stored in the archive folder in your log directory. This should prevent this problem from ever happening again since they will never get to that crucial 2GB point.

camperjohn wrote:

So what do you need programmed? A gallery creator? Need something debugged? How about a stock market prediction program? I'll write that for you!

Thanks! Just participating on ozzu and helping others with problems that you know the answers to from time to time is all that I need

October 16th, 2009, 8:37 am

So you don't need a stock market prediction program?

Guess I will just have to write that one myself then!

October 16th, 2009, 8:38 am

Weird, when I clicked post advanced reply, it posted, then posted again...