Apache Error Conditional logs

  • camperjohn
  • Guru
  • Guru
  • User avatar
  • Posts: 1127
  • Loc: San Diego

Post 3+ Months Ago

Does anyone have experience with the conditional logs of apache?

I have LOTS of 404 errors, and dont want to log those:
[Sat Apr 17 21:15:25 2010] [error] [client 195.42.102.25] File does not exist: /home/photof/public_html/cats/random.htlm

Is there a way I can get apache to log all errors except those?
http://httpd.apache.org/docs/2.2/logs.html

What environment variable do you think I could use to regex against to acheive this goal?
  • Don2007
  • Web Master
  • Web Master
  • Don2007
  • Posts: 4924
  • Loc: NY

Post 3+ Months Ago

The site you posted said that there is no way to customize the logs but 2 sites I found says it can be done. You decide.

http://www.logfilter.org/

http://www.sudleyplace.com/pipederrorlogs.html
  • joebert
  • Fart Bubbles
  • Genius
  • User avatar
  • Posts: 13504
  • Loc: Florida

Post 3+ Months Ago

I looked through Apache manual pages for SetInvIf, Conditional Logging, Environment Variables. I looked through CGI specification pages. I even looked at the Environment (as well as the rest of the) output from a phpinfo() call. I can't seem to find any environment variable that holds the HTTP status code.

It's starting to look like there are 3 options here. The first two being pretty much things Don pointed out.

1) Log the 404 lines anyways and filter them later. Though I suspect this defeats part of the purpose of you wanting to skip logging these. I seem to remember you posting something about large log files crashing your server awhile back.

2) Use piped logs, which basically means you have to keep a process running for the purpose of filtering out unwanted lines.

3) Dig through the Apache source and see where the HTTP status gets set. Maybe there will be a clue about some undocumented variable. If there's nothing available, maybe it would be a trivial task to add something and recompile.
  • camperjohn
  • Guru
  • Guru
  • User avatar
  • Posts: 1127
  • Loc: San Diego

Post 3+ Months Ago

Here is what I found as suggested on another forum:

ErrorLog "|/bin/grep -v \'File does not exist:\' |/bin/grep -v \'not found or unable to stat\' >> /var/log/httpd/error_log"

Works great. Pipe into grep, and -v if it isn't found.

Now, requests go into access_log only, and not into both access_log and error_log
  • joebert
  • Fart Bubbles
  • Genius
  • User avatar
  • Posts: 13504
  • Loc: Florida

Post 3+ Months Ago

Any noticeable performance hit ?
  • camperjohn
  • Guru
  • Guru
  • User avatar
  • Posts: 1127
  • Loc: San Diego

Post 3+ Months Ago

It isn't really a performance hit I am looking for, I was trying to avoid having the logs track 404 errors and duplicating it in the access_logs. After only 10 days since deleting the access_logs,it's up to 242 megs in size. That means 2.4 gigs in 100 days? I just don't want that duplicated in error_logs.

Most of my traffic is 404 generated, even though I deliver a valid page, it is initiated from a 404 call.

eg: http://feedginger.com/ozzu-is-cool.html
  • joebert
  • Fart Bubbles
  • Genius
  • User avatar
  • Posts: 13504
  • Loc: Florida

Post 3+ Months Ago

I had a feeling your motive had something to do with log file size.

I'm guessing there's no noticeable change to performance by piping the logs through grep if you're still doing it a day later here.

I wonder how it works exactly. Does it start a new process for grep with every page request ?
  • camperjohn
  • Guru
  • Guru
  • User avatar
  • Posts: 1127
  • Loc: San Diego

Post 3+ Months Ago

I think it does start a new grep. If it was causing problems I could just make a cron job to filter the error_log each day.

Google is currently slurping content from my server at a made pace, and it seems to be keeping up just fine. New Dual Xeon Quad Core has an uptime of 0.0 0.0 0.0 and it not even blinking at 20,000 visitors a day.
  • joebert
  • Fart Bubbles
  • Genius
  • User avatar
  • Posts: 13504
  • Loc: Florida

Post 3+ Months Ago

I'm assuming you meant load instead of uptime.

One of these days I'll have to get the rest of the computers I have at home to beat on the old 2.2 Ghz Celeron server next to me with and without such piped logs using Apache Bench and see if I can spot a difference in the number of requests-per-second.
  • camperjohn
  • Guru
  • Guru
  • User avatar
  • Posts: 1127
  • Loc: San Diego

Post 3+ Months Ago

joebert wrote:
I'm assuming you meant load instead of uptime.


I don't know the difference. I typed load in shell and it isn't a command I have.

My old server would send me emails every 3 months telling me it's getting to 80% harddrive space, and it was mostly LOG files that were in the gigs and gigs. This is why I wanted piped logs to avoid logging things I already know.
  • joebert
  • Fart Bubbles
  • Genius
  • User avatar
  • Posts: 13504
  • Loc: Florida

Post 3+ Months Ago

Quote:
I don't know the difference. I typed load in shell and it isn't a command I have.


Those three "0.0" numbers you get when you use the uptime command are the "load" or "load average". :)
  • ideyatech
  • Born
  • Born
  • ideyatech
  • Posts: 2

Post 3+ Months Ago

I don't know much about condition logs. But, if you are running an old version of the server, I would recommend getting the newer version of it.

Post Information

  • Total Posts in this topic: 12 posts
  • Users browsing this forum: No registered users and 17 guests
  • You cannot post new topics in this forum
  • You cannot reply to topics in this forum
  • You cannot edit your posts in this forum
  • You cannot delete your posts in this forum
  • You cannot post attachments in this forum
 
 

© 1998-2014. Ozzu® is a registered trademark of Unmelted, LLC.