Logging - Text File to Syslog

  • humbletech99
  • Proficient
  • Proficient
  • User avatar
  • Posts: 300

Post 3+ Months Ago

I have some databases which log to a flat text file and I don't think that this version can log to syslog, so I'd like to stream the log file into syslog myself. I figure that keeping something like this running should work:
Code: [ Select ]
tail -f /path/to/logfile | logger

or
Code: [ Select ]
logger -f /path/to/logfile


My question is whether or not there is a better way of doing this?

I think that if this process is restarted, and the logfile is always quite large, that it will read at least some of it again and duplicate that information into syslog which I'd like to avoid.

Is there a better way, some program perhaps that will remember it's position in the logfile and if crashed/restarted will pick up again from that exact point without missing anything or reinserting the same log information again, possibly also handling log truncation or rotation? Does logger -f have an intelligence on this?
  • Melvinchi
  • Novice
  • Novice
  • Melvinchi
  • Posts: 19
  • Loc: Rush City, MN

Post 3+ Months Ago

The way I would do that would be have the service log the file by a strict date format like 010808.log or something like that.
After that I would use cron to run a perl or php script to read the file and put it in a database. Then you could just view the log from a database. And have it all in one place :)
With this you could even allow you to view this information from the web or from SSH or Telnet.

or could sit there and cat the file and look at it that way :D
  • humbletech99
  • Proficient
  • Proficient
  • User avatar
  • Posts: 300

Post 3+ Months Ago

the program is Oracle 9i so it can't log to syslog according to my oracle trainer, nor can it log to different log files in this way without shutting down a 24x7 database for each name change, so again a no-no.

logger -f seems to read in a whole file in one go and not tail it, so that doesn't work.

tail -f could work but any time it's restarted it could miss log information or put in the same information twice.


I really do need an intelligent logging program. Do I really have to write one myself?

Surely somebody in the world has written this already...
  • this213
  • Guru
  • Guru
  • User avatar
  • Posts: 1260
  • Loc: ./

Post 3+ Months Ago

Do you mean something like this?
Code: [ Select ]
#!/bin/sh

MSG_LOG=/var/log/messages
ORACLE_SID=your_sid
FACILITY=local0
PRIORITY=notice

tail -f ${MSG_LOG} | egrep -v \
'^[A-Z][a-z]{2} [A-Z][a-z]{2} [ 1-3][1-9] [0-9]{2}:[0-9]{2}:[0-9]{2} [0-9]{4}$'\
| logger -t oracle/${ORACLE_SID} -p ${FACILITY}.${PRIORITY}
  1. #!/bin/sh
  2. MSG_LOG=/var/log/messages
  3. ORACLE_SID=your_sid
  4. FACILITY=local0
  5. PRIORITY=notice
  6. tail -f ${MSG_LOG} | egrep -v \
  7. '^[A-Z][a-z]{2} [A-Z][a-z]{2} [ 1-3][1-9] [0-9]{2}:[0-9]{2}:[0-9]{2} [0-9]{4}$'\
  8. | logger -t oracle/${ORACLE_SID} -p ${FACILITY}.${PRIORITY}

I can't take credit for this myself, found it online *quite* some time ago
  • humbletech99
  • Proficient
  • Proficient
  • User avatar
  • Posts: 300

Post 3+ Months Ago

this is no different than the original tail -f logfile | logger ....

you've just surrounded it with vars...

The point is, if it restarts, it does not remember where it left off. You will either miss things or will insert the same messages in to syslog.

It is not restart-safe is the point.
  • Daemonguy
  • Moderator
  • Web Master
  • User avatar
  • Posts: 2700
  • Loc: Somewhere outside the box in Sarasota, FL.

Post 3+ Months Ago

Out of curiosity, what version of what database are you using?

Have you considered letting it log to a db table? Then you can syslog to the same db using something like sysslog-ng?

The second question is just off the cuff.

Perhaps once we know more about the database itself, the answer will come more quickly.
  • this213
  • Guru
  • Guru
  • User avatar
  • Posts: 1260
  • Loc: ./

Post 3+ Months Ago

humbletech99 wrote:
this is no different than the original tail -f logfile | logger ....

you've just surrounded it with vars...

The point is, if it restarts, it does not remember where it left off. You will either miss things or will insert the same messages in to syslog.

It is not restart-safe is the point.

That's why I asked if that was what you were trying to do. If your concern is getting repeat records, just write a perl script to parse it - you could do something as pedestrian as saving the last record in a text file and making the script start from that point.

Realistically, if I were to put this much thought into logging something, it would certainly be going into a database somewhere where I could have centralized management, compressed storage and some form of lookup facility other than grep and tail - but that's just me.
  • humbletech99
  • Proficient
  • Proficient
  • User avatar
  • Posts: 300

Post 3+ Months Ago

above I mentioned that the program (ie Database) is Oracle 9i.

I want it to log to syslog so that I can control it from there, it's no use to me in a table.

Also, I cannot change the logging on the database as this would require a restart of a 24x7 database breaking all the long running computation it does and would severely disrupt the business...

So... I'm stuck with a text file and my need to intelligently following that in real time piping it to | logger...

Post Information

  • Total Posts in this topic: 8 posts
  • Users browsing this forum: No registered users and 30 guests
  • You cannot post new topics in this forum
  • You cannot reply to topics in this forum
  • You cannot edit your posts in this forum
  • You cannot delete your posts in this forum
  • You cannot post attachments in this forum
 
 

© 1998-2014. Ozzu® is a registered trademark of Unmelted, LLC.