I'm looking for a high level view of how to get a program to tail a file continuously, not at intervals, and actually act on the data it has...whiles till continuously monitoring the file.

I'm running into a mental block trying to figure out how to keep the program monitoring, which a while(1=1) loop would certainly accomplish, but how do you act on the data you've gathered, while still monitoring the file...

If you exit the loop to act on the data, how do you still keep tabs on where you are in the file, it seems to me these need to be done in parallel...what am I missing?

If you need a more concise or clear answer please let me know, I'm having a hard time getting my mind wrapped around this. Thoughts?

Thanks,

-Rob

asked 10 Jun '10, 20:55

rfelsburg's gravatar image

rfelsburg ♦
6061618
accept rate: 25%

edited 20 Jul '10, 19:15




Make a filter and use tail. You could use the filter like so:

$ tail -f some.log | monitor

Where monitor is your filter. In the monitor app, simply use fgets() to read from STDIN. fgets() reads from a handle and returns a complete line (ending in a new line). If there's no full ine yet, it simply waits. So, you call fgets(), wait for a line, act on it, then jump back to gfets() again.

This follows the UNIX philosophy. Do one thing and do it well. Use tail to follow the file. Use your monitor just to act on the lines.

link

answered 10 Jun '10, 22:28

Sander%20Marechal's gravatar image

Sander Marechal
398116
accept rate: 29%

I don't suppose you have a simple example of using fgets()? For example, I'm tailing a log file and any time the user "aws" appears, I want to write that to separate file. thanks!

(11 Jun '10, 15:39) Andy

No, I don't have an example ready. I did easily find several examples in Google that show how to use fgets and how to read data piped to an application (creating a filter), but no example that combines them both.

(20 Jun '10, 07:59) Sander Marechal

Not exactly what I was looking for but still a good answer.

I was looking for a little higher level not specifics as this isn't going to be in bash. I implemented in perl, and used file::tail.

Thanks though.

-Rob

(13 Jul '10, 18:57) rfelsburg ♦

Well, I wasn't recommending just bash. I was recommending splitting the tailing and the monitoring and acting on data. That way you could also use your monitoring app easily on text files (cat test.txt | monitor), on the output of other programs (grep something | sed some-sed-magic | monitor), or even e-mail, etcetera.

(15 Jul '10, 09:19) Sander Marechal

I understand, I was thinking more self contained loop, that could run as a daemon, not a crontab or something of that nature. I think if it were a simple one off I probably would have split the tail and monitoring itself into separate scripts. Thanks

(21 Jul '10, 20:46) rfelsburg ♦

there's "inotail" util can be used instead of "tail" if your kernel has inotify enabled(most default kernels from modern distros do). you can use it with the way Sander Marechal shown, or you can code your own app using inotify functions on C or other lang providing inotify API(say, python).

link

answered 11 Jun '10, 12:38

Web31337's gravatar image

Web31337
317111
accept rate: 11%

You might also find the command "tee" useful. It not only splits an output stream, but can write to any named pipe rather than just to a file. That would make real-time monitoring easier since you can parse the pipe by character if you wish.

See man(1) tee or http://en.wikipedia.org/wiki/Tee_%28command%29

Good Luck.

link

answered 27 Jul '10, 01:47

DBA's gravatar image

DBA
2264
accept rate: 23%

One thing to be aware of is that -- if you're using fgets, you should call setlinebuf to set line-buffering so that you don't get stuck waiting for time-critical data to clear the buffers.

link

answered 06 Mar '13, 01:16

darkonc's gravatar image

darkonc
1
accept rate: 0%

Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "Title")
  • image?![alt text](/path/img.jpg "Title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Tags:

×90
×16
×9
×7

Asked: 10 Jun '10, 20:55

Seen: 3,127 times

Last updated: 10 Apr '13, 23:00

powered by OSQA