Hi Roger,

Thx a ton!

It did work!

Thx to Stanley too, for helping me out!

I had already written a auto edition code in C for this, but it was giving me some errors.

although its not that great, but would like to contribute it to the netsaint-users group, when ready and tested. I hope thats allowed.

Thx again to both of you.

Regds,

Amol Sapkal

GTL Ltd.

 netsaint-users-request <at> lists.sourceforge.net wrote:

Send Netsaint-users mailing list submissions to
netsaint-users <at> lists.sourceforge.net

To subscribe or unsubscribe via the World Wide Web, visit
https://lists.sourceforge.net/lists/listinfo/netsaint-users
or, via email, send a message with subject or body 'help' to
netsaint-users-request <at> lists.sourceforge.net

You can reach the person managing the list at
netsaint-users-admin <at> lists.sourceforge.net

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Netsaint-users digest..."


Today's Topics:

1. Re: System time problem with NetSaint (Roger Flemming/SYD/CEtv)
2. nrpe-1.5 on AIX 4.3.3 (Tyson Harker)

--__--__--

Message: 1
To: netsaint-users <at> lists.sourceforge.net
Cc: amol sapkal
Subject: Re: [netsaint] System time problem with NetSaint
From: "Roger Flemmi ng/SYD/CEtv"
Date: Mon, 12 Aug 2002 18:07:27 +1100

This is a multipart message in MIME format.
--=_alternative 0027337FCA256C13_=
Content-Type: text/plain; charset="us-ascii"

Hello Amol,

I am sure many people will answer your question but I haven't seen any
yet, so here goes. Also I hope the list doesn't mind that this post is
basically all Perl, very little Netsaint!

> Dear Sir,
> Thx for the solution!
> But im unable to decipher the command here..
> perl -i.bak -pe 's#^\[(\d+)\]#$1 - Your_Time_Correction(eg 2*3600)#e'
netsaint.log

It's a Perl one-liner script.
-i.bak says edit in place, backing up the original with extension .bak.
-e says the first argument is the actual script to execute, instead of the
script's name. -p says take in a file (the second argument) line by line,
putting each line into the default argument $_, execute the script, and
printout $_. Thus the file (in this case netsaint.log) will have each line
transformed by the contents of that script, with the original saved as
netsaint.log.bak. The script itself use the pattern-match-and-substitute
operator s/// to search for the start of a line (^) immediately followed
by a literal left bracket (\[) followed by one or more digits (\d+) and a
right bracket (\]). This is then replaced by $1 - Your_Time_Correction(eg
2*3600), where $1 is anything matched inside the parenthesis earlier (i.e.
the digits) and the rest was meant to be replaced by you with your actual
correction factor. That is, if the error was 2 hours 3 seconds, you'd
actually put #$1 - 7203#. The 'e' right at the end says the replacement
string is to be evaluated as an expression, i.e. replace the matched
pattern with the digits minus the correction factor.

Note there may be a small error here: while the date gets corrected, the
square brackets will vanish. It might be better to put
perl -i.bak -pe 's#^\[(\d+)\]#"[".($1 - 7200)."]"#e' netsaint.log
which should keep the square brackets

> is it necessary that it has to be in multiples of 3600 (1 hr)

No. The time argument is simply seconds. By 'eg 2*3600' he meant put 7200
if you want 2 hours.

> also pls tell me where this command has to be executed.

If Perl has been set up correctly on your system you should be able to
execute it from anywhere. Note that if you are not in the directory where
netsaint.log lives, you will need to give its full path.

> Also, I was unable to find any relations between the 10 digit time
stamps (eg. 10288....), and the corresponding time value. Please help me
to know the logic behind the generation of these time stamps. I dont find
any common link between the sequence of the time stamps. Is it using some
knind of algorithm?

These time stamps are a standard Unix internal way of representing time,
commonly known as t_time or epoch time. They represent the number of
non-leap seconds past midnight, 1st January 1970, UTC (usually as a signed
32 bit integer) which is roughly how old Unix is. Yes this does mean you
will need to get a 64 bit machine before the 2038 crisis! This date format
has many advantages:
- it is constant through out the world, i.e. time zone independent;
- it is very compact (one 32 bit int for seconds, minutes, hours, days,
months, and years);
- apart from the word size problem in 2038, it runs forever into the past
and future without any funny glitches or historical weirdness;
- it is very easy to do arithmetic in it, while most other formats are
horribly confusing; and
- it is very portable.
On the down side, it is hard for humans to read, but while we're talking
Perl, you can convert epoch dates to human readable format in your own
timezone , with:

perl -e 'print scalar localtime 1029132827'

(Substituting the number in question for the one I entered.) Similarly if
you want the current epoch date just go:

perl -e 'print time'

[...]
Stanley wrote:
> perl -i.bak -pe 's#^\[(\d+)\]#$1 - Your_Time_Correction(eg 2*3600)#e'
netsaint.log
>
> may be something like what you want, provided you know that the log
> entries are __all__ are out by the same constant amount.[...]

Or if there is a time before/after which they are bad by a constant
amount, you could try something like:

perl -i.bak -pe 's#^\[(\d+)\]#$1<1028132827 ? $1 - 7200 : $1#e'
netsaint.log

Substituting the epoch date where it all went wrong for 1028132827 and the
actual error for 7200. May still be some hand editing at the cross-over
point if the bad dates went forward in time, though.


Cheers,
Roger

********************************** *****************************************************************************************************************************************
This email and any files transmitted with it, are confidential and is intended solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error, please notify the system manager.

This footnote also confirms that this email message has been scanned by AUSTAR Communications content and virus scanning applications
for the presence of computer viruses.
***************************************************************************************************************************************************************************


--=_alternative 0027337FCA256C13_=
Content-Type: text/html; charset="us-ascii"



Hello Amol,



I am sure many people will answer your question but I have n't seen any yet, so here goes. Also I hope the list doesn't mind that this post is basically all Perl, very little Netsaint!



> Dear Sir,

> Thx for the solution!

> But im unable to decipher the command here..

> perl -i.bak -pe 's#^\[(\d+)\]#$1 - Your_Time_Correction(eg 2*3600)#e' netsaint.log



It's a Perl one-liner script.

-i.bak says edit in place, backing up the original with extension .bak.

-e says the first argument is the actual script to execute, instead of the script's name. -p says take in a file (the second argument) line by line, putting each line into the default argument $_, execute the script, and printout $_. Thus the file (in this case netsaint.log) will have each line transformed by the contents of that script, with the original saved as netsaint.log.bak. The script itself use the pattern-match-and-substitute operator s/// to search for the start of a line (^) immediately followed by a literal left bracket (\[) followed by one or more digits (\d+) and a right bracket (\]). This is then replaced by $1 - Your_Time_Correction(eg 2*3600), where $1 is anything matched inside the parenthesis earlier (i.e. the digits) and the rest was meant to be replaced by you with your actual correction factor. That is, if the error was 2 hours 3 seconds, you'd actually put #$1 - 7203#. The 'e' right at the end says the replacement string is to be evaluated as an expression, i.e. replace the matched pattern with the digits minus the correction factor.



Note there may be a small error here: while the date gets corrected, the square brackets will vanish. It might be better to p ut

 perl -i.bak -pe 's#^\[(\d+)\]#"[".($1 - 7200)."]"#e' netsaint.log

which should keep the square brackets



> is it necessary that it has to be in multiples of 3600 (1 hr)



No. The time argument is simply seconds. By 'eg 2*3600' he meant put 7200 if you want 2 hours.



> also pls tell me where this command has to be executed.



If Perl has been set up correctly on your system you should be able to execute it from anywhere. Note that if you are not in the directory where netsaint.log lives, you will need to give its full path.



> Also, I was unable to find any relations between the 10 digit time stamps (eg. 10288....), and the corre sponding time value. Please help me to know the logic behind the generation of these time stamps. I dont find any common link between the sequence of the time stamps. Is it using some knind of algorithm?



These time stamps are a standard Unix internal way of representing time, commonly known as t_time or epoch time. They represent the number of non-leap seconds past midnight, 1st January 1970, UTC (usually as a signed 32 bit integer) which is roughly how old Unix is. Yes this does mean you will need to get a 64 bit machine before the 2038 crisis! This date format has many advantages:

- it is constant through out the world, i.e. time zone independent;

- it is very compact (one 32 bit int for seconds, minutes, hours, days, months, and years);

- apart from the word size problem in 2038, it runs forever into the past and future without any funny glitches or historical weirdness;

- it is very easy to do arithmetic in it, while most other formats are horribly confusing; and

- it is very portable.

On the down side, it is hard for humans to read, but while we're talking Perl, you can convert epoch dates to human readable format in your own timezone, with:



perl -e 'print scalar localtime 1029132827'



(Substituting the number in question for the one I entered.) Similarly if you want the current epoch date just go:



perl -e 'print time'



[...]

Stanley wrote:

> perl -i. bak -pe 's#^\[(\d+)\]#$1 - Your_Time_Correction(eg 2*3600)#e' netsaint.log

>

> may be something like what you want, provided you know that the log

> entries are __all__ are out by the same constant amount.[...]



Or if there is a time before/after which they are bad by a constant amount, you could try something like:



perl -i.bak -pe 's#^\[(\d+)\]#$1<1028132827 ? $1 - 7200 : $1#e' netsaint.log



Substituting the epoch date where it all went wrong for 1028132827 and the actual error for 7200. May still be some hand editing at the cross-over point if the bad dates went forward in time, though.





Cheers,

Roger



***************************************************************************************************************************************************************************

This email and any files transmitted with it, are confidential and is intended solely for the use of the individual or entity to whom they are addressed.

If you have received this email in error, please notify the system manager.



This footnote also confirms that this email message has been scanned by AUSTAR Communications content and virus scanning applications

for the presence of computer viruses.

***************************************************************************************************************************************************************************



--=_alternative 0027337FCA256C13_=--


--__--__--

Message: 2
From: Tyson Harker
To: "'netsaint-users <at> lists.sourceforge.net'"

Date: Mon, 12 Aug 2002 08:49:03 -0700
Subject: [netsaint] nrpe-1.5 on AIX 4.3.3

I had everything running great and with no problems, then I found out that
the nrpe daemon wouldnt allow me to "down" the software that my company uses
so I decided to go in and play with the permissions that nrpe runs under.
After getting no luck, I decided to put everything back as it was, but when
I go to run the daemon useing ./nrpe -d /etc/nrpe.cfg it never runs. I go a
grep for the process and it never shows up. So I wiped out every instance
of nrpe that I could find then reconfigured it and I still get the same
problem. Does anyone have any idea why I suddenly cannot run this?

-----Original Message-----
From: netsaint-users-request <at> lists.sourceforge.net
[mailto:netsaint-users-request <at> lists.sourceforge.net]
Sent: Saturday, August 10, 2002 12:03 PM
To: netsaint-users <at> lists.s ourceforge.net
Subject: Netsaint-users digest, Vol 1 #1431 - 3 msgs


Send Netsaint-users mailing list submissions to
netsaint-users <at> lists.sourceforge.net

To subscribe or unsubscribe via the World Wide Web, visit
https://lists.sourceforge.net/lists/listinfo/netsaint-users
or, via email, send a message with subject or body 'help' to
netsaint-users-request <at> lists.sourceforge.net

You can reach the person managing the list at
netsaint-users-admin <at> lists.sourceforge.net

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Netsaint-users digest..."


Today's Topics:

1. Re: System time problem with NetSaint (amol sapkal)
2. Detecting anomalies in time series. Was: checking for glacial changes
(Stanley Hopcroft)
3. Re: Detecting anomalies in time series. Was: checking for glacial
changes (Stanley Hopcroft)

-- __--__--

Message: 1
Date: Fri, 9 Aug 2002 15 :45:38 -0700 (PDT)
From: amol sapkal
Subject: Re: [netsaint] System time problem with NetSaint
To: Stanley Hopcroft ,
netsaint-users <at> lists.sourceforge.net
Cc: amolsapkal <at> yahoo.com

--0-1857066914-1028933138=:31770
Content-Type: text/plain; charset=us-ascii


Dear Sir,
Thx for the solution!
But im unable to decipher the command here..
perl -i.bak -pe 's#^\[(\d+)\]#$1 - Your_Time_Correction(eg 2*3600)#e'
netsaint.log

is it necessary that it has to be in multiples of 3600 (1 hr)

also pls tell me where this command has to be executed.



Also, I was unable to find any relations between the 10 digit time stamps
(eg. 10288....), and the corresponding time value. Please help me to know
the logic behind the generation of these time stamps. I dont find any common
link between the sequence of the time stamps. Is it using some knind of
algo rithm?


Will manual edition help, once the relation is known?


PS : I got 2 log files automatically generated for that day (8 august), one
with the name..
netsaint-08-08-2002-00.log
and other with
netsaint-08-08-2002-18.log
The system date change was approximately 18-19 hrs ahead of the original
time. Is the value 18 in the 2nd log file to do anything with it?
your kind help is appreciated.
Warm Regds,
Amol Sapkal
GTL Ltd., Mumbai


Stanley Hopcroft wrote:Dear Sir,



On Thu, Aug 08, 2002 at 02:20:52PM -0700, amol sapkal wrote:
>
> Hi All,
>
> I'm facing a strange but yet significant problem, where in my machine's
system timing was changed unknowingly. The NetSaint logs now show me wrong
log entries.
>
> Is there anyway to get back the correct entries?
>

I thhink that the only way of getting them back is editing them with the
swiss arm y chainsaw, making sure that you are careful not to take your
legs off.

> Do I need to edit the netsaint logs manually.
>

perl -i.bak -pe 's#^\[(\d+)\]#$1 - Your_Time_Correction(eg 2*3600)#e'
netsaint.log

may be something like what you want, provided you know that the log
entries are __all__ are out by the same constant amount.

(in any case, you should try your auto edit program [if you choose to
use one] on a copy of the log. -i.bak will give you the original log
with a '.bak' suffix/extension).

Otherwise, you'll have to identify when the log got the bad timestamps
and then fix that bit.

> I have restored the correct system timing now.

It really sounds like you must

. identify the bad bits in the log
. cut them out and save them somewhere
. correct them
. put the corrected bits back in

This is feasable (I have done something like it) but messy and risky if
you g et lost.

Yours sincerely.

--
------------------------------------------------------------------------
Stanley Hopcroft
------------------------------------------------------------------------

'...No man is an island, entire of itself; every man is a piece of the
continent, a part of the main. If a clod be washed away by the sea,
Europe is the less, as well as if a promontory were, as well as if a
manor of thy friend's or of thine own were. Any man's death diminishes
me, because I am involved in mankind; and therefore never send to know
for whom the bell tolls; it tolls for thee...'

from Meditation 17, J Donne.


---------------------------------
Do You Yahoo!?
HotJobs, a Yahoo! service - Search Thousands of New Jobs
--0-1857066914-1028933138=:31770
Content-Type: text/html; charset=us-ascii

Dear Sir,

Thx for the solution!

But im unable to decipher the command here..

perl -i.bak -pe 's#^\[(\d+)\]#$1 - Your_Time_Correction(eg 2*3600)#e'

netsaint.log

is it necessary that it has to be in multiples of 3600 (1 hr)

also pls tell me where this command has to be executed.

 

Also, I was unable to find any relations between the 10 digit time stamps
(eg. 10288....), and the corresponding time value. Please help me to know
the logic behind the generation of these time stamps. I dont find any common
link between the sequence of the time stamps. Is it using some knind of
algorithm?

Will manual edition help, once the relation is known?

PS : I got 2 log files automatically generated for that day (8 august),
one with the name..

netsaint-08-08-2002-00.log

and other with

netsaint-08-08-2002-18.log

The system date change was approximately 18-19 hrs ahead of the original
time. Is the value 18 in the 2nd log file to do anything with it?

your kind help is appreciated.

Warm Regds,

Amol Sapkal

GTL Ltd., Mumbai

 

 Stanley Hopcroft
wrote:

Dear Sir,



On Thu, Aug 08, 2002 at 02:20:52PM
-0700, amol sapkal wrote:
>
> Hi All,
>
> I'm
facing a strange but yet significant problem, where in my machine's system
timing was changed unknowingly. The NetSaint logs now show me wrong log
entries.
>
> Is there anyway to get back the correct
entries?
>

I thhink that the only way of getting them back is
editing them with the
swiss army chainsaw, making sure that you are
careful not to take your
legs off.

> Do I need to edit the
netsaint logs manually.
>

perl -i.bak -pe 's#^\[(\d+)\]#$1 -
Your_Time_Correction(eg 2*3600)#e'
netsaint.log

may be something
like what you want, provided you know that the log
entries are __all__
are out by the same constant amount.

(in any case, you should try
your auto edit program [if you c hoose to
use one] on a copy of the
log. -i.bak will give you the original log
with a '.bak'
suffix/extension).

Otherwise, you'll have to identify when the log
got the bad timestamps
and then fix that bit.

> I have
restored the correct system timing now.

It really sounds like you
must

. identify the bad bits in the log
. cut them out and save
them somewhere
. correct them
. put the corrected bits back
in

This is feasable (I have done something like it) but messy and
risky if
you get lost.

Yours sincerely.

--

------------------------------------------------------------------------

Stanley
Hopcroft
----------------------------------------------------------------
--------

'...No man is an island, entire of itself; every man is a
piece of the
continent, a part of the main. If a clod be washed away by
the sea,
Europe is the less, as well as if a promontory were, as well as
if a
manor of thy friend's or of thine own were. Any man's death
diminishes
>me, because I am involved in mankind; and therefore never send to
know
for whom the bell tolls; it tolls for thee...'

from
Meditation 17, J Donne.

Do You
Yahoo!?

href="http://rd.yahoo.com/careers/mailsig/new/*http://www.hotjobs.com">HotJo
bs, a Yahoo! service - Search Thousands of New Jobs
--0-1857066914-1028933138=:31770--


-- __--__--

Message: 2
Date: Sat, 10 Aug 2002 09:20:36 +1000
From: Stanley Hopcroft
To: netsaint-users <at> lists.sourceforge.net
Subject: [netsaint] Detecting anomalies in time series. Was: checking for
glacial changes

Dear Ladies and Gentlmen,

I am writing with hopefully the last words about this matter. This may
be an important matter that my poor words are unfortunately obfuscating.

On Fri, Aug 09, 2002 at 07:36:48AM -0400, Bishop, Dean wrote:
> maybe you could use something like tcpdump to capture specific network
> traffic??
>
> This assumes that this is a network app.
>

Collecting the data is not an issue - sorry, former letters m ade this
unclear.

The problem is that for the monitor to make sense of time series data
and especially to recognise anomalies.

Applications are straightforward :-

. InOctet time series: does it look right
. Mails/time_units : is the mail server actually working
. Syn packets : is this a DOS
. Number processed : is the processing system alive ?
. VMStat, SAR etc : is the box Ok


So given a time series collected by some method and most familiarly
represented (for people) as a graph, how does one recognise if the graph
looks Ok ?

The Cricket people have phrased it better than I can at

http://cricket.sourceforge.net/aberrant

(There is also a paper by Luca Deri at http://luca.ntop.org/ADS.pdf -
Senior Deri is the ntop author - about this matter but it doesn't seem
to be specific enough to help me. I probably should read it again)

In a nutshell, you have a whole bunch of time series data (that you
probably graph), how can one automatically conclude that a dataset
indicates a fault ?

This I think is a reasonable thing for Netsaint to take an interest in.

The Cricket approach to anomaly detection is to

. have a modified RRD with a prediction Consolidation function in it
. check the actual values against the predicted (and the confidence
intervals/error bands)

Has anyone had any experience with a time series anomaly detector
(Cricket based or otherwise) ?

Yours sincerely.

--
------------------------------------------------------------------------
Stanley Hopcroft
------------------------------------------------------------------------

'...No man is an island, entire of itself; every man is a piece of the
continent, a part of the main. If a clod be washed away by the sea,
Europe is the less, as well as if a promontory were, as well as if a
manor of thy friend's or of thine own were. An y man's death diminishes
me, because I am involved in mankind; and therefore never send to know
for whom the bell tolls; it tolls for thee...'

from Meditation 17, J Donne.


-- __--__--

Message: 3
Date: Sat, 10 Aug 2002 13:07:57 +1000
From: Stanley Hopcroft
To: netsaint-users <at> lists.sourceforge.net
Subject: Re: [netsaint] Detecting anomalies in time series. Was: checking
for glacial changes

Dear Ladies and Gentlemen,

The development branch of rrdtool (1.1.x) apparently has the new RRAs
HWPREDICT (and friends, see the Cricket articles) for anomaly detection
in time series.

Has anyone any experience with them ?

Yours sincerely.

--
------------------------------------------------------------------------
Stanley Hopcroft
------------------------------------------------------------------------

'...No man is an island, entire of itself; every man is a piece of the
continent, a part of the main. If a clod be washed away by the sea,
Europe is the less, as well as if a promontory were, as well as if a
manor of thy friend's or of thine own were. Any man's death diminishes
me, because I am involved in mankind; and therefore never send to know
for whom the bell tolls; it tolls for thee...'

from Meditation 17, J Donne.



-- __--__--

_______________________________________________
Netsaint-users mailing list
Netsaint-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/netsaint-users


End of Netsaint-users Digest



--__--__--

_______________________________________________
Netsaint-users mailing list
Netsaint-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/netsaint-users


End of Netsaint-users Digest