Re: merging nmsg records by timestamp?

Dave Plonka wrote:
> I have the need to merge sets of nmsg records (e.g., in files) so
> that they are interleaved according to the timestamps in the records
> so that they can be processed inorder as if they were originally
> received that way.
> 
> One of the sets is of dnsqr nmsg records, so the timestamp I'd be
> using fields such as response_time_sec and response_time_nsec to
> compare for sorting.  (Since different nmsg records have different
> names for these fields, I would have to configure that based on type.)
> 
> Basically, I want to do what mergecap does with pcap files.  Has anyone
> already done this or have an idea for a quick-and-possibly-dirty solution?

hi, dave:

the response_time_sec and response_time_nsec fields are actually arrays
that contain the timestamp for each packet in the response_packet field
(which is also an array).  these three fields can have arrays of more
than one value if the DNS response arrived via a fragmented EDNS
message.  (similarly, there are query_time_sec, query_time_nsec, and
query_packet fields which are also arrays, although UDP DNS query
messages should never be fragmented, iirc.)

there is also a generic NMSG-level timestamp field, and in the
nmsg/isc/dnsqr.c:dnsqr_to_message() function there's the following code:

        if (dnsqr->n_query_time_sec > 0) {
                ts.tv_sec = dnsqr->query_time_sec[0];
                ts.tv_nsec = dnsqr->query_time_nsec[0];
                nmsg_message_set_time(m, &ts);
        } else if (dnsqr->n_response_time_sec > 0) {
                ts.tv_sec = dnsqr->response_time_sec[0];
                ts.tv_nsec = dnsqr->response_time_nsec[0];
                nmsg_message_set_time(m, &ts);
        }

this copies the dnsqr query packet timestamp to the NMSG-level
timestamp, unless there was no query timestamp (i.e., the dnsqr message
is of type UDP_UNSOLICITED_RESPONSE), in which case the timestamp of the
first response packet is used instead.

it would be possible to write a fully generic "nmsg-merge" utility that
merged multiple NMSG files together based on the NMSG-level timestamp,
but this wouldn't quite do what you asked, since you want to merge based
on response timestamp and not query timestamp.  if you can handle query
timestamp sorted data, though, it would be relatively straight forward
to write an "nmsg-merge" utility.  (in fact i have something already
implemented that could be adapted to do this.)

if you must have the output sorted by response timestamp, though, you'd
need to do it in two stages.  the first would need to sort each input
file by response timestamp, and the second would need to do the actual
merging.  it would probably be easier to have the first stage also
overwrite the NMSG-level timestamp with the response timestamp so the
second stage could be fully generic "nmsg-merge" utility i just
described.

hope this helps.

--

-- 
Robert Edmonds
edmonds <at> isc.org