Monitoring DNS Queries with tcpdump

I’ve been fighting an issue at work the last few days where the response times on our web servers would skyrocket at random times (i.e., non-peak times). All of our regular monitoring seemed to indicate everything was fine up until the time of the spike. The only lead I had to go on was a couple of name lookup timed out errors in the Apache error logs, so I decided to set up some ad hoc monitoring on a few of the servers with tcpdump.

tcpdump -vvv -s 0 -l -n port 53

This outputs each DNS query and response on a separate line, but I wanted to do some correlation between the two to find out how long each query was taking, and which queries (if any) weren’t getting responses. So I wrote a little script to do it. It reads the tcpdump output on standard input and correlates queries and responses. If you’re cating a log file to the script, it will buffer its output and correlate queries without a response. If you’re piping the output of tcpdump directly to the script, it will only print responses (run the script in “follow” mode with -f).

Here’s some sample output from the script:

18:03:12.378415 A          1.040 ms !
18:08:40.528361 A         160.063 ms !!!
18:13:45.418993 A           0.849 ms
18:18:01.400179 A    0.733 ms
18:29:46.485955 A    api.internal                    - ms **********

The fields are: time, query type, query subject, response time, and a logarithmic “histogram” — more !s means a longer response time. Failed queries (queries that receive no response at all) are marked with *s.

Tagged , ,

4 thoughts on “Monitoring DNS Queries with tcpdump

  1. Could you provide some usage examples of this script? Just lost on how to direct tcpdump to this script (it may be a lack of sleep, and not enough coffee..)

  2. Try this:

    sudo tcpdump -vvv -s 0 -l -n port 53 | php parse-tcpdump-udp-port-53.php -f

    Alternatively, if you want to see queries without replies, try this:

    sudo tcpdump -vvv -s 0 -l -n port 53 > log
    cat log | php parse-tcpdump-udp-port-53.php

    I edited the post to add the -n flag, which tells tcpdump itself not to do name lookups, as that would add queries that the system under observation wouldn’t have made on its own.

    I also updated the script to handle mutli-line tcpdump output. (I originally wrote this post against CentOS 5 or 6, but I just tried it with Ubuntu 12.04 LTS with tcpdump 4.2.1 and it didn’t work out of the box.)

  3. Excellent bro! Also, I am having this issue:
    people are hijacking some blocked web pages by DNS throttling, which means that they are essentially generating more “DNS traffic” in order to get to some pages… Do you happen to know how to reduce this? Maybe applying some QoS marking through DSCP and reducing the BW for these type of packets?

  4. Hello Sir,
    The above was simple and good approach and design sir and help me for my final year project .
    please can you help me a script or snippet of code for displaying
    Source IP,Destination IP , Domain name , mac address . I am facing lots of difficulty in it. please can you help for above.
    Hope for positive reply..

Leave a Reply

Your email address will not be published. Required fields are marked *