When a service "can't connect," the app logs and the network rarely tell the same story. tcpdump settles the argument: it shows what actually went over the wire, not what the client library claims happened. Learning to read three packet signatures is usually enough to split a network problem from an application problem in under a minute.
One ground rule first: only capture on systems you own or are explicitly authorized to assess. A pcap can hold credentials, tokens, and personal data, so treat capture files as sensitive and delete them when you're done. This is defensive infrastructure validation, not snooping.
Capture just enough, and nothing else
The mistake most people make is capturing everything and drowning in it. Filter at the source instead.
sudo tcpdump -i eth0 -n host 10.0.4.20 and port 443 -c 20
Breaking that down:
-
-i eth0— the interface to listen on (anyworks if you're unsure). -
-n— don't resolve IPs or ports to names, so DNS lag never distorts the output. -
host 10.0.4.20andport 443— a BPF filter: only packets to/from that host on that port. -
-c 20— stop after 20 packets so the terminal stays readable.
Add tcp to drop everything else, or write the raw capture to disk for later analysis:
sudo tcpdump -i eth0 -n host 10.0.4.20 and tcp -w handshake.pcap
-w handshake.pcap writes packets untouched — open it later in the same tool or in Wireshark. The point is to reproduce the failing request while this runs, then read back exactly what the kernel saw.
Signature 1: a healthy connection
A working TCP connection always opens with the three-way handshake — SYN, SYN-ACK, ACK:
10.0.1.5.51420 > 10.0.4.20.443: Flags [S], seq 12345
10.0.4.20.443 > 10.0.1.5.51420: Flags [S.], seq 98765, ack 12346
10.0.1.5.51420 > 10.0.4.20.443: Flags [.], ack 98766
[S] is SYN, [S.] is SYN-ACK, [.] is a bare ACK. Three lines, both directions, done. If you see this, the network path and the listening service are both fine — your problem is above the transport layer: TLS, auth, a slow query, a 500. Stop blaming the firewall and go read the app.
Signature 2: connection refused
10.0.1.5.51422 > 10.0.4.20.443: Flags [S], seq 22222
10.0.4.20.443 > 10.0.1.5.51422: Flags [R.], seq 0, ack 22223
The SYN gets an immediate RST ([R.]). That's a fast, deliberate "no." The packet reached the host, but nothing is listening on that port — the process is down, bound to the wrong interface (127.0.0.1 instead of 0.0.0.0), or a proxy is rejecting it. Notice the speed: refusals come back in milliseconds. Check that the service is up and bound correctly.
Signature 3: timeout / silence
10.0.1.5.51424 > 10.0.4.20.443: Flags [S], seq 33333
10.0.1.5.51424 > 10.0.4.20.443: Flags [S], seq 33333
10.0.1.5.51424 > 10.0.4.20.443: Flags [S], seq 33333
Your SYN goes out, gets retransmitted after ~1s, ~2s, ~4s — and nothing ever comes back. No RST, no SYN-ACK, just your own side talking to a wall. That silence is the signature of a dropped packet: a security group, iptables rule, missing route, or a host that's simply gone. A refusal answers; a firewall drop stays quiet. That distinction alone routes the ticket to the right team.
Don't forget DNS
Half of "network" outages are name resolution. DNS rides UDP/53, and a healthy lookup is one query, one response:
10.0.1.5.40311 > 10.0.0.2.53: A? api.internal.example. (37)
10.0.0.2.53 > 10.0.1.5.40311: A 10.0.4.20 (53)
Query out, answer back with an address. If the query leaves and no response returns, you have a resolver problem, not a connectivity problem — and every downstream connect() will fail for the wrong-looking reason. I walk through this end to end in the full tcpdump packet-analysis lesson.
Capturing inside a container
Containers don't get raw-socket access by default. The wrong fix is --privileged, which hands the container broad host-level capabilities — and never mount the Docker socket to work around it. Grant exactly the one capability packet capture needs:
services:
debug:
image: nicolaka/netshoot
cap_add:
- NET_RAW
network_mode: "service:app"
NET_RAW is enough for tcpdump; sharing the target's network namespace lets you watch its traffic without touching the host. There's a container-specific walkthrough in the Docker packet-capture lesson.
The habit
Observe the packets, form a hypothesis (handshake completes → look higher; RST → nothing listening; silence → something dropping), then test and validate the same way. The capture is evidence, not a guess.
If you want the guided version, start with the tcpdump lesson in the networking series. The site has free, hands-on Kali learning paths built for DevOps engineers — Kali as a portable troubleshooting toolbox, not a hacking course.
