6 · ARP, ICMP, DHCP, DNS and the life of a packet Network

ARP request/reply · ICMP types · DHCP DORA and relay · DNS resolution chain · IPv6 NDP/SLAAC · three rehearsed 'what happens when' walk-throughs

Why it matters for NPE. 'Two hosts just booted, how do they start talking', 'ping across two networks' and 'type facebook.com into a browser' are the most-reported network questions after BGP and TCP. These four protocols are the moving parts.

Primer: four helper protocols, one story

A host that has just booted knows nothing. DHCP gives it an address, a mask, a gateway and a DNS server. DNS turns a name into an address. ARP (or IPv6 NDP) turns the next hop's IP into a MAC so a frame can be built. ICMP is how the network reports back when something goes wrong, and it is what ping and traceroute are made of. Every "what happens when" question is these four in order, then TCP, then the application.

Watch

Reading beats video for two things: the record-type table at Cloudflare, DNS records, and the IANA ICMP registry for the types and codes below.

ARP: IP on this link → MAC

ARP sits directly on Ethernet (EtherType 0x0806), has no IP header, and therefore never crosses a router. A host ARPs only for addresses in its own subnet or for its next hop.

• Who learns what. The target learns the asker's mapping from the request (sender fields) and the asker learns the target's from the reply. Every switch on the path learns both source MACs. One exchange fills three tables.
• Gratuitous ARP is an unsolicited announcement: sender IP = target IP = my own IP, broadcast. Hosts send it at boot to detect a duplicate address, and a VIP owner sends it after failover (VRRP, keepalived, a VM that moved) so every neighbor overwrites its cached MAC immediately instead of waiting for a timeout.
• Cache timing. Linux neighbor entries go REACHABLE for about 30 s (randomised 15-45 s), then STALE. A STALE entry is still used, but the next send triggers DELAY (5 s) then PROBE (3 unicast ARPs, 1 s apart). No answer means FAILED and the frame is dropped. Cisco routers cache for 4 hours by default, which is why a changed NIC can be unreachable from the router for hours unless the entry is cleared or the new host sends a gratuitous ARP.
• Stale or wrong entry. Frames are built to a MAC nobody owns. The switch has aged that MAC out, so it floods the frame; nothing answers, so TCP retransmits and ping shows timeouts while every other host on the subnet works. A duplicate IP shows as the entry flapping between two MACs and intermittent loss. Fix: ip neigh flush dev eth0 or clear arp-cache, then find the cause. ARP has no authentication, so a forged reply redirects traffic (ARP spoofing). Dynamic ARP inspection on switches is the counter.
• Proxy ARP. A router answers ARP for addresses it can route to. Useful for odd legacy setups, confusing in troubleshooting because the router's MAC shows up for a host that is not local.

$ ip neigh show
10.1.1.1  dev eth0 lladdr 00:1c:73:aa:bb:01 REACHABLE
10.1.1.20 dev eth0 lladdr bb:bb:bb:bb:bb:bb STALE
10.1.1.99 dev eth0  FAILED                      <-- probed three times, nobody answered

$ tcpdump -eni eth0 arp
aa:aa:aa:aa:aa:aa > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806): Request who-has 10.1.1.20 tell 10.1.1.10
bb:bb:bb:bb:bb:bb > aa:aa:aa:aa:aa:aa, ethertype ARP (0x0806): Reply 10.1.1.20 is-at bb:bb:bb:bb:bb:bb

ICMP: the network's error channel

ICMP rides directly on IP (protocol 1), has no ports, and is generated by routers and hosts, not by applications. Error messages carry the IP header plus the first 8 bytes of the packet that caused them, so the sender can match the error to a socket.

• ping sends type 8 with an id (the process) and an incrementing sequence number. Loss shows as gaps in the sequence. A reply from a different address than you pinged is a type 3 from a router in the path. ICMP being rate-limited or filtered at a hop does not mean user traffic is lost there.
• traceroute sends probes with TTL 1, 2, 3 and so on. Each router whose TTL hits zero returns type 11 code 0 from its ingress interface address, which is how you learn the path. Linux traceroute sends UDP to ports 33434 and up, so the destination answers with type 3 code 3 and that ends the trace. Windows tracert and traceroute -I use ICMP echo, and the destination answers with an echo reply. traceroute -T -p 443 uses TCP SYNs and gets through firewalls that drop UDP and ICMP echo.
• Three stars at one hop and normal replies after it mean that hop does not generate ICMP, not that it drops traffic. Stars from a hop to the end mean the path is broken there or the return path is.

DHCP: DORA

DHCP runs over UDP, client port 68, server port 67. The client has no address yet, so it sends from 0.0.0.0 to the broadcast address, and the switch floods it. Every message carries the same 32-bit transaction ID (xid) and the client's MAC (chaddr) so the client can match replies without an IP.

• Relay agent. Broadcasts stop at the router, so in any routed network the client's gateway runs a relay (ip helper-address, dhcrelay). It receives the broadcast, writes its own interface address into the giaddr field, and unicasts the message to the server. The server picks the scope from giaddr, replies unicast to the relay, and the relay hands it to the client. No relay, no address: that is the first thing to check when a whole new subnet gets no leases.
• Lease timers. At T1 = 50% of the lease the client unicasts a Request to the server that gave it the lease (renew). At T2 = 87.5% it broadcasts a Request to any server (rebind). At 100% it must stop using the address and go back to Discover. A short lease is a fine mitigation during migrations; it also multiplies server load.
• 169.254.x.x (APIPA, link-local) means the client sent Discovers and nothing answered. Causes: port in the wrong VLAN, trunk missing that VLAN, relay not configured, scope exhausted, server down, or DHCP snooping blocking the server port. It is never a routable address, and two 169.254 hosts on one segment can still talk to each other, which is a useful clue.
• Rogue server. Any host can answer Discovers. DHCP snooping on switches marks one port as trusted and drops Offers from the rest.

$ tcpdump -ni eth0 -v udp port 67 or udp port 68
IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from aa:aa:aa:aa:aa:aa, xid 0x3f2a1c, DHCP-Message: Discover
IP 10.1.1.1.67 > 255.255.255.255.68: BOOTP/DHCP, Reply, xid 0x3f2a1c, Your-IP 10.1.1.10, DHCP-Message: Offer, Server-ID 10.1.1.1, Lease-Time 86400, Subnet-Mask 255.255.255.0, Default-Gateway 10.1.1.1, Domain-Name-Server 10.1.1.53
IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from aa:aa:aa:aa:aa:aa, xid 0x3f2a1c, DHCP-Message: Request, Requested-IP 10.1.1.10, Server-ID 10.1.1.1
IP 10.1.1.1.67 > 255.255.255.255.68: BOOTP/DHCP, Reply, xid 0x3f2a1c, Your-IP 10.1.1.10, DHCP-Message: ACK

DNS: name → record

• TTL and caching. Every record carries a TTL in seconds. The recursive resolver, the OS stub and the browser each cache for up to that long, so a change propagates only after the old TTL expires. Lower the TTL a day before a migration, change, then raise it. Negative answers are cached for the SOA minimum (RFC 2308).
• NXDOMAIN vs SERVFAIL vs NODATA. NXDOMAIN (rcode 3) is an authoritative "this name does not exist": a typo, a deleted record, or you are querying the wrong zone. SERVFAIL (rcode 2) is the resolver saying "I could not get an answer": authoritative servers unreachable, lame delegation, DNSSEC validation failure, or a timeout. NOERROR with an empty answer means the name exists but has no record of that type (you asked for AAAA and there is only an A). REFUSED (rcode 5) is policy: the server will not recurse for you.
• Transport. UDP 53 by default. Classic limit 512 bytes; EDNS0 raises it (1232 is the common safe size). If the answer does not fit, the server sets the TC bit and the client retries over TCP 53. Zone transfers (AXFR/IXFR) always use TCP. DoT is TCP 853, DoH is HTTPS 443. A firewall that allows only UDP 53 breaks large answers and transfers.
• Message format. 16-bit ID, flags (QR query/response, AA authoritative, TC truncated, RD recursion desired, RA recursion available, AD authenticated), counts, then question, answer, authority and additional sections. Random IDs and source ports are the defence against cache poisoning; DNSSEC is the real fix.

$ dig www.facebook.com

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 51283
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; QUESTION SECTION:
;www.facebook.com.              IN      A

;; ANSWER SECTION:
www.facebook.com.        3600   IN      CNAME   star-mini.c10r.facebook.com.
star-mini.c10r.facebook.com. 60 IN      A       157.240.22.35

;; Query time: 18 msec
;; SERVER: 10.1.1.53#53(10.1.1.53) (UDP)

Read it top down: status is the rcode. flags: qr rd ra means a response, recursion was asked for and is available; no aa because a cache answered, not the authority. The TTL column is what is left in the cache (60 s on the A record: Meta moves traffic fast). Query time near 0 ms is a cache hit; SERVER shows which resolver you actually used. Useful variants: dig @8.8.8.8 name to bypass the local resolver, dig +trace name to walk root → TLD → authoritative yourself, dig +short, dig -x 157.240.22.35, dig name AAAA.

IPv6 equivalents

ICMPv6 is not optional in IPv6: block types 133-137 and nothing on the link works; block type 2 (Packet Too Big) and PMTUD fails, and IPv6 routers never fragment. ip -6 neigh shows the neighbor cache with the same REACHABLE/STALE states as IPv4.

Three rehearsed walk-throughs

Interview questions

Traps

• Saying "the host ARPs for the destination IP" when the destination is in another subnet. It ARPs for the gateway.
• Saying the switch has an ARP table. Switches keep a MAC-to-port table; the ARP cache is on hosts and routers. An L3 switch has both.
• Getting DHCP backwards: the client uses port 68 and sends from 0.0.0.0; the server uses port 67. The Request is broadcast too, not unicast.
• Claiming DHCP broadcasts cross routers "because the router forwards them". They do not; a relay agent rewrites and unicasts them, and giaddr is how the server knows the subnet.
• Treating NXDOMAIN and SERVFAIL as the same "DNS is down". They point at different culprits.
• Saying DNS is UDP only, or that TTL is in minutes. TTL is seconds and TCP is part of the protocol.
• Saying IPv6 uses ARP or broadcast. It uses ICMPv6 NDP and multicast, and the gateway comes from the RA, not from DHCPv6.
• Reading traceroute stars as packet loss at that hop. Loss shows in mtr as sustained loss from a hop to the end, not as one silent hop.

Scenario

A user says "the internal wiki is down". You find curl https://wiki.corp.example hangs for five seconds then says "could not resolve host", but curl -k https://10.20.30.40 returns the wiki login page. Other users are fine. Walk through it.

← 5 · File handling and log parsing · all topics · 7 · Sorting, Big-O and top-k with heaps →