8 · TCP vs UDP: handshake, state, loss and congestion control Network

3-way handshake, seq/ack, FIN vs RST, retransmission, flow vs congestion control, MTU/MSS/PMTUD

Why it matters for NPE. TCP is the protocol most candidates are asked to explain in depth. Expect follow-ups on what you'd see in a capture when things go wrong.

Primer: TCP gives you a reliable byte stream; UDP gives you a datagram and a shrug

TCP numbers every byte, acknowledges what arrived, retransmits what didn't, keeps data in order, and slows down when the receiver or the network can't keep up. UDP adds only ports and a checksum to IP; anything else (ordering, retries) is the application's job, which is exactly why DNS, DHCP, QUIC and most real-time traffic use it. The interviewer wants to hear that you know which fields and timers make TCP's promises true and what you'd see on the wire when they fail.

Watch

Reading: RFC 9293 §3.5 (state diagram) and RFC 5681 §3.1-3.2 (slow start, congestion avoidance, fast retransmit).

Headers

Connection lifecycle

• Why three messages? Both sides must prove they can send and receive and must exchange their random initial sequence numbers. Two would leave the server unsure the client received its SYN+ACK.
• Why random ISNs? So a stale segment from an old connection on the same 4-tuple isn't mistaken for new data, and so off-path attackers can't guess sequence numbers.
• Half-open and SYN floods. The server keeps state after the SYN+ACK; floods exhaust the backlog. SYN cookies encode the state in the ISN instead.
• TIME_WAIT lets late segments die and makes sure the final ACK can be resent. Lots of TIME_WAIT sockets on a busy client is normal, not a leak.
• FIN vs RST. FIN is a polite "I have no more data", each direction closes independently (half-close). RST aborts immediately: sent when a segment arrives for a port with no listener, or when an application closes with unread data, or by a firewall that rejects rather than drops.

Reliability: timers and duplicate ACKs

• Retransmission timeout (RTO) is computed from smoothed RTT and its variance (RFC 6298: RTO = SRTT + 4×RTTVAR, min 200 ms on Linux, initial 1 s). On timeout the segment is resent and the RTO doubles (exponential backoff).
• Fast retransmit. If the receiver gets an out-of-order segment it re-ACKs the last in-order byte. Three duplicate ACKs tell the sender a segment was lost before the timer fires. With SACK it knows exactly which one.
• Out of order arrival is buffered by the receiver and acknowledged cumulatively once the gap fills. Reordering looks like loss to the sender if it is severe (spurious retransmits).

Flow control vs congestion control

Throughput ≈ window / RTT. A 64 KB window over a 100 ms RTT gives at most 5.2 Mbit/s regardless of link speed. That is the answer to "the link is 10 Gbit/s, why is one transfer slow across the ocean": the bandwidth-delay product exceeds the window, or loss keeps cwnd small (Mathis: throughput ≈ MSS / (RTT × √loss)).

MTU, MSS and the black hole

Ethernet MTU 1500 → IPv4 TCP MSS 1460. If a path has a smaller MTU (a VPN or MPLS tunnel, say 1400) and the packet has DF set, the router drops it and sends ICMP Fragmentation Needed (type 3 code 4). The sender then shrinks its segments: that is Path MTU Discovery. If a firewall blocks that ICMP, large segments vanish silently: the handshake works, small requests work, big responses hang. Fixes: allow ICMP type 3/4, lower the MTU, or clamp MSS on the edge router (--clamp-mss-to-pmtu). IPv6 routers never fragment; PMTUD is mandatory.

TCP vs UDP: when to choose which

What you'd see in a capture or in ss

$ ss -tni state established '( dport = :443 )'
ESTAB 0 0 10.1.1.10:51234 203.0.113.5:443  cubic wscale:7,7 rto:204 rtt:3.1/0.5 mss:1448 cwnd:10 bytes_sent:4210 retrans:0/2 ...

$ tcpdump -ni eth0 'tcp port 443 and (tcp[tcpflags] & (tcp-syn|tcp-rst) != 0)'
IP 10.1.1.10.51234 > 203.0.113.5.443: Flags [S], seq 1234, win 64240, options [mss 1460,sackOK,wscale 7]
IP 203.0.113.5.443 > 10.1.1.10.51234: Flags [S.], seq 5678, ack 1235, win 65160, options [mss 1460,sackOK,wscale 7]
IP 10.1.1.10.51234 > 203.0.113.5.443: Flags [.], ack 1, win 502

Interview questions

Traps

• Saying ACK numbers count packets. They count bytes; the ACK is the next expected byte.
• Saying "TCP is slower because it's reliable" without naming the mechanisms (handshake RTT, head-of-line blocking, cwnd ramp).
• Saying the server sends FIN "to close the connection". Each direction closes separately; CLOSE_WAIT piling up means your app never called close().
• Claiming UDP has no checksum. It does (optional in IPv4, mandatory in IPv6).
• Confusing the receive window (flow control, advertised) with the congestion window (sender's private estimate).
• Forgetting that DNS also uses TCP (answers over 512/1232 bytes, zone transfers), so "DNS is UDP" is incomplete.
• Calling BGP a "layer 3 protocol". It is an application that runs over TCP port 179.

Scenario

Users in California say a service in Europe is "slow". Pings show 150 ms RTT and 0.5% loss. The web team says the server is fine. What do you say and do?

← 7 · Sorting, Big-O and top-k with heaps · all topics · 9 · Strings, two pointers and sliding windows →