19 · Troubleshooting walkthroughs Network

1. Scope. What does "unreachable" mean: ping, SSH, the web page, and the exact error? By name or by IP? Since when, and what changed? Then the three splits: one client or all clients, one server or all servers on that subnet, by IP or by name.
   • Name fails and IP works: DNS. dig, getent hosts, /etc/hosts. Done, different problem.
   • Many clients, one server: the server or its edge. One client, one server: the client, or something specific to that pair (ACL, route, host firewall).
   • One client, every server: the client's own stack. Start bottom-up on the client.
2. Client, bottom-up. ip -br link (UP and LOWER_UP), ip addr (real address, not 169.254), ip route get 10.9.5.5 (which gateway and interface), ip neigh for that gateway (REACHABLE), ping gateway. If the gateway answers, the local link, VLAN and ARP are fine and the problem is at or beyond the gateway.
3. Same subnet or remote? Same subnet: arping -I eth0 10.9.5.5. FAILED or no reply means a layer 2 fault: server down, wrong VLAN, cable, or two devices claiming the IP (two MACs reply). Remote: traceroute -n 10.9.5.5. The last responding hop is the furthest router that has both a route forward and a route back to you. The fault is at the next hop: no route to 10.9.0.0/16, an ACL, or no return route to the client's subnet.
4. Server side (your shell, or a neighbour host on the same subnet). ip -br link carrier, ip addr right IP and mask, ip route default gateway, nft list ruleset. Then the decisive test: tcpdump -ni eth0 icmp and host <client> while the client pings.
   • Requests arrive, replies leave: the return path is broken. The server's route back is wrong, or a stateful firewall sees only one direction (asymmetric routing).
   • Requests arrive, no reply: the server's host firewall drops them, or rp_filter drops packets whose source is not routable back through the arriving interface.
   • Nothing arrives: the drop is upstream: an ACL, a missing route on an intermediate router, or the server's switch port in the wrong VLAN. Split again at the midpoint router.
5. Is ping even the right test? ICMP may be filtered while the service is fine. nc -zv 10.9.5.5 22 or curl to the real port. "Server unreachable" and "ping blocked" are different incidents.
6. Server down? Console or out-of-band: power, kernel panic, NIC driver, a reboot with a changed MAC while a neighbour holds a stale ARP entry. Check ip neigh on the gateway for 10.9.5.5.
7. Report. "Confirmed: client stack and gateway fine, traceroute stops at R2. Ruled out: DNS, client firewall. Testing: route on R2 and return route on R3 next." Name the layer each test covered.

1. Baseline. What is normal RTT between A and B (light in fibre is roughly 1 ms per 100 km one way, so trans-Atlantic is 70 to 80 ms RTT) and what is it now? Is it all of region A or one site? Did synthetic monitoring see it, or only users? Since when, and what changed: a maintenance, a link flap, a BGP policy change, a new tunnel, a traffic shift?
2. Measure both directions. mtr -rwzc 200 -n -T -P 443 from A toward B and from B toward A. TCP mode to the real port because ICMP is often deprioritised. Paths are often asymmetric, so one direction can be clean while the other is bad.
3. Read mtr from the last hop backwards. Find the first hop whose loss and jitter persist to the destination; ignore hops whose loss vanishes downstream (ICMP rate limiting). A latency jump that persists means congestion at that hop or a path change (compare the hop list and AS path to yesterday's; a reroute through a longer path is latency with zero loss). Jitter (high StDev, high Wrst) with loss means queueing. Loss with flat latency means a bad link (CRC errors), a policer, or a black hole for some flows.
4. Hop by hop at the suspect. Interface counters on both ends of that link: input errors and CRCs (dirty optic, bad cable, failing transceiver), output drops (queue full, congestion), utilisation graphs. On Linux: ethtool -S, ip -s link. On routers: the interface counters and queue drops. Errors climbing means replace the optic; drops with 95% utilisation means capacity or a traffic shift.
5. Congestion vs loss vs reordering from the TCP side. ss -tni on the server for the affected flows: retrans climbing with cwnd small is loss; rtt growing before loss appears is a filling queue (congestion); dup ACKs and SACK blocks with no actual retransmits needed, or D-SACK, is reordering, which happens when parallel paths have different delays. Throughput ≈ MSS / (RTT × √loss), so even 0.5% loss at 150 ms kills a single flow (topic 8).
6. MTU. Small transfers fine, big ones stall: ping -M do -s 1472 across the path. A new tunnel or MPLS segment with a smaller MTU and ICMP blocked is a black hole for large packets only, which users describe as "slow and flaky".
7. ECMP: only some flows suffer. If some users are fine and others on the same path are not, suspect one member of an ECMP group or a LAG. Hashing on the 5-tuple sends each flow down one member, so a bad member hurts 1/N of flows. Test per flow by varying the source port: for p in $(seq 40000 40015); do mtr -rwc 50 -n -T -P 443 -L $p 10.9.5.5 | tail -1; done (or traceroute -T -p 443 --sport=$p). If two of sixteen show loss and the hop list for those shows a different member IP, you have found the bad link. Drain it (cost it out, shut the member), then fix the optic.
8. Mitigate, then root-cause. Shift traffic off the bad path (local preference, cost-out, drain the link), confirm with the same mtr, keep the before and after. Then the postmortem: why did monitoring not catch it, and why did the link not auto-drain on errors.

1. Define slow. Time to first byte, or full page load? Everyone, or one region or ISP? Every request, or intermittently? Which URL? Since when, and what changed?
2. Reproduce with a timing breakdown from near the users and from near the server:

   curl -s -o /dev/null -w 'dns %{time_namelookup} tcp %{time_connect} tls %{time_appconnect} ttfb %{time_starttransfer} total %{time_total} code %{http_code}\n' https://www.example.com/

   Subtract adjacent values and the bucket names the suspect:
   • DNS large: slow or failing resolver, long CNAME chain, or geo-DNS sending users to a far edge.
   • TCP (connect minus namelookup) large: path RTT, or 1 s / 3 s steps, which are SYN retransmits, so a SYN is being lost.
   • TLS (appconnect minus connect) large: two RTTs (TLS 1.2 without resumption), OCSP fetch, or an overloaded TLS terminator.
   • TTFB (starttransfer minus appconnect) large: the server or something behind it (database, upstream API). This is the only bucket that is the server's.
   • Transfer (total minus starttransfer) large: the network during the download: loss, small windows, MTU.
3. If transfer is slow, capture. tcpdump -w on the client and look for: retransmissions and dup ACKs (loss on the path; find it with mtr, scenario 2); window 0 from the client (the client or a proxy is not reading, so not a network problem); a small advertised window or no window scaling (buffer tuning); the first large response segment retransmitted over and over while the handshake and small requests are fine (PMTUD black hole; fix with MSS clamping or allow ICMP type 3 code 4). On the server, ss -tni shows retrans, cwnd and rtt per connection.
4. If TTFB is slow, "CPU is fine" is not enough. Which server: the edge, the app tier, or the database? uptime and top for load, wa (disk) and si (softirq). Test each backend directly with curl --resolve host:443:backendIP: one slow backend behind a load balancer makes one request in N slow, which users report as "sometimes slow".
5. CDN and DNS geo. Compare dig from the users' resolver with yours. If users' resolver maps them to a far edge (a resolver change, EDNS client subnet missing, a stale geo map), every connection pays a long RTT and the server is innocent. curl --resolve to the near edge versus the far edge quantifies it.
6. Hand off with numbers. "DNS 20 ms, TCP 85 ms, TLS 90 ms, TTFB 1.9 s from Frankfurt against backend 3 only" sends the right team to the right place.

1. What the shape tells you. Pair-specific failure clears the core network, B's whole stack and A's whole stack. The fault is in how A and B see each other: addressing, ARP, a host firewall, a route, or a per-pair filter.
2. Masks. ip addr on both. If A is 10.1.1.10/24 and B is 10.1.1.20/25, A treats B as on-link and ARPs directly, while B may treat A as remote and send its replies to the gateway (which may or may not forward them back onto the same segment). One direction works, the other depends on the gateway. A mask mismatch produces exactly "most things work, this pair does not".
3. ARP. ip neigh show 10.1.1.20 on A. FAILED: B is not answering A on this link. REACHABLE with the wrong MAC: stale ARP after B's NIC or VM was replaced; ip neigh del and retry. arping -I eth0 10.1.1.20: two MACs replying means a duplicate IP; whichever answered last wins in each host's cache, so some hosts reach the real B and some reach the impostor. Check B for a PERMANENT entry pointing at A's old MAC.
4. Host firewall on B. nft list ruleset or iptables -L -n -v on B, looking for a rule that names A's address. The most common real-world cause here is fail2ban or a similar tool that banned A after failed logins. Watch the DROP counter while A retries. Check A's OUTPUT chain too.
5. VLAN. If A and B both reach remote hosts through their gateways but not each other, their switch ports may be in different VLANs that both carry a 10.1.1.0/24 address plan, or B's port was moved. ARP broadcasts do not cross VLANs. Check the switch: which VLAN each port is in, and on which port B's MAC was learned.
6. ACL or security group. A per-host ACL, micro-segmentation policy or cloud security group that allows the subnet but not this pair. Prove it the same way as always: capture on B. A's packets arrive and no reply leaves means B (firewall, rp_filter). A's packets never arrive means something between: ACL on the switch, VLAN, or A sending them to the wrong place.
7. Routes. ip route get 10.1.1.20 on A. It should say dev eth0 with no via. A stray /32 (a VPN client, a container network, an old static) sends that one destination somewhere else. Also ip rule for policy routing. Repeat on B toward A: the return path needs the same check.
8. Decision table. ARP FAILED → L2 or VLAN. Wrong MAC → stale or duplicate. Packets arrive at B, no reply → B's firewall or rp_filter. Works one way only → mask mismatch or asymmetric route. Everything looks right but a /32 appears in ip route get → fix the route.

dig returns an address; curl or nc -zv to it fails. DNS and the service are independent paths: the resolver is reachable, the server is not. First, is the address right? Compare with ip addr on the server; a stale record, split-horizon DNS or an /etc/hosts entry sends you to the wrong host. Then the failure type: refused (RST) means reachable but nothing listening on that address and port, so ss -tlnp on the server (bound to 127.0.0.1?); timeout means the SYN is dropped, so route, firewall or host down, which is the next item. Also check the address family: an AAAA record with broken IPv6 makes the client try v6 first; curl -4 versus -6 separates them.

tcpdump on the client shows [S] retransmitted at 1, 2, 4, 8 s and nothing returning. Four possibilities, split by a capture on the server. (a) The SYN never reaches the server: no route, an ACL or firewall dropping silently, wrong VLAN. (b) It arrives and the server's host firewall drops it: tcpdump on the server shows the SYN, no SYN-ACK, DROP counter moves. (c) It arrives, the server answers, and the SYN-ACK is lost on the way back: wrong default route on the server, asymmetric path through a stateful firewall, rp_filter. (d) The listen backlog is full, so the kernel drops SYNs (nstat -az TcpExtListenDrops). An RST would be a different story: reachable, port closed.

1. "What changed" is given: the deploy. Ask what it touched: application, config, number of backends, a kernel or network change. Ask whether the failing eighth is random or sticks to some users or some source addresses.
2. One in eight is a ratio that smells like one bad member out of eight: one backend behind the load balancer, or one ECMP or LAG member. Random per request points to a backend; sticky per source address points to a hashed path.
3. Test each backend directly with curl --resolve api.example.com:443:backendIP -w for all eight. One timing out or returning errors is the answer; check it with ss -tlnp, its logs and uptime. If all eight are fine from inside, sweep source ports through the load balancer and the path (scenario 2) to find a bad link.
4. Mitigate as soon as you find it: drain that backend or link. Confirm the failure rate drops to zero with the same test. Then root-cause: why did the health check not catch it, and roll back or fix the deploy.
5. Status along the way: "Confirmed: 1/8 timeouts since the deploy. Hypothesis: one backend. Testing each directly now; drain the bad one as soon as identified."