5 · File handling and log parsing Code

with open, line streaming, split vs csv, json, validation, aggregation, two-file joins · the NPE signature question and its follow-ups

Why it matters for NPE. The official guide names file handling first, and 7 of 22 firsthand reports describe a file task: join two CSVs, count word frequency, parse a delimited file too big for RAM. The follow-ups are always the same: complexity, dictionary size, missing rows or columns, try/except, and 'what if the file is 20 GB'.

Primer: the shape of every file question

Read a file line by line. Clean each line. Split it into fields. Validate. Update a small amount of state (usually a dictionary keyed by the thing you're asked about). Produce a sorted report. Then survive the follow-ups: complexity, how big the dictionary gets, a missing row or column, malformed lines, and "what if the file doesn't fit in memory". Firsthand reports describe exactly this pattern: join two CSVs on a key and sort by a computed value, count word frequency in a file, read a %-delimited fortune file and handle one bigger than RAM.

from collections import defaultdict, Counter

def report(path):
    counts = Counter()
    bad = []                                   # (line_no, reason)
    with open(path, encoding='utf-8') as f:   # streams; nothing loaded at once
        for line_no, line in enumerate(f, 1):
            line = line.strip()
            if not line or line.startswith('#'):
                continue
            parts = line.split(',')
            if len(parts) != 2:
                bad.append((line_no, 'field count'))
                continue
            host, status = (p.strip() for p in parts)
            if status.upper() == 'DOWN':
                counts[host.lower()] += 1
    return counts, bad

Watch

The file object

Always with open(...): the file closes even on an exception. Say encoding='utf-8' and, if asked about garbage bytes, errors='replace'. Open in 'rb' when you need byte offsets that survive multi-byte characters.

Cleaning and splitting

line.rstrip('\n')          # just the newline; keeps leading/trailing spaces in data
line.strip()               # all surrounding whitespace, the usual choice
line.split()               # any run of whitespace, empties dropped: good for log lines
line.split(',')            # exact delimiter, empties kept: good for CSV without quotes
line.split(',', 2)         # at most 2 splits: message fields that contain commas
key, _, value = line.partition('=')   # first '=' only; always 3 parts

import csv
with open(path, newline='') as f:
    for row in csv.DictReader(f):              # handles quoted commas, header → keys
        row['bytes']                            # str! convert yourself

import json
rec = json.loads(line)                          # JSON Lines: one object per line
data = json.load(f)                             # one document: loads it all

import re
m = re.match(r'(\S+) (\S+) \[(.*?)\] "(\w+) (\S+)', line)   # only when split() can't

Validation and the exception policy

Decide out loud what "bad input" means and what you do with it. Three acceptable policies: skip and count (default for reports), skip and record the line number (so someone can fix the data), or fail fast (for config files where silence is dangerous). Pick one, say it, move on.

try:
    n = int(fields[1])
except (ValueError, IndexError):
    bad.append(line_no)
    continue
if n < 0:
    bad.append(line_no); continue

Catch specific exceptions. A bare except: hides KeyboardInterrupt and real bugs, and interviewers notice. Check types too: '5' == 5 is False, and None from dict.get will crash int().

Aggregation patterns

The answer to "how big does the dictionary get" is the number of distinct keys, not the number of lines. If keys are unbounded (every line has a unique request ID) say so and propose a fix (bucket by minute, external sort, approximate counting).

Worked example: the two-file join (the "dinosaur" pattern)

File 1: NAME,LEG_LENGTH,DIET. File 2: NAME,STRIDE_LENGTH,STANCE. Print the bipedal dinosaurs from fastest to slowest, where speed = ((STRIDE_LENGTH / LEG_LENGTH) − 1) × sqrt(LEG_LENGTH × g), g = 9.8.

import csv, math
G = 9.8

def read_rows(path):
    with open(path, newline='') as f:
        for row in csv.DictReader(f):
            yield row

legs = {}                                            # name → leg length (file 1 is small, index it)
for row in read_rows('dino1.csv'):
    try:
        legs[row['NAME']] = float(row['LEG_LENGTH'])
    except (KeyError, ValueError):
        continue                                     # policy: skip malformed rows

speeds = []
for row in read_rows('dino2.csv'):
    name = row.get('NAME')
    if row.get('STANCE') != 'bipedal' or name not in legs:
        continue                                     # missing in file 1: can't compute, skip
    try:
        stride = float(row['STRIDE_LENGTH'])
    except (KeyError, ValueError):
        continue
    leg = legs[name]
    if leg <= 0:
        continue
    speeds.append((name, (stride / leg - 1) * math.sqrt(leg * G)))

for name, _ in sorted(speeds, key=lambda t: -t[1]):
    print(name)

Complexity. O(a + b) to read both files, O(m log m) to sort the m bipedal matches, O(a) space for the index. Missing column? row['LEG_LENGTH'] raises KeyError, caught by the policy, or check the header once with reader.fieldnames and fail fast. Name in file 2 but not file 1? name not in legs skips it; say you'd log it. File 1 huge too? Index whichever is smaller, or sort both by name externally and merge like lab 9.

When the file is bigger than RAM

• Line-oriented aggregation: already fine, you only hold the state dict. Point this out.
• Random access to records (the fortune-cookie file): make one pass recording f.tell() at the start of each record into a list of offsets (8 bytes per record, not the record itself), then f.seek(offsets[random.randrange(len(offsets))]) and read until the delimiter. Two passes, O(records) memory for offsets only. Read in 'rb' so offsets are exact.
• Sorting a huge file: external sort, chunks that fit in memory sorted and written, then a k-way merge with heapq.merge.
• Grouping with unbounded keys: partition by hash of the key into several files, then process each.
• Output could be huge: write it as you go instead of building a list.
• Mention generators: def lines(path): with open(path) as f: yield from f lets you chain filters lazily.

Reading a fortune file: records separated by a delimiter line

def fortunes(path):
    buf = []
    with open(path, encoding='utf-8') as f:
        for line in f:
            if line.rstrip('\n') == '%':
                if buf:
                    yield ''.join(buf)
                    buf = []
            else:
                buf.append(line)
    if buf:
        yield ''.join(buf)                 # the last record may lack a trailing delimiter

import random
print(random.choice(list(fortunes('fortunes.txt'))))      # small file
# big file: reservoir sampling, one pass, O(1) memory:
pick = None
for i, fortune in enumerate(fortunes('fortunes.txt'), 1):
    if random.randrange(i) == 0:
        pick = fortune

Interview questions

Traps

• Forgetting the trailing newline: 'DOWN\n' != 'DOWN'. Strip first.
• split(',') on a quoted CSV field: "edge,west",100 becomes three fields.
• Counting lines instead of distinct keys when asked about memory.
• Building a list of every row "just in case" and then claiming O(1) memory.
• A bare except:, or catching Exception and silently continuing without counting.
• Comparing strings from the file with ints from the question without converting.
• Normalising inconsistently: counting WEB-01 and web-01 separately when they are the same host.
• Using f.read().split('\n') on a file you were told is large.

Do before marking this topic done

1. Labs 1-6 below, timed, in a blank editor. Compare with the expected JSON only after your run.
2. Word frequency: given a text file, print the 10 most common words (case-insensitive, punctuation stripped) with counts. Then answer out loud: complexity, dictionary size, what if the file is 50 GB.
3. Reimplement the dinosaur join from the description above without looking. Add the missing-column check.
4. Fortune file: implement both the offsets approach and reservoir sampling.

Practice: linked problems and labs

Logged attempts feed the tracker. Open the problem in a new tab, solve in a blank editor, then log honestly.

• LC 819Most Common WordEasy
• LC 1108Defanging an IP AddressEasy
• LC 1941Check if All Characters Have Equal Number of OccurrencesEasy
• LC 165Compare Version NumbersMedium
• LC 468Validate IP AddressMedium
• LC 811Subdomain Visit CountMedium
• LC 937Reorder Data in Log FilesMedium
• LC 71Simplify PathMedium
• LC 388Longest Absolute File PathMedium
• LC 751IP to CIDRMedium
• lab 1Count host failures from a fileLab
• lab 2Aggregate quoted CSV fieldsLab
• lab 3Keep the latest health recordLab
• lab 4Find the busiest clientsLab
• lab 5Compare two configuration snapshotsLab
• lab 6Track consecutive failures per hostLab

← 4 · IPv4, CIDR and subnetting · all topics · 6 · ARP, ICMP, DHCP, DNS and the life of a packet →