04 - The interfaces that matter: io.Reader/Writer

Objectives: Understand what io.Reader and io.Writer are and why the Go standard library is built around them. See how safeheaders-go uses these interfaces for streaming compression, JSON parsing, and image loading. Learn why io.LimitReader is a first-class safety tool, not an afterthought. Estimated time: 20 minutes.

What this actually means (plain English)

No jargon — here's what the ideas in this lesson actually mean, and why they matter.

• io.Reader = "a garden hose you can plug into any tap." Anything that produces bytes — a file, a network socket, a compressed stream, a []byte already in memory — can satisfy the io.Reader contract, so your code works with all of them identically.
• io.Writer = "a drain that doesn't care what pipe feeds it." Anything that absorbs bytes — a file, a network connection, an in-memory buffer — implements io.Writer, so the same CompressStream function can write to a disk file or a network connection without any changes.
• bytes.Reader / bytes.Buffer = "a travel adapter between the plug-in-the-wall world and the streaming world." bytes.NewReader wraps a plain []byte so it satisfies io.Reader; bytes.Buffer does the reverse — it collects streaming writes and hands back a []byte when you call buf.Bytes().
• io.Copy = "a bucket brigade, not a dump truck." It moves data from a Reader to a Writer 32 KB at a time, so a 2 GB file never fully materialises in RAM — only the current bucket is ever in flight.
• io.LimitReader = "a turnstile at the door of your memory." Wrapping any untrusted reader with io.LimitReader(r, N) means downstream code is physically incapable of reading byte N+1, making decompression-bomb protection a one-liner rather than a convention.
• Small interfaces compose = "LEGO bricks that snap together without custom connectors." Because io.Reader has only one method (Read), every concrete type in the standard library already satisfies it, and you can stack wrappers — LimitReader(TeeReader(r, log), max) — without writing any new type, as LoadStream demonstrates with TeeReader + MultiReader.

Why it matters: the moment untrusted input enters your program, you need to decide how much memory you're willing to spend on it. The io interfaces let you make that decision once, at the boundary, using io.LimitReader — rather than allocating a huge slice and hoping for the best.

See it — how io.LimitReader guards a streaming decompress.

---

The two-method contract

An interface in Go is a kind of contract: it names one or more actions (called methods — think of a method as a function, i.e. a named, reusable chunk of instructions, that happens to belong to a particular kind of value) and says "anything that can perform these actions counts as satisfying this contract," no matter what it actually is underneath. A byte is the basic unit computers store data in — roughly one letter or number's worth of raw data — and []byte means "a list of bytes," which Go calls a slice (a resizable, ordered sequence of values, similar in spirit to a numbered list you can grow or shrink).

// from the Go standard library — shown here for reference
type Reader interface {
    Read(p []byte) (n int, err error)
}

type Writer interface {
    Write(p []byte) (n int, err error)
}

In plain terms: this defines two contracts. Anything that has a Read method — that can fill a []byte handed to it with the next chunk of data and report back how much it filled — counts as an io.Reader. Anything with a Write method — that can accept a []byte and absorb (consume) it — counts as an io.Writer. That's the whole contract. Read fills p and returns (hands back to whoever called it) how many bytes it put there, plus any error — a special value that signals something went wrong, or a specific "end of data" signal called io.EOF. Write drains p and returns how many bytes it consumed. Everything else in the io package (a package is a named, reusable bundle of pre-written Go code — here, the built-in "io" bundle that ships with the language, part of what's called the standard library, the set of ready-made packages that come with Go itself) — Copy, ReadAll, LimitReader, TeeReader, MultiReader — is built on these two methods.

---

bytes.Reader and bytes.Buffer — the in-memory bridges

Most functions in safeheaders-go receive raw []byte from the caller (the piece of code that runs — "calls" or "invokes" — a function, here meaning the caller already loaded or received a blob of bytes). They bridge to the streaming world with bytes.NewReader:

From miniz-go/miniz.go:

func ExtractArchive(data []byte) ([]ZipFile, error) {
    r, err := zip.NewReader(bytes.NewReader(data), int64(len(data)))
    // ...
}

In plain terms: this function takes a plain []byte (data already sitting in memory) and wraps it so it can be handed to zip.NewReader, which expects the streaming-style interface rather than a raw byte list. zip.NewReader needs an io.ReaderAt (a positioned reader — one that can jump to a specific spot, or offset, in the data rather than only reading forward). bytes.NewReader provides it from a plain []byte. No copy, no temp file.

The write side uses bytes.Buffer as the accumulator:

func CreateArchive(files []FileEntry) ([]byte, error) {
    var buf bytes.Buffer
    w := zip.NewWriter(&buf)
    // ... write entries ...
    return buf.Bytes(), nil
}

In plain terms: buf is an empty holding area (a buffer — a chunk of memory used to temporarily collect data as it arrives); the zip writer sends its output into that buffer piece by piece, and at the end buf.Bytes() hands back everything collected as one plain []byte. bytes.Buffer satisfies io.Writer (in plain terms: it has a Write method, so it counts as one), so zip.NewWriter can write directly into it. When you're done, buf.Bytes() hands back the accumulated slice.

---

CompressStream — when you don't want to buffer at all

CompressData (in miniz-go/miniz.go) reads a []byte, compresses it, returns a []byte. Simple. But if your file is 2 GB, you don't want two copies of it in memory at once. That's what CompressStream is for:

// from miniz-go/miniz.go
func CompressStream(dst io.Writer, src io.Reader) error {
    if dst == nil || src == nil {
        return errors.New("nil reader or writer")
    }
    w, err := flate.NewWriter(dst, flate.BestCompression)
    if err != nil {
        return fmt.Errorf("create compressor: %w", err)
    }
    if _, err := io.Copy(w, src); err != nil {
        _ = w.Close()
        return fmt.Errorf("compress stream: %w", err)
    }
    if err := w.Close(); err != nil {
        return fmt.Errorf("finalize compression: %w", err)
    }
    return nil
}

In plain terms: this function reads from whatever src is (a file, a network connection, memory — it doesn't care) and writes the compressed result into whatever dst is, checking at each step whether something went wrong and, if so, reporting that back immediately instead of continuing. io.Copy(w, src) is the key line. It loops internally (repeats the same small step over and over on its own, without the calling code writing the loop itself), reading up to 32 KB from src and writing it to the compressor w, until src returns io.EOF (the built-in signal meaning "there is no more data to read"). The total memory in flight at any moment is just those 32 KB — not the whole input.

The caller can pass an os.File as src and a network connection as dst, and the function works unchanged. That's the composition payoff.

---

io.LimitReader — the safety fence

A decompression bomb is a compressed file that expands to gigabytes. Without a fence, io.ReadAll on the decompressed stream will happily allocate (reserve a chunk of the computer's memory to hold data — every time a program needs space to store something, it has to allocate that space first) until the process is killed.

Here is the helper that protects DecompressData and ExtractArchive in miniz-go/miniz.go:

// from miniz-go/miniz.go
func readAllLimited(r io.Reader, limit int64) ([]byte, error) {
    src := r
    if limit > 0 {
        // +1 so we can distinguish "exactly at the limit" from "over it".
        src = io.LimitReader(r, limit+1)
    }
    data, err := io.ReadAll(src)
    if err != nil {
        return nil, fmt.Errorf("read: %w", err)
    }
    if limit > 0 && int64(len(data)) > limit {
        return nil, fmt.Errorf("decompressed size exceeds %d-byte limit (adjust MaxDecompressedSize)", limit)
    }
    return data, nil
}

In plain terms: this function reads everything from r, but refuses to read more than limit+1 bytes no matter how much data is actually available; if it hits that ceiling, it treats the input as too big and hands back an error instead of the data. The +1 trick is intentional: io.LimitReader stops at exactly limit+1 bytes. If io.ReadAll returns limit+1 bytes, you know the real stream was larger and you return an error. If it returns <= limit bytes, you're within budget.

DecompressStream uses the same pattern on the fly, without a helper:

// from miniz-go/miniz.go
func DecompressStream(dst io.Writer, src io.Reader) error {
    r := flate.NewReader(src)
    defer r.Close()

    var reader io.Reader = r
    if MaxDecompressedSize > 0 {
        reader = io.LimitReader(r, MaxDecompressedSize+1)
    }
    n, err := io.Copy(dst, reader)
    if err != nil {
        return fmt.Errorf("decompress stream: %w", err)
    }
    if MaxDecompressedSize > 0 && n > MaxDecompressedSize {
        return fmt.Errorf("decompressed size exceeds %d-byte limit (adjust MaxDecompressedSize)", MaxDecompressedSize)
    }
    return nil
}

The aggregate cap across all ZIP entries is enforced in ExtractArchive by tracking total and computing a perEntryLimit = MaxDecompressedSize - total for each entry — see the full code in miniz-go/miniz.go. A multi-entry bomb that stays under the per-entry threshold still hits the aggregate ceiling.

The global cap is a shared variable

MaxDecompressedSize is a package-level var, readable by all goroutines without a lock. The doc comment in the source says: "It must be set before any concurrent decompression begins; it is read without synchronization, so mutating it while a decompress is in flight is a data race." Set it once at startup and leave it alone.

In plain terms: a package-level var is a value declared once for the whole program, so every part shares the same one (rather than each function keeping its own copy). Goroutines are Go's lightweight units of work that run at the same time — "running many things at once" is concurrency. A lock (often called a mutex) lets only one goroutine touch shared data at a time, so two don't collide. A data race is what happens when two of them read and change the same memory at once with no coordination: the result becomes unpredictable.

---

UnmarshalStream and MarshalStream — JSON over a Reader/Writer

cjson-go/cjson.go shows the same pattern for JSON. The all-in-memory variants take []byte:

// from cjson-go/cjson.go
func Unmarshal(data []byte, v interface{}) error {
    if len(data) == 0 {
        return errors.New("empty JSON data")
    }
    if err := json.Unmarshal(data, v); err != nil {
        return fmt.Errorf("unmarshal error: %w", err)
    }
    return nil
}

In plain terms: this function takes a whole block of JSON text already sitting in memory as data, and fills in the value v with what it parsed from that text, reporting an error if the text is empty or malformed. The streaming variants take io.Reader / io.Writer:

// from cjson-go/cjson.go
func UnmarshalStream(r io.Reader, v interface{}) error {
    decoder := json.NewDecoder(r)
    if err := decoder.Decode(v); err != nil {
        return fmt.Errorf("stream unmarshal error: %w", err)
    }
    return nil
}

func MarshalStream(w io.Writer, v interface{}) error {
    encoder := json.NewEncoder(w)
    if err := encoder.Encode(v); err != nil {
        return fmt.Errorf("stream marshal error: %w", err)
    }
    return nil
}

In plain terms: UnmarshalStream reads JSON text directly from a stream (rather than requiring it all in memory first) and fills in v; MarshalStream does the reverse, writing v back out as JSON text directly into a stream. json.NewDecoder reads from the io.Reader incrementally (a little at a time, as it goes, rather than all at once). You can pass it an HTTP response body directly, and the JSON parser reads from the network as it goes — no need to slurp the whole body first.

UnmarshalStream does not impose a size limit

The doc comment in the source is explicit: "It does not impose a size limit, so for untrusted input callers MUST wrap r in an io.LimitReader (or http.MaxBytesReader) to bound memory." Always wrap before passing an untrusted reader:

limited := io.LimitReader(resp.Body, 4<<20) // 4 MiB
err := cjsongo.UnmarshalStream(limited, &result)

---

LoadStream — streaming image decode with a header peek

stb-image-go/stb_image.go shows a slightly more involved pattern: it needs to read the image header (to check dimensions) and then re-read the full stream for decoding — but it has only one io.Reader, and reading consumes bytes (once you've read a byte from a stream, it's gone from that stream — there's no rewinding without special help, which is the problem this pattern solves).

// from stb-image-go/stb_image.go
func LoadStream(r io.Reader) (image.Image, error) {
    if r == nil {
        return nil, errors.New("nil reader")
    }
    if MaxImagePixels > 0 {
        // Tee the header bytes consumed by DecodeConfig into a buffer so the full
        // image can still be decoded (DecodeConfig reads only the header).
        var header bytes.Buffer
        cfg, _, cfgErr := image.DecodeConfig(io.TeeReader(r, &header))
        if cfgErr == nil && cfg.Width > 0 && cfg.Height > 0 &&
            int64(cfg.Width)*int64(cfg.Height) > int64(MaxImagePixels) {
            return nil, fmt.Errorf("image %dx%d exceeds the %d-pixel decode limit (adjust MaxImagePixels)",
                cfg.Width, cfg.Height, MaxImagePixels)
        }
        // Replay the consumed header, then the remainder of the stream.
        r = io.MultiReader(&header, r)
    }
    img, _, err := image.Decode(r)
    // ...
}

In plain terms: the function first peeks at just the image's header (a small chunk at the start that describes its width and height) to reject anything absurdly large, while simultaneously saving a copy of those header bytes; then it glues that saved copy back onto the front of the stream so the real decoder can read the file from the very beginning, as if nothing had been peeked at all. Three io helpers work together here:

Helper	Role
io.TeeReader(r, &header)	Reads from r; every byte also goes into header
bytes.Buffer (header)	Captures the bytes DecodeConfig consumed
io.MultiReader(&header, r)	Presents the captured header, then the remaining stream, as one Reader

The result: image.DecodeConfig gets the header; the full image.Decode sees the complete stream from the beginning, with no seeking and no extra allocation of the full body.

---

When to use streaming vs. all-in-memory

Situation	Prefer
Small, known-size input (< a few MB)	[]byte all-in-memory — simpler code
Large or variable-size input	io.Reader streaming — constant memory
Must check limits before allocating	io.LimitReader or header-peek pattern
Output destination varies (file, network, buffer)	io.Writer — caller picks the destination
Processing each JSON item independently	UnmarshalArrayParallel with MaxArrayItems cap

---

Try it

Run the miniz streaming round-trip test to see CompressStream / DecompressStream in action:

cd miniz-go && go test -v -run TestStream

Expected outcome: the test compresses a known string, decompresses it back through DecompressStream, and asserts the result equals the original. You should see PASS with no allocation complaints. If you want to confirm the limit fires, also run:

go test -v -run TestDecompress

which exercises DecompressData with readAllLimited and checks that oversized output is rejected.

Compose for free

Because all of these functions speak io.Reader/io.Writer, you can stack them at the call site without touching library code:

// Read from an HTTP response, cap at 8 MiB, decompress on the fly,
// write into a file — no intermediate allocation.
limited := io.LimitReader(resp.Body, 8<<20)
if err := minizgo.DecompressStream(outFile, limited); err != nil {
    log.Fatal(err)
}

---

Key takeaways

• io.Reader and io.Writer are the universal language of byte streams in Go; any concrete source or sink can satisfy them with one method each.
• bytes.NewReader and bytes.Buffer bridge the []byte world to the streaming world — they appear in almost every file in this repo.
• io.Copy moves data in fixed-size chunks; it never loads the full stream into memory and is the right default for large transfers.
• io.LimitReader is a one-line safety check at the trust boundary; wrap any untrusted reader with it before passing it downstream — UnmarshalStream in cjson-go/cjson.go explicitly documents this requirement.
• When you need to peek at a stream header without consuming it, combine io.TeeReader (to capture bytes as they're read) with io.MultiReader (to replay them) — the pattern used by LoadStream in stb-image-go/stb_image.go.