05 · Tokenizing text: the jsmn JSON tokenizer¶

Objectives: Understand the difference between a tokenizer and a full parser, and see how jsmn-go represents a JSON document as a flat array of tokens linked by parent indices — with zero per-token heap allocation during the walk. Learn how to use the Parser, read token metadata, and choose the right Config for untrusted input. Estimated time: 20 minutes.

What this actually means (plain English)¶

No jargon — here's what the ideas in this lesson actually mean, and why they matter.

Tokenizer vs parser = "a surveyor who marks where the rooms are on a blueprint, versus the contractor who actually builds them." jsmn-go only records where each JSON value starts and ends in the raw bytes; it does not build objects, fill structs, or validate business rules — encoding/json.Unmarshal is the contractor that works on top of that blueprint.
Flat array as a tree = "a family-tree on an index card, where each person's card says their parent's card number, not a mobile hanging from the ceiling." jsmn-go fills one pre-allocated []Token slice; the hierarchy is encoded entirely in each token's ParentIdx integer, so no heap node is allocated per JSON value.
Start/End are byte offsets, not copies = "a sticky note that says 'the treasure is between pages 7 and 12' — it doesn't photocopy the pages." The Token holds Start and End indices so you slice the original []byte yourself; no string is ever duplicated in memory.
Size is child count, not byte count = "the headcount at a dinner table, not the table's width in centimetres." For an Object or Array token, Size tells you how many direct children it has, which lets you pre-size a map or loop without scanning the slice again; for String or Primitive tokens it is always 0.
Config is a safety fuse = "the circuit breaker in your electrical panel — you pick the amperage before you plug anything in." DefaultConfig caps input at 100 MB and 1 000 000 tokens; StrictConfig tightens those to 10 MB / 100 000; the tokenizer returns a sentinel error (ErrInputTooLarge, ErrTooManyTokens) if untrusted input trips the breaker.
Why it matters: Tokenizing without allocating per-value is the foundation of fast, memory-bounded JSON processing. Every other module in this workspace (cjson-go, cgltf-go) sits on top of this idea.

See it — a tree with no nodes. The input below tokenizes into one flat []Token. The tree shape is not a graph of allocated objects — it lives entirely in each token's ParentIdx (the curved arrows). Start/End are byte offsets into the original input, so no value is ever copied.

The Token struct¶

All the information jsmn-go records about one JSON value lives in four fields. A "field" here just means one named slot of data bundled together — and the bundle itself is called a struct (in plain terms: a struct is a labeled container that groups several related pieces of data under one name, so you can pass them around as a single unit instead of separately). From jsmn-go/jsmn.go:

type Token struct {
    Type      TokenType // Object | Array | String | Primitive
    Start     int       // byte offset of the first character (inclusive)
    End       int       // byte offset after the last character (exclusive)
    Size      int       // number of direct children (0 for strings/primitives)
    ParentIdx int       // index of the enclosing token in the flat slice (-1 = top-level)
}

In plain terms: this defines a Token shape — every token carries five labeled values: its Type (what kind of JSON thing it is), where it starts and ends in the raw text (as "byte offsets" — a byte is the basic unit of data a computer stores text in, so an offset is simply "count this many bytes from the start of the document"), how many children it has, and which other token contains it.

Start and End follow the usual Go half-open convention (Go is the programming language this project is written in — a "half-open" range includes its start point but stops just short of its end point, the same way "pages 7 up to but not including 12" would): json[tok.Start:tok.End] gives you the raw bytes of that value, including any surrounding quotes for strings. (Here json[a:b] is called "slicing" — in plain terms, it means "read out the piece of this data that sits between position a and position b, without copying or modifying the original data.")

The four TokenType values match the four structural elements of JSON:

Constant	Matches
`Object`	`{ … }`
`Array`	`[ … ]`
`String`	`"…"` (keys and string values)
`Primitive`	numbers, `true`, `false`, `null`

How the flat array forms a tree¶

Consider the tiny document:

{"name":"Alice","age":30}

After parsing, the token slice looks like this (indices 0–4):

idx  Type       Start  End  Size  ParentIdx
 0   Object       0    25    2      -1
 1   String       2     6    0       0     ← key "name"
 2   String       9    14    0       0     ← value "Alice"
 3   String      17    20    0       0     ← key "age"
 4   Primitive   23    25    0       0     ← value 30

In plain terms: each row is one token, and you can rebuild the whole nested document just by reading the ParentIdx column — token 1 ("name") and token 2 ("Alice") both point back to token 0 (the outer object), so you know they belong to it, without any tree of connected objects ever being built in memory.

No heap node was allocated for the object or any of its values. ("The heap" is a region of a program's memory set aside for data whose size or lifetime isn't known up front, as opposed to "the stack," which is a much cheaper region used for short-lived, fixed-size data. "Allocating" a node there means reserving a chunk of that memory for a brand-new piece of data — something jsmn-go deliberately avoids doing per token.) The whole tree is read from a single []Token slice that was filled in one pass over the input bytes. (A "slice" in Go is a resizable, ordered list of values — here, a list of Token structs sitting next to each other in memory.)

To extract the raw bytes of a token you just slice the original input:

// "Alice" — includes the surrounding quotes
raw := json[tokens[2].Start:tokens[2].End]

// strip quotes for a string token
content := json[tokens[2].Start:tokens[2].End] // e.g. `"Alice"`
unquoted := json[tokens[2].Start+1 : tokens[2].End-1] // e.g. `Alice`

In plain terms: both lines just read a small window of bytes out of the original input — the first keeps the quote marks, the second nudges the start and end points inward by one byte each to drop them.

Creating a Parser and walking tokens¶

The simplest path: NewParser + Parse — "calling" (or "invoking") a function means running it, and a function is a named, reusable block of instructions that does one job; you can hand it inputs and it hands back ("returns") a result when it's done. From jsmn-go/jsmn.go:

// NewParser creates a new parser with space for numTokens.
func NewParser(numTokens int) *Parser {
    return &Parser{
        tokens: make([]Token, numTokens),
    }
}

// Parse tokenizes the JSON input, returning the number of tokens or an error.
func (p *Parser) Parse(json []byte) (int, error) { … }

// Tokens returns the parsed tokens (slice up to toknext).
func (p *Parser) Tokens() []Token {
    return p.tokens[:p.toknext]
}

In plain terms: NewParser builds a fresh Parser with room for numTokens tokens already reserved; Parse reads the given JSON bytes and fills that space, handing back either how many tokens it found or an error; Tokens just returns the portion of that reserved space that was actually filled in.

A minimal usage pattern:

package main

import (
    "fmt"
    jsmngo "github.com/yourorg/safeheaders-go/jsmn-go"
)

func main() {
    input := []byte(`{"lang":"Go","year":2009}`)

    p := jsmngo.NewParser(32)
    n, err := p.Parse(input)
    if err != nil {
        panic(err)
    }

    for i, tok := range p.Tokens()[:n] {
        raw := input[tok.Start:tok.End]
        fmt.Printf("[%d] type=%-10s size=%d parent=%d  %s\n",
            i, tok.Type, tok.Size, tok.ParentIdx, raw)
    }
}

In plain terms: this is a complete, runnable program (a "package main" file that Go's tool will "compile" — turn from human-readable source text into a program the machine can execute — and then run). The import block pulls in outside code the program depends on: here, a package for printing text (fmt) and the jsmn-go package itself, referred to elsewhere in this lesson as a "package" or "module," which is just a named, reusable bundle of Go code someone else wrote. The program builds a Parser with room for 32 tokens, parses a small JSON string, and then loops over each token it found, printing its index, type, size, parent, and raw bytes.

Running this prints one line per token, making the flat-tree structure visible.

Walking the tree by parent index¶

Because every token records its parent's index (its position, or "index," inside the token slice — the first item is index 0, the second is index 1, and so on), you can filter children of any node with a plain loop — no recursion, no stack. ("Recursion" is when a function calls itself to solve a smaller piece of the same problem; a "stack," in this sense, is the bookkeeping the computer would normally need to remember where to return to at each nested call. Because the parent links already exist, none of that bookkeeping is needed here.)

// direct children of token at parentIdx
func directChildren(tokens []jsmngo.Token, parentIdx int) []int {
    var children []int
    for i, tok := range tokens {
        if tok.ParentIdx == parentIdx {
            children = append(children, i)
        }
    }
    return children
}

In plain terms: this function walks every token in order, and whenever a token's ParentIdx matches the one you asked about, it adds that token's index to a growing list; once it's checked everything, it hands that list back to whoever called it.

For a deeply nested document this linear scan can be replaced by a single pass that builds an adjacency list — still no extra allocations on the token side.

Key/value pairing in objects

In jsmn-go, object keys and their values appear consecutively in the token slice: key at index i, value at i + 1. Both have the same ParentIdx (the object). You can iterate object fields with a step of 2:

for i := 1; i < len(tokens); i += 2 {
    key   := input[tokens[i].Start+1 : tokens[i].End-1]   // strip quotes
    value := input[tokens[i+1].Start : tokens[i+1].End]
    fmt.Printf("%s => %s\n", key, value)
}

In plain terms: this loop skips through the tokens two at a time — treating each pair as one key and the value right after it — and prints them as key => value.

This works because the parser emits tokens in document order.

Choosing a Config¶

For production code that accepts user-supplied JSON, use ParseWithConfig from jsmn-go/config.go instead of the bare Parse method.

// DefaultConfig: 100 MB input, 1 000 000 tokens
func DefaultConfig() *Config { … }

// StrictConfig: 10 MB input, 100 000 tokens — for untrusted input
func StrictConfig() *Config { … }

// UnlimitedConfig: no caps — use only in benchmarks or controlled pipelines
func UnlimitedConfig() *Config { … }

In plain terms: these are three ready-made settings profiles you can hand to the parser — one with generous limits, one with tight limits for input you don't trust, and one with no limits at all. (A "benchmark," mentioned above, is simply a timed test that measures how fast or how much memory a piece of code uses — not a test of correctness. A "pipeline" is a sequence of processing steps where the output of one step feeds into the next.)

Usage:

import (
    "context"
    jsmngo "github.com/yourorg/safeheaders-go/jsmn-go"
)

tokens, err := jsmngo.ParseWithConfig(context.Background(), input, jsmngo.StrictConfig())
if err != nil {
    // could be ErrInputTooLarge, ErrTooManyTokens, ErrEmptyInput, or a parse error
    log.Fatalf("tokenize: %v", err)
}

In plain terms: this calls the safer parsing entry point with the strict limits applied, and if anything goes wrong — the input was too big, too complex, empty, or just malformed — it stops the program and prints the error.

The three sentinel errors let callers distinguish limit violations from malformed JSON: (a "sentinel error" is simply a specific, pre-defined error value the code deliberately returns so that calling code — "the caller" — can check exactly which problem occurred, rather than just knowing that something went wrong.)

switch {
case errors.Is(err, jsmngo.ErrInputTooLarge):
    http.Error(w, "payload too large", http.StatusRequestEntityTooLarge)
case errors.Is(err, jsmngo.ErrTooManyTokens):
    http.Error(w, "document too complex", http.StatusBadRequest)
case err != nil:
    http.Error(w, "invalid JSON", http.StatusBadRequest)
}

In plain terms: this checks which sentinel error came back and replies to the web request with the matching, specific HTTP status code instead of a generic failure.

UnlimitedConfig in production

UnlimitedConfig exists for benchmarks and for ParseParallel's internal fast-path. Passing it to a public HTTP handler means a 2 GB JSON blob will attempt to fill a token slice of that size — or exhaust heap and OOM (in plain terms: use up all of the memory the program is allowed to reserve and get shut down by the operating system — "OOM" stands for "out of memory"). Always prefer DefaultConfig or StrictConfig on the boundary.

How `allocToken` avoids a hard capacity limit¶

Early versions required you to pre-size the token slice exactly. The current implementation in jsmn-go/jsmn.go grows automatically:

func (p *Parser) allocToken(tok Token) error {
    if p.toknext >= len(p.tokens) {
        // grow instead of returning an error
        p.tokens = append(p.tokens, Token{})
    }
    p.tokens[p.toknext] = tok
    if p.toksuper != -1 {
        p.tokens[p.toksuper].Size++
    }
    p.toknext++
    return nil
}

In plain terms: whenever the reserved token space runs out, this function grows it with append (Go's built-in way of adding to the end of a slice, reserving more memory automatically if there isn't room) instead of failing; it then stores the new token, bumps its parent's child count, and moves the "next empty slot" pointer forward by one.

append doubles capacity when it needs to grow (standard Go amortisation — in plain terms, doubling the size each time means most append calls are nearly free, and only occasionally does one trigger an expensive reallocation, so the average cost per item stays low), so the total number of allocations is O(log n), not O(n) (this "Big-O" notation is just a shorthand for how the cost of an operation grows as the input grows larger — O(log n) grows far more slowly than O(n) as n increases). The Config.MaxTokens cap in ParseWithConfig is enforced after Parse returns, not inside allocToken — so the growth is still bounded by the config when you use the safe entry point.

Try it¶

Try it

From the repo root, run the jsmn-go test suite (a "test" is a small piece of code, written by the developers, whose only job is to check that another piece of code behaves the way it's supposed to — a "test suite" is the whole collection of these checks for a project):

cd jsmn-go && go test ./... -v -count=1

Expected outcome: all tests pass; output includes lines like --- PASS: TestParse, --- PASS: TestParseWithConfig, and --- PASS: TestParallelTokensMatchSerial confirming that parallel and serial tokenization produce identical token slices for the same input.

Try it — race detector

The parallel tokenizer uses goroutines (a goroutine is Go's lightweight unit of concurrent work — a piece of code that can run at the same time as other pieces, letting the program do several things "in parallel" instead of strictly one after another; this overall style is called concurrency). Verify no data races (a "data race" is a bug where two of these concurrently running pieces of code read and write the same piece of memory at the same time without coordinating, producing unpredictable results):

cd jsmn-go && go test ./... -race -count=1

Expected outcome: PASS with no DATA RACE reports. The parallel path in jsmn-go/parallel.go was designed to share no mutable state between workers; each chunk gets its own Parser instance.

Under the hood: how `Parse` handles closing brackets¶

When the parser sees } or ], it closes the current container by writing the current position into the open container token's End field, then climbs up via ParentIdx. From jsmn-go/jsmn.go:

case '}', ']':
    if p.toksuper != -1 {
        p.tokens[p.toksuper].End = p.pos + 1
        p.toksuper = p.tokens[p.toksuper].ParentIdx
    }
    p.pos++

In plain terms: when the parser hits a closing } or ], it records the current reading position as the end of whichever container is currently open, then moves its "which container am I in" tracker up to that container's parent — and finally advances one byte forward through the input.

p.toksuper is effectively a cursor pointing at the innermost open container. Opening a { or [ pushes a new token and updates toksuper to point at it. Closing } or ] seals that token and pops toksuper back to the parent — without any explicit stack allocation, because the parent chain is already encoded in ParentIdx.

At the end of Parse, any token whose End is still -1 (a top-level primitive that extends to the end of input) gets End = len(json), and the parser returns an error if any container is still open:

if p.toksuper != -1 {
    return 0, errors.New("unclosed object or array")
}

In plain terms: if, after reading the whole document, some container was never closed, the function returns an error saying so instead of returning a result — this is what "the function returns" means throughout this lesson: it stops running and hands a value back to whoever called it.

Lesson 12 — how the parallel path fans per-chunk parsers across goroutines and merges the token slices (including the ParentIdx rebasing fix that was a real production bug).
Lesson 14 — the channel-buffer deadlock that lurked inside the parallel worker pool, and the watchdog test that catches it.

Key takeaways¶

A tokenizer records where values are (byte offsets + type); it does not copy or interpret them. That is intentionally all jsmn-go does.
The flat []Token array encodes a full tree through ParentIdx integer links — zero per-node heap allocation during traversal.
Start/End are half-open byte offsets into the original input slice; slice them yourself, pay nothing extra.
Size on an Object or Array token is its direct child count, not its byte size — useful for pre-allocating a map or slice before you walk the children.
Always use ParseWithConfig with StrictConfig (or at least DefaultConfig) on any input you did not generate yourself; the sentinel errors (ErrInputTooLarge, ErrTooManyTokens) give you clean HTTP status codes.