Handling Unknown JSON Fields in Go Without Losing Data

A Go struct describes the keys you expect. What the decoder does with the keys you did not expect is a separate question with three reasonable answers: ignore them, reject them, or keep them. The default is to ignore, and it is the right default for some programs and a source of quiet bugs in others.

I wrote the same small user type three ways and tried inputs that carry extra members, to see what each choice costs. The code is short, and the interesting parts are the places where the choices interact with custom decoders.

The default: unknown members vanish

var k Known
json.Unmarshal([]byte(in), &k)
out, _ := json.Marshal(k)
// {"name":"Ana","plan":"pro","seats":3}

beta and flags were gone, and nothing said so. For a service that only reads documents this is fine, and it is what lets a producer add fields without breaking you. For a service that reads a document, changes a field and writes it back, it is data loss: the next writer sees a document without beta. This is how a client with an older model strips fields added by a newer one, and it is why event consumers that forward messages should treat unknown fields as something to pass through (event payloads in JSON and schema evolution has the compatibility matrix).

The same default hides typos. A key spelled plna was accepted and the field stayed empty:

json.Unmarshal([]byte(`{"name":"Ana","plna":"pro"}`), &k3)
// k3.Plan == ""

For a configuration file this is the worst case: the program starts, uses the default plan and nobody knows why.

Rejecting: DisallowUnknownFields

The decoder has a switch for strictness:

d := json.NewDecoder(strings.NewReader(`{"name":"Ana","plna":"pro"}`))
d.DisallowUnknownFields()
err := d.Decode(&k)
// json: unknown field "plna"

The typo now fails and names the key. Unmarshal, the function, has no such option; you need a Decoder. It also applied through an embedded struct:

type Outer struct {
    Known
    Tag string `json:"tag"`
}
// {"name":"A","tag":"t","seats":1,"zzz":1}  ->  json: unknown field "zzz"

Use it for configuration, for request bodies where a stray field means a client bug, and in tests that decode fixtures (a fixture with a field the model lacks usually means the model fell behind). Do not use it for a consumer of events or of a third-party API, where a new optional field would then take you down. The error returns one key at a time, so a request with three typos reports only the first. If the client needs the full list, decode into a map and compare the keys yourself.

Keeping: capture the rest

To pass unknown members through, you need somewhere to put them. The pattern is a type with a custom UnmarshalJSON that decodes twice:

type User struct {
    Known
    Extra map[string]json.RawMessage `json:"-"`
}

func (u *User) UnmarshalJSON(b []byte) error {
    if err := json.Unmarshal(b, &u.Known); err != nil {
        return err
    }
    var all map[string]json.RawMessage
    if err := json.Unmarshal(b, &all); err != nil {
        return err
    }
    for _, k := range []string{"name", "plan", "seats"} {
        delete(all, k)
    }
    if len(all) > 0 {
        u.Extra = all
    }
    return nil
}

The first pass fills the declared fields. The second reads every member as raw bytes, deletes the known keys and keeps the rest. On the sample input it held two keys, and a matching MarshalJSON that merges Extra back into the output produced:

{"beta":true,"flags":{"x":[1,2]},"name":"Ana","plan":"pro","seats":3}

Nothing was lost. Two things changed: the keys came out in alphabetical order, because the merge went through a map and Marshal sorts map keys, and the cost is two parses of every document. For a hot path, measure it. The deleted-keys list also duplicates the struct's tags, which is a way to introduce a bug when a field is added. Derive it from the struct with reflection if you do this in many places.

Strict and custom do not combine

The last test is the one that surprised me. A decoder with DisallowUnknownFields applied to the User type above accepted an unknown member:

d := json.NewDecoder(strings.NewReader(`{"name":"Ana","zzz":1}`))
d.DisallowUnknownFields()
var u User
d.Decode(&u)       // err == nil, len(u.Extra) == 1

The strictness setting belongs to the decoder, and a custom UnmarshalJSON receives only bytes. Inside it, the call to json.Unmarshal is a fresh, lenient decode, so the setting does not reach it. Any type with its own UnmarshalJSON is a gap in a strict decode, which includes any wrapper type your code defines. The unknown-field check runs only for the fields that the standard decoder itself fills. If you rely on strictness, test it with a type that has a custom decoder in the middle, and do the check inside the custom method with a new Decoder that has the flag set.

Testing that strictness holds

Because the strict setting can be bypassed by a custom decoder, and because a typo in a struct tag fails silently, the test that matters is a negative one. For every type that is meant to be strict, keep a fixture with one extra key and assert that decoding it returns an error. For a type with a custom UnmarshalJSON, include a second fixture that puts the unknown key inside the part the custom method handles. These tests are short and cheap, and they catch the two ways this goes wrong: the day someone adds a custom decoder to a strict type, and the day a tag is misspelled so a field never receives its key.

A similar test for tolerant types is worth having. It decodes a fixture with extra members and checks that the known fields are still filled, so an upgrade to strict mode does not slip in without a decision. Name the fixtures after the behavior, unknown_member_rejected and unknown_member_ignored, and the next reader can see which choice was made on purpose.

Choosing

Where the choice is strict, decode into a struct for the types and still validate ranges and lengths afterwards (validating request JSON before it reaches the database). For decoding basics and tags, see parsing JSON in Go with struct tags. The JSON to Go converter generates a first struct from a sample. The Decoder methods are documented at pkg.go.dev/encoding/json#Decoder, and the Go team's notes on a rewrite of the package, including unknown member handling, are in the encoding/json/v2 proposal discussion.

When ignoring unknown fields is the right default

You are a consumer of someone else's data. A provider adds a field; your program should keep working. This is the whole reason for the default.

The struct is a view. If you read a few fields from a large document and never write it back, extra members cost nothing and are not your concern.

Version skew is expected. In a rolling deploy two versions of a service exchange messages. The old one must ignore what the new one adds.

The input is validated elsewhere. If a schema check ran before decoding, it has already rejected what it does not allow, and the decoder does not need to repeat the job.