Repository navigation
encoding/json/jsontext: add errors to Token accessors for numbers #77666
Description
Proposal Details
This was proposed in go-json-experiment/json#158 . An implementation of this proposal is also present there.
Background
I was using jsontext to implement a Unmarshaler for my type. But I found the The current jsontext.Token APIs about number parsing are confusing and easy to misuse.
See also #63397 (comment)
At first glance, it seems very natural to use Token.Float() in the custom unmarshaler. However, it is not.
Specifically, I'm not satisfied with the following behavior (and cannot be disabled):
token, _ := jsontext.NewDecoder(bytes.NewReader([]byte("1e500"))).ReadToken()
token.Float() // returns 1.7976931348623157e+308token, _ := jsontext.NewDecoder(bytes.NewReader([]byte(`"Infinity"`))).ReadToken()
token.Float() // returns +InfThese are not standard, and not consistent with the json package itself. But a new user of jsontext package may not realize this, and finally implement an undesired unmarshaler. e.g. unmarshal {"total":1e1000,"used":1e500} into
type Amount struct {
Total float64 `json:"total"`
Used float64 `json:"used"`
}Total - Used could return 0 silently if the unmarshaler is using Token.Float()
So I think Token.Float(), etc. are not suitable for custom unmarshaler.
Proposed API Changes
Separate API for decoding and encoding
flowchart LR
Decoder -->|ReadToken| RawToken -->|jsontext.Raw| Token -->|WriteToken| Encoder
RawToken -->|ParseInt| int((int64)) -->|jsontext.Int| Token
Added:
// RawToken is similar to [Token], and is returned by [Decoder.ReadToken].
//
// Use [Raw] to convert it to [Token] for [Encoder.WriteToken].
type RawToken struct {...}
func (RawToken) ParseFloat(bits int) (float64, error)
func (RawToken) ParseInt(bits int) (int64, error)
func (RawToken) ParseUint(bits int) (uint64, error)
// Raw wraps a [RawToken] as [Token], for passing through the freshly decoded token to [Encoder].
func Raw(RawToken) Token
// Raw returns the [RawToken] embedded.
// It panics if the token is not created with [Raw].
func (t Token) Raw() RawToken {ParseXxx functions should work like strconv.ParseXxx. The only difference is that it only needs to support JSON format, so that the implementation may be more efficient. Compared with the current Token.Float() etc., it will not silently exhibit any non-standard behavior:
- It will not parse
"Infinity"as float value Infinity (error should be returned instead) - It will not parse
1e500asmath.MaxFloat64ormath.MaxInt64(error should be returned instead) - It will not truncate fractional component when parsing int (error should be returned instead)
Changed: Decoder.ReadToken should now return RawToken instead of Token
// ReadToken reads the next [RawToken], advancing the read offset.
// The returned token is only valid until the next Peek, Read, or Skip call.
// It returns [io.EOF] if there are no more tokens.
func (d *Decoder) ReadToken() (RawToken, error)Behavior changes to existing APIs:
When passing an Infinity, -Infinity, NaN float token to Encoder.WriteToken, it should now return an error, rather than writing JSON string "Infinity", "-Infinity", "NaN", which is not standard.
The following functions are greatly simplified to only act as accessors to Token values passed to corresponding constructor. Much like reflect.Value.Float()
// Float returns the floating-point value for a JSON number.
// It panics if the token is not created with [Float].
func (Token) Float() float64
// Int returns the signed integer value for a JSON number.
// It panics if the token is not created with [Int].
func (Token) Int() int64
// Uint returns the unsigned integer value for a JSON number.
// It panics if the token is not created with [Uint].
func (Token) Uint() uint64As a result of this change, only the passthrough use case will become a little more complex (need enc.WriteToken(jsontext.Raw(tok)), rather than enc.WriteToken(tok). I think this use case is very rare.
Other currently supported patterns that are disabled by this proposal, such as jsontext.Float(123.456).Int(), should not have valid use cases, IMO.
RawToken returned by decoder does not have Float() method, so it is impossible to misuse.
Convertion between JSON string "Infinity", "-Infinity", "NaN" and Go float is now an encoding/json feature exclusively (controlled by nonfinite format option). jsontext package will not do such convertion. User can write his own convert function easily.
Alternatives
Use strconv
tok, _ := dec.ReadToken()
f, err := strconv.ParseFloat(tok.String(), 64)It will take non-trivial efforts to realize that JSON number is a subset of Go float. This usage is very hard to discover.
And we cannot take the advantage of that json-float is actually simpler than go-float (in the future).
Add ParseFloat method to Token
ParseXxx is actually only useful for decoding. But if added to existing Token type, we will be forced to determine what to do with jsontext.Float(123.456).ParseInt().
cc @dsnet
Activity
gabyhelp commentedon Feb 18, 2026
Related Issues
Related Code Changes
(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)
prattmic commentedon Feb 18, 2026
cc @dsnet
neild commentedon Mar 12, 2026
We discussed this in the JSONv2 working group.
Consensus is that it is an error to convert a JSON number to a type, when the number cannot be represented by that type. For example, it is an error to convert "1.5" to an int64, or "1e500" to a float64.
Decoder.ReadTokenreturns a token and an error, but it does not parse numbers. This means that it may return a token which is a syntactically valid JSON number, but which cannot be represented by a float64.Token.Float,Token.Int, andToken.Uintall parse a JSON number into a Go type. This parse operation can fail. Since it can fail, these methods should return an error to report failures.We therefore feel we should change these methods to:
Token.Float,Token.Int, andToken.Uintwill return an error whenstrconv.ParseFloat,strconv.ParseInt, andstrconv.ParseUintwould return an error, respectively.On an out-of-range error,
Token.Floatwill return+Infor-Infas well as an error, matchingstrconv.Float.Token.IntandToken.Uintwill return the same values they do today as well as an error.Any use case which doesn't care about errors can just ignore the error and get the same results as today, with the exception that out-of-range floats will return infinity rather than saturating the float64 range.
Parse errors will wrap the
strconverror, and can be tested witherrors.Is(err, strconv.ErrRange)or similar.Token.Float,Token.Int, andToken.Uintwill still panic if used with a non-number token.Other
Tokenmethods, such asToken.Bool, will not return errors, becauseReadTokenwill never return a boolean token which cannot be converted into a Gobool.Tokenaccessors only return errors when the set of valid JSON values for a token kind is larger than the set of valid Go values for the same kind.json.Unmarshalwill return an error if you try to unmarshal an out-of-range number such as1e500into afloat64field.apparentlymart commentedon Mar 13, 2026
All of what's in the previous comment is reasonable and unsurprising to me, except for one detail:
The comment seems to be mainly focused on float values whose magnitude is out of range, i.e. if the required exponent is too large to represent.
Is the intention to also return an error if conversion to float would lose precision?
I ask because I wrote a library which had a similar need to this and I had originally made a decision that I'd treat both magnitude and precision in the same way, returning an error if either would not fit. But I found that didn't work out so well in practice because the JSON serialization of numbers is in decimal while the
float64representation is binary, and so it failed whenever a finite decimal representation needed an infinitely-sized binary representation.The compromise I settled on was to accept silently losing precision, producing the best possible binary approximation of the given decimal number. That matched the expectations of users of my library well enough, but
encoding/jsonhas a broader set of use-cases so perhaps that compromise isn't appropriate here.A very general solution would be to follow the lead of
math/bigand return something equivalent tobig.Accuracyfrom theFloatmethod, but that seems excessive for a function that also needs to return an error. Maybe it'd work to return the most accurate possible representation along with a matchable error so that callers can decide for themselves whether to tolerate the inaccurate representation. It could be the same matchable error as when the needed exponent is too large, and then the caller could test whether the returned value is an infinity or not to distinguish the two situations. 🤔neild commentedon Mar 13, 2026
For floats, we intend to match
strconv.ParseFloat.ParseFloatreturns an error for out-of-range values, but not for loss of precision, soToken.Floatwill do the same.[-]proposal: encoding/json/jsontext: separate number parsing API[/-][+]proposal: encoding/json/jsontext: add errors to Token accessors for numbers[/+][-]proposal: encoding/json/jsontext: add errors to Token accessors for numbers[/-][+]encoding/json/jsontext: add errors to Token accessors for numbers[/+]gopherbot commentedon Apr 29, 2026
Change https://go.dev/cl/772360 mentions this issue:
encoding/json/jsontext: report errors for numeric Token accessorsgopherbot commentedon May 4, 2026
Change https://go.dev/cl/773961 mentions this issue:
encoding/json/jsontext: use custom wrapper type for Token accessor errorsencoding/json/jsontext: use custom wrapper type for Token accessor er…