| Age | Commit message (Collapse) | Author |
|---|
|
This reduce the struct encodeState size from 56 to 16 bytes (-42),
struct structFields size from 40 to 24 bytes (-16), and struct field size
from 136 to 128 bytes (-8).
Benchmark memory profiling before and after,
File: json.test
Type: alloc_space
Time: Jan 25, 2024 at 4:51am (WIB)
Showing nodes accounting for -4.01GB, 4.66% of 85.95GB total
Dropped 64 nodes (cum <= 0.43GB)
flat flat% sum% cum cum%
-4.25GB 4.94% 4.94% -4.01GB 4.67% encoding/json.Marshal
0.19GB 0.22% 4.73% 0.24GB 0.28% encoding/json.mapEncoder.encode
0.05GB 0.057% 4.67% 0.05GB 0.057% reflect.copyVal
0.04GB 0.049% 4.62% 0.04GB 0.049% bytes.growSlice
-0.04GB 0.045% 4.66% -0.04GB 0.045% encoding/json.RawMessage.MarshalJSON
0 0% 4.66% 0.04GB 0.046% bytes.(*Buffer).WriteString
0 0% 4.66% 0.04GB 0.049% bytes.(*Buffer).grow
0 0% 4.66% -0.04GB 0.045% encoding/json.(*Encoder).Encode
0 0% 4.66% 0.20GB 0.23% encoding/json.(*encodeState).marshal
0 0% 4.66% 0.20GB 0.23% encoding/json.(*encodeState).reflectValue
|
|
This reduce the UnmarshalTypeError size from 64 to 56 bytes (-8 bytes),
and decodeState from 128 to 96 (-32 bytes).
|
|
Instead of re-slicing the string for checking the value, use single
index variable.
Benchmark result,
name old time/op new time/op delta
NumberIsValid-8 19.0ns ± 0% 14.5ns ± 1% -23.70% (p=0.008 n=5+5)
name old alloc/op new alloc/op delta
NumberIsValid-8 0.00B 0.00B ~ (all equal)
name old allocs/op new allocs/op delta
NumberIsValid-8 0.00 0.00 ~ (all equal)
Change-Id: I4698c5db134998f83ff47fb3add6a04ba6ec3aa0
|
|
Due to the lack of MarshalWrite in the v1 API,
it is unfortunately common to see:
json.NewEncoder(out).Encode(in)
where a single-use Encoder is constructed and thrown away.
This performed acceptably in v1 since every call to Encode
used a globally pooled encoder resource.
Prior to this change, the v1-in-v2 implementation relied
on a bytes.Buffer cached only for the lifetime of the Encoder
object itself. Thus, a single-use Encoder does not benefit.
Modify the wrapper implementation to use the internal
pooled encoder from v2 and use the intermediate buffer
to write directly to the output io.Writer.
We assume that the user-provided io.Writer never leaks the buffer,
but this assumption was already held in the v1 implementation.
We are not increasing the surface area of data corruption risk.
Performance of v1 to v1-in-v2 (before the pool fix):
name old time/op new time/op delta
NewEncoderEncode-32 30.2ms ± 4% 28.3ms ± 9% -6.19% (p=0.002 n=9+10)
name old alloc/op new alloc/op delta
NewEncoderEncode-32 7.64MB ± 0% 28.37MB ± 0% +271.23% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
NewEncoderEncode-32 200k ± 0% 100k ± 0% -49.99% (p=0.000 n=9+10)
Interestingly, v1-in-2 is slightly faster,
but the amount of allocated memory is massive.
Performance of v1 to v1-in-v2 (after the pool fix):
name old time/op new time/op delta
NewEncoderEncode-32 30.2ms ± 4% 24.0ms ± 7% -20.36% (p=0.000 n=9+10)
name old alloc/op new alloc/op delta
NewEncoderEncode-32 7.64MB ± 0% 4.09MB ± 3% -46.52% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
NewEncoderEncode-32 200k ± 0% 100k ± 0% -50.00% (p=0.000 n=9+9)
Now, the v1-in-v2 implementation is better than v1 on all metrics.
Fixes #75026
Change-Id: I50c975b1d5b8da806e46bc627966b0a39c1817eb
Reviewed-on: https://go-review.googlesource.com/c/go/+/740660
Reviewed-by: Michael Pratt <mpratt@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
|
|
Change-Id: Id5520757e4d73e56e533e4de4f5f303105c4339e
Reviewed-on: https://go-review.googlesource.com/c/go/+/738180
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Auto-Submit: David Symonds <dsymonds@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: David Symonds <dsymonds@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Joseph Tsai <joetsai@digital-static.net>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
|
|
Followon for CL 721160.
Change-Id: I9c22c5e99c9084e24047c77d20717c5b46165cde
Reviewed-on: https://go-review.googlesource.com/c/go/+/732220
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Michael Pratt <mpratt@google.com>
Reviewed-by: Sean Liao <sean@liao.dev>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
|
|
The json/v2 working group decided to commit to no default representation
for time.Duration for the foreseeable future.
Thus, we can remove the issue reference in the error message.
If JavaScript (TC39) formally adopts the Temporal.Duration type,
which uses ISO 8601 as the JSON representation for a duration,
we may consider changing the default.
Switching from no default to some default in the future
is generally a compatible change.
Updates #71631
Change-Id: I6f71a0fa97dcefba56aab0c0ddf445d9d38f6e58
Reviewed-on: https://go-review.googlesource.com/c/go/+/735080
Reviewed-by: Michael Pratt <mpratt@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
Auto-Submit: Joseph Tsai <joetsai@digital-static.net>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
|
|
Add constants for each possible Kind value (KindNull, KindTrue, etc.).
Leave the values unchanged ('n', 't', etc.).
Update documentation to reference the symbols.
Fixes #71756
Change-Id: Ib33b2ad9ee55f6f547d9e6a1c5a7f00c8400d3d3
Reviewed-on: https://go-review.googlesource.com/c/go/+/728420
Auto-Submit: Damien Neil <dneil@google.com>
Reviewed-by: Joseph Tsai <joetsai@digital-static.net>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
|
|
For #75431
Change-Id: Iafefe952d3c1837e2f4c8c24cae96945d9e5abbf
Reviewed-on: https://go-review.googlesource.com/c/go/+/728380
Reviewed-by: Joseph Tsai <joetsai@digital-static.net>
Auto-Submit: Damien Neil <dneil@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
|
|
Historically, Decoder.More reports true when the next read will
return an error. Adjust the v2 Decoder to follow this behavior.
Fixes #76467
Change-Id: I03bfa391e4e89ada8ca869db43c1d0bb63cc0413
Reviewed-on: https://go-review.googlesource.com/c/go/+/728300
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Auto-Submit: Damien Neil <dneil@google.com>
Reviewed-by: Joseph Tsai <joetsai@digital-static.net>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
|
|
The linknames have been removed in the latest commits:
isValidNumber, typeFields:
https://github.com/bytedance/sonic/commit/908af5dfaf7d509db1f2c2b1c43f4df728db80a6
unquoteBytes:
https://github.com/bytedance/sonic/commit/8e9090a84be10962046197f1867d297d3e73020a
For #67401
Change-Id: I30b057137932fcf267014ab5470c417298765249
Reviewed-on: https://go-review.googlesource.com/c/go/+/721160
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Joseph Tsai <joetsai@digital-static.net>
Reviewed-by: Junyang Shao <shaojunyang@google.com>
Reviewed-by: Mark Freeman <markfreeman@google.com>
|
|
CL 708495 mass migrated the stdlib to use errors.AsType.
It rewrote the documentation, but uses the non-pointer type
of SemanticError or SyntacticError, which is invalid since
the Error method is declared on the pointer receiver.
Also, call errors.AsType on a separate line for readability.
Change-Id: I2eaf59614e2b58aa4bc8669ab81ce64168c54f13
Reviewed-on: https://go-review.googlesource.com/c/go/+/714660
Reviewed-by: David Chase <drchase@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Johan Brandhorst-Satzkorn <johan.brandhorst@gmail.com>
Auto-Submit: Joseph Tsai <joetsai@digital-static.net>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
|
|
Previously, we put a jsontext.Encoder (or Decoder)
back into a pool with minimal reset logic.
This was semantically safe since we always did a full reset
after obtaining an Encoder/Decoder back out of the pool.
However, this meant that so long as an Encoder/Decoder was
alive in the pool, any application data referenced by the coder
would be kept alive longer than necessary.
Explicitly, clear such fields so that application data
can be more aggressively garbage collected.
Performance:
name old time/op new time/op delta
Unmarshal/Bool-32 52.0ns ± 3% 50.3ns ± 3% -3.30% (p=0.001 n=10+10)
Marshal/Bool-32 55.4ns ± 3% 54.4ns ± 2% -1.75% (p=0.006 n=10+9)
This only impacts the performance of discrete Marshal/Unmarshal calls.
For the simplest possible call (i.e., to marsha/unmarshal a bool),
there is a 1-2ns slow down. This is an appropriate slowdown
for an improvement in memory utilization.
Change-Id: I5e7d7827473773e53a9dcb3d7fe9052a75481e9f
Reviewed-on: https://go-review.googlesource.com/c/go/+/713640
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Johan Brandhorst-Satzkorn <johan.brandhorst@gmail.com>
TryBot-Bypass: Damien Neil <dneil@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Auto-Submit: Damien Neil <dneil@google.com>
|
|
This is semantically identical and just a cleanup.
Prior to #63397, JSON object names were sorted according to UTF-16
to match the semantic of RFC 8785, but there were a number of
objections in the discussion to using that as the sorting order.
In https://github.com/go-json-experiment/json/pull/121,
we switched to sorting by UTF-8, which matches the behavior
of v1 and avoids an option to toggle the behavior.
However, we should have deleted the stringSlice.Sort method
and just directly called slices.Sort.
From a principled perspective, both UTF-16 and UTF-8 are
reasonable ways to sort JSON object names.
RFC 8259 specifies that the entire JSON text is encoded as UTF-8.
However, the way JSON strings are encoded requires escaping
Unicode codepoints according to UTF-16 surragate halves
(a quirk of JavaScript inherited by JSON).
Thus, JSON is inconsistently both UTF-8 and UTF-16.
Change-Id: Id92b5cc20efe4201827e9d3fccf24ccf894d3e60
Reviewed-on: https://go-review.googlesource.com/c/go/+/713522
Reviewed-by: Johan Brandhorst-Satzkorn <johan.brandhorst@gmail.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
TryBot-Bypass: Damien Neil <dneil@google.com>
Auto-Submit: Damien Neil <dneil@google.com>
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Reviewed-by: Damien Neil <dneil@google.com>
|
|
The jsontext package represents the location of JSON errors
using a JSON Pointer (RFC 6901). This uses the JSON type system.
Unfortunately the v1 json.UnmarshalTypeError assumes a Go struct-based
mechanism for reporting the location of errors
(and has historically never been implemented correctly since
it was a weird mix of both JSON and Go namespaces; see #43126).
Trying to map a JSON Pointer into UnmarshalTypeError.{Struct,Field}
is difficult to get right without teaching jsontext
about the Go type system.
To reduce the probability of misleading errors,
check whether the last token looks like a JSON array index
and if so, elide the phrase "into Go struct field".
Fixes #74801
Change-Id: Id2088ffb9c339a9238ed38c90223d86a89422842
Reviewed-on: https://go-review.googlesource.com/c/go/+/710676
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
|
|
Originally, DefaultOptionsV1 and DefaultOptionsV2 represented
the full set of all options with specific ones set to true or false.
However, there are certain options such as WithIndent or WithMarshalers
that are neither v1 or v2 specific.
At some point we removed whitespace related options from the set:
https://github.com/go-json-experiment/json/pull/26
This avoids DefaultOptionsV1 or DefaultOptionsV2 from affecting
any previously set whitespace. However, why are whitespace options
special and thus excluded from the set? What about Marshalers?
As a more principaled way to address this, we restrict
DefaultOptionsV1 and DefaultOptionsV2 to only be the options
where the default setting changes between v1 and v2.
All other options are unpopulated.
This avoids a panic with GetOption(DefaultOptionsV2, WithMarshalers)
since DefaultOptionsV2 previously had the presence bit for
Marshalers set to true, but had no actual value.
Now, the presence bit is set to false, so the value is not consulted.
Fixes #75149
Change-Id: I30b45abd35404578b4135cc3bad1a1a2993cb0cf
Reviewed-on: https://go-review.googlesource.com/c/go/+/710878
Reviewed-by: Johan Brandhorst-Satzkorn <johan.brandhorst@gmail.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
|
|
The fully streaming UnmarshalJSONFrom method and UnmarshalFromFunc
introduce an edge case where they can encounter EOF in the stream,
where it should be reported upstream as EOF rather than
ErrUnexpectedEOF or be wrapped within a SemanticError.
This is not possible with other unmarshal methods since the
"json" package would read the appropriate JSON value
before calling the custom method or function.
To avoid custom unmarshal methods from encountering EOF,
check whether the stream is already at EOF for top-level values
before calling the custom method.
Also, when wrapping EOF within a SemanticError, convert it
to ErrUnexpectedEOF to better indicate that this is unexpected.
Fixes #75802
Change-Id: I001396734b7e95b5337f77b71326284974ee730a
Reviewed-on: https://go-review.googlesource.com/c/go/+/710877
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Johan Brandhorst-Satzkorn <johan.brandhorst@gmail.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
Reviewed-by: Carlos Amedee <carlos@golang.org>
|
|
This change replaces most occurrences (in code as well as in comments) of
errors.As with errors.AsType. It leaves the errors package and vendored
code untouched.
Change-Id: I3bde73f318a0b408bdb8f5a251494af15a13118a
GitHub-Last-Rev: 8aaaa36a5a12d2a6a90c6d51680464e1a3115139
GitHub-Pull-Request: golang/go#75698
Reviewed-on: https://go-review.googlesource.com/c/go/+/708495
Auto-Submit: Michael Pratt <mpratt@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Alan Donovan <adonovan@google.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
|
|
The legacy parsing of quoted numbers in v1 was according to
the Go grammar for a number, rather than
the JSON grammar for a number.
The former is a superset of the latter.
This is a historical mistake, but usages exist that depend on it.
We already have branches for StringifyWithLegacySemantics
to handle quoted nulls, so we can expand it to handle this.
Fixes #75619
Change-Id: Ic07802539b7cbe0e1f53bd0f7e9bb344a8447203
Reviewed-on: https://go-review.googlesource.com/c/go/+/709615
Reviewed-by: Damien Neil <dneil@google.com>
Auto-Submit: Joseph Tsai <joetsai@digital-static.net>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
|
|
The Decoder.InputOffset method was always ambiguous about the exact
offset returned since anything between the end of the previous token
to the start of the next token could be a valid result.
Empirically, it seems that the behavior was to report
the end of the previous token unless Decoder.More is called,
in which case it reports the start of the next token.
This is an odd semantic since a relatively side-effect free method
like More is not quite so side-effect free.
However, our goal is to preserve historical v1 semantic when possible
regardless of whether it made sense.
Note that jsontext.Decoder.InputOffset consistently always reports
the end of the previous token. Users can explicitly choose the
exact position they want by inspecting the UnreadBuffer.
Fixes #75468
Change-Id: I1e946e83c9d29dfc09f2913ff8d6b2b80632f292
Reviewed-on: https://go-review.googlesource.com/c/go/+/703856
Reviewed-by: Damien Neil <dneil@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Junyang Shao <shaojunyang@google.com>
|
|
Use the correct spelling of the Gregorian calendar.
Change-Id: I7e1d2974d38d5d3ded64f98852ee726c7b84be22
Reviewed-on: https://go-review.googlesource.com/c/go/+/704595
Reviewed-by: Joseph Tsai <joetsai@digital-static.net>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
Auto-Submit: Emmanuel Odeke <emmanuel@orijtech.com>
|
|
When the json package calls
Marshaler, MarshalerTo, Unmarshaler, or UnmarshalerFrom methods
and a SemanticError is returned, it will automatically
annotate the error with context.
Document this behavior.
Change-Id: Id8e775a7c1c2a6ffc29ea518913591915e8aff87
Reviewed-on: https://go-review.googlesource.com/c/go/+/701956
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
|
|
Updates #62121
Change-Id: Ic3c4fe84a5dacfd8270aba0d5dd59f83f0a9030f
Reviewed-on: https://go-review.googlesource.com/c/go/+/701955
Reviewed-by: Mark Freeman <markfreeman@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
|
|
Flags struct has field Values but in the comments use Value.
Fix it to correct name Values.
Change-Id: Ib47e62538599a788c69fda27a7e2a97b8cf73263
Reviewed-on: https://go-review.googlesource.com/c/go/+/701415
Reviewed-by: Joseph Tsai <joetsai@digital-static.net>
Reviewed-by: Michael Pratt <mpratt@google.com>
Auto-Submit: Joseph Tsai <joetsai@digital-static.net>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
|
|
This change makes the fast path for ASCII characters inlineable in
DecodeRune and DecodeRuneInString and removes most instances of manual
inlining at call sites.
Here are some benchmark results (no change to allocations):
goos: darwin
goarch: amd64
pkg: unicode/utf8
cpu: Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz
│ old │ new │
│ sec/op │ sec/op vs base │
DecodeASCIIRune-8 2.4545n ± 2% 0.6253n ± 2% -74.52% (p=0.000 n=20)
DecodeJapaneseRune-8 3.988n ± 1% 4.023n ± 1% +0.86% (p=0.050 n=20)
DecodeASCIIRuneInString-8 2.4675n ± 1% 0.6264n ± 2% -74.61% (p=0.000 n=20)
DecodeJapaneseRuneInString-8 3.992n ± 1% 4.001n ± 1% ~ (p=0.625 n=20)
geomean 3.134n 1.585n -49.43%
Note: when #61502 gets resolved, DecodeRune and DecodeRuneInString should
be reverted to their idiomatic implementations.
Fixes #31666
Updates #48195
Change-Id: I4be25c4f52417dc28b3a7bd72f1b04018470f39d
GitHub-Last-Rev: 2e352a0045027e059be79cdb60241b5cf35fec71
GitHub-Pull-Request: golang/go#75181
Reviewed-on: https://go-review.googlesource.com/c/go/+/699675
Reviewed-by: Sean Liao <sean@liao.dev>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
|
|
There is no need to explicitly pass in the options
since this contained within the Encoder or Decoder struct
ever since https://github.com/go-json-experiment/json/pull/163.
Thus, remove it as an argument and fetch it from the coder.
This only modifies code that is compiled in under goexperiment.jsonv2.
Change-Id: I6c928b864bf7869889d7ee7d5c1d396fbe71296b
Reviewed-on: https://go-review.googlesource.com/c/go/+/695278
Reviewed-by: Damien Neil <dneil@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
|
|
There is a fast-path optimization for marshaling an any type
that should be semantically identical to when the optimization
is not active (i.e., optimizeCommon is false).
Unfortunately, the optimization accidentally allows NaN,
which this change fixes.
The source of this discrepency is that Encoder.WriteToken(Float(math.NaN()))
emits a JSON string with "NaN", rather than report an error.
The rationale for this behavior is because we needed to decide what to do
with Float(math.NaN()), whether it would return an error, panic, or allow it.
To keep the API simpler (no errors) and less sharp (no panics), we permitted NaN.
The fact that WriteToken allowed it is a logical extension of that decision,
but we could decide to disallow it at least within WriteToken.
As things stand, it is already inconsistent between json/v2 and jsontext, where
json/v2 rejects NaN by default in Marshal, but jsontext allows it in WriteToken.
This only modifies code that is compiled under goexperiment.jsonv2.
Fixes #74797
Change-Id: Ib0708cfbf93c2b059c0a85e4c4544c0604573448
Reviewed-on: https://go-review.googlesource.com/c/go/+/695276
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
|
|
The type reported in a ErrCycle is the wrong type due to a typo.
This discrepency was detected by setting optimizeCommon to false
and running the tests.
This only modifies code that is compiled in under goexperiment.jsonv2.
Change-Id: I68268f5c719d8b79a67424a35ed0647adf12288c
Reviewed-on: https://go-review.googlesource.com/c/go/+/695277
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
|
|
The Indent function preserves trailing whitespace,
while the v1 emulation under v2 implementation accidentally dropped it.
There was prior logic that attempted to preserve it,
but it did not work correctly since it ran in a defer and
accidentally mutated the dst input argument rather than the output argument.
Move the logic to the end and avoid a defer.
Also, add a test to both v1 and v1in2 to codify this behavior.
This only modifies code that is compiled in under goexperiment.jsonv2.
Updates #13520
Fixes #74806
Change-Id: I22b1a8da5185eb969e2a8a111b625d3752cfcbe8
Reviewed-on: https://go-review.googlesource.com/c/go/+/692195
Reviewed-by: Sean Liao <sean@liao.dev>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: David Chase <drchase@google.com>
Auto-Submit: Sean Liao <sean@liao.dev>
|
|
When EOF is encountered within jsontext.Decoder stream without
starting to parse any token, UnmarshalDecode should report EOF,
rather than converting it into ErrUnexpectedEOF.
This fixes a regression introduced by https://go.dev/cl/689919.
This change only affects code compiled under goexperiment.jsonv2.
Fixes #74835
Change-Id: I7e8e57ab11b462c422c538503ed8c6b91ead53bd
Reviewed-on: https://go-review.googlesource.com/c/go/+/692175
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Sean Liao <sean@liao.dev>
Reviewed-by: David Chase <drchase@google.com>
Reviewed-by: Jake Bailey <jacob.b.bailey@gmail.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Auto-Submit: Joseph Tsai <joetsai@digital-static.net>
|
|
The jsontext.Decoder.ReadToken method reports a non-EOF error,
if the token stream is truncated and does not form a valid JSON value.
In contrast, the v1 json.Decoder.Token method would report EOF
so long as the input was a prefix of some valid JSON value.
Modify json.Decoder.Token to preserve historical behavior.
This only modifies code that is compiled in under goexperiment.jsonv2.
Updates #69449
Fixes #74750
Change-Id: Ifd281c46f118f0e748076013fefc7659f77c56ed
Reviewed-on: https://go-review.googlesource.com/c/go/+/689516
Reviewed-by: Damien Neil <dneil@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Joseph Tsai <joetsai@digital-static.net>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
|
|
This does the equivalent of CL 681177 for the Encoder.
It preserves the internal buffer between resets.
Change-Id: I5e9353b6d7755e067d4f9a4d1ea3d8f056253027
Reviewed-on: https://go-review.googlesource.com/c/go/+/690375
Reviewed-by: Johan Brandhorst-Satzkorn <johan.brandhorst@gmail.com>
Auto-Submit: Joseph Tsai <joetsai@digital-static.net>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
|
|
When operating under v1 semantics in the v2 implementation,
a extra data error should take precedence over any semantic error
that could theoretically occur within the value itself.
This change only affects code compiled under goexperiment.jsonv2.
Fixes #74614
Change-Id: I055a606b053fa66b0c766ae205487b8290109285
Reviewed-on: https://go-review.googlesource.com/c/go/+/689919
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
Auto-Submit: Michael Knyszek <mknyszek@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
|
|
There were minor and unnecessary error text changes
when v1 was implemented using v2.
Reduce divergences if possible.
Of the cases reported in #74713, there are no more differences for:
v1: json: cannot unmarshal number into Go value of type chan int
v2: json: cannot unmarshal number into Go value of type chan int
and
v1: json: cannot unmarshal number into Go value of type error
v2: json: cannot unmarshal number into Go value of type error
However, there is a difference between:
v1: json: cannot unmarshal string into Go struct field .F.V of type int
v2: json: cannot unmarshal string into Go struct field S.F.V of type int
For reasons unclear, the v1 logic was always inconsistent about
whether it could properly record the root struct type,
while the v1 emulation layer under v2 is always able to.
This only modifies code that is compiled in under goexperiment.jsonv2.
Fixes #74713
Change-Id: I9e87323b1810130cb929288fdd86aff4be82d5f2
Reviewed-on: https://go-review.googlesource.com/c/go/+/689918
Reviewed-by: Damien Neil <dneil@google.com>
Auto-Submit: Joseph Tsai <joetsai@digital-static.net>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
|
|
The Decoder.Reset method is not preserving the internal buffer between
resets, causing buffer capacity to be lost and resulting in unnecessary
allocations when reusing decoders. This is particularly problematic when
decoding many small messages.
This commit fixes the Reset method to preserve the internal buffer. It
makes sure aliasing is removed if the buffer currently points to an
internal byte slice of a bytes.Buffer. It adds a TestDecoderReset test
structured into subtests to better validate the different scenarios.
Change-Id: Ia685bff47034598224489173bb7f2ffd48e89da5
GitHub-Last-Rev: 462ddc936409ee3da6d75def7d9de59eb3c65f8d
GitHub-Pull-Request: golang/go#74120
Reviewed-on: https://go-review.googlesource.com/c/go/+/681177
Reviewed-by: Sean Liao <sean@liao.dev>
Reviewed-by: Joseph Tsai <joetsai@digital-static.net>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
Auto-Submit: Joseph Tsai <joetsai@digital-static.net>
|
|
WARNING: This commit contains breaking changes
for those already using GOEXPERIMENT=jsonv2.
This decomposes FormatBytesWithLegacySemantics as:
* FormatBytesWithLegacySemantics
* FormatByteArrayAsArray
* ParseBytesWithLooseRFC4648
This decomposes FormatTimeWithLegacySemantics as:
* FormatDurationAsNano
* ParseTimeWithLooseRFC3339
In particular, it splits out specific behaviors from the option
that may need to be specified on a finer-grain level.
FormatByteArrayAsArray and FormatDurationAsNano are targeted
to just the default representation of a [N]byte or time.Duration type.
Both of these are not necessary if the `format` tag is explicitly specified.
However, we want to isolate their behavior from other behaviors that used to
be part of FormatBytesWithLegacySemantics and FormatTimeWithLegacySemantics.
ParseBytesWithLooseRFC4648 and ParseTimeWithLooseRFC3339 are targeted
to just historically buggy parsing according to the relevant RFCs,
which may need to be enabled by some services for backwards compatibility.
While FormatTimeWithLegacySemantics is deleted, we still need
FormatBytesWithLegacySemantics to configure highly esoteric
aspects of how v1 used to handle byte slices.
We rename OmitEmptyWithLegacyDefinition as OmitEmptyWithLegacySemantics
to be consistent with other options with the WithLegacySemantics suffix.
Updates #71497
Change-Id: Ic660515fb086fe3af237135f195736de99c2bd33
Reviewed-on: https://go-review.googlesource.com/c/go/+/685395
Auto-Submit: Joseph Tsai <joetsai@digital-static.net>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Johan Brandhorst-Satzkorn <johan.brandhorst@gmail.com>
|
|
This follows up CL 684315 with an expanded section in the v2 doc.
Updates #14750
Updates #71845
Change-Id: I1ffa97e030f5f2b709e8142028e3c8e0e38b80ce
Reviewed-on: https://go-review.googlesource.com/c/go/+/685195
Auto-Submit: Joseph Tsai <joetsai@digital-static.net>
Reviewed-by: Damien Neil <dneil@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
|
|
In the presence of invalid UTF-8, the AllowInvalidUTF8 option
allows such bytes to be present, but silently mangles them
using the Unicode replacement character.
The v2 default is to emit the replacement character verbatim
(which is valid UTF-8 and exactly what it is for).
However, the v1 behavior has historically been to emit
the escaped form of the replacement character.
This behavior was introduced in https://go.dev/cl/11211045
where the documentation says that it is:
replacing invalid bytes with the Unicode replacement rune U+FFFD
but the implementation actually replaces it with
the escaped form of the Unicode replacement rune.
Given that the documentation differs from the implementation,
the actual behavior is likely an oversight.
Given how esoteric of behavior this is,
we change the v1in2 behavior to avoid the unnecesary escaping
and drop support for EscapeInvalidUTF8.
This does not violate the Go compatibility agreement since
we do not document what the exact syntactic output is.
Also, there has already been prior precedence for changing the output:
* [encoding/json: encode \b and \f as '\b' and '\f' in JSON strings](https://go.dev/cl/521675)
* [encoding/json: encode \n in strings as "\n", not "\u000A"](https://go.dev/cl/4678046)
* [encoding/json: encode \t as \t instead of \u0009](https://go.dev/cl/162340043)
* [encoding/json: use standard ES6 formatting for numbers during marshal](https://go.dev/cl/30371)
Fixes #74551
Change-Id: Ib59a873c44713d302f1f6ab103ffba2520d63276
Reviewed-on: https://go-review.googlesource.com/c/go/+/687116
Auto-Submit: Joseph Tsai <joetsai@digital-static.net>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Johan Brandhorst-Satzkorn <johan.brandhorst@gmail.com>
Reviewed-by: Carlos Amedee <carlos@golang.org>
|
|
In the event that the input is just JSON whitespace,
the underlying jsontext.Decoder treats this as an empty stream
and reports io.EOF.
The logic in unmarshalFull simply casted io.EOF as io.ErrUnexpectedEOF,
which is inconsistent with how all other io.ErrUnexpectedEOF are reported,
which are wrapped within a jsontext.SyntacticError.
Do the same thing for consistency.
We add a v1 test (without goexperiment.jsonv2) to verify that
the behavior is identical to how v1 has always behaved.
We add a v1in2 test (with goexperiment.jsonv2) to verify that
the v1in2 behavior correctly replicates historical v1 behavior.
We also fix a faulty check in v1 Decoder.Decode,
where it tried to detect errUnexpectedEnd and
return an unwrapped io.ErrUnexpectedEOF error.
This is the exact semantic that v1 has always done
in streaming Decoder.Decode (but not non-streaming Unmarshal).
There is a prior bug reported in #25956 about this inconsistency,
but we aim to preserve historical v1 behavior to reduce
the probability of churn when v1 is re-implemented in terms of v2.
Fixes #74548
Change-Id: Ibca52c3699ff3c09141e081c85f853781a86ec8e
Reviewed-on: https://go-review.googlesource.com/c/go/+/687115
Auto-Submit: Joseph Tsai <joetsai@digital-static.net>
Reviewed-by: Carlos Amedee <carlos@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
|
|
This logic was added in October, 2021:
https://github.com/go-json-experiment/json/commit/0b3bd4e1ed96587be346b7f964d6bb3fcfed65f4
before the introduction of bytes.Buffer.AvailableBuffer in March, 2023.
https://go.dev/cl/474635
Updates #71845
Change-Id: I96800e1ba8fce15cc78316779db4ddcd4fe1d510
Reviewed-on: https://go-review.googlesource.com/c/go/+/685136
Reviewed-by: Damien Neil <dneil@google.com>
Auto-Submit: Joseph Tsai <joetsai@digital-static.net>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
|
|
The jsonopts.Struct.join method unfortunately escapes
the receiver because it is passed to JoinUnknownOption,
which is a dynamically implemented function.
This affects jsontext.Encoder.reset and jsontext.Decoder.reset,
which relied on a local jsonopts.Struct to temporarily store
prior options such that it would have to be heap allocated.
Adjust the signature of JoinUnknownOption to avoid pointers
so that nothing escape.
This is a regression from
https://github.com/go-json-experiment/json/pull/163
Performance:
name old time/op new time/op delta
Marshal/Bool-32 72.1ns ± 2% 51.3ns ± 1% -28.77% (p=0.000 n=10+9)
name old allocs/op new allocs/op delta
Marshal/Bool-32 2.00 ± 0% 1.00 ± 0% -50.00% (p=0.000 n=10+10)
Updates #71845
Change-Id: Ife500d82d3d2beb13652553a4ffdf882c136f5a0
Reviewed-on: https://go-review.googlesource.com/c/go/+/685135
Auto-Submit: Joseph Tsai <joetsai@digital-static.net>
Reviewed-by: Damien Neil <dneil@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
|
|
Add a section to the package doc which details the security
considerations of using encoding/json, in particular with respect to
parser misalignment issues.
Additionally, clarify previously ambiguous statement in the Unmarshal
doc about how case is used when matching keys in objects, and add a note
about how duplicate keys are handled.
Fixes #14750
Change-Id: I66f9b845efd98c86a684d7333b3aa8a456564922
Reviewed-on: https://go-review.googlesource.com/c/go/+/684315
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Joseph Tsai <joetsai@digital-static.net>
Auto-Submit: Roland Shoemaker <roland@golang.org>
Reviewed-by: Damien Neil <dneil@google.com>
|
|
Updates #71845
Change-Id: Ie099e7ac77293696fd9e69559487e27f4b70ab3f
Reviewed-on: https://go-review.googlesource.com/c/go/+/684416
Auto-Submit: Joseph Tsai <joetsai@digital-static.net>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
|
|
WARNING: This commit contains a breaking change.
This is permissible since jsontext is experimental and
not subject to the Go 1 compatibility agreement.
Existing callers of UnusedBuffer should use AvailableBuffer instead.
Updates #71497
Change-Id: Ib080caf306d545a8fb038e57f0817b18dd0f91cf
Reviewed-on: https://go-review.googlesource.com/c/go/+/683897
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Johan Brandhorst-Satzkorn <johan.brandhorst@gmail.com>
Reviewed-by: Damien Neil <dneil@google.com>
Auto-Submit: Joseph Tsai <joetsai@digital-static.net>
|
|
This follows the precedent set by:
bufio.Writer.AvailableBuffer
bytes.Buffer.AvailableBuffer
both with methods that return a zero-length buffer that
is intended to only be used with a following Write call.
This keeps the older UnusedBuffer method around so that
at least one commit that has both methods for migration purposes.
Updates #71497
Change-Id: I3815f593e09f645280ae5ad9cbdd63a6c147123b
Reviewed-on: https://go-review.googlesource.com/c/go/+/683896
Reviewed-by: Damien Neil <dneil@google.com>
TryBot-Bypass: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Johan Brandhorst-Satzkorn <johan.brandhorst@gmail.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
|
|
There is a non-public zstd decoder in the stdlib (CL 473356) and
also zstd compressed testdata already present.
Delete testdata/code.json.gz and
instead use internal/jsontest/testdata/golang_source.json.zst,
which has exactly the same content:
$ cat internal/jsontest/testdata/golang_source.json.zst | zstd -d | sha1sum
3f70b6fd429f4aba3e8e1c3e5a294c8f2e219a6e -
$ cat testdata/code.json.gz | zstd -d | sha1sum
3f70b6fd429f4aba3e8e1c3e5a294c8f2e219a6e -
This will reduce the size of the final Go release by 118KB.
Updates #71845
Change-Id: I6da2df27bd260befc0a44c6bc0255365be0a5b0f
Reviewed-on: https://go-review.googlesource.com/c/go/+/525516
Auto-Submit: Damien Neil <dneil@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Bypass: Damien Neil <dneil@google.com>
|
|
Based on the discussion in #71631, it is hotly contested
whether the default JSON representation for a Go time.Duration
should be the time.Duration.String format or
a particular profile of ISO 8601.
Regardless of the default, it seems clear that we should
at least support ISO 8601 if specified via a format flag.
Note that this CL does not alter the default representation.
Unfortunately, ISO 8601 is a large and evolving standard
with many optional extensions and optional restrictions.
Thus, the term "ISO 8601 duration" unfortunately does not
resolve to a particular grammar, nor one that is stable.
However, there is precedence that we can follow in this matter.
JSON finds its heritage in JavaScript and
JavaScript is adding a Temporal.Duration type whose default
JSON representation is ISO 8601.
There is a well-specified grammar for their particular
profile of ISO 8601, which is documented at:
https://tc39.es/proposal-temporal/#prod-Duration
This particular CL adds support for ISO 8601 according to
the exact same grammar that JavaScript uses.
While Temporal.Duration is technically still a proposal,
it is already in stage 3 of the TC39 proposal process
(i.e., "no changes to the proposal are expected"
and "has been recommended for implementation")
and therefore close to final adoption.
One major concern with ISO 8601 is that it supports
nominal date units like years, months, weeks, and days
that do not have an accurate meaning without being
anchored to a particular point in time and place on Earth.
Fortunately, JavaScript (by default) avoids producing
Temporal.Duration values with nominal units unless
arithmetic in JavaScript explicitly sets a largestUnits
value that is larger than "hours". In the Go implementation,
we support syntactically parsing the full ISO 8601 grammar
(according to JavaScript), but semantically report an error if
nominal units are present. This ensures that ISO 8601 durations
remain accurate so long as they only use the accurate units
of hours, minutes, or seconds.
Updates #71631
Change-Id: I983593662f2150461ebc486a5acfeb72f0286939
Reviewed-on: https://go-review.googlesource.com/c/go/+/682403
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Reviewed-by: Damien Neil <dneil@google.com>
Auto-Submit: Joseph Tsai <joetsai@digital-static.net>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
|
|
In this blog:
https://blog.trailofbits.com/2025/06/17/unexpected-security-footguns-in-gos-parsers/
the concern was raised that whenever "-" is combined with other options,
the "-" is intepreted as as a name, rather than an ignored field,
which may go contrary to user expectation.
Static analysis demonstrates that there are ~2k instances of `json:"-,omitempty"
in the wild, where almost all of them intended for the field to be ignored.
To prevent this footgun, reject any tags that has "-," as a prefix
and warn the user to choose one of the reasonable alternatives.
The documentation of json/v2 already suggests `json:"'-'"`
as the recommended way to explicitly specify dash as the name.
See Example_fieldNames for example usages of the single-quoted literal.
Update the v1 json documentation to suggest the same thing.
Updates #71497
Change-Id: I7687b6eecdf82a5d894d057c78a4a90af4f5a6e4
Reviewed-on: https://go-review.googlesource.com/c/go/+/683175
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Johan Brandhorst-Satzkorn <johan.brandhorst@gmail.com>
Reviewed-by: Damien Neil <dneil@google.com>
Auto-Submit: Joseph Tsai <joetsai@digital-static.net>
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
|
|
The default representation of a time.Duration is still undecided.
In order to keep the future open, report an error on a time.Duration
without an explicit format flag provided.
Updates #71631
Change-Id: I08248404ff6551723851417c8188a13f53c61937
Reviewed-on: https://go-review.googlesource.com/c/go/+/682455
Auto-Submit: Joseph Tsai <joetsai@digital-static.net>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
|
|
RFC 8259, section 2 uses the term "begin-array" amd "begin-object"
rather than "start array" or "start object".
Be consistent in our documentation.
Change-Id: I172eb354c5e64b84c74bd662b1e581424e719a32
Reviewed-on: https://go-review.googlesource.com/c/go/+/683155
Auto-Submit: Joseph Tsai <joetsai@digital-static.net>
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Johan Brandhorst-Satzkorn <johan.brandhorst@gmail.com>
|
