aboutsummaryrefslogtreecommitdiff
path: root/src/encoding/json
AgeCommit message (Collapse)Author
2020-02-26encoding/json: consolidate the isSpace functioncodesoap
The new code is easier to read, and practically equivalent in terms of performance. name old time/op new time/op delta CodeUnmarshal-2 166ms ± 1% 166ms ± 1% ~ (p=0.863 n=11+10) CodeUnmarshalReuse-2 139ms ± 1% 139ms ± 1% ~ (p=0.050 n=10+12) UnmarshalString-2 1.08µs ± 1% 1.07µs ± 1% -0.64% (p=0.001 n=10+11) UnmarshalFloat64-2 1.01µs ± 1% 1.01µs ± 1% ~ (p=0.280 n=12+11) UnmarshalInt64-2 850ns ± 0% 851ns ± 0% ~ (p=0.455 n=11+12) name old speed new speed delta CodeUnmarshal-2 11.7MB/s ± 1% 11.7MB/s ± 1% ~ (p=0.904 n=11+10) CodeUnmarshalReuse-2 14.0MB/s ± 1% 14.0MB/s ± 1% +0.40% (p=0.041 n=10+12) name old alloc/op new alloc/op delta CodeUnmarshal-2 3.28MB ± 0% 3.28MB ± 0% ~ (p=0.907 n=10+11) CodeUnmarshalReuse-2 2.19MB ± 0% 2.19MB ± 0% ~ (p=0.306 n=12+12) UnmarshalString-2 192B ± 0% 192B ± 0% ~ (all equal) UnmarshalFloat64-2 180B ± 0% 180B ± 0% ~ (all equal) UnmarshalInt64-2 176B ± 0% 176B ± 0% ~ (all equal) name old allocs/op new allocs/op delta CodeUnmarshal-2 92.7k ± 0% 92.7k ± 0% ~ (all equal) CodeUnmarshalReuse-2 80.4k ± 0% 80.4k ± 0% ~ (all equal) UnmarshalString-2 2.00 ± 0% 2.00 ± 0% ~ (all equal) UnmarshalFloat64-2 2.00 ± 0% 2.00 ± 0% ~ (all equal) UnmarshalInt64-2 1.00 ± 0% 1.00 ± 0% ~ (all equal) Change-Id: I6d5a48c624d436551409a17c21542e26d29e26b3 GitHub-Last-Rev: 7d81961688b5ee3a7e4718188c0eaf3413521f97 GitHub-Pull-Request: golang/go#37385 Reviewed-on: https://go-review.googlesource.com/c/go/+/220581 Reviewed-by: Daniel Martí <mvdan@mvdan.cc> Reviewed-by: Joe Tsai <joetsai@google.com> Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org>
2020-02-24encoding/json: limit max nesting depthJordan Liggitt
Limit the maximum nesting depth when parsing to protect against stack overflow, permitted by https://tools.ietf.org/html/rfc7159#section-9 A nesting depth limit of 10,000 was chosen to be a conservative balance between avoiding stack overflow and avoiding impacting legitimate JSON documents. 10,000 is less than 1% of the experimental stack depth limit with the default stack size: * On 64-bit systems, the default stack limit is 1GB, which allows ~2,800,000 frames of recursive parsing * On 32-bit systems, the default stack limit is 250MB, which allows ~1,100,000 frames of recursive parsing Fixes #31789 Change-Id: I4f5a90e89dcb4ab1a957ad9d02e1fa0efafaccf6 Reviewed-on: https://go-review.googlesource.com/c/go/+/199837 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
2020-02-22encoding/json: fix broken link to json.orgJon Johnson
This updates the link to a newer image. Change-Id: Ibdfe8c57d9217a325bcfde98cb6f952ca63d588a GitHub-Last-Rev: f5970ba395781c0d299dfdaa1ed75e46125b8268 GitHub-Pull-Request: golang/go#36938 Reviewed-on: https://go-review.googlesource.com/c/go/+/217297 Reviewed-by: Toshihiro Shiino <shiino.toshihiro@gmail.com> Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
2019-11-11encoding/json: error when encoding a pointer cycleDaniel Martí
Otherwise we'd panic with a stack overflow. Most programs are in control of the data being encoded and can ensure there are no cycles, but sometimes it's not that simple. For example, running a user's html template with script tags can easily result in crashes if the user can find a pointer cycle. Adding the checks via a map to every ptrEncoder.encode call slowed down the benchmarks below by a noticeable 13%. Instead, only start doing the relatively expensive pointer cycle checks if we're many levels of pointers deep in an encode state. A threshold of 1000 is small enough to capture pointer cycles before they're a problem (the goroutine stack limit is currently 1GB, and I needed close to a million levels to reach it). Yet it's large enough that reasonable uses of the json encoder only see a tiny 1% slow-down due to the added ptrLevel field and check. name old time/op new time/op delta CodeEncoder-8 2.34ms ± 1% 2.37ms ± 0% +1.05% (p=0.000 n=10+10) CodeMarshal-8 2.42ms ± 1% 2.44ms ± 0% +1.10% (p=0.000 n=10+10) name old speed new speed delta CodeEncoder-8 829MB/s ± 1% 820MB/s ± 0% -1.04% (p=0.000 n=10+10) CodeMarshal-8 803MB/s ± 1% 795MB/s ± 0% -1.09% (p=0.000 n=10+10) name old alloc/op new alloc/op delta CodeEncoder-8 43.1kB ± 8% 42.5kB ±10% ~ (p=0.989 n=10+10) CodeMarshal-8 1.99MB ± 0% 1.99MB ± 0% ~ (p=0.254 n=9+6) name old allocs/op new allocs/op delta CodeEncoder-8 0.00 0.00 ~ (all equal) CodeMarshal-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) Finally, add a few tests to ensure that the code handles the edge cases properly. Fixes #10769. Change-Id: I73d48e0cf6ea140127ea031f2dbae6e6a55e58b8 Reviewed-on: https://go-review.googlesource.com/c/go/+/187920 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bjørn Erik Pedersen <bjorn.erik.pedersen@gmail.com> Reviewed-by: Andrew Bonventre <andybons@golang.org>
2019-10-31encoding/json: avoid work when unquoting strings, take 2Daniel Martí
This is a re-submission of CL 151157, since it was reverted in CL 190909 due to an introduced crash found by a fuzzer. The revert CL included regression tests, while this CL includes a fixed version of the original change. In particular, what we forgot in the original optimization was that we still need the length and trailing quote checks at the beginning of unquoteBytes. Without those, we could end up in a crash later on. We can work out how many bytes can be unquoted trivially in rescanLiteral, which already iterates over a string's bytes. Removing the extra loop in unquoteBytes simplifies the function and speeds it up, especially when decoding simple strings, which are common. While at it, we can remove the check that s[0]=='"', since all call sites already meet that condition. name old time/op new time/op delta CodeDecoder-8 10.6ms ± 2% 10.5ms ± 1% -1.01% (p=0.004 n=20+10) name old speed new speed delta CodeDecoder-8 183MB/s ± 2% 185MB/s ± 1% +1.02% (p=0.003 n=20+10) Updates #28923. Change-Id: I8c6b13302bcd86a364bc998d72451332c0809cde Reviewed-on: https://go-review.googlesource.com/c/go/+/190659 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Peter Weinberger <pjw@google.com>
2019-10-28encoding/json: remove allocation when using a Marshaler with value receiverPhil Pearl
If we marshal a non-pointer struct field whose type implements Marshaler with a non-pointer receiver, then we avoid an allocation if we take the address of the field before casting it to an interface. name old time/op new time/op delta EncodeMarshaler-8 104ns ± 1% 92ns ± 2% -11.72% (p=0.001 n=7+7) name old alloc/op new alloc/op delta EncodeMarshaler-8 36.0B ± 0% 4.0B ± 0% -88.89% (p=0.000 n=8+8) name old allocs/op new allocs/op delta EncodeMarshaler-8 2.00 ± 0% 1.00 ± 0% -50.00% (p=0.000 n=8+8) Test coverage already looks good enough for this change. TestRefValMarshal already covers all possible combinations of value & pointer receivers on value and pointer struct fields. Change-Id: I6fc7f72396396d98f9a90c3c86e813690f41c099 Reviewed-on: https://go-review.googlesource.com/c/go/+/203608 Reviewed-by: Daniel Martí <mvdan@mvdan.cc> Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-10-27encoding/json: improve performance of CompactPhil Pearl
This change improves performance of Compact by using a sync.Pool to allow re-use of a scanner. This also has the side-effect of removing an allocation for each field that implements Marshaler when marshalling JSON. name old time/op new time/op delta EncodeMarshaler-8 118ns ± 2% 104ns ± 1% -12.21% (p=0.001 n=7+7) name old alloc/op new alloc/op delta EncodeMarshaler-8 100B ± 0% 36B ± 0% -64.00% (p=0.000 n=8+8) name old allocs/op new allocs/op delta EncodeMarshaler-8 3.00 ± 0% 2.00 ± 0% -33.33% (p=0.000 n=8+8) Change-Id: Ic70c61a0a6354823da5220f5aad04b94c054f233 Reviewed-on: https://go-review.googlesource.com/c/go/+/200864 Reviewed-by: Daniel Martí <mvdan@mvdan.cc> Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-10-16encoding/json: correct caller's name in encoding errorsEugene Kalinin
1. Change mapencode.encode to use fmt.Error rather than MarshalerError. MarshalerError refer to MarshalJSON, but mapencode.encode does not use that. 2. Add sourceFunc field to MarshalerError to record the name of the function that creates the error, so that the Error method can report it correctly. Fixes #29753 Change-Id: I186c2fac8470ae2f9e300501de3730face642230 Reviewed-on: https://go-review.googlesource.com/c/go/+/184119 Run-TryBot: Ian Lance Taylor <iant@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-10-11encoding/json: add Decoder.InputOffset for offset accessFrancesco Renzi
This makes Decoder.offset public while renaming it to Decoder.InputOffset to match encoding/xml Decoder API Code changes made by Adam Stankiewicz [sheerun@sher.pl] Fixes #29688 Change-Id: I86dbfd2b2da80160846e92bfa580c53d8d45e2db Reviewed-on: https://go-review.googlesource.com/c/go/+/200677 Run-TryBot: Johan Brandhorst <johan.brandhorst@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-10-10encoding/json: support TextUnmarshaler for map keys with string underlying typesCuong Manh Le
When unmarshaling to a map, the map's key type must either be a string, an integer, or implement encoding.TextUnmarshaler. But for a user defined type, reflect.Kind will not distinguish between the static type and the underlying type. In: var x MyString = "x" t := reflect.TypeOf(x) println(t.Kind() == reflect.String) the Kind of x is still reflect.String, even though the static type of x is MyString. Moreover, checking for the map's key type is a string occurs first, so even if the map key type MyString implements encoding.TextUnmarshaler, it will be ignored. To fix the bug, check for encoding.TextUnmarshaler first. Fixes #34437 Change-Id: I780e0b084575e1dddfbb433fe03857adf71d05fb Reviewed-on: https://go-review.googlesource.com/c/go/+/200237 Run-TryBot: Cuong Manh Le <cuong.manhle.vn@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
2019-10-10encoding/json: stop escaping U+2028 and U+2029 in CompactTom Thorogood
Compact has been inconsistently escaping only some problematic characters (U+2028 and U+2029), but not others (<, > and &). This change addresses this inconsistency by removing the escaping of U+2028 and U+2029. Callers who need to escape the output of Compact should use HTMLEscape which escapes <, >, &, U+2028 and U+2029. Fixes #34070 Fixes #30357 Updates #5836 Change-Id: Icfce7691d2b8b1d9b05ba7b64d2d1e4f3b67871b GitHub-Last-Rev: 38859fe3e2fd586bbd45175c2742f7b123836bf3 GitHub-Pull-Request: golang/go#34804 Reviewed-on: https://go-review.googlesource.com/c/go/+/200217 Reviewed-by: Daniel Martí <mvdan@mvdan.cc> Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-09-16encoding/json: validate strings when decoding into NumberLucas Bremgartner
Unmarshaling a string into a json.Number should first check that the string is a valid Number. If not, we should fail without decoding it. Fixes #14702 Change-Id: I286178e93df74ad63c0a852c3f3489577072cf47 GitHub-Last-Rev: fe69bb68eed06d056639f440d2daf4bb7c99013b GitHub-Pull-Request: golang/go#34272 Reviewed-on: https://go-review.googlesource.com/c/go/+/195045 Reviewed-by: Daniel Martí <mvdan@mvdan.cc> Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-09-16encoding/json: make Number with the ,string option marshal with quotesLucas Bremgartner
Add quotes when marshaling a json.Number with the string option set via a struct tag. This ensures that the resulting json can be unmarshaled into the source struct without error. Fixes #34268 Change-Id: Ide167d9dec77019554870b5957b37dc258119d81 GitHub-Last-Rev: dde81b71208be01c253bb87dbb6f81ac6e0785be GitHub-Pull-Request: golang/go#34269 Reviewed-on: https://go-review.googlesource.com/c/go/+/195043 Reviewed-by: Daniel Martí <mvdan@mvdan.cc> Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-09-11encoding/json: encode nil encoding.TextMarshaler instance as "null"William Poussier
Fixes #34235. Change-Id: Ia3795fd18860530fa6a4b171545f525e784ffdcb GitHub-Last-Rev: 1a319c452857818f7aaf22ef46823b43ca9b2276 GitHub-Pull-Request: golang/go#34238 Reviewed-on: https://go-review.googlesource.com/c/go/+/194642 Reviewed-by: Daniel Martí <mvdan@mvdan.cc> Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-09-11encoding/json: fix and optimize marshal for quoted stringLucas Bremgartner
Since Go 1.2 every string can be marshaled to JSON without error even if it contains invalid UTF-8 byte sequences. Therefore there is no need to use Marshal again for the only reason of enclosing the string in double quotes. Not using Marshal here also removes the error check as there has not been a way for Marshal to fail anyway. name old time/op new time/op delta Issue34127-4 360ns ± 3% 200ns ± 3% -44.56% (p=0.008 n=5+5) name old alloc/op new alloc/op delta Issue34127-4 56.0B ± 0% 40.0B ± 0% -28.57% (p=0.008 n=5+5) name old allocs/op new allocs/op delta Issue34127-4 3.00 ± 0% 2.00 ± 0% -33.33% (p=0.008 n=5+5) Fixes #34154 Change-Id: Ib60dc11980f9b20d8bef2982de7168943d632263 GitHub-Last-Rev: 9b0ac1d4c5318b6bf9ed7930320f2bd755f9939c GitHub-Pull-Request: golang/go#34127 Reviewed-on: https://go-review.googlesource.com/c/go/+/193604 Reviewed-by: Daniel Martí <mvdan@mvdan.cc> Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-09-10encoding/json: don't reset before returning buffer to poolandig
Reset is already performed when retrieving from pool Change-Id: Ia810dd18d3e55a1565a5ad435a00d1e46724576c GitHub-Last-Rev: d9df74a4aeb86e5d292c9fc33568a3c9a64a967d GitHub-Pull-Request: golang/go#34195 Reviewed-on: https://go-review.googlesource.com/c/go/+/194338 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
2019-09-10encoding/json: don't indirect pointers when decoding nullRomain Baugue
The indirect method checked the type of the child when indirecting a pointer. If the current value is a pointer and we are decoding null, we can skip this entirely and return early, avoiding the whole descent. Fixes #31776 Change-Id: Ib8b2a2357572c41f56fceac59b5a858980f3f65e Reviewed-on: https://go-review.googlesource.com/c/go/+/174699 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
2019-09-09src: gofmt -sAinar Garipov
Change-Id: I56d7eeaf777ac30886ee77428ca1ac72b77fbf7d Reviewed-on: https://go-review.googlesource.com/c/go/+/193849 Run-TryBot: Dave Cheney <dave@cheney.net> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-09-09encoding/json: clarify Unmarshal behavior for map keysEli Bendersky
This is a documentation-only change Fixes #33298 Change-Id: I816058a872b57dc868dff11887214d9de92d9342 Reviewed-on: https://go-review.googlesource.com/c/go/+/188821 Reviewed-by: Daniel Martí <mvdan@mvdan.cc> Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-09-02std: remove unused bits of code all over the placeDaniel Martí
Some were never used, and some haven't been used for years. One exception is net/http's readerAndCloser, which was only used in a test. Move it to a test file. While at it, remove a check in regexp that could never fire; the field is an uint32, so it can never be negative. Change-Id: Ia2200f6afa106bae4034045ea8233b452f38747b Reviewed-on: https://go-review.googlesource.com/c/go/+/192621 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-09-02encoding/json: fix scanner byte offset on scanEnditchyny
scanEnd is delayed one byte so we decrement the scanner bytes count by 1 to ensure that this value is correct in the next call of Decode. Fixes #32399 Change-Id: I8c8698e7f95bbcf0373aceaa05319819eae9d86f GitHub-Last-Rev: 0ac25d8de23d38c7ac577faddc6983571023f561 GitHub-Pull-Request: golang/go#32598 Reviewed-on: https://go-review.googlesource.com/c/go/+/182117 Reviewed-by: Daniel Martí <mvdan@mvdan.cc> Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-09-01encoding/json: revert Compact HTML escaping documentationTom Thorogood
This partly reverts CL 173417 as it incorrectly documented that Compact performed HTML escaping and the output was safe to embed inside HTML <script> tags. This has never been true. Although Compact does escape U+2028 and U+2029, it doesn't escape <, > or &. Compact is thus only performing a subset of HTML escaping and it's output is not safe to embed inside HTML <script> tags. A more complete fix would be for Compact to either never perform any HTML escaping, as it was prior to CL 10883045, or to actually perform the same HTML escaping as HTMLEscape. Neither change is likely safe enough for go1.13. Updates #30357 Change-Id: I912f0fe9611097d988048b28228c4a5b985080ba GitHub-Last-Rev: aebababc9233c5705785b225377e80096d4bb8c4 GitHub-Pull-Request: golang/go#33427 Reviewed-on: https://go-review.googlesource.com/c/go/+/188717 Reviewed-by: Daniel Martí <mvdan@mvdan.cc> Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-09-01encoding/json: fix panic for nil instances of TextMarshaler in map keysWilliam Poussier
This change adds a a check in the encodeWithString.resolve method to ensure that a reflect.Value with kind Ptr is not nil before the type assertion to TextMarshaler. If the value is nil, the method returns a nil error, and the map key encodes to an empty string. Fixes #33675 Change-Id: I0a04cf690ae67006f6a9c5f8cbb4cc99d236bca8 GitHub-Last-Rev: 6c987c90846f854e21814dbfb3a073605ec8a94c GitHub-Pull-Request: golang/go#33700 Reviewed-on: https://go-review.googlesource.com/c/go/+/190697 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
2019-08-27encoding/json: remove unnecessary isValidNumber callDaniel Martí
The decoder called this function to check numbers being decoded into a json.Number. However, these can't be quoted as strings, so the tokenizer has already verified they are valid JSON numbers. Verified this by adding a test with such an input. As expected, it produces a syntax error, not the fmt.Errorf - that line could never execute. Since the only remaining non-test caller of isvalidnumber is in encode.go, move the function there. This change should slightly reduce the amount of work when decoding into json.Number, though that isn't very common nor part of any current benchmarks. Change-Id: I67a1723deb3d18d5b542d6dd35f3ae56a43f23eb Reviewed-on: https://go-review.googlesource.com/c/go/+/184817 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-08-27encoding/json: fix the broken "overwriting of data" testsDaniel Martí
Because TestUnmarshal actually allocates a new value to decode into using ptr's pointer type, any existing data is thrown away. This was harmless in alomst all of the test cases, minus the "overwriting of data" ones added in 2015 in CL 12209. I spotted that nothing covered decoding a JSON array with few elements into a slice which already had many elements. I initially assumed that the code was buggy or that some code could be removed, when in fact there simply wasn't any code covering the edge case. Move those two tests to TestPrefilled, which already served a very similar purpose. Remove the map case, as TestPrefilled already has plenty of prefilled map cases. Moreover, we no longer reset an entire map when decoding, as per the godoc: To unmarshal a JSON object into a map, Unmarshal first establishes a map to use. If the map is nil, Unmarshal allocates a new map. Otherwise Unmarshal reuses the existing map, keeping existing entries. Finally, to ensure that ptr is used correctly in the future, make TestUnmarshal error if it's anything other than a pointer to a zero value. That is, the only correct use should be new(type). Don't rename the ptr field, as that would be extremely noisy and cause unwanted merge conflicts. Change-Id: I41e3ecfeae42d877ac5443a6bd622ac3d6c8120c Reviewed-on: https://go-review.googlesource.com/c/go/+/185738 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
2019-08-21Revert "encoding/json: avoid work when unquoting strings"Filippo Valsorda
This reverts CL 151157. CL 151157 introduced a crash when decoding into ",string" fields. It came with a moderate speedup, so at this stage of the release cycle let's just revert it, and reapply it in Go 1.14 with the fix in CL 190659. Also applied the test cases from CL 190659. Updates #33728 Change-Id: Ie46e2bc15224b251888580daf6b79d5865f3878e Reviewed-on: https://go-review.googlesource.com/c/go/+/190909 Run-TryBot: Andrew Bonventre <andybons@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Andrew Bonventre <andybons@golang.org>
2019-08-18encoding/json: fix format string in the Fuzz funcDmitry Vyukov
Currently test build fails with: $ go test -tags=gofuzz encoding/json encoding/json/fuzz.go:36:4: Println call has possible formatting directive %s FAIL encoding/json [build failed] Change-Id: I23aef44a421ed0e7bcf48b74ac5a8c6768a4841b Reviewed-on: https://go-review.googlesource.com/c/go/+/190698 Run-TryBot: Ian Lance Taylor <iant@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-07-31encoding/json: clarify Marshal behavior for string keys of mapsEli Bendersky
This is a documentation-only change. Fixes #28827 Change-Id: Ife9ab997809048784f35872b09905bc209a05eff Reviewed-on: https://go-review.googlesource.com/c/go/+/188417 Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-07-06encoding/json: obey SetEscapeHTML in all MarshalJSON casesDaniel Martí
It wasn't obeyed in the case where the MarshalJSON method uses a pointer receiver, and the encoder grabs the address of a value to find that method. addrMarshalerEncoder is the function that does this work, but it ignored opts.escapeHTML. Here's the before and after of the added test case, which was failing before the fix. Now the two cases are correct and consistent. {"NonPtr":"<str>","Ptr":"\u003cstr\u003e"} {"NonPtr":"<str>","Ptr":"<str>"} Fixes #32896. Change-Id: Idc53077ece074973558bd3bb5ad036380db0d02c Reviewed-on: https://go-review.googlesource.com/c/go/+/184757 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com> Reviewed-by: Caleb Spare <cespare@gmail.com>
2019-05-22all: shorten some testsRuss Cox
Shorten some of the longest tests that run during all.bash. Removes 7r 50u 21s from all.bash. After this change, all.bash is under 5 minutes again on my laptop. For #26473. Change-Id: Ie0460aa935808d65460408feaed210fbaa1d5d79 Reviewed-on: https://go-review.googlesource.com/c/go/+/177559 Run-TryBot: Russ Cox <rsc@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-05-04all: add Unwrap and Is methods to various error typesDamien Neil
Add Unwrap methods to types which wrap an underlying error: "encodinc/csv".ParseError "encoding/json".MarshalerError "net/http".transportReadFromServerError "net".OpError "net".DNSConfigError "net/url".Error "os/exec".Error "signal/internal/pty".PtyError "text/template".ExecError Add os.ErrTemporary. A case could be made for putting this error value in package net, since no exported error types in package os include a Temporary method. However, syscall errors returned from the os package do include this method. Add Is methods to error types with a Timeout or Temporary method, making errors.Is(err, os.Err{Timeout,Temporary}) equivalent to testing the corresponding method: "context".DeadlineExceeded "internal/poll".TimeoutError "net".adrinfoErrno "net".OpError "net".DNSError "net/http".httpError "net/http".tlsHandshakeTimeoutError "net/pipe".timeoutError "net/url".Error Updates #30322 Updates #29934 Change-Id: I409fb20c072ea39116ebfb8c7534d493483870dc Reviewed-on: https://go-review.googlesource.com/c/go/+/170037 Run-TryBot: Damien Neil <dneil@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Marcel van Lohuizen <mpvl@golang.org>
2019-04-30encoding/json: fix Unmarshal hang on recursive pointersLE Manh Cuong
indirect walks down v until it gets to a non-pointer. But it does not handle the case when v is a pointer to itself, like in: var v interface{} v = &v Unmarshal(b, v) So just stop immediately if we see v is a pointer to itself. Fixes #31740 Change-Id: Ie396264119e24d70284cd9bf76dcb2050babb069 Reviewed-on: https://go-review.googlesource.com/c/go/+/174337 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Daniel Martí <mvdan@mvdan.cc> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-04-29encoding/json: add a Fuzz functionRomain Baugue
Adds a sample Fuzz test function to package encoding/json following the guidelines defined in #31309, based on https://github.com/dvyukov/go-fuzz-corpus/blob/master/json/json.go Fixes #31309 Updates #19109 Change-Id: I5fe04d9a5f41c0de339f8518dae30896ec14e356 Reviewed-on: https://go-review.googlesource.com/c/go/+/174058 Reviewed-by: Dmitry Vyukov <dvyukov@google.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Dmitry Vyukov <dvyukov@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-04-23encoding/json: document HTML escaping in CompactRuss Cox
Make explicit that Compact does HTML escaping. Fixes #30357. Change-Id: I4648f8f3e907d659db977d07253f716df6e07d7b Reviewed-on: https://go-review.googlesource.com/c/go/+/173417 Run-TryBot: Russ Cox <rsc@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-04-23encoding/json: index names for the struct decoderDaniel Martí
In the common case, structs have a handful of fields and most inputs match struct field names exactly. The previous code would do a linear search over the fields, stopping at the first exact match, and otherwise using the first case insensitive match. This is unfortunate, because it means that for the common case, we'd do a linear search with bytes.Equal. Even for structs with only two or three fields, that is pretty wasteful. Worse even, up until the exact match was found via the linear search, all previous fields would run their equalFold functions, which aren't cheap even in the simple case. Instead, cache a map along with the field list that indexes the fields by their name. This way, a case sensitive field search doesn't involve a linear search, nor does it involve any equalFold func calls. This patch should also slightly speed up cases where there's a case insensitive match but not a case sensitive one, as then we'd avoid calling bytes.Equal on all the fields. Though that's not a common case, and there are no benchmarks for it. name old time/op new time/op delta CodeDecoder-8 11.0ms ± 0% 10.6ms ± 1% -4.42% (p=0.000 n=9+10) name old speed new speed delta CodeDecoder-8 176MB/s ± 0% 184MB/s ± 1% +4.62% (p=0.000 n=9+10) name old alloc/op new alloc/op delta CodeDecoder-8 2.28MB ± 0% 2.28MB ± 0% ~ (p=0.725 n=10+10) name old allocs/op new allocs/op delta CodeDecoder-8 76.9k ± 0% 76.9k ± 0% ~ (all equal) Updates #28923. Change-Id: I9929c1f06c76505e5b96914199315dbdaae5dc76 Reviewed-on: https://go-review.googlesource.com/c/go/+/172918 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-04-22encoding/json: avoid work when unquoting stringsDaniel Martí
We can work out how many bytes can be unquoted trivially in rescanLiteral, which already iterates over a string's bytes. Removing the extra loop in unquoteBytes simplifies the function and speeds it up, especially when decoding simple strings, which are common. While at it, we can remove unnecessary checks like len(s)<2 and s[0]=='"'. Add a comment explaining why. name old time/op new time/op delta CodeDecoder-8 11.2ms ± 0% 11.1ms ± 1% -1.63% (p=0.000 n=9+10) name old speed new speed delta CodeDecoder-8 173MB/s ± 0% 175MB/s ± 1% +1.66% (p=0.000 n=9+10) Updates #28923. Change-Id: I2436a3a7f8148a2f7a6a4cdbd7dec6b32ef5e20c Reviewed-on: https://go-review.googlesource.com/c/go/+/151157 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-04-13encoding/json: remove a bounds check in readValueDaniel Martí
readValue is a hot function, clocking in at ~13% flat CPU use in CodeDecoder. In particular, looping over the bytes is slow. That's partially because the code contains a bounds check at the start of the loop. The source of the problem is that scanp is a signed integer, and comes from a field, so the compiler doesn't know that it's non-negative. Help it with a simple and comparatively cheap hint. While at it, use scanp as the index variable directly, removing the need for a duplicate index variable which is later added back into scanp. name old time/op new time/op delta CodeDecoder-8 11.3ms ± 1% 11.2ms ± 1% -0.98% (p=0.000 n=9+9) name old speed new speed delta CodeDecoder-8 172MB/s ± 1% 174MB/s ± 1% +0.99% (p=0.000 n=9+9) Updates #28923. Change-Id: I138f83babdf316fc97697cc18f595c3403c1ddb7 Reviewed-on: https://go-review.googlesource.com/c/go/+/170939 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-04-05encoding/json: use SetBytes in UnmarshalReuse benchmarkDaniel Martí
This was the only benchmark missing the SetBytes call, as spotted earlier by Bryan. It's not required to make the benchmark useful, but it can still be a good way to see how its speed is affected by the reduced allocations: name time/op CodeUnmarshal-8 12.1ms ± 1% CodeUnmarshalReuse-8 11.4ms ± 1% name speed CodeUnmarshal-8 161MB/s ± 1% CodeUnmarshalReuse-8 171MB/s ± 1% name alloc/op CodeUnmarshal-8 3.28MB ± 0% CodeUnmarshalReuse-8 1.94MB ± 0% name allocs/op CodeUnmarshal-8 92.7k ± 0% CodeUnmarshalReuse-8 77.6k ± 0% While at it, remove some unnecessary empty lines. Change-Id: Ib2bd92d5b3237b8f3092e8c6f863dab548fee2f5 Reviewed-on: https://go-review.googlesource.com/c/go/+/170938 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-04-03encoding/json: speed up tokenization of literalsDaniel Martí
Decoder.Decode and Unmarshal actually scan the input bytes twice - the first time to check for syntax errors and the length of the value, and the second to perform the decoding. It's in the second scan that we actually tokenize the bytes. Since syntax errors aren't a possibility, we can take shortcuts. In particular, literals such as quoted strings are very common in JSON, so we can avoid a lot of work by special casing them. name old time/op new time/op delta CodeDecoder-8 10.3ms ± 1% 9.1ms ± 0% -11.89% (p=0.002 n=6+6) UnicodeDecoder-8 342ns ± 0% 283ns ± 0% -17.25% (p=0.000 n=6+5) DecoderStream-8 239ns ± 0% 230ns ± 0% -3.90% (p=0.000 n=6+5) CodeUnmarshal-8 11.0ms ± 0% 9.8ms ± 0% -11.45% (p=0.002 n=6+6) CodeUnmarshalReuse-8 10.3ms ± 0% 9.0ms ± 0% -12.72% (p=0.004 n=5+6) UnmarshalString-8 104ns ± 0% 92ns ± 0% -11.35% (p=0.002 n=6+6) UnmarshalFloat64-8 93.2ns ± 0% 87.6ns ± 0% -6.01% (p=0.010 n=6+4) UnmarshalInt64-8 74.5ns ± 0% 71.5ns ± 0% -3.91% (p=0.000 n=5+6) name old speed new speed delta CodeDecoder-8 189MB/s ± 1% 214MB/s ± 0% +13.50% (p=0.002 n=6+6) UnicodeDecoder-8 40.9MB/s ± 0% 49.5MB/s ± 0% +20.96% (p=0.002 n=6+6) CodeUnmarshal-8 176MB/s ± 0% 199MB/s ± 0% +12.93% (p=0.002 n=6+6) Updates #28923. Change-Id: I7a5e2aef51bd4ddf2004aad24210f6f50e01eaeb Reviewed-on: https://go-review.googlesource.com/c/go/+/151042 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-18encoding/json: fix performance regression in the decoderDaniel Martí
In golang.org/cl/145218, a feature was added where the JSON decoder would keep track of the entire path to a field when reporting an UnmarshalTypeError. However, we all failed to check if this affected the benchmarks - myself included, as a reviewer. Below are the numbers comparing the CL's parent with itself, once it was merged: name old time/op new time/op delta CodeDecoder-8 12.9ms ± 1% 28.2ms ± 2% +119.33% (p=0.002 n=6+6) name old speed new speed delta CodeDecoder-8 151MB/s ± 1% 69MB/s ± 3% -54.40% (p=0.002 n=6+6) name old alloc/op new alloc/op delta CodeDecoder-8 2.74MB ± 0% 109.39MB ± 0% +3891.83% (p=0.002 n=6+6) name old allocs/op new allocs/op delta CodeDecoder-8 77.5k ± 0% 168.5k ± 0% +117.30% (p=0.004 n=6+5) The reason why the decoder got twice as slow is because it now allocated ~40x as many objects, which puts a lot of pressure on the garbage collector. The reason is that the CL concatenated strings every time a nested field was decoded. In other words, practically every field generated garbage when decoded. This is hugely wasteful, especially considering that the vast majority of JSON decoding inputs won't return UnmarshalTypeError. Instead, use a stack of fields, and make sure to always use the same backing array, to ensure we only need to grow the slice to the maximum depth once. The original CL also introduced a bug. The field stack string wasn't reset to its original state when reaching "d.opcode == scanEndObject", so the last field in a decoded struct could leak. For example, an added test decodes a list of structs, and encoding/json before this CL would fail: got: cannot unmarshal string into Go struct field T.Ts.Y.Y.Y of type int want: cannot unmarshal string into Go struct field T.Ts.Y of type int To fix that, simply reset the stack after decoding every field, even if it's the last. Below is the original performance versus this CL. There's a tiny performance hit, probably due to the append for every decoded field, but at least we're back to the usual ~150MB/s. name old time/op new time/op delta CodeDecoder-8 12.9ms ± 1% 13.0ms ± 1% +1.25% (p=0.009 n=6+6) name old speed new speed delta CodeDecoder-8 151MB/s ± 1% 149MB/s ± 1% -1.24% (p=0.009 n=6+6) name old alloc/op new alloc/op delta CodeDecoder-8 2.74MB ± 0% 2.74MB ± 0% +0.00% (p=0.002 n=6+6) name old allocs/op new allocs/op delta CodeDecoder-8 77.5k ± 0% 77.5k ± 0% +0.00% (p=0.002 n=6+6) Finally, make all of these benchmarks report allocs by default. The decoder ones are pretty sensitive to generated garbage, so ReportAllocs would have made the performance regression more obvious. Change-Id: I67b50f86b2e72f55539429450c67bfb1a9464b67 Reviewed-on: https://go-review.googlesource.com/c/go/+/167978 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-03-08all: simplify multiple for loopsDaniel Martí
If a for loop has a simple condition and begins with a simple "if x { break; }"; we can simply add "!x" to the loop's condition. While at it, simplify a few assignments to use the common pattern "x := staticDefault; if cond { x = otherValue(); }". Finally, simplify a couple of var declarations. Change-Id: I413982c6abd32905adc85a9a666cb3819139c19f Reviewed-on: https://go-review.googlesource.com/c/go/+/165342 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-03-05all: join a few chained ifsDaniel Martí
I had been finding these over a year or so, but none were big enough changes to warrant CLs. They're a handful now, so clean them all up in a single commit. The smaller bodies get a bit simpler, but most importantly, the larger bodies get unindented. Change-Id: I5707a6fee27d4c9ff9efd3d363af575d7a4bf2aa Reviewed-on: https://go-review.googlesource.com/c/go/+/165340 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-03-05encoding/json: add Path to UnmarshalTypeErrorLE Manh Cuong
When parsing nested object, UnmarshalTypeError does not contain actual path to nested field in original JSON. This commit change Field to contain the full path to that field. One can get the Field name by stripping all the leading path elements. Fixes #22369 Change-Id: I6969cc08abe8387a351e3fb2944adfaa0dccad2a Reviewed-on: https://go-review.googlesource.com/c/go/+/145218 Reviewed-by: Daniel Martí <mvdan@mvdan.cc> Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-02-27encoding/json: remove use of DeepEqual for testing errorsMarcel van Lohuizen
Comparing errors using DeepEqual breaks if frame information is added as proposed in Issue #29934. Updates #29934. Change-Id: Ib430c9ddbe588dd1dd51314c408c74c07285e1ff Reviewed-on: https://go-review.googlesource.com/c/162179 Run-TryBot: Marcel van Lohuizen <mpvl@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Russ Cox <rsc@golang.org> Reviewed-by: Damien Neil <dneil@google.com>
2019-02-26encoding/json: add example for json.HTMLEscapeKetan Parmar
Change-Id: Ib00fcfd46eae27eea0a3d4cab4406f4c461fb57b Reviewed-on: https://go-review.googlesource.com/c/160517 Reviewed-by: Andrew Bonventre <andybons@golang.org> Run-TryBot: Andrew Bonventre <andybons@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-01-15encoding/json: add comment for mashalererror structGuilhermeCaruso
Change-Id: Iaabbfe5a4c1bbedd19d4087f1b79e5a38bdd3878 GitHub-Last-Rev: 55c91fc19074dacc66623aa7ff2286b11ccd5340 GitHub-Pull-Request: golang/go#29752 Reviewed-on: https://go-review.googlesource.com/c/157958 Reviewed-by: Ian Lance Taylor <iant@golang.org>
2018-12-12encoding/xml, encoding/json: docs and examples using custom marshalersLeigh McCulloch
Both the encoding/xml and encoding/json packages support custom marshalers for JSON and XML, as well as the basic encoding.TextMarshaler and encoding.TextUnmarshaler interfaces, but the docs and examples for these are missing. There are docs for how to use encoding.TextMarshaler and encoding.TextUnmarshaler in encoding/json, but not encoding/xml. There are no examples for how to use them with either json or xml. This commit includes docs for encoding/xml and examples for both encoding/json and encoding/xml. There is an example using custom marshalers MarshalJSON and UnmarshalJSON in encoding/json, but not MarshalXML and UnmarshalXML in encoding/json. These docs are more so necessary for encoding/xml because the complexities of XML documents is significantly greater than JSON documents which more often leads to the need for custom marshaling. The encoding/json package includes an example of how to write a custom marshaler, and this commit includes the same example for the xml package. All examples are mirrored off the existing custom marshaler example in encoding/json. Fixes #6859 Change-Id: Ic93abc27c0b4d5e48dea6ede4e20b1bedca4ab39 Reviewed-on: https://go-review.googlesource.com/c/76350 Run-TryBot: Ian Lance Taylor <iant@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2018-10-28encoding/json: add example to Validhearot
Change-Id: I411483d76a2ca91cd15ff42ae1adb9134486d183 Reviewed-on: https://go-review.googlesource.com/c/145278 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2018-10-16encoding/json: always verify we can get a field's valueDaniel Martí
Calling .Interface on a struct field's reflect.Value isn't always safe. For example, if that field is an unexported anonymous struct. We only descended into this branch if the struct type had any methods, so this bug had gone unnoticed for a few release cycles. Add the check, and add a simple test case. Fixes #28145. Change-Id: I02f7e0ab9a4a0c18a5e2164211922fe9c3d30f64 Reviewed-on: https://go-review.googlesource.com/c/141537 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2018-10-16encoding/json: fix "data changed underfoot?" panicDaniel Martí
Given a program as follows: data := []byte(`{"F": { "a": 2, "3": 4 }}`) json.Unmarshal(data, &map[string]map[int]int{}) The JSON package should error, as "a" is not a valid integer. However, we'd encounter a panic: panic: JSON decoder out of sync - data changing underfoot? The reason was that decodeState.object would return a nil error on encountering the invalid map key string, while saving the key type error for later. This broke if we were inside another object, as we would abruptly end parsing the nested object, leaving the decoder in an unexpected state. To fix this, simply avoid storing the map element and continue decoding the object, to leave the decoder state exactly as if we hadn't seen an invalid key type. This affected both signed and unsigned integer keys, so fix both and add two test cases. Updates #28189. Change-Id: I8a6204cc3ff9fb04ed769df7a20a824c8b94faff Reviewed-on: https://go-review.googlesource.com/c/142518 Reviewed-by: Ian Lance Taylor <iant@golang.org>
generated by cgit v1.3 (git 2.53.0) at 2026-04-15 12:42:12 +0000