aboutsummaryrefslogtreecommitdiff
path: root/src/encoding
AgeCommit message (Collapse)Author
2019-09-16encoding/json: make Number with the ,string option marshal with quotesLucas Bremgartner
Add quotes when marshaling a json.Number with the string option set via a struct tag. This ensures that the resulting json can be unmarshaled into the source struct without error. Fixes #34268 Change-Id: Ide167d9dec77019554870b5957b37dc258119d81 GitHub-Last-Rev: dde81b71208be01c253bb87dbb6f81ac6e0785be GitHub-Pull-Request: golang/go#34269 Reviewed-on: https://go-review.googlesource.com/c/go/+/195043 Reviewed-by: Daniel Martí <mvdan@mvdan.cc> Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-09-11encoding/json: encode nil encoding.TextMarshaler instance as "null"William Poussier
Fixes #34235. Change-Id: Ia3795fd18860530fa6a4b171545f525e784ffdcb GitHub-Last-Rev: 1a319c452857818f7aaf22ef46823b43ca9b2276 GitHub-Pull-Request: golang/go#34238 Reviewed-on: https://go-review.googlesource.com/c/go/+/194642 Reviewed-by: Daniel Martí <mvdan@mvdan.cc> Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-09-11encoding/json: fix and optimize marshal for quoted stringLucas Bremgartner
Since Go 1.2 every string can be marshaled to JSON without error even if it contains invalid UTF-8 byte sequences. Therefore there is no need to use Marshal again for the only reason of enclosing the string in double quotes. Not using Marshal here also removes the error check as there has not been a way for Marshal to fail anyway. name old time/op new time/op delta Issue34127-4 360ns ± 3% 200ns ± 3% -44.56% (p=0.008 n=5+5) name old alloc/op new alloc/op delta Issue34127-4 56.0B ± 0% 40.0B ± 0% -28.57% (p=0.008 n=5+5) name old allocs/op new allocs/op delta Issue34127-4 3.00 ± 0% 2.00 ± 0% -33.33% (p=0.008 n=5+5) Fixes #34154 Change-Id: Ib60dc11980f9b20d8bef2982de7168943d632263 GitHub-Last-Rev: 9b0ac1d4c5318b6bf9ed7930320f2bd755f9939c GitHub-Pull-Request: golang/go#34127 Reviewed-on: https://go-review.googlesource.com/c/go/+/193604 Reviewed-by: Daniel Martí <mvdan@mvdan.cc> Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-09-11encoding/base32: increase performance and code reuseSven Taute
Add benchmarks for the Encode/Decode functions operating on []byte and increase decoding performance by removing the calls to strings.Map/bytes.Map and reusing the newline filtering code that is used by NewDecoder. Cut allocations in half for DecodeString. Comparison using the new benchmarks: name old time/op new time/op delta Encode 16.7µs ± 1% 17.0µs ± 2% +2.25% (p=0.000 n=9+9) EncodeToString 21.1µs ± 1% 20.9µs ± 1% -0.96% (p=0.000 n=10+10) Decode 141µs ± 1% 54µs ± 1% -61.51% (p=0.000 n=10+10) DecodeString 81.4µs ± 0% 54.7µs ± 1% -32.79% (p=0.000 n=9+10) name old speed new speed delta Encode 492MB/s ± 1% 481MB/s ± 2% -2.19% (p=0.000 n=9+9) EncodeToString 389MB/s ± 1% 392MB/s ± 1% +0.97% (p=0.000 n=10+10) Decode 93.0MB/s ± 1% 241.6MB/s ± 1% +159.82% (p=0.000 n=10+10) DecodeString 161MB/s ± 0% 240MB/s ± 1% +48.78% (p=0.000 n=9+10) Change-Id: Id53633514a9e14ecd0389d52114b2b8ca64370cb GitHub-Last-Rev: f4be3cf55caf5b89d76d14b7f32422faff39e3c3 GitHub-Pull-Request: golang/go#30376 Reviewed-on: https://go-review.googlesource.com/c/go/+/163598 Run-TryBot: Ian Lance Taylor <iant@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-09-10encoding/json: don't reset before returning buffer to poolandig
Reset is already performed when retrieving from pool Change-Id: Ia810dd18d3e55a1565a5ad435a00d1e46724576c GitHub-Last-Rev: d9df74a4aeb86e5d292c9fc33568a3c9a64a967d GitHub-Pull-Request: golang/go#34195 Reviewed-on: https://go-review.googlesource.com/c/go/+/194338 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
2019-09-10encoding/json: don't indirect pointers when decoding nullRomain Baugue
The indirect method checked the type of the child when indirecting a pointer. If the current value is a pointer and we are decoding null, we can skip this entirely and return early, avoiding the whole descent. Fixes #31776 Change-Id: Ib8b2a2357572c41f56fceac59b5a858980f3f65e Reviewed-on: https://go-review.googlesource.com/c/go/+/174699 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
2019-09-09encoding/asn1: handle ASN1's string type BMPStringsergeilem
This code enables handling of ASN1's string type BMPString, used in some digital signatures. Parsing code taken from golang.org/x/crypto/pkcs12. Change-Id: Ibeae9cf4d8ae7c18f8b5420ad9244a16e117ff6b GitHub-Last-Rev: 694525351411f2ec3982a6bf4ac33be892ce1b12 GitHub-Pull-Request: golang/go#26690 Reviewed-on: https://go-review.googlesource.com/c/go/+/126624 Run-TryBot: Andrew Bonventre <andybons@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Andrew Bonventre <andybons@golang.org>
2019-09-09src: gofmt -sAinar Garipov
Change-Id: I56d7eeaf777ac30886ee77428ca1ac72b77fbf7d Reviewed-on: https://go-review.googlesource.com/c/go/+/193849 Run-TryBot: Dave Cheney <dave@cheney.net> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-09-09encoding/json: clarify Unmarshal behavior for map keysEli Bendersky
This is a documentation-only change Fixes #33298 Change-Id: I816058a872b57dc868dff11887214d9de92d9342 Reviewed-on: https://go-review.googlesource.com/c/go/+/188821 Reviewed-by: Daniel Martí <mvdan@mvdan.cc> Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-09-08all: fix typosAinar Garipov
Use the following (suboptimal) script to obtain a list of possible typos: #!/usr/bin/env sh set -x git ls-files |\ grep -e '\.\(c\|cc\|go\)$' |\ xargs -n 1\ awk\ '/\/\// { gsub(/.*\/\//, ""); print; } /\/\*/, /\*\// { gsub(/.*\/\*/, ""); gsub(/\*\/.*/, ""); }' |\ hunspell -d en_US -l |\ grep '^[[:upper:]]\{0,1\}[[:lower:]]\{1,\}$' |\ grep -v -e '^.\{1,4\}$' -e '^.\{16,\}$' |\ sort -f |\ uniq -c |\ awk '$1 == 1 { print $2; }' Then, go through the results manually and fix the most obvious typos in the non-vendored code. Change-Id: I3cb5830a176850e1a0584b8a40b47bde7b260eae Reviewed-on: https://go-review.googlesource.com/c/go/+/193848 Reviewed-by: Robert Griesemer <gri@golang.org>
2019-09-02std: remove unused bits of code all over the placeDaniel Martí
Some were never used, and some haven't been used for years. One exception is net/http's readerAndCloser, which was only used in a test. Move it to a test file. While at it, remove a check in regexp that could never fire; the field is an uint32, so it can never be negative. Change-Id: Ia2200f6afa106bae4034045ea8233b452f38747b Reviewed-on: https://go-review.googlesource.com/c/go/+/192621 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-09-02encoding/json: fix scanner byte offset on scanEnditchyny
scanEnd is delayed one byte so we decrement the scanner bytes count by 1 to ensure that this value is correct in the next call of Decode. Fixes #32399 Change-Id: I8c8698e7f95bbcf0373aceaa05319819eae9d86f GitHub-Last-Rev: 0ac25d8de23d38c7ac577faddc6983571023f561 GitHub-Pull-Request: golang/go#32598 Reviewed-on: https://go-review.googlesource.com/c/go/+/182117 Reviewed-by: Daniel Martí <mvdan@mvdan.cc> Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-09-01encoding/json: revert Compact HTML escaping documentationTom Thorogood
This partly reverts CL 173417 as it incorrectly documented that Compact performed HTML escaping and the output was safe to embed inside HTML <script> tags. This has never been true. Although Compact does escape U+2028 and U+2029, it doesn't escape <, > or &. Compact is thus only performing a subset of HTML escaping and it's output is not safe to embed inside HTML <script> tags. A more complete fix would be for Compact to either never perform any HTML escaping, as it was prior to CL 10883045, or to actually perform the same HTML escaping as HTMLEscape. Neither change is likely safe enough for go1.13. Updates #30357 Change-Id: I912f0fe9611097d988048b28228c4a5b985080ba GitHub-Last-Rev: aebababc9233c5705785b225377e80096d4bb8c4 GitHub-Pull-Request: golang/go#33427 Reviewed-on: https://go-review.googlesource.com/c/go/+/188717 Reviewed-by: Daniel Martí <mvdan@mvdan.cc> Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-09-01encoding/json: fix panic for nil instances of TextMarshaler in map keysWilliam Poussier
This change adds a a check in the encodeWithString.resolve method to ensure that a reflect.Value with kind Ptr is not nil before the type assertion to TextMarshaler. If the value is nil, the method returns a nil error, and the map key encodes to an empty string. Fixes #33675 Change-Id: I0a04cf690ae67006f6a9c5f8cbb4cc99d236bca8 GitHub-Last-Rev: 6c987c90846f854e21814dbfb3a073605ec8a94c GitHub-Pull-Request: golang/go#33700 Reviewed-on: https://go-review.googlesource.com/c/go/+/190697 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
2019-08-28encoding/xml: rename fInnerXml to fInnerXMLEddie Scholtz
Per the code review guidelines: "Words in names that are initialisms or acronyms have a consistent case." Change-Id: I347b02d2f48455f2cbbc040191ba197e3e8f23fc Reviewed-on: https://go-review.googlesource.com/c/go/+/191970 Reviewed-by: Daniel Martí <mvdan@mvdan.cc> Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-08-27encoding/json: remove unnecessary isValidNumber callDaniel Martí
The decoder called this function to check numbers being decoded into a json.Number. However, these can't be quoted as strings, so the tokenizer has already verified they are valid JSON numbers. Verified this by adding a test with such an input. As expected, it produces a syntax error, not the fmt.Errorf - that line could never execute. Since the only remaining non-test caller of isvalidnumber is in encode.go, move the function there. This change should slightly reduce the amount of work when decoding into json.Number, though that isn't very common nor part of any current benchmarks. Change-Id: I67a1723deb3d18d5b542d6dd35f3ae56a43f23eb Reviewed-on: https://go-review.googlesource.com/c/go/+/184817 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-08-27encoding/json: fix the broken "overwriting of data" testsDaniel Martí
Because TestUnmarshal actually allocates a new value to decode into using ptr's pointer type, any existing data is thrown away. This was harmless in alomst all of the test cases, minus the "overwriting of data" ones added in 2015 in CL 12209. I spotted that nothing covered decoding a JSON array with few elements into a slice which already had many elements. I initially assumed that the code was buggy or that some code could be removed, when in fact there simply wasn't any code covering the edge case. Move those two tests to TestPrefilled, which already served a very similar purpose. Remove the map case, as TestPrefilled already has plenty of prefilled map cases. Moreover, we no longer reset an entire map when decoding, as per the godoc: To unmarshal a JSON object into a map, Unmarshal first establishes a map to use. If the map is nil, Unmarshal allocates a new map. Otherwise Unmarshal reuses the existing map, keeping existing entries. Finally, to ensure that ptr is used correctly in the future, make TestUnmarshal error if it's anything other than a pointer to a zero value. That is, the only correct use should be new(type). Don't rename the ptr field, as that would be extremely noisy and cause unwanted merge conflicts. Change-Id: I41e3ecfeae42d877ac5443a6bd622ac3d6c8120c Reviewed-on: https://go-review.googlesource.com/c/go/+/185738 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
2019-08-21Revert "encoding/json: avoid work when unquoting strings"Filippo Valsorda
This reverts CL 151157. CL 151157 introduced a crash when decoding into ",string" fields. It came with a moderate speedup, so at this stage of the release cycle let's just revert it, and reapply it in Go 1.14 with the fix in CL 190659. Also applied the test cases from CL 190659. Updates #33728 Change-Id: Ie46e2bc15224b251888580daf6b79d5865f3878e Reviewed-on: https://go-review.googlesource.com/c/go/+/190909 Run-TryBot: Andrew Bonventre <andybons@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Andrew Bonventre <andybons@golang.org>
2019-08-18encoding/json: fix format string in the Fuzz funcDmitry Vyukov
Currently test build fails with: $ go test -tags=gofuzz encoding/json encoding/json/fuzz.go:36:4: Println call has possible formatting directive %s FAIL encoding/json [build failed] Change-Id: I23aef44a421ed0e7bcf48b74ac5a8c6768a4841b Reviewed-on: https://go-review.googlesource.com/c/go/+/190698 Run-TryBot: Ian Lance Taylor <iant@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-07-31encoding/json: clarify Marshal behavior for string keys of mapsEli Bendersky
This is a documentation-only change. Fixes #28827 Change-Id: Ife9ab997809048784f35872b09905bc209a05eff Reviewed-on: https://go-review.googlesource.com/c/go/+/188417 Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-07-28all: change some function documentation to be more idiomaticDominik Honnef
Change-Id: I932de9bb061a8ba3332ef03207983e8b98d6f1e5 Reviewed-on: https://go-review.googlesource.com/c/go/+/187918 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-07-06encoding/json: obey SetEscapeHTML in all MarshalJSON casesDaniel Martí
It wasn't obeyed in the case where the MarshalJSON method uses a pointer receiver, and the encoder grabs the address of a value to find that method. addrMarshalerEncoder is the function that does this work, but it ignored opts.escapeHTML. Here's the before and after of the added test case, which was failing before the fix. Now the two cases are correct and consistent. {"NonPtr":"<str>","Ptr":"\u003cstr\u003e"} {"NonPtr":"<str>","Ptr":"<str>"} Fixes #32896. Change-Id: Idc53077ece074973558bd3bb5ad036380db0d02c Reviewed-on: https://go-review.googlesource.com/c/go/+/184757 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com> Reviewed-by: Caleb Spare <cespare@gmail.com>
2019-05-22encoding/gob: properly ignore errorsLeon Klingele
Change-Id: I8827cef0f57459384329c50c51795350da0ede4b GitHub-Last-Rev: c9ad9e12b5a0fff47c21a8c299b762b64b8c9c7c GitHub-Pull-Request: golang/go#30010 Reviewed-on: https://go-review.googlesource.com/c/go/+/160434 Run-TryBot: Bryan C. Mills <bcmills@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-05-22all: shorten some testsRuss Cox
Shorten some of the longest tests that run during all.bash. Removes 7r 50u 21s from all.bash. After this change, all.bash is under 5 minutes again on my laptop. For #26473. Change-Id: Ie0460aa935808d65460408feaed210fbaa1d5d79 Reviewed-on: https://go-review.googlesource.com/c/go/+/177559 Run-TryBot: Russ Cox <rsc@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-05-21all: remove PEM-encoded private keys from testsRuss Cox
Gerrit is complaining about pushes that affect these files and forcing people to use -o nokeycheck, which defeats the point of the check. Hide the keys from this kind of scan by marking them explicitly as testing keys. This is a little annoying but better than training everyone who ever edits one of these test files to reflexively override the Gerrit check. The only remaining keys explicitly marked as private instead of testing are in examples, and there's not much to do about those. Hopefully they are not edited as much. Change-Id: I4431592b5266cb39fe6a80b40e742d97da803a0b Reviewed-on: https://go-review.googlesource.com/c/go/+/178178 Run-TryBot: Russ Cox <rsc@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-05-09encoding/gob: rename encBuffer.WriteByte to writeByteRuss Cox
Renaming the method makes clear, both to readers and to vet, that this method is not the implementation of io.ByteWriter. Working toward making the tree vet-safe instead of having so many exceptions in cmd/vet/all/whitelist. For #31916. Change-Id: I5b509eb7f0118d5f2d3c6e352ff2849cd5a3071e Reviewed-on: https://go-review.googlesource.com/c/go/+/176110 Run-TryBot: Russ Cox <rsc@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-05-06all: simplify code using "gofmt -s -w"Shulhan
Most changes are removing redundant declaration of type when direct instantiating value of map or slice, e.g. []T{T{}} become []T{{}}. Small changes are removing the high order of subslice if its value is the length of slice itself, e.g. T[:len(T)] become T[:]. The following file is excluded due to incompatibility with go1.4, - src/cmd/compile/internal/gc/ssa.go Change-Id: Id3abb09401795ce1e6da591a89749cba8502fb26 Reviewed-on: https://go-review.googlesource.com/c/go/+/166437 Run-TryBot: Dave Cheney <dave@cheney.net> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-05-04all: add Unwrap and Is methods to various error typesDamien Neil
Add Unwrap methods to types which wrap an underlying error: "encodinc/csv".ParseError "encoding/json".MarshalerError "net/http".transportReadFromServerError "net".OpError "net".DNSConfigError "net/url".Error "os/exec".Error "signal/internal/pty".PtyError "text/template".ExecError Add os.ErrTemporary. A case could be made for putting this error value in package net, since no exported error types in package os include a Temporary method. However, syscall errors returned from the os package do include this method. Add Is methods to error types with a Timeout or Temporary method, making errors.Is(err, os.Err{Timeout,Temporary}) equivalent to testing the corresponding method: "context".DeadlineExceeded "internal/poll".TimeoutError "net".adrinfoErrno "net".OpError "net".DNSError "net/http".httpError "net/http".tlsHandshakeTimeoutError "net/pipe".timeoutError "net/url".Error Updates #30322 Updates #29934 Change-Id: I409fb20c072ea39116ebfb8c7534d493483870dc Reviewed-on: https://go-review.googlesource.com/c/go/+/170037 Run-TryBot: Damien Neil <dneil@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Marcel van Lohuizen <mpvl@golang.org>
2019-04-30encoding/gob: adding missing fuzz skip to one of the fuzz testsBrad Fitzpatrick
It's slow & often times out randomly on longtest builders. Not useful. Fixes #31517 Change-Id: Icedbb0c94fbe43d04e8b47d5785ac61c5e2d8750 Reviewed-on: https://go-review.googlesource.com/c/go/+/174522 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-04-30encoding/csv: add a Fuzz functionRomain Baugue
Adds a sample Fuzz test function to package encoding/csv based on https://github.com/dvyukov/go-fuzz-corpus/blob/master/csv/main.go Updates #19109 Updates #31309 Change-Id: Ieb0cb6caa1df72dbb7e29df4bdeed0bfa91187d3 Reviewed-on: https://go-review.googlesource.com/c/go/+/174302 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-04-30encoding/json: fix Unmarshal hang on recursive pointersLE Manh Cuong
indirect walks down v until it gets to a non-pointer. But it does not handle the case when v is a pointer to itself, like in: var v interface{} v = &v Unmarshal(b, v) So just stop immediately if we see v is a pointer to itself. Fixes #31740 Change-Id: Ie396264119e24d70284cd9bf76dcb2050babb069 Reviewed-on: https://go-review.googlesource.com/c/go/+/174337 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Daniel Martí <mvdan@mvdan.cc> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-04-29encoding/json: add a Fuzz functionRomain Baugue
Adds a sample Fuzz test function to package encoding/json following the guidelines defined in #31309, based on https://github.com/dvyukov/go-fuzz-corpus/blob/master/json/json.go Fixes #31309 Updates #19109 Change-Id: I5fe04d9a5f41c0de339f8518dae30896ec14e356 Reviewed-on: https://go-review.googlesource.com/c/go/+/174058 Reviewed-by: Dmitry Vyukov <dvyukov@google.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Dmitry Vyukov <dvyukov@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-04-23encoding/json: document HTML escaping in CompactRuss Cox
Make explicit that Compact does HTML escaping. Fixes #30357. Change-Id: I4648f8f3e907d659db977d07253f716df6e07d7b Reviewed-on: https://go-review.googlesource.com/c/go/+/173417 Run-TryBot: Russ Cox <rsc@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-04-23encoding/json: index names for the struct decoderDaniel Martí
In the common case, structs have a handful of fields and most inputs match struct field names exactly. The previous code would do a linear search over the fields, stopping at the first exact match, and otherwise using the first case insensitive match. This is unfortunate, because it means that for the common case, we'd do a linear search with bytes.Equal. Even for structs with only two or three fields, that is pretty wasteful. Worse even, up until the exact match was found via the linear search, all previous fields would run their equalFold functions, which aren't cheap even in the simple case. Instead, cache a map along with the field list that indexes the fields by their name. This way, a case sensitive field search doesn't involve a linear search, nor does it involve any equalFold func calls. This patch should also slightly speed up cases where there's a case insensitive match but not a case sensitive one, as then we'd avoid calling bytes.Equal on all the fields. Though that's not a common case, and there are no benchmarks for it. name old time/op new time/op delta CodeDecoder-8 11.0ms ± 0% 10.6ms ± 1% -4.42% (p=0.000 n=9+10) name old speed new speed delta CodeDecoder-8 176MB/s ± 0% 184MB/s ± 1% +4.62% (p=0.000 n=9+10) name old alloc/op new alloc/op delta CodeDecoder-8 2.28MB ± 0% 2.28MB ± 0% ~ (p=0.725 n=10+10) name old allocs/op new allocs/op delta CodeDecoder-8 76.9k ± 0% 76.9k ± 0% ~ (all equal) Updates #28923. Change-Id: I9929c1f06c76505e5b96914199315dbdaae5dc76 Reviewed-on: https://go-review.googlesource.com/c/go/+/172918 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-04-22encoding/json: avoid work when unquoting stringsDaniel Martí
We can work out how many bytes can be unquoted trivially in rescanLiteral, which already iterates over a string's bytes. Removing the extra loop in unquoteBytes simplifies the function and speeds it up, especially when decoding simple strings, which are common. While at it, we can remove unnecessary checks like len(s)<2 and s[0]=='"'. Add a comment explaining why. name old time/op new time/op delta CodeDecoder-8 11.2ms ± 0% 11.1ms ± 1% -1.63% (p=0.000 n=9+10) name old speed new speed delta CodeDecoder-8 173MB/s ± 0% 175MB/s ± 1% +1.66% (p=0.000 n=9+10) Updates #28923. Change-Id: I2436a3a7f8148a2f7a6a4cdbd7dec6b32ef5e20c Reviewed-on: https://go-review.googlesource.com/c/go/+/151157 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-04-13encoding/json: remove a bounds check in readValueDaniel Martí
readValue is a hot function, clocking in at ~13% flat CPU use in CodeDecoder. In particular, looping over the bytes is slow. That's partially because the code contains a bounds check at the start of the loop. The source of the problem is that scanp is a signed integer, and comes from a field, so the compiler doesn't know that it's non-negative. Help it with a simple and comparatively cheap hint. While at it, use scanp as the index variable directly, removing the need for a duplicate index variable which is later added back into scanp. name old time/op new time/op delta CodeDecoder-8 11.3ms ± 1% 11.2ms ± 1% -0.98% (p=0.000 n=9+9) name old speed new speed delta CodeDecoder-8 172MB/s ± 1% 174MB/s ± 1% +0.99% (p=0.000 n=9+9) Updates #28923. Change-Id: I138f83babdf316fc97697cc18f595c3403c1ddb7 Reviewed-on: https://go-review.googlesource.com/c/go/+/170939 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-04-05encoding/json: use SetBytes in UnmarshalReuse benchmarkDaniel Martí
This was the only benchmark missing the SetBytes call, as spotted earlier by Bryan. It's not required to make the benchmark useful, but it can still be a good way to see how its speed is affected by the reduced allocations: name time/op CodeUnmarshal-8 12.1ms ± 1% CodeUnmarshalReuse-8 11.4ms ± 1% name speed CodeUnmarshal-8 161MB/s ± 1% CodeUnmarshalReuse-8 171MB/s ± 1% name alloc/op CodeUnmarshal-8 3.28MB ± 0% CodeUnmarshalReuse-8 1.94MB ± 0% name allocs/op CodeUnmarshal-8 92.7k ± 0% CodeUnmarshalReuse-8 77.6k ± 0% While at it, remove some unnecessary empty lines. Change-Id: Ib2bd92d5b3237b8f3092e8c6f863dab548fee2f5 Reviewed-on: https://go-review.googlesource.com/c/go/+/170938 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-04-03encoding/json: speed up tokenization of literalsDaniel Martí
Decoder.Decode and Unmarshal actually scan the input bytes twice - the first time to check for syntax errors and the length of the value, and the second to perform the decoding. It's in the second scan that we actually tokenize the bytes. Since syntax errors aren't a possibility, we can take shortcuts. In particular, literals such as quoted strings are very common in JSON, so we can avoid a lot of work by special casing them. name old time/op new time/op delta CodeDecoder-8 10.3ms ± 1% 9.1ms ± 0% -11.89% (p=0.002 n=6+6) UnicodeDecoder-8 342ns ± 0% 283ns ± 0% -17.25% (p=0.000 n=6+5) DecoderStream-8 239ns ± 0% 230ns ± 0% -3.90% (p=0.000 n=6+5) CodeUnmarshal-8 11.0ms ± 0% 9.8ms ± 0% -11.45% (p=0.002 n=6+6) CodeUnmarshalReuse-8 10.3ms ± 0% 9.0ms ± 0% -12.72% (p=0.004 n=5+6) UnmarshalString-8 104ns ± 0% 92ns ± 0% -11.35% (p=0.002 n=6+6) UnmarshalFloat64-8 93.2ns ± 0% 87.6ns ± 0% -6.01% (p=0.010 n=6+4) UnmarshalInt64-8 74.5ns ± 0% 71.5ns ± 0% -3.91% (p=0.000 n=5+6) name old speed new speed delta CodeDecoder-8 189MB/s ± 1% 214MB/s ± 0% +13.50% (p=0.002 n=6+6) UnicodeDecoder-8 40.9MB/s ± 0% 49.5MB/s ± 0% +20.96% (p=0.002 n=6+6) CodeUnmarshal-8 176MB/s ± 0% 199MB/s ± 0% +12.93% (p=0.002 n=6+6) Updates #28923. Change-Id: I7a5e2aef51bd4ddf2004aad24210f6f50e01eaeb Reviewed-on: https://go-review.googlesource.com/c/go/+/151042 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-18encoding/json: fix performance regression in the decoderDaniel Martí
In golang.org/cl/145218, a feature was added where the JSON decoder would keep track of the entire path to a field when reporting an UnmarshalTypeError. However, we all failed to check if this affected the benchmarks - myself included, as a reviewer. Below are the numbers comparing the CL's parent with itself, once it was merged: name old time/op new time/op delta CodeDecoder-8 12.9ms ± 1% 28.2ms ± 2% +119.33% (p=0.002 n=6+6) name old speed new speed delta CodeDecoder-8 151MB/s ± 1% 69MB/s ± 3% -54.40% (p=0.002 n=6+6) name old alloc/op new alloc/op delta CodeDecoder-8 2.74MB ± 0% 109.39MB ± 0% +3891.83% (p=0.002 n=6+6) name old allocs/op new allocs/op delta CodeDecoder-8 77.5k ± 0% 168.5k ± 0% +117.30% (p=0.004 n=6+5) The reason why the decoder got twice as slow is because it now allocated ~40x as many objects, which puts a lot of pressure on the garbage collector. The reason is that the CL concatenated strings every time a nested field was decoded. In other words, practically every field generated garbage when decoded. This is hugely wasteful, especially considering that the vast majority of JSON decoding inputs won't return UnmarshalTypeError. Instead, use a stack of fields, and make sure to always use the same backing array, to ensure we only need to grow the slice to the maximum depth once. The original CL also introduced a bug. The field stack string wasn't reset to its original state when reaching "d.opcode == scanEndObject", so the last field in a decoded struct could leak. For example, an added test decodes a list of structs, and encoding/json before this CL would fail: got: cannot unmarshal string into Go struct field T.Ts.Y.Y.Y of type int want: cannot unmarshal string into Go struct field T.Ts.Y of type int To fix that, simply reset the stack after decoding every field, even if it's the last. Below is the original performance versus this CL. There's a tiny performance hit, probably due to the append for every decoded field, but at least we're back to the usual ~150MB/s. name old time/op new time/op delta CodeDecoder-8 12.9ms ± 1% 13.0ms ± 1% +1.25% (p=0.009 n=6+6) name old speed new speed delta CodeDecoder-8 151MB/s ± 1% 149MB/s ± 1% -1.24% (p=0.009 n=6+6) name old alloc/op new alloc/op delta CodeDecoder-8 2.74MB ± 0% 2.74MB ± 0% +0.00% (p=0.002 n=6+6) name old allocs/op new allocs/op delta CodeDecoder-8 77.5k ± 0% 77.5k ± 0% +0.00% (p=0.002 n=6+6) Finally, make all of these benchmarks report allocs by default. The decoder ones are pretty sensitive to generated garbage, so ReportAllocs would have made the performance regression more obvious. Change-Id: I67b50f86b2e72f55539429450c67bfb1a9464b67 Reviewed-on: https://go-review.googlesource.com/c/go/+/167978 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-03-13encoding/base64: speed up the decoderDaniel Martí
Most of the decoding time is spent in the first Decode loop, since the rest of the function only deals with the few remaining bytes. Any unnecessary work done in that loop body matters tremendously. One such unnecessary bottleneck was the use of the enc.decodeMap table. Since enc is a pointer receiver, and the field is used within the non-inlineable function decode64, the decoder must perform a nil check at every iteration. To fix that, move the enc.decodeMap uses to the parent function, where we can lift the nil check outside the loop. That gives roughly a 15% speed-up. The function no longer performs decoding per se, so rename it. While at it, remove the now unnecessary receivers. An unfortunate side effect of this change is that the loop now contains eight bounds checks on src instead of just one. However, not having to slice src plus the nil check removal well outweigh the added cost. The other piece that made decode64 slow was that it wasn't inlined, and had multiple branches. Use a simple bitwise-or trick suggested by Roger Peppe, and collapse the rest of the bitwise logic into a single expression. Inlinability and the reduced branching give a further 10% speed-up. Finally, add these two functions to TestIntendedInlining, since we want them to stay inlinable. Apply the same refactor to decode32 for consistency, and to let 32-bit architectures see a similar performance gain for large inputs. name old time/op new time/op delta DecodeString/2-8 47.3ns ± 1% 45.8ns ± 0% -3.28% (p=0.002 n=6+6) DecodeString/4-8 55.8ns ± 2% 51.5ns ± 0% -7.71% (p=0.004 n=5+6) DecodeString/8-8 64.9ns ± 0% 61.7ns ± 0% -4.99% (p=0.004 n=5+6) DecodeString/64-8 238ns ± 0% 198ns ± 0% -16.54% (p=0.002 n=6+6) DecodeString/8192-8 19.5µs ± 0% 14.6µs ± 0% -24.96% (p=0.004 n=6+5) name old speed new speed delta DecodeString/2-8 84.6MB/s ± 1% 87.4MB/s ± 0% +3.38% (p=0.002 n=6+6) DecodeString/4-8 143MB/s ± 2% 155MB/s ± 0% +8.41% (p=0.004 n=5+6) DecodeString/8-8 185MB/s ± 0% 195MB/s ± 0% +5.29% (p=0.004 n=5+6) DecodeString/64-8 369MB/s ± 0% 442MB/s ± 0% +19.78% (p=0.002 n=6+6) DecodeString/8192-8 560MB/s ± 0% 746MB/s ± 0% +33.27% (p=0.004 n=6+5) Updates #19636. Change-Id: Ib839577b0e3f5a2bb201f5cae580c61365d92894 Reviewed-on: https://go-review.googlesource.com/c/go/+/151177 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Reviewed-by: roger peppe <rogpeppe@gmail.com>
2019-03-08all: simplify multiple for loopsDaniel Martí
If a for loop has a simple condition and begins with a simple "if x { break; }"; we can simply add "!x" to the loop's condition. While at it, simplify a few assignments to use the common pattern "x := staticDefault; if cond { x = otherValue(); }". Finally, simplify a couple of var declarations. Change-Id: I413982c6abd32905adc85a9a666cb3819139c19f Reviewed-on: https://go-review.googlesource.com/c/go/+/165342 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-03-08encoding/gob: update documentation in doc.go for wireTypeRob Pike
It was just out of date. Fixes #30656 Change-Id: I1fab7dd93091865a8240769eca5dd19cdbc78b81 Reviewed-on: https://go-review.googlesource.com/c/go/+/166177 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-05all: join a few chained ifsDaniel Martí
I had been finding these over a year or so, but none were big enough changes to warrant CLs. They're a handful now, so clean them all up in a single commit. The smaller bodies get a bit simpler, but most importantly, the larger bodies get unindented. Change-Id: I5707a6fee27d4c9ff9efd3d363af575d7a4bf2aa Reviewed-on: https://go-review.googlesource.com/c/go/+/165340 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-03-05encoding/base64: do not slice past output unnecessarilyRuss Cox
Base64-encoding 32 bytes results in a 44-byte string. While in general a 44-byte string might decode to 33 bytes, if you take a 44-byte string that actually only encodes 32 bytes, and you try to decode it into 32 bytes, that should succeed. Instead it fails trying to do a useless dst[33:] slice operation. Delete that slice operation. Noticed while preparing CL 156322. Change-Id: I8024bf28a65e2638675b980732b2ff91c66c62cf Reviewed-on: https://go-review.googlesource.com/c/go/+/164628 Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-03-05encoding/hex: simplify encoder arithmeticDaniel Martí
Two additions are faster than two multiplications and one addition. The code seems simpler to me too, as it's more obvious that we advance two destination bytes for each source byte. name old time/op new time/op delta Encode/256-8 374ns ± 0% 331ns ± 0% -11.44% (p=0.008 n=5+5) Encode/1024-8 1.47µs ± 0% 1.29µs ± 0% -11.89% (p=0.004 n=6+5) Encode/4096-8 5.85µs ± 1% 5.15µs ± 0% -11.89% (p=0.004 n=6+5) Encode/16384-8 23.3µs ± 0% 20.6µs ± 0% -11.68% (p=0.004 n=6+5) Change-Id: Iabc63616c1d9fded55fa668ff41dd49efeaa2ea4 Reviewed-on: https://go-review.googlesource.com/c/go/+/151198 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: roger peppe <rogpeppe@gmail.com>
2019-03-05encoding/json: add Path to UnmarshalTypeErrorLE Manh Cuong
When parsing nested object, UnmarshalTypeError does not contain actual path to nested field in original JSON. This commit change Field to contain the full path to that field. One can get the Field name by stripping all the leading path elements. Fixes #22369 Change-Id: I6969cc08abe8387a351e3fb2944adfaa0dccad2a Reviewed-on: https://go-review.googlesource.com/c/go/+/145218 Reviewed-by: Daniel Martí <mvdan@mvdan.cc> Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-03-04encoding/csv: document that Writer is bufferedBrian Kessler
Add documentation that individual Write calls are buffered and copy documentation from bufio.Writer notifying the user to call Flush and Error when all writes are complete. Remove reference to "file" since the implementation is general and allows any io.Writer. Fixes #30045 Change-Id: I50165470e548f296494e764707fbabe36c665015 Reviewed-on: https://go-review.googlesource.com/c/160680 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-03encoding/base64: lift nil check out of encode loopDaniel Martí
Most of the encoding time is spent in the first Encode loop, since the rest of the function only deals with the few remaining bytes. Any unnecessary work done in that loop body matters tremendously. One such unnecessary bottleneck was the use of the enc.encode table. Since enc is a pointer receiver, and the field is first used within the loop, the encoder must perform a nil check at every iteration. Add a dummy use of the field before the start of the loop, to move the nil check there. After that line, the compiler now knows that enc can't be nil, and thus the hot loop is free of nil checks. name old time/op new time/op delta EncodeToString-4 14.7µs ± 0% 13.7µs ± 1% -6.53% (p=0.000 n=10+10) name old speed new speed delta EncodeToString-4 559MB/s ± 0% 598MB/s ± 1% +6.99% (p=0.000 n=10+10) Updates #20206. Change-Id: Icbb523a7bd9e470a8be0a448d1d78ade97ed4ff6 Reviewed-on: https://go-review.googlesource.com/c/151158 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-02encoding/pem: skip whitespace work on most inputsDaniel Martí
encoding/base64 already skips \r and \n when decoding, so this package must only deal with spaces and tabs. Those aren't nearly as common, so we can add a fast path with bytes.ContainsAny to skip the costly alloc and filtering code. name old time/op new time/op delta Decode-8 279µs ± 0% 259µs ± 1% -7.07% (p=0.002 n=6+6) name old speed new speed delta Decode-8 319MB/s ± 0% 343MB/s ± 1% +7.61% (p=0.002 n=6+6) name old alloc/op new alloc/op delta Decode-8 164kB ± 0% 74kB ± 0% -54.90% (p=0.002 n=6+6) name old allocs/op new allocs/op delta Decode-8 12.0 ± 0% 11.0 ± 0% -8.33% (p=0.002 n=6+6) Change-Id: Idfca8700c52f46eb70a4a7e0d2db3bf0124e4699 Reviewed-on: https://go-review.googlesource.com/c/155964 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-02encoding/hex: simplify decoder arithmeticDaniel Martí
Remove all multiplications and divisions from the main decoding loop. name old time/op new time/op delta Decode/256-8 323ns ± 0% 293ns ± 0% -9.29% (p=0.000 n=5+4) Decode/1024-8 1.26µs ± 0% 1.14µs ± 0% -9.48% (p=0.000 n=6+5) Decode/4096-8 4.99µs ± 0% 4.51µs ± 0% -9.55% (p=0.002 n=6+6) Decode/16384-8 20.0µs ± 0% 18.1µs ± 0% -9.54% (p=0.002 n=6+6) name old speed new speed delta Decode/256-8 791MB/s ± 0% 872MB/s ± 0% +10.34% (p=0.002 n=6+6) Decode/1024-8 814MB/s ± 0% 899MB/s ± 0% +10.48% (p=0.004 n=6+5) Decode/4096-8 821MB/s ± 0% 908MB/s ± 0% +10.55% (p=0.002 n=6+6) Decode/16384-8 821MB/s ± 0% 908MB/s ± 0% +10.54% (p=0.002 n=6+6) Change-Id: Ie9f91242ce04c130a77c1184379e3b9de38fe713 Reviewed-on: https://go-review.googlesource.com/c/151199 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
generated by cgit v1.3-5-g45d5 (git 2.53.0) at 2026-04-17 17:06:37 +0000