aboutsummaryrefslogtreecommitdiff
path: root/src/encoding
AgeCommit message (Collapse)Author
2020-08-06[release-branch.go1.13-security] encoding/binary: read at most ↵Katie Hockman
MaxVarintLen64 bytes in ReadUvarint This CL ensures that ReadUvarint consumes only a limited amount of input (instead of an unbounded amount). On some inputs, ReadUvarint could read an arbitrary number of bytes before deciding to return an overflow error. After this CL, ReadUvarint returns that same overflow error sooner, after reading at most MaxVarintLen64 bytes. Fix authored by Robert Griesemer and Filippo Valsorda. Thanks to Diederik Loerakker, Jonny Rhea, Raúl Kripalani, and Preston Van Loon for reporting this. Fixes CVE-2020-16845 Change-Id: Ie0cb15972f14c38b7cf7af84c45c4ce54909bb8f Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/812099 Reviewed-by: Filippo Valsorda <valsorda@google.com> Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/812324
2019-09-02[release-branch.go1.13] encoding/json: revert Compact HTML escaping ↵Tom Thorogood
documentation This partly reverts CL 173417 as it incorrectly documented that Compact performed HTML escaping and the output was safe to embed inside HTML <script> tags. This has never been true. Although Compact does escape U+2028 and U+2029, it doesn't escape <, > or &. Compact is thus only performing a subset of HTML escaping and it's output is not safe to embed inside HTML <script> tags. A more complete fix would be for Compact to either never perform any HTML escaping, as it was prior to CL 10883045, or to actually perform the same HTML escaping as HTMLEscape. Neither change is likely safe enough for go1.13. Fixes #34006 Updates #30357 Change-Id: I912f0fe9611097d988048b28228c4a5b985080ba GitHub-Last-Rev: aebababc9233c5705785b225377e80096d4bb8c4 GitHub-Pull-Request: golang/go#33427 Reviewed-on: https://go-review.googlesource.com/c/go/+/188717 Reviewed-by: Daniel Martí <mvdan@mvdan.cc> Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> (cherry picked from commit 79669dc705aa8fcddc8f9407c568e4317d619dfe) Reviewed-on: https://go-review.googlesource.com/c/go/+/192747 Reviewed-by: Andrew Bonventre <andybons@golang.org> Run-TryBot: Andrew Bonventre <andybons@golang.org>
2019-08-21Revert "encoding/json: avoid work when unquoting strings"Filippo Valsorda
This reverts CL 151157. CL 151157 introduced a crash when decoding into ",string" fields. It came with a moderate speedup, so at this stage of the release cycle let's just revert it, and reapply it in Go 1.14 with the fix in CL 190659. Also applied the test cases from CL 190659. Updates #33728 Change-Id: Ie46e2bc15224b251888580daf6b79d5865f3878e Reviewed-on: https://go-review.googlesource.com/c/go/+/190909 Run-TryBot: Andrew Bonventre <andybons@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Andrew Bonventre <andybons@golang.org>
2019-08-18encoding/json: fix format string in the Fuzz funcDmitry Vyukov
Currently test build fails with: $ go test -tags=gofuzz encoding/json encoding/json/fuzz.go:36:4: Println call has possible formatting directive %s FAIL encoding/json [build failed] Change-Id: I23aef44a421ed0e7bcf48b74ac5a8c6768a4841b Reviewed-on: https://go-review.googlesource.com/c/go/+/190698 Run-TryBot: Ian Lance Taylor <iant@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-07-31encoding/json: clarify Marshal behavior for string keys of mapsEli Bendersky
This is a documentation-only change. Fixes #28827 Change-Id: Ife9ab997809048784f35872b09905bc209a05eff Reviewed-on: https://go-review.googlesource.com/c/go/+/188417 Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-07-28all: change some function documentation to be more idiomaticDominik Honnef
Change-Id: I932de9bb061a8ba3332ef03207983e8b98d6f1e5 Reviewed-on: https://go-review.googlesource.com/c/go/+/187918 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-07-06encoding/json: obey SetEscapeHTML in all MarshalJSON casesDaniel Martí
It wasn't obeyed in the case where the MarshalJSON method uses a pointer receiver, and the encoder grabs the address of a value to find that method. addrMarshalerEncoder is the function that does this work, but it ignored opts.escapeHTML. Here's the before and after of the added test case, which was failing before the fix. Now the two cases are correct and consistent. {"NonPtr":"<str>","Ptr":"\u003cstr\u003e"} {"NonPtr":"<str>","Ptr":"<str>"} Fixes #32896. Change-Id: Idc53077ece074973558bd3bb5ad036380db0d02c Reviewed-on: https://go-review.googlesource.com/c/go/+/184757 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com> Reviewed-by: Caleb Spare <cespare@gmail.com>
2019-05-22encoding/gob: properly ignore errorsLeon Klingele
Change-Id: I8827cef0f57459384329c50c51795350da0ede4b GitHub-Last-Rev: c9ad9e12b5a0fff47c21a8c299b762b64b8c9c7c GitHub-Pull-Request: golang/go#30010 Reviewed-on: https://go-review.googlesource.com/c/go/+/160434 Run-TryBot: Bryan C. Mills <bcmills@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-05-22all: shorten some testsRuss Cox
Shorten some of the longest tests that run during all.bash. Removes 7r 50u 21s from all.bash. After this change, all.bash is under 5 minutes again on my laptop. For #26473. Change-Id: Ie0460aa935808d65460408feaed210fbaa1d5d79 Reviewed-on: https://go-review.googlesource.com/c/go/+/177559 Run-TryBot: Russ Cox <rsc@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-05-21all: remove PEM-encoded private keys from testsRuss Cox
Gerrit is complaining about pushes that affect these files and forcing people to use -o nokeycheck, which defeats the point of the check. Hide the keys from this kind of scan by marking them explicitly as testing keys. This is a little annoying but better than training everyone who ever edits one of these test files to reflexively override the Gerrit check. The only remaining keys explicitly marked as private instead of testing are in examples, and there's not much to do about those. Hopefully they are not edited as much. Change-Id: I4431592b5266cb39fe6a80b40e742d97da803a0b Reviewed-on: https://go-review.googlesource.com/c/go/+/178178 Run-TryBot: Russ Cox <rsc@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-05-09encoding/gob: rename encBuffer.WriteByte to writeByteRuss Cox
Renaming the method makes clear, both to readers and to vet, that this method is not the implementation of io.ByteWriter. Working toward making the tree vet-safe instead of having so many exceptions in cmd/vet/all/whitelist. For #31916. Change-Id: I5b509eb7f0118d5f2d3c6e352ff2849cd5a3071e Reviewed-on: https://go-review.googlesource.com/c/go/+/176110 Run-TryBot: Russ Cox <rsc@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-05-06all: simplify code using "gofmt -s -w"Shulhan
Most changes are removing redundant declaration of type when direct instantiating value of map or slice, e.g. []T{T{}} become []T{{}}. Small changes are removing the high order of subslice if its value is the length of slice itself, e.g. T[:len(T)] become T[:]. The following file is excluded due to incompatibility with go1.4, - src/cmd/compile/internal/gc/ssa.go Change-Id: Id3abb09401795ce1e6da591a89749cba8502fb26 Reviewed-on: https://go-review.googlesource.com/c/go/+/166437 Run-TryBot: Dave Cheney <dave@cheney.net> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-05-04all: add Unwrap and Is methods to various error typesDamien Neil
Add Unwrap methods to types which wrap an underlying error: "encodinc/csv".ParseError "encoding/json".MarshalerError "net/http".transportReadFromServerError "net".OpError "net".DNSConfigError "net/url".Error "os/exec".Error "signal/internal/pty".PtyError "text/template".ExecError Add os.ErrTemporary. A case could be made for putting this error value in package net, since no exported error types in package os include a Temporary method. However, syscall errors returned from the os package do include this method. Add Is methods to error types with a Timeout or Temporary method, making errors.Is(err, os.Err{Timeout,Temporary}) equivalent to testing the corresponding method: "context".DeadlineExceeded "internal/poll".TimeoutError "net".adrinfoErrno "net".OpError "net".DNSError "net/http".httpError "net/http".tlsHandshakeTimeoutError "net/pipe".timeoutError "net/url".Error Updates #30322 Updates #29934 Change-Id: I409fb20c072ea39116ebfb8c7534d493483870dc Reviewed-on: https://go-review.googlesource.com/c/go/+/170037 Run-TryBot: Damien Neil <dneil@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Marcel van Lohuizen <mpvl@golang.org>
2019-04-30encoding/gob: adding missing fuzz skip to one of the fuzz testsBrad Fitzpatrick
It's slow & often times out randomly on longtest builders. Not useful. Fixes #31517 Change-Id: Icedbb0c94fbe43d04e8b47d5785ac61c5e2d8750 Reviewed-on: https://go-review.googlesource.com/c/go/+/174522 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-04-30encoding/csv: add a Fuzz functionRomain Baugue
Adds a sample Fuzz test function to package encoding/csv based on https://github.com/dvyukov/go-fuzz-corpus/blob/master/csv/main.go Updates #19109 Updates #31309 Change-Id: Ieb0cb6caa1df72dbb7e29df4bdeed0bfa91187d3 Reviewed-on: https://go-review.googlesource.com/c/go/+/174302 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-04-30encoding/json: fix Unmarshal hang on recursive pointersLE Manh Cuong
indirect walks down v until it gets to a non-pointer. But it does not handle the case when v is a pointer to itself, like in: var v interface{} v = &v Unmarshal(b, v) So just stop immediately if we see v is a pointer to itself. Fixes #31740 Change-Id: Ie396264119e24d70284cd9bf76dcb2050babb069 Reviewed-on: https://go-review.googlesource.com/c/go/+/174337 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Daniel Martí <mvdan@mvdan.cc> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-04-29encoding/json: add a Fuzz functionRomain Baugue
Adds a sample Fuzz test function to package encoding/json following the guidelines defined in #31309, based on https://github.com/dvyukov/go-fuzz-corpus/blob/master/json/json.go Fixes #31309 Updates #19109 Change-Id: I5fe04d9a5f41c0de339f8518dae30896ec14e356 Reviewed-on: https://go-review.googlesource.com/c/go/+/174058 Reviewed-by: Dmitry Vyukov <dvyukov@google.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Dmitry Vyukov <dvyukov@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-04-23encoding/json: document HTML escaping in CompactRuss Cox
Make explicit that Compact does HTML escaping. Fixes #30357. Change-Id: I4648f8f3e907d659db977d07253f716df6e07d7b Reviewed-on: https://go-review.googlesource.com/c/go/+/173417 Run-TryBot: Russ Cox <rsc@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-04-23encoding/json: index names for the struct decoderDaniel Martí
In the common case, structs have a handful of fields and most inputs match struct field names exactly. The previous code would do a linear search over the fields, stopping at the first exact match, and otherwise using the first case insensitive match. This is unfortunate, because it means that for the common case, we'd do a linear search with bytes.Equal. Even for structs with only two or three fields, that is pretty wasteful. Worse even, up until the exact match was found via the linear search, all previous fields would run their equalFold functions, which aren't cheap even in the simple case. Instead, cache a map along with the field list that indexes the fields by their name. This way, a case sensitive field search doesn't involve a linear search, nor does it involve any equalFold func calls. This patch should also slightly speed up cases where there's a case insensitive match but not a case sensitive one, as then we'd avoid calling bytes.Equal on all the fields. Though that's not a common case, and there are no benchmarks for it. name old time/op new time/op delta CodeDecoder-8 11.0ms ± 0% 10.6ms ± 1% -4.42% (p=0.000 n=9+10) name old speed new speed delta CodeDecoder-8 176MB/s ± 0% 184MB/s ± 1% +4.62% (p=0.000 n=9+10) name old alloc/op new alloc/op delta CodeDecoder-8 2.28MB ± 0% 2.28MB ± 0% ~ (p=0.725 n=10+10) name old allocs/op new allocs/op delta CodeDecoder-8 76.9k ± 0% 76.9k ± 0% ~ (all equal) Updates #28923. Change-Id: I9929c1f06c76505e5b96914199315dbdaae5dc76 Reviewed-on: https://go-review.googlesource.com/c/go/+/172918 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-04-22encoding/json: avoid work when unquoting stringsDaniel Martí
We can work out how many bytes can be unquoted trivially in rescanLiteral, which already iterates over a string's bytes. Removing the extra loop in unquoteBytes simplifies the function and speeds it up, especially when decoding simple strings, which are common. While at it, we can remove unnecessary checks like len(s)<2 and s[0]=='"'. Add a comment explaining why. name old time/op new time/op delta CodeDecoder-8 11.2ms ± 0% 11.1ms ± 1% -1.63% (p=0.000 n=9+10) name old speed new speed delta CodeDecoder-8 173MB/s ± 0% 175MB/s ± 1% +1.66% (p=0.000 n=9+10) Updates #28923. Change-Id: I2436a3a7f8148a2f7a6a4cdbd7dec6b32ef5e20c Reviewed-on: https://go-review.googlesource.com/c/go/+/151157 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-04-13encoding/json: remove a bounds check in readValueDaniel Martí
readValue is a hot function, clocking in at ~13% flat CPU use in CodeDecoder. In particular, looping over the bytes is slow. That's partially because the code contains a bounds check at the start of the loop. The source of the problem is that scanp is a signed integer, and comes from a field, so the compiler doesn't know that it's non-negative. Help it with a simple and comparatively cheap hint. While at it, use scanp as the index variable directly, removing the need for a duplicate index variable which is later added back into scanp. name old time/op new time/op delta CodeDecoder-8 11.3ms ± 1% 11.2ms ± 1% -0.98% (p=0.000 n=9+9) name old speed new speed delta CodeDecoder-8 172MB/s ± 1% 174MB/s ± 1% +0.99% (p=0.000 n=9+9) Updates #28923. Change-Id: I138f83babdf316fc97697cc18f595c3403c1ddb7 Reviewed-on: https://go-review.googlesource.com/c/go/+/170939 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-04-05encoding/json: use SetBytes in UnmarshalReuse benchmarkDaniel Martí
This was the only benchmark missing the SetBytes call, as spotted earlier by Bryan. It's not required to make the benchmark useful, but it can still be a good way to see how its speed is affected by the reduced allocations: name time/op CodeUnmarshal-8 12.1ms ± 1% CodeUnmarshalReuse-8 11.4ms ± 1% name speed CodeUnmarshal-8 161MB/s ± 1% CodeUnmarshalReuse-8 171MB/s ± 1% name alloc/op CodeUnmarshal-8 3.28MB ± 0% CodeUnmarshalReuse-8 1.94MB ± 0% name allocs/op CodeUnmarshal-8 92.7k ± 0% CodeUnmarshalReuse-8 77.6k ± 0% While at it, remove some unnecessary empty lines. Change-Id: Ib2bd92d5b3237b8f3092e8c6f863dab548fee2f5 Reviewed-on: https://go-review.googlesource.com/c/go/+/170938 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-04-03encoding/json: speed up tokenization of literalsDaniel Martí
Decoder.Decode and Unmarshal actually scan the input bytes twice - the first time to check for syntax errors and the length of the value, and the second to perform the decoding. It's in the second scan that we actually tokenize the bytes. Since syntax errors aren't a possibility, we can take shortcuts. In particular, literals such as quoted strings are very common in JSON, so we can avoid a lot of work by special casing them. name old time/op new time/op delta CodeDecoder-8 10.3ms ± 1% 9.1ms ± 0% -11.89% (p=0.002 n=6+6) UnicodeDecoder-8 342ns ± 0% 283ns ± 0% -17.25% (p=0.000 n=6+5) DecoderStream-8 239ns ± 0% 230ns ± 0% -3.90% (p=0.000 n=6+5) CodeUnmarshal-8 11.0ms ± 0% 9.8ms ± 0% -11.45% (p=0.002 n=6+6) CodeUnmarshalReuse-8 10.3ms ± 0% 9.0ms ± 0% -12.72% (p=0.004 n=5+6) UnmarshalString-8 104ns ± 0% 92ns ± 0% -11.35% (p=0.002 n=6+6) UnmarshalFloat64-8 93.2ns ± 0% 87.6ns ± 0% -6.01% (p=0.010 n=6+4) UnmarshalInt64-8 74.5ns ± 0% 71.5ns ± 0% -3.91% (p=0.000 n=5+6) name old speed new speed delta CodeDecoder-8 189MB/s ± 1% 214MB/s ± 0% +13.50% (p=0.002 n=6+6) UnicodeDecoder-8 40.9MB/s ± 0% 49.5MB/s ± 0% +20.96% (p=0.002 n=6+6) CodeUnmarshal-8 176MB/s ± 0% 199MB/s ± 0% +12.93% (p=0.002 n=6+6) Updates #28923. Change-Id: I7a5e2aef51bd4ddf2004aad24210f6f50e01eaeb Reviewed-on: https://go-review.googlesource.com/c/go/+/151042 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-18encoding/json: fix performance regression in the decoderDaniel Martí
In golang.org/cl/145218, a feature was added where the JSON decoder would keep track of the entire path to a field when reporting an UnmarshalTypeError. However, we all failed to check if this affected the benchmarks - myself included, as a reviewer. Below are the numbers comparing the CL's parent with itself, once it was merged: name old time/op new time/op delta CodeDecoder-8 12.9ms ± 1% 28.2ms ± 2% +119.33% (p=0.002 n=6+6) name old speed new speed delta CodeDecoder-8 151MB/s ± 1% 69MB/s ± 3% -54.40% (p=0.002 n=6+6) name old alloc/op new alloc/op delta CodeDecoder-8 2.74MB ± 0% 109.39MB ± 0% +3891.83% (p=0.002 n=6+6) name old allocs/op new allocs/op delta CodeDecoder-8 77.5k ± 0% 168.5k ± 0% +117.30% (p=0.004 n=6+5) The reason why the decoder got twice as slow is because it now allocated ~40x as many objects, which puts a lot of pressure on the garbage collector. The reason is that the CL concatenated strings every time a nested field was decoded. In other words, practically every field generated garbage when decoded. This is hugely wasteful, especially considering that the vast majority of JSON decoding inputs won't return UnmarshalTypeError. Instead, use a stack of fields, and make sure to always use the same backing array, to ensure we only need to grow the slice to the maximum depth once. The original CL also introduced a bug. The field stack string wasn't reset to its original state when reaching "d.opcode == scanEndObject", so the last field in a decoded struct could leak. For example, an added test decodes a list of structs, and encoding/json before this CL would fail: got: cannot unmarshal string into Go struct field T.Ts.Y.Y.Y of type int want: cannot unmarshal string into Go struct field T.Ts.Y of type int To fix that, simply reset the stack after decoding every field, even if it's the last. Below is the original performance versus this CL. There's a tiny performance hit, probably due to the append for every decoded field, but at least we're back to the usual ~150MB/s. name old time/op new time/op delta CodeDecoder-8 12.9ms ± 1% 13.0ms ± 1% +1.25% (p=0.009 n=6+6) name old speed new speed delta CodeDecoder-8 151MB/s ± 1% 149MB/s ± 1% -1.24% (p=0.009 n=6+6) name old alloc/op new alloc/op delta CodeDecoder-8 2.74MB ± 0% 2.74MB ± 0% +0.00% (p=0.002 n=6+6) name old allocs/op new allocs/op delta CodeDecoder-8 77.5k ± 0% 77.5k ± 0% +0.00% (p=0.002 n=6+6) Finally, make all of these benchmarks report allocs by default. The decoder ones are pretty sensitive to generated garbage, so ReportAllocs would have made the performance regression more obvious. Change-Id: I67b50f86b2e72f55539429450c67bfb1a9464b67 Reviewed-on: https://go-review.googlesource.com/c/go/+/167978 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-03-13encoding/base64: speed up the decoderDaniel Martí
Most of the decoding time is spent in the first Decode loop, since the rest of the function only deals with the few remaining bytes. Any unnecessary work done in that loop body matters tremendously. One such unnecessary bottleneck was the use of the enc.decodeMap table. Since enc is a pointer receiver, and the field is used within the non-inlineable function decode64, the decoder must perform a nil check at every iteration. To fix that, move the enc.decodeMap uses to the parent function, where we can lift the nil check outside the loop. That gives roughly a 15% speed-up. The function no longer performs decoding per se, so rename it. While at it, remove the now unnecessary receivers. An unfortunate side effect of this change is that the loop now contains eight bounds checks on src instead of just one. However, not having to slice src plus the nil check removal well outweigh the added cost. The other piece that made decode64 slow was that it wasn't inlined, and had multiple branches. Use a simple bitwise-or trick suggested by Roger Peppe, and collapse the rest of the bitwise logic into a single expression. Inlinability and the reduced branching give a further 10% speed-up. Finally, add these two functions to TestIntendedInlining, since we want them to stay inlinable. Apply the same refactor to decode32 for consistency, and to let 32-bit architectures see a similar performance gain for large inputs. name old time/op new time/op delta DecodeString/2-8 47.3ns ± 1% 45.8ns ± 0% -3.28% (p=0.002 n=6+6) DecodeString/4-8 55.8ns ± 2% 51.5ns ± 0% -7.71% (p=0.004 n=5+6) DecodeString/8-8 64.9ns ± 0% 61.7ns ± 0% -4.99% (p=0.004 n=5+6) DecodeString/64-8 238ns ± 0% 198ns ± 0% -16.54% (p=0.002 n=6+6) DecodeString/8192-8 19.5µs ± 0% 14.6µs ± 0% -24.96% (p=0.004 n=6+5) name old speed new speed delta DecodeString/2-8 84.6MB/s ± 1% 87.4MB/s ± 0% +3.38% (p=0.002 n=6+6) DecodeString/4-8 143MB/s ± 2% 155MB/s ± 0% +8.41% (p=0.004 n=5+6) DecodeString/8-8 185MB/s ± 0% 195MB/s ± 0% +5.29% (p=0.004 n=5+6) DecodeString/64-8 369MB/s ± 0% 442MB/s ± 0% +19.78% (p=0.002 n=6+6) DecodeString/8192-8 560MB/s ± 0% 746MB/s ± 0% +33.27% (p=0.004 n=6+5) Updates #19636. Change-Id: Ib839577b0e3f5a2bb201f5cae580c61365d92894 Reviewed-on: https://go-review.googlesource.com/c/go/+/151177 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Reviewed-by: roger peppe <rogpeppe@gmail.com>
2019-03-08all: simplify multiple for loopsDaniel Martí
If a for loop has a simple condition and begins with a simple "if x { break; }"; we can simply add "!x" to the loop's condition. While at it, simplify a few assignments to use the common pattern "x := staticDefault; if cond { x = otherValue(); }". Finally, simplify a couple of var declarations. Change-Id: I413982c6abd32905adc85a9a666cb3819139c19f Reviewed-on: https://go-review.googlesource.com/c/go/+/165342 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-03-08encoding/gob: update documentation in doc.go for wireTypeRob Pike
It was just out of date. Fixes #30656 Change-Id: I1fab7dd93091865a8240769eca5dd19cdbc78b81 Reviewed-on: https://go-review.googlesource.com/c/go/+/166177 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-05all: join a few chained ifsDaniel Martí
I had been finding these over a year or so, but none were big enough changes to warrant CLs. They're a handful now, so clean them all up in a single commit. The smaller bodies get a bit simpler, but most importantly, the larger bodies get unindented. Change-Id: I5707a6fee27d4c9ff9efd3d363af575d7a4bf2aa Reviewed-on: https://go-review.googlesource.com/c/go/+/165340 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-03-05encoding/base64: do not slice past output unnecessarilyRuss Cox
Base64-encoding 32 bytes results in a 44-byte string. While in general a 44-byte string might decode to 33 bytes, if you take a 44-byte string that actually only encodes 32 bytes, and you try to decode it into 32 bytes, that should succeed. Instead it fails trying to do a useless dst[33:] slice operation. Delete that slice operation. Noticed while preparing CL 156322. Change-Id: I8024bf28a65e2638675b980732b2ff91c66c62cf Reviewed-on: https://go-review.googlesource.com/c/go/+/164628 Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-03-05encoding/hex: simplify encoder arithmeticDaniel Martí
Two additions are faster than two multiplications and one addition. The code seems simpler to me too, as it's more obvious that we advance two destination bytes for each source byte. name old time/op new time/op delta Encode/256-8 374ns ± 0% 331ns ± 0% -11.44% (p=0.008 n=5+5) Encode/1024-8 1.47µs ± 0% 1.29µs ± 0% -11.89% (p=0.004 n=6+5) Encode/4096-8 5.85µs ± 1% 5.15µs ± 0% -11.89% (p=0.004 n=6+5) Encode/16384-8 23.3µs ± 0% 20.6µs ± 0% -11.68% (p=0.004 n=6+5) Change-Id: Iabc63616c1d9fded55fa668ff41dd49efeaa2ea4 Reviewed-on: https://go-review.googlesource.com/c/go/+/151198 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: roger peppe <rogpeppe@gmail.com>
2019-03-05encoding/json: add Path to UnmarshalTypeErrorLE Manh Cuong
When parsing nested object, UnmarshalTypeError does not contain actual path to nested field in original JSON. This commit change Field to contain the full path to that field. One can get the Field name by stripping all the leading path elements. Fixes #22369 Change-Id: I6969cc08abe8387a351e3fb2944adfaa0dccad2a Reviewed-on: https://go-review.googlesource.com/c/go/+/145218 Reviewed-by: Daniel Martí <mvdan@mvdan.cc> Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-03-04encoding/csv: document that Writer is bufferedBrian Kessler
Add documentation that individual Write calls are buffered and copy documentation from bufio.Writer notifying the user to call Flush and Error when all writes are complete. Remove reference to "file" since the implementation is general and allows any io.Writer. Fixes #30045 Change-Id: I50165470e548f296494e764707fbabe36c665015 Reviewed-on: https://go-review.googlesource.com/c/160680 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-03encoding/base64: lift nil check out of encode loopDaniel Martí
Most of the encoding time is spent in the first Encode loop, since the rest of the function only deals with the few remaining bytes. Any unnecessary work done in that loop body matters tremendously. One such unnecessary bottleneck was the use of the enc.encode table. Since enc is a pointer receiver, and the field is first used within the loop, the encoder must perform a nil check at every iteration. Add a dummy use of the field before the start of the loop, to move the nil check there. After that line, the compiler now knows that enc can't be nil, and thus the hot loop is free of nil checks. name old time/op new time/op delta EncodeToString-4 14.7µs ± 0% 13.7µs ± 1% -6.53% (p=0.000 n=10+10) name old speed new speed delta EncodeToString-4 559MB/s ± 0% 598MB/s ± 1% +6.99% (p=0.000 n=10+10) Updates #20206. Change-Id: Icbb523a7bd9e470a8be0a448d1d78ade97ed4ff6 Reviewed-on: https://go-review.googlesource.com/c/151158 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-02encoding/pem: skip whitespace work on most inputsDaniel Martí
encoding/base64 already skips \r and \n when decoding, so this package must only deal with spaces and tabs. Those aren't nearly as common, so we can add a fast path with bytes.ContainsAny to skip the costly alloc and filtering code. name old time/op new time/op delta Decode-8 279µs ± 0% 259µs ± 1% -7.07% (p=0.002 n=6+6) name old speed new speed delta Decode-8 319MB/s ± 0% 343MB/s ± 1% +7.61% (p=0.002 n=6+6) name old alloc/op new alloc/op delta Decode-8 164kB ± 0% 74kB ± 0% -54.90% (p=0.002 n=6+6) name old allocs/op new allocs/op delta Decode-8 12.0 ± 0% 11.0 ± 0% -8.33% (p=0.002 n=6+6) Change-Id: Idfca8700c52f46eb70a4a7e0d2db3bf0124e4699 Reviewed-on: https://go-review.googlesource.com/c/155964 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-02encoding/hex: simplify decoder arithmeticDaniel Martí
Remove all multiplications and divisions from the main decoding loop. name old time/op new time/op delta Decode/256-8 323ns ± 0% 293ns ± 0% -9.29% (p=0.000 n=5+4) Decode/1024-8 1.26µs ± 0% 1.14µs ± 0% -9.48% (p=0.000 n=6+5) Decode/4096-8 4.99µs ± 0% 4.51µs ± 0% -9.55% (p=0.002 n=6+6) Decode/16384-8 20.0µs ± 0% 18.1µs ± 0% -9.54% (p=0.002 n=6+6) name old speed new speed delta Decode/256-8 791MB/s ± 0% 872MB/s ± 0% +10.34% (p=0.002 n=6+6) Decode/1024-8 814MB/s ± 0% 899MB/s ± 0% +10.48% (p=0.004 n=6+5) Decode/4096-8 821MB/s ± 0% 908MB/s ± 0% +10.55% (p=0.002 n=6+6) Decode/16384-8 821MB/s ± 0% 908MB/s ± 0% +10.54% (p=0.002 n=6+6) Change-Id: Ie9f91242ce04c130a77c1184379e3b9de38fe713 Reviewed-on: https://go-review.googlesource.com/c/151199 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-02encoding/base64: remove ineffectual assignment in testLeon Klingele
Change-Id: I4a0d5b2f76138895567939920fa5d83cbdec17d2 GitHub-Last-Rev: 061d9d1d5655a6a9d8371f08d2f77a0ed7a495cc GitHub-Pull-Request: golang/go#30008 Reviewed-on: https://go-review.googlesource.com/c/160432 Reviewed-by: Bryan C. Mills <bcmills@google.com> Run-TryBot: Bryan C. Mills <bcmills@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-03-02encoding/base32: remove ineffectual assignment in testLeon Klingele
Change-Id: I8aaa3d1d2797f3ace34bc09f5123538f6a77efce GitHub-Last-Rev: 2758c462041ff5e444651b7927d53e809d2efe4d GitHub-Pull-Request: golang/go#30009 Reviewed-on: https://go-review.googlesource.com/c/160433 Reviewed-by: Bryan C. Mills <bcmills@google.com> Run-TryBot: Bryan C. Mills <bcmills@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-02-27encoding/json: remove use of DeepEqual for testing errorsMarcel van Lohuizen
Comparing errors using DeepEqual breaks if frame information is added as proposed in Issue #29934. Updates #29934. Change-Id: Ib430c9ddbe588dd1dd51314c408c74c07285e1ff Reviewed-on: https://go-review.googlesource.com/c/162179 Run-TryBot: Marcel van Lohuizen <mpvl@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Russ Cox <rsc@golang.org> Reviewed-by: Damien Neil <dneil@google.com>
2019-02-26encoding/base32: simplify and speed up decoderDaniel Martí
First, we can lift the enc.decodeMap nil check out of the loop. Second, we can make it clear to the compiler that 'in := src[0]' doesn't need a bounds check, by making len(src)==0 a single if check that always stops the loop. This is by far the largest speed-up. Third, we can use a dst slice index instead of reslicing dst, which removes work from the loop body. While at it, we can merge the two 'switch dlen' pieces of code, which simplifies the code and doesn't affect performance. name old time/op new time/op delta DecodeString-8 80.2µs ± 0% 67.5µs ± 0% -15.81% (p=0.002 n=6+6) name old speed new speed delta DecodeString-8 163MB/s ± 0% 194MB/s ± 0% +18.78% (p=0.002 n=6+6) Change-Id: Iefeaae94c03453f8760452b1da706a77b3522718 Reviewed-on: https://go-review.googlesource.com/c/154422 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-26encoding/json: add example for json.HTMLEscapeKetan Parmar
Change-Id: Ib00fcfd46eae27eea0a3d4cab4406f4c461fb57b Reviewed-on: https://go-review.googlesource.com/c/160517 Reviewed-by: Andrew Bonventre <andybons@golang.org> Run-TryBot: Andrew Bonventre <andybons@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-01-15encoding/json: add comment for mashalererror structGuilhermeCaruso
Change-Id: Iaabbfe5a4c1bbedd19d4087f1b79e5a38bdd3878 GitHub-Last-Rev: 55c91fc19074dacc66623aa7ff2286b11ccd5340 GitHub-Pull-Request: golang/go#29752 Reviewed-on: https://go-review.googlesource.com/c/157958 Reviewed-by: Ian Lance Taylor <iant@golang.org>
2018-12-28encoding/gob: mention that Encoder and Decoder are safe for concurrent useMostyn Bramley-Moore
Fixes #29416 Change-Id: I24364bfee77aceace53f85f1046ef4d73f8feebb Change-Id: I24364bfee77aceace53f85f1046ef4d73f8feebb GitHub-Last-Rev: ad9f31145763dc16f53dd9f3154667b162759f69 GitHub-Pull-Request: golang/go#29417 Reviewed-on: https://go-review.googlesource.com/c/155742 Reviewed-by: Ian Lance Taylor <iant@golang.org>
2018-12-12encoding/xml, encoding/json: docs and examples using custom marshalersLeigh McCulloch
Both the encoding/xml and encoding/json packages support custom marshalers for JSON and XML, as well as the basic encoding.TextMarshaler and encoding.TextUnmarshaler interfaces, but the docs and examples for these are missing. There are docs for how to use encoding.TextMarshaler and encoding.TextUnmarshaler in encoding/json, but not encoding/xml. There are no examples for how to use them with either json or xml. This commit includes docs for encoding/xml and examples for both encoding/json and encoding/xml. There is an example using custom marshalers MarshalJSON and UnmarshalJSON in encoding/json, but not MarshalXML and UnmarshalXML in encoding/json. These docs are more so necessary for encoding/xml because the complexities of XML documents is significantly greater than JSON documents which more often leads to the need for custom marshaling. The encoding/json package includes an example of how to write a custom marshaler, and this commit includes the same example for the xml package. All examples are mirrored off the existing custom marshaler example in encoding/json. Fixes #6859 Change-Id: Ic93abc27c0b4d5e48dea6ede4e20b1bedca4ab39 Reviewed-on: https://go-review.googlesource.com/c/76350 Run-TryBot: Ian Lance Taylor <iant@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2018-11-19encoding/pem: test getLine does not include trailing whitespaceSanthosh Kumar Tekuri
Change-Id: I7a1046f5e0aedbbdd1106a616de410fe4e0cb7d8 Reviewed-on: https://go-review.googlesource.com/c/92295 Run-TryBot: Ian Lance Taylor <iant@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2018-11-02all: use "reports whether" consistently in the few places that didn'tBrad Fitzpatrick
Go documentation style for boolean funcs is to say: // Foo reports whether ... func Foo() bool (rather than "returns true if") This CL also replaces 4 uses of "iff" with the same "reports whether" wording, which doesn't lose any meaning, and will prevent people from sending typo fixes when they don't realize it's "if and only if". In the past I think we've had the typo CLs updated to just say "reports whether". So do them all at once. (Inspired by the addition of another "returns true if" in CL 146938 in fd_plan9.go) Created with: $ perl -i -npe 's/returns true if/reports whether/' $(git grep -l "returns true iff" | grep -v vendor) $ perl -i -npe 's/returns true if/reports whether/' $(git grep -l "returns true if" | grep -v vendor) Change-Id: Ided502237f5ab0d25cb625dbab12529c361a8b9f Reviewed-on: https://go-review.googlesource.com/c/147037 Reviewed-by: Ian Lance Taylor <iant@golang.org>
2018-10-28encoding/json: add example to Validhearot
Change-Id: I411483d76a2ca91cd15ff42ae1adb9134486d183 Reviewed-on: https://go-review.googlesource.com/c/145278 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2018-10-22encoding/gob: delete out of memory testRob Pike
Now that the library allows much larger data, it can kill machines with less memory. Fixes #28321 Change-Id: I98e1a5fdf812fd75adfb22bf01542423de405fe2 Reviewed-on: https://go-review.googlesource.com/c/143817 Reviewed-by: Ian Lance Taylor <iant@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-22encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machinesRob Pike
A little shift magic makes it easy to adjust the maximum buffer size on machines with larger integers. Fixes #27635 Change-Id: I1f26b07a363fbb9730df2377052475fa88bbb781 Reviewed-on: https://go-review.googlesource.com/c/143678 Run-TryBot: Rob Pike <r@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2018-10-16encoding/json: always verify we can get a field's valueDaniel Martí
Calling .Interface on a struct field's reflect.Value isn't always safe. For example, if that field is an unexported anonymous struct. We only descended into this branch if the struct type had any methods, so this bug had gone unnoticed for a few release cycles. Add the check, and add a simple test case. Fixes #28145. Change-Id: I02f7e0ab9a4a0c18a5e2164211922fe9c3d30f64 Reviewed-on: https://go-review.googlesource.com/c/141537 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2018-10-16encoding/json: fix "data changed underfoot?" panicDaniel Martí
Given a program as follows: data := []byte(`{"F": { "a": 2, "3": 4 }}`) json.Unmarshal(data, &map[string]map[int]int{}) The JSON package should error, as "a" is not a valid integer. However, we'd encounter a panic: panic: JSON decoder out of sync - data changing underfoot? The reason was that decodeState.object would return a nil error on encountering the invalid map key string, while saving the key type error for later. This broke if we were inside another object, as we would abruptly end parsing the nested object, leaving the decoder in an unexpected state. To fix this, simply avoid storing the map element and continue decoding the object, to leave the decoder state exactly as if we hadn't seen an invalid key type. This affected both signed and unsigned integer keys, so fix both and add two test cases. Updates #28189. Change-Id: I8a6204cc3ff9fb04ed769df7a20a824c8b94faff Reviewed-on: https://go-review.googlesource.com/c/142518 Reviewed-by: Ian Lance Taylor <iant@golang.org>
generated by cgit v1.3-5-g9baa (git 2.53.0) at 2026-04-17 16:01:23 +0000