go, branch go1.14.5

[release-branch.go1.14-security] go1.14.5

2020-07-14T14:10:30Z

Change-Id: Ic2fb9041f6610dd5ce54185e674ab4b62fabf3b5 Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/794130 Reviewed-by: Katie Hockman

[release-branch.go1.14-security] crypto/x509: respect VerifyOptions.KeyUsages on Windows

2020-07-14T12:24:21Z

When using the platform verifier on Windows (because Roots is nil) we were always enforcing server auth EKUs if DNSName was set, and none otherwise. If an application was setting KeyUsages, they were not being respected. Started correctly surfacing IncompatibleUsage errors from the system verifier, as those are the ones applications will see if they are affected by this change. Also refactored verify_test.go to make it easier to add tests for this, and replaced the EKULeaf chain with a new one that doesn't have a SHA-1 signature. Thanks to Niall Newman for reporting this. Fixes #39360 Fixes CVE-2020-14039 Change-Id: If5c00d615f2944f7d57007891aae1307f9571c32 Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/774414 Reviewed-by: Katie Hockman Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/793511 Reviewed-by: Filippo Valsorda

[release-branch.go1.14-security] net/http: synchronize "100 Continue" write and Handler writes

2020-07-13T20:58:01Z

The expectContinueReader writes to the connection on the first Request.Body read. Since a Handler might be doing a read in parallel or before a write, expectContinueReader needs to synchronize with the ResponseWriter, and abort if a response already went out. The tests will land in a separate CL. Fixes #34902 Fixes CVE-2020-15586 Change-Id: Icdd8dd539f45e8863762bd378194bb4741e875fc Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/793350 Reviewed-by: Filippo Valsorda (cherry picked from commit b5e504f4a07c572744b228fa1b10e3989c4c44f3) Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/793500

[release-branch.go1.14] go1.14.4

2020-06-01T19:34:54Z

Change-Id: I0daa397bee2ad754fc4860e1365c982de232f171 Reviewed-on: https://go-review.googlesource.com/c/go/+/235919 Run-TryBot: Dmitri Shuralyov TryBot-Result: Gobot Gobot Reviewed-by: Andrew Bonventre

[release-branch.go1.14] cmd/doc: fix merging comments in -src mode

2020-05-28T23:17:18Z

These changes fix go doc -src mode that vomits comments from random files if filesystem does not sort files by name. The issue was with parse.ParseDir using the Readdir order of files, which varies between platforms and filesystem implementations. Another option is to merge comments using token.FileSet.Iterate order in cmd/doc, but since ParseDir is mostly used in go doc, I’ve opted for smaller change because it’s unlikely to break other uses or cause any perfomance issues. Example (macOS APFS): `go doc -src net.ListenPacket` Fixes #38993 Change-Id: I7f9f368c7d9ccd9a2cbc48665f2cb9798c7b3a3f GitHub-Last-Rev: 654fb450421266a0bb64518016944db22bd681e3 GitHub-Pull-Request: golang/go#36104 Reviewed-on: https://go-review.googlesource.com/c/go/+/210999 Run-TryBot: Rob Pike TryBot-Result: Gobot Gobot Reviewed-by: Rob Pike (cherry picked from commit 585e31df63f6879c03b285711de6f9dcba1f2cb0) Reviewed-on: https://go-review.googlesource.com/c/go/+/235579 Run-TryBot: Andrew Bonventre Reviewed-by: Andrew Gerrand

[release-branch.go1.14] encoding/json: don't mangle strings in an edge case when decoding

2020-05-27T20:03:49Z

The added comment contains some context. The original optimization assumed that each call to unquoteBytes (or unquote) followed its corresponding call to rescanLiteral. Otherwise, unquoting a literal might use d.safeUnquote from another re-scanned literal. Unfortunately, this assumption is wrong. When decoding {"foo": "bar"} into a map[T]string where T implements TextUnmarshaler, the sequence of calls would be as follows: 1) rescanLiteral "foo" 2) unquoteBytes "foo" 3) rescanLiteral "bar" 4) unquoteBytes "foo" (for UnmarshalText) 5) unquoteBytes "bar" Note that the call to UnmarshalText happens in literalStore, which repeats the work to unquote the input string literal. But, since that happens after we've re-scanned "bar", we're using the wrong safeUnquote field value. In the added test case, the second string had a non-zero number of safe bytes, and the first string had none since it was all non-ASCII. Thus, "safely" unquoting a number of the first string's bytes could cut a rune in half, and thus mangle the runes. A rather simple fix, without a full revert, is to only allow one use of safeUnquote per call to unquoteBytes. Each call to rescanLiteral when we have a string is soon followed by a call to unquoteBytes, so it's no longer possible for us to use the wrong index. Also add a test case from #38126, which is the same underlying bug, but affecting the ",string" option. Before the fix, the test would fail, just like in the original two issues: --- FAIL: TestUnmarshalRescanLiteralMangledUnquote (0.00s) decode_test.go:2443: Key "开源" does not exist in map: map[开��:12345开源] decode_test.go:2458: Unmarshal unexpected error: json: invalid use of ,string struct tag, trying to unmarshal "\"aaa\tbbb\"" into string Fixes #38106. For #38105. For #38126. Change-Id: I761e54924e9a971a4f9eaa70bbf72014bb1476e6 Reviewed-on: https://go-review.googlesource.com/c/go/+/226218 Run-TryBot: Daniel Martí TryBot-Result: Gobot Gobot Reviewed-by: Joe Tsai (cherry picked from commit 55361a26177b3faf151a1d35467db5d403b51f22) Reviewed-on: https://go-review.googlesource.com/c/go/+/233057 Run-TryBot: Dmitri Shuralyov Reviewed-by: Daniel Martí

[release-branch.go1.14] encoding/json: properly encode strings with ",string" again

2020-05-27T19:22:19Z

golang.org/cl/193604 fixed one bug when one encodes a string with the ",string" option: if SetEscapeHTML(false) is used, we should not be using HTML escaping for the inner string encoding. The CL correctly fixed that. The CL also tried to speed up this edge case. By avoiding an entire new call to Marshal, the new Issue34127 benchmark reduced its time/op by 45%, and lowered the allocs/op from 3 to 2. However, that last optimization wasn't correct: Since Go 1.2 every string can be marshaled to JSON without error even if it contains invalid UTF-8 byte sequences. Therefore there is no need to use Marshal again for the only reason of enclosing the string in double quotes. JSON string encoding isn't just about adding quotes and taking care of invalid UTF-8. We also need to escape some characters, like tabs and newlines. The new code failed to do that. The bug resulted in the added test case failing to roundtrip properly; before our fix here, we'd see an error: invalid use of ,string struct tag, trying to unmarshal "\"\b\f\n\r\t\"\\\"" into string If you pay close attention, you'll notice that the special characters like tab and newline are only encoded once, not twice. When decoding with the ",string" option, the outer string decode works, but the inner string decode fails, as we are now decoding a JSON string with unescaped special characters. The fix we apply here isn't to go back to Marshal, as that would re-introduce the bug with SetEscapeHTML(false). Instead, we can use a new encode state from the pool - it results in minimal performance impact, and even reduces allocs/op further. The performance impact seems fair, given that we need to check the entire string for characters that need to be escaped. name old time/op new time/op delta Issue34127-8 89.7ns ± 2% 100.8ns ± 1% +12.27% (p=0.000 n=8+8) name old alloc/op new alloc/op delta Issue34127-8 40.0B ± 0% 32.0B ± 0% -20.00% (p=0.000 n=8+8) name old allocs/op new allocs/op delta Issue34127-8 2.00 ± 0% 1.00 ± 0% -50.00% (p=0.000 n=8+8) Instead of adding another standalone test, we convert an existing "string tag" test to be table-based, and add another test case there. One test case from the original CL also had to be amended, due to the same problem - when escaping '<' due to SetEscapeHTML(true), we need to end up with double escaping, since we're using ",string". Fixes #38178. For #38173. Change-Id: I2b0df9e4f1d3452fff74fe910e189c930dde4b5b Reviewed-on: https://go-review.googlesource.com/c/go/+/226498 Run-TryBot: Daniel Martí TryBot-Result: Gobot Gobot Reviewed-by: Joe Tsai (cherry picked from commit b1a48af7e8ee87cc46e1bbb07f81ac4853e0f27b) Reviewed-on: https://go-review.googlesource.com/c/go/+/233037 Reviewed-by: Daniel Martí Run-TryBot: Dmitri Shuralyov

[release-branch.go1.14] runtime: disable preemption in startTemplateThread

2020-05-27T17:55:13Z

When a locked M wants to start a new M, it hands off to the template thread to actually call clone and start the thread. The template thread is lazily created the first time a thread is locked (or if cgo is in use). stoplockedm will release the P (_Pidle), then call handoffp to give the P to another M. In the case of a pending STW, one of two things can happen: 1. handoffp starts an M, which does acquirep followed by schedule, which will finally enter _Pgcstop. 2. handoffp immediately enters _Pgcstop. This only occurs if the P has no local work, GC work, and no spinning M is required. If handoffp starts an M, and must create a new M to do so, then newm will simply queue the M on newmHandoff for the template thread to do the clone. When a stop-the-world is required, stopTheWorldWithSema will start the stop and then wait for all Ps to enter _Pgcstop. If the template thread is not fully created because startTemplateThread gets stopped, then another stoplockedm may queue an M that will never get created, and the handoff P will never leave _Pidle. Thus stopTheWorldWithSema will wait forever. A sequence to trigger this hang when STW occurs can be visualized with two threads: T1 T2 ------------------------------- ----------------------------- LockOSThread LockOSThread haveTemplateThread == 0 startTemplateThread haveTemplateThread = 1 newm haveTemplateThread == 1 preempt -> schedule g.m.lockedExt++ gcstopm -> _Pgcstop g.m.lockedg = ... park g.lockedm = ... return ... (any code) preempt -> schedule stoplockedm releasep -> _Pidle handoffp startm (first 3 handoffp cases) newm g.m.lockedExt != 0 Add to newmHandoff, return park Note that the P in T2 is stuck sitting in _Pidle. Since the template thread isn't running, the new M will not be started complete the transition to _Pgcstop. To resolve this, we disable preemption around the assignment of haveTemplateThread and the creation of the template thread in order to guarantee that if handTemplateThread is set then the template thread will eventually exist, in the presence of stops. For #38931 Fixes #38933 Change-Id: I50535fbbe2f328f47b18e24d9030136719274191 Reviewed-on: https://go-review.googlesource.com/c/go/+/232978 Run-TryBot: Michael Pratt Reviewed-by: Austin Clements TryBot-Result: Gobot Gobot (cherry picked from commit 11b3730a02c93fd5745bfd977156541a9033759b) Reviewed-on: https://go-review.googlesource.com/c/go/+/234885 Reviewed-by: Cherry Zhang

[release-branch.go1.14] syscall: preserve Windows file permissions for O_CREAT|O_TRUNC

2020-05-23T21:08:37Z

On Windows, calling syscall.Open(file, O_CREAT|O_TRUNC, 0) for a file that already exists would change the file to be read-only. That is not how the Unix syscall.Open behaves, so avoid it on Windows by calling CreateFile twice if necessary. For #38225 Fixes #39158 Change-Id: I70097fca8863df427cc8a97b9376a9ffc69c6318 Reviewed-on: https://go-review.googlesource.com/c/go/+/234534 Run-TryBot: Ian Lance Taylor TryBot-Result: Gobot Gobot Reviewed-by: Alex Brainman (cherry picked from commit 567556d78657326c99b8fa84ec2a5ee511a0941b) Reviewed-on: https://go-review.googlesource.com/c/go/+/234686 Reviewed-by: Emmanuel Odeke

[release-branch.go1.14] go1.14.3

2020-05-14T19:18:34Z

Change-Id: I8bd94ad129f0f229a7994ea7dd5a4309821dac4c Reviewed-on: https://go-review.googlesource.com/c/go/+/234000 Run-TryBot: Andrew Bonventre TryBot-Result: Gobot Gobot Reviewed-by: Dmitri Shuralyov