aboutsummaryrefslogtreecommitdiff
path: root/src/net/http/transfer.go
AgeCommit message (Collapse)Author
2025-11-18Revert "net/http: do not discard body content when closing it within request ↵Nicholas S. Husin
handlers" This reverts commit cb0d9980f5721715ebb73dd2e580eaa11c2ddee2. Reason for revert: the old behavior seems to be relied on by current users, e.g. https://github.com/connectrpc/connect-go/blob/cb2e11fb88c9a61804043355a619c12d4a30a1a5/protocol_connect.go#L837. For #75933 Change-Id: I996280238e5c70a8d760a0b31e3a13c6a44b8616 Reviewed-on: https://go-review.googlesource.com/c/go/+/721761 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Damien Neil <dneil@google.com> Auto-Submit: Nicholas Husin <nsh@golang.org> Reviewed-by: Nicholas Husin <husin@google.com>
2025-11-14net/http: do not discard body content when closing it within request handlersNicholas S. Husin
(*body).Close() internally tries to discard the content of a request body up to 256 KB. We rely on this behavior to allow connection re-use, by calling (*body).Close() when our request handler exits. Unfortunately, this causes an unfortunate side-effect where we would prematurely try to discard a body content when (*body).Close() is called from within a request handler. There should not be a good reason for (*body).Close() to do this when called from within a request handler. As such, this CL modifies (*body).Close() to not discard body contents when called from within a request handler. Note that when a request handler exits, it will still try to discard the body content for connection re-use. For #75933 Change-Id: I71d2431a540579184066dd35d3da49d6c85c3daf Reviewed-on: https://go-review.googlesource.com/c/go/+/720380 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Nicholas Husin <husin@google.com> Reviewed-by: Damien Neil <dneil@google.com>
2024-09-13net/http: make use of maps.CopyJes Cok
Change-Id: If8e02760fe1fb095ecebc306fd98662a51a1f66f GitHub-Last-Rev: 7b8099f1fed04271c121cdde1c3abd8822955368 GitHub-Pull-Request: golang/go#69426 Reviewed-on: https://go-review.googlesource.com/c/go/+/612736 Reviewed-by: Tim King <taking@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Ian Lance Taylor <iant@google.com> Auto-Submit: Ian Lance Taylor <iant@google.com>
2024-07-29net/http: don't write body for HEAD responses in Response.WriteDamien Neil
Fixes #62015 Change-Id: I88c5427f85e740d5b956942bb1c2727dac2935ea Reviewed-on: https://go-review.googlesource.com/c/go/+/601238 Reviewed-by: Michael Knyszek <mknyszek@google.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-03-21net/http: use slices to simplify the codeapocelipes
"strSliceContains" is replaced by "slices.Contains". Replace "sort.Strings" with "slices.Sort" since it becomes a wrapper of "slices.Sort" from Go 1.22. "headerSorter" no longer has to implement "sort.Interface". We use "slice.SortFunc" to sort kvs. Change-Id: Ic29b4c3db147c16079575eca7ad6ff6c0f581188 GitHub-Last-Rev: 78221d5aa223a259a89860b672f39a34897df253 GitHub-Pull-Request: golang/go#66440 Reviewed-on: https://go-review.googlesource.com/c/go/+/573275 Reviewed-by: Ian Lance Taylor <iant@google.com> Reviewed-by: qiulaidongfeng <2645477756@qq.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Damien Neil <dneil@google.com> Auto-Submit: Ian Lance Taylor <iant@golang.org> Reviewed-by: Emmanuel Odeke <emmanuel@orijtech.com> TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Emmanuel Odeke <emmanuel@orijtech.com>
2024-03-09internal/godebugs: test for use of IncNonDefaultRuss Cox
A few recent godebugs are missing IncNonDefault uses. Test for that, so that people remember to do it. Filed bugs for the missing ones. For #66215. For #66216. For #66217. Change-Id: Ia3fd10fd108e1b003bb30a8bc2f83995c768fab6 Reviewed-on: https://go-review.googlesource.com/c/go/+/570275 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Damien Neil <dneil@google.com>
2024-02-14net/http: reject requests with invalid Content-Length headersAndy Pan
According to RFC 9110 and RFC 9112, invalid "Content-Length" headers might involve request smuggling or response splitting, which could also cause security failures. Currently, `net/http` ignores all "Content-Length" headers when there is a "Transfer-Encoding" header and forward the message anyway while other mainstream HTTP implementations such as Apache Tomcat, Nginx, HAProxy, Node.js, Deno, Tornado, etc. reject invalid Content-Length headers regardless of the presence of a "Transfer-Encoding" header and only forward chunked-encoding messages with either valid "Content-Length" headers or no "Content-Length" headers. Fixes #65505 Change-Id: I73af2ee0785137e56c7546a4cce4a5c5c348dbc5 Reviewed-on: https://go-review.googlesource.com/c/go/+/561075 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Bryan Mills <bcmills@google.com> Reviewed-by: Damien Neil <dneil@google.com>
2024-01-10net: add available godoc linkcui fliter
Change-Id: Ib7c4baf0247c421954aedabfbb6a6af8a08a8936 Reviewed-on: https://go-review.googlesource.com/c/go/+/540021 Reviewed-by: Damien Neil <dneil@google.com> Run-TryBot: shuang cui <imcusg@gmail.com> TryBot-Result: Gopher Robot <gobot@golang.org> Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
2023-11-21net/http: use pointers to array for copyBufPoolJorropo
This is inspired by CL 539915, I'm only submitting now that CL 456435 has been merged. This divide the number of objects kept alive by the heap by two and remove the slice header allocation in New and in the put back. Change-Id: Ibcd5166bac5a37f365a533e09a28f3b79f81ad58 Reviewed-on: https://go-review.googlesource.com/c/go/+/543515 Reviewed-by: Damien Neil <dneil@google.com> Auto-Submit: Damien Neil <dneil@google.com> Reviewed-by: qiulaidongfeng <2645477756@qq.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Michael Knyszek <mknyszek@google.com>
2023-11-09net/http: use copyBufPool in transferWriter.doBodyCopy()Michał Matczuk
This is a followup to CL 14177. It applies copyBufPool optimization to transferWriter.doBodyCopy(). The function is used every time Request or Response is written. Without this patch for every Request and Response processed, if there is a body, we need to allocate and GC a 32k buffer. This is quickly causing GC pressure. Fixes #57202 Change-Id: I4c30e1737726ac8d9937846106efd02effbae300 GitHub-Last-Rev: 908573cdbe2e8b6f91ce026cf8632ff5f2c41110 GitHub-Pull-Request: golang/go#57205 Reviewed-on: https://go-review.googlesource.com/c/go/+/456435 Reviewed-by: Damien Neil <dneil@google.com> Reviewed-by: Bryan Mills <bcmills@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: qiulaidongfeng <2645477756@qq.com>
2023-08-11net/http: disallow empty Content-Length headerMauri de Souza Meneguzzo
The Content-Length must be a valid numeric value, empty values should not be accepted. See: https://www.rfc-editor.org/rfc/rfc9110.html#name-content-length Fixes #61679 Change-Id: Icbcd933087fe5e50199b62ff34c58bf92a09d3d4 GitHub-Last-Rev: 932e46b55b54d5f2050453bcaa50e9476c8559fd GitHub-Pull-Request: golang/go#61865 Reviewed-on: https://go-review.googlesource.com/c/go/+/517336 Reviewed-by: Damien Neil <dneil@google.com> Auto-Submit: Bryan Mills <bcmills@google.com> Reviewed-by: Bryan Mills <bcmills@google.com> Run-TryBot: Damien Neil <dneil@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>
2023-02-14all: fix some commentscui fliter
Change-Id: I16ec916b47de2f417b681c8abff5a1375ddf491b Reviewed-on: https://go-review.googlesource.com/c/go/+/468055 Run-TryBot: Ian Lance Taylor <iant@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Auto-Submit: Ian Lance Taylor <iant@google.com> Reviewed-by: Ian Lance Taylor <iant@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
2022-11-18all: add missing periods in commentscui fliter
Change-Id: I69065f8adf101fdb28682c55997f503013a50e29 Reviewed-on: https://go-review.googlesource.com/c/go/+/449757 Auto-Submit: Ian Lance Taylor <iant@google.com> Reviewed-by: Joedian Reid <joedian@golang.org> Reviewed-by: Keith Randall <khr@google.com> Reviewed-by: Keith Randall <khr@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Joedian Reid <joedian@golang.org> Run-TryBot: Ian Lance Taylor <iant@google.com> Reviewed-by: Ian Lance Taylor <iant@google.com>
2022-09-21net/http: accept HEAD requests with a bodyDamien Neil
RFC 7231 permits HEAD requests to contain a body, although it does state there are no defined semantics for payloads of HEAD requests and that some servers may reject HEAD requests with a payload. Accept HEAD requests with a body. Fix a bug where a HEAD request with a chunked body would interpret the body as the headers for the next request on the connection. For #53960. Change-Id: I83f7112fdedabd6d6291cd956151d718ee6942cd Reviewed-on: https://go-review.googlesource.com/c/go/+/418614 Run-TryBot: Damien Neil <dneil@google.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Reviewed-by: Cherry Mui <cherryyz@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>
2022-06-29net/http: don't strip whitespace from Transfer-Encoding headersDamien Neil
Do not accept "Transfer-Encoding: \rchunked" as a valid TE header setting chunked encoding. Thanks to Zeyu Zhang (https://www.zeyu2001.com/) for identifying the issue. Fixes #53188 Fixes CVE-2022-1705 Change-Id: I1a16631425159267f2eca68056b057192a7edf6c Reviewed-on: https://go-review.googlesource.com/c/go/+/409874 Reviewed-by: Roland Shoemaker <roland@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2022-05-17net/http: allow sending 1xx responsesKévin Dunglas
Currently, it's not possible to send informational responses such as 103 Early Hints or 102 Processing. This patch allows calling WriteHeader() multiple times in order to send informational responses before the final one. If the status code is in the 1xx range, the current content of the header map is also sent. Its content is not removed after the call to WriteHeader() because the headers must also be included in the final response. The Chrome and Fastly teams are starting a large-scale experiment to measure the real-life impact of the 103 status code. Using Early Hints is proposed as a (partial) alternative to Server Push, which are going to be removed from Chrome: https://groups.google.com/a/chromium.org/g/blink-dev/c/K3rYLvmQUBY/m/21anpFhxAQAJ Being able to send this status code from servers implemented using Go would help to see if implementing it in browsers is worth it. Fixes #26089 Fixes #36734 Updates #26088 Change-Id: Ib7023c1892c35e8915d4305dd7f6373dbd00a19d GitHub-Last-Rev: 06d749d3454aa35c177a50ce4a25715df21fd742 GitHub-Pull-Request: golang/go#42597 Reviewed-on: https://go-review.googlesource.com/c/go/+/269997 Reviewed-by: Damien Neil <dneil@google.com> Reviewed-by: Ian Lance Taylor <iant@google.com>
2022-05-03io: NopCloser forward WriterTo implementations if the reader supports itJorropo
This patch also include related fixes to net/http. io_test.go don't test reading or WritingTo of the because the logic is simple. NopCloser didn't even had direct tests before. Fixes #51566 Change-Id: I1943ee2c20d0fe749f4d04177342ce6eca443efe GitHub-Last-Rev: a6b9af4e945a6903735a74aa185e2d1c4c2e2cef GitHub-Pull-Request: golang/go#52340 Reviewed-on: https://go-review.googlesource.com/c/go/+/400236 Run-TryBot: Ian Lance Taylor <iant@golang.org> Run-TryBot: Ian Lance Taylor <iant@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@google.com> Reviewed-by: Benny Siegert <bsiegert@gmail.com> Auto-Submit: Ian Lance Taylor <iant@google.com>
2022-04-11all: gofmt main repoRuss Cox
[This CL is part of a sequence implementing the proposal #51082. The design doc is at https://go.dev/s/godocfmt-design.] Run the updated gofmt, which reformats doc comments, on the main repository. Vendored files are excluded. For #51082. Change-Id: I7332f099b60f716295fb34719c98c04eb1a85407 Reviewed-on: https://go-review.googlesource.com/c/go/+/384268 Reviewed-by: Jonathan Amsterdam <jba@google.com> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2021-12-13all: gofmt -w -r 'interface{} -> any' srcRuss Cox
And then revert the bootstrap cmd directories and certain testdata. And adjust tests as needed. Not reverting the changes in std that are bootstrapped, because some of those changes would appear in API docs, and we want to use any consistently. Instead, rewrite 'any' to 'interface{}' in cmd/dist for those directories when preparing the bootstrap copy. A few files changed as a result of running gofmt -w not because of interface{} -> any but because they hadn't been updated for the new //go:build lines. Fixes #49884. Change-Id: Ie8045cba995f65bd79c694ec77a1b3d1fe01bb09 Reviewed-on: https://go-review.googlesource.com/c/go/+/368254 Trust: Russ Cox <rsc@golang.org> Run-TryBot: Russ Cox <rsc@golang.org> Reviewed-by: Robert Griesemer <gri@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org>
2021-11-07net/http: fix spelling in documentationjiahua wang
Change-Id: I8b0924300eafe27de98975512a78a6527a92e446 Reviewed-on: https://go-review.googlesource.com/c/go/+/354729 Reviewed-by: Ian Lance Taylor <iant@golang.org> Trust: Damien Neil <dneil@google.com>
2021-09-02net/http: fix hang in probing for a zero-length request bodyDamien Neil
Fix a hang that occurs when making a request and all of the following apply: * The request method is one of GET, HEAD, DELETE, OPTIONS, PROPFIND, or SEARCH. * The Request.Body is non-nil. * The content length is not set, or is set to -1. * Transfer-Encoding: chunked is not set. * The request body does not respond to a read within 200ms. In this case, we give up on probing for a zero-length body and send the request while the probe completes in the background. Fix a bug in the io.Reader wrapping the in-flight probe: It should return io.EOF after the probe completes, but does not. Fixes #47568. Change-Id: I7f9188c96e1210055df68424081af927006e4816 Reviewed-on: https://go-review.googlesource.com/c/go/+/340256 Trust: Damien Neil <dneil@google.com> Run-TryBot: Damien Neil <dneil@google.com> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Ingo Oeser <nightlyone@googlemail.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2021-05-10net/http: switch HTTP1 to ASCII equivalents of string functionsRoberto Clapis
The current implementation uses UTF-aware functions like strings.EqualFold and strings.ToLower. This could, in some cases, cause http smuggling. Change-Id: I0e76a993470a1e1b1b472f4b2859ea0a2b22ada0 Reviewed-on: https://go-review.googlesource.com/c/go/+/308009 Run-TryBot: Filippo Valsorda <filippo@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Trust: Roberto Clapis <roberto@golang.org> Reviewed-by: Filippo Valsorda <filippo@golang.org>
2020-10-20all: update references to symbols moved from io/ioutil to ioRuss Cox
The old ioutil references are still valid, but update our code to reflect best practices and get used to the new locations. Code compiled with the bootstrap toolchain (cmd/asm, cmd/dist, cmd/compile, debug/elf) must remain Go 1.4-compatible and is excluded. Also excluded vendored code. For #41190. Change-Id: I6d86f2bf7bc37a9d904b6cee3fe0c7af6d94d5b1 Reviewed-on: https://go-review.googlesource.com/c/go/+/263142 Trust: Russ Cox <rsc@golang.org> Run-TryBot: Russ Cox <rsc@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
2020-10-16net/http: ensure Request.Body.Close is called once and only onceRoss Light
Makes *Request.write always close the body, so that callers no longer have to close the body on returned errors, which was the trigger for double-close behavior. Fixes #40382 Change-Id: I128f7ec70415f240d82154cfca134b3f692191e3 Reviewed-on: https://go-review.googlesource.com/c/go/+/257819 Reviewed-by: Damien Neil <dneil@google.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Trust: Damien Neil <dneil@google.com> Trust: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Damien Neil <dneil@google.com> TryBot-Result: Go Bot <gobot@golang.org>
2020-08-24net/http: set Content-Length:0 for empty PATCH requests as with POST, PATCHSegflow
Sets Content-Length:0 for nil bodies in PATCH requests, as we already do for POST and PUT requests. RFC 2616 mentions that unless a method’s Content-Length is forbidden it can send one. In the wild, we’ve found that Microsoft Azure’s DataLake Gen2 storage API https://docs.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/update deliberately rejects PATCH requests without a Content-Length, yet there is no workaround for setting that header when trying to flush the content of a file which was uploaded in a previous request. Fixes #40978 Change-Id: Ib0a623b907d827a1c5ee431dca3c41024fa291c5 GitHub-Last-Rev: 12a3903f2bc22bcc4f5f8e2abcc3922b612b8871 GitHub-Pull-Request: golang/go#40991 Reviewed-on: https://go-review.googlesource.com/c/go/+/250039 Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
2020-05-31net/http: reject HTTP/1.1 Content-Length with sign in responsePaschalis Tsilias
Enforces section 14.13 of RFC 2616 so that Content-Length header values with a sign such as "+5" will be rejected. Updates #39017 Change-Id: Icce9f00d03c8475fe704b33f9bed9089ff8802f0 Reviewed-on: https://go-review.googlesource.com/c/go/+/234817 Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
2020-05-27net/http: handle body rewind in HTTP/2 connection loss betterRuss Cox
In certain cases the HTTP/2 stack needs to resend a request. It obtains a fresh body to send by calling req.GetBody. This call was missing from the path where the HTTP/2 round tripper returns ErrSkipAltProtocol, meaning fall back to HTTP/1.1. The result was that the HTTP/1.1 fallback request was sent with no body at all. This CL changes that code path to rewind the body before falling back to HTTP/1.1. But rewinding the body is easier said than done. Some requests have no GetBody function, meaning the body can't be rewound. If we need to rewind and can't, that's an error. But if we didn't read anything, we don't need to rewind. So we have to track whether we read anything, with a new ReadCloser wrapper. That in turn requires adding to the couple places that unwrap Body values to look at the underlying implementation. This CL adds the new rewinding code in the main retry loop as well. The new rewindBody function also takes care of closing the old body before abandoning it. That was missing in the old rewind code. Thanks to Aleksandr Razumov for CL 210123 and to Jun Chen for CL 234358, both of which informed this CL. Fixes #32441. Change-Id: Id183758526c087c6b179ab73cf3b61ed23a2a46a Reviewed-on: https://go-review.googlesource.com/c/go/+/234894 Run-TryBot: Russ Cox <rsc@golang.org> Reviewed-by: Damien Neil <dneil@google.com> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2020-05-06net/http: use ASCII space trimming throughoutFilippo Valsorda
Security hardening against HTTP request smuggling. Thank you to ZeddYu for reporting this issue. Change-Id: I98bd9f8ffe58360fc3bca9dc5d9a106773e55373 Reviewed-on: https://go-review.googlesource.com/c/go/+/231419 Reviewed-by: Katie Hockman <katie@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2020-05-06net/http: only support "chunked" in inbound Transfer-Encoding headersFilippo Valsorda
This is a security hardening measure against HTTP request smuggling. Thank you to ZeddYu for reporting this issue. We weren't parsing things correctly anyway, allowing "identity" to be combined with "chunked", and ignoring any Transfer-Encoding header past the first. This is a delicate security surface that already broke before, just be strict and don't add complexity to support cases not observed in the wild (nginx removed "identity" support [1] and multiple TE header support [2]) and removed by RFC 7230 (see page 81). It'd probably be good to also drop support for anything other than "chunked" in outbound TE headers, as "identity" is not a thing anymore, and we are probably off-spec for anything other than "chunked", but it should not be a security concern, so leaving it for now. See #38867. [1]: https://hg.nginx.org/nginx/rev/fe5976aae0e3 [2]: https://hg.nginx.org/nginx/rev/aca005d232ff Change-Id: If17d0827f9c6167a0b19a158e2bc5844ec803288 Reviewed-on: https://go-review.googlesource.com/c/go/+/231418 Reviewed-by: Katie Hockman <katie@golang.org>
2020-05-01net/http: remove badStringError, make some unexported structs non-comparableBrad Fitzpatrick
Reduces binary size by 4K, not counting the http2 changes (in CL 231119) that'll be bundled into this package in the future. Updates golang/go#38782 Change-Id: Id360348707e076b8310a8f409e412d68dd2394b2 Reviewed-on: https://go-review.googlesource.com/c/go/+/231118 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2020-01-22Revert "net/http: support gzip, x-gzip Transfer-Encodings"Filippo Valsorda
This reverts commit e6c12c3d0296251f1d5a96ebde811dbfd4a914fe. Reason for revert: the assumption that a T-E of "gzip" implies "chunked" seems incorrect. The RFC does state that one "MUST apply chunked as the final transfer coding" but that should be interpreted to mean that a "chunked" encoding must be listed as the last one, not that one should be assumed to be there if not. This is confirmed by the alternative option to chunking on the server side being to "terminate the message by closing the connection". The issue seems confirmed by the fact that the code in the body of #29162 fails with the following error: net/http: HTTP/1.x transport connection broken: http: failed to gunzip body: unexpected EOF This late in the cycle, revert rather than fix, also because we don't apparently have tests for the correct behavior. Change-Id: I920ec928754cd8e96a06fb7ff8a53316c0f959e5 Reviewed-on: https://go-review.googlesource.com/c/go/+/215757 Run-TryBot: Filippo Valsorda <filippo@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Katie Hockman <katie@golang.org> Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
2019-11-15all: fix a bunch of misspellingsVille Skyttä
Change-Id: I5b909df0fd048cd66c5a27fca1b06466d3bcaac7 GitHub-Last-Rev: 778c5d21311abee09a5fbda2e4005a5fd4cc3f9f GitHub-Pull-Request: golang/go#35624 Reviewed-on: https://go-review.googlesource.com/c/go/+/207421 Run-TryBot: Ian Lance Taylor <iant@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-11-08net/http: support gzip, x-gzip Transfer-EncodingsEmmanuel T Odeke
Support "gzip" aka "x-gzip" as a transfer-encoding for requests and responses as per RFC 7230 Section 3.3.1. "gzip" and "x-gzip" are equivalents as requested by RFC 7230 Section 4.2.3. Transfer-Encoding is an on-fly property of the body that can be applied by proxies, other servers and basically any intermediary to transport the content e.g. across data centers or backends/machine to machine that need compression. For this change, "gzip" is both explicitly and implicitly combined with transfer-encoding "chunked" in an ordering such as: Transfer-Encoding: gzip, chunked and NOT Transfer-Encoding: chunked, gzip Obviously the latter form is counter-intuitive for streaming. Thus "chunked" is the last value to appear in that transfer-encoding header, if explicitly included. When parsing the response, the chunked body is concatenated as "chunked" does, before finally being decompressed as "gzip". A chunked and compressed body would typically look like this: <LENGTH_1>\r\n<CHUNK_1_GZIPPED_BODY>\r\n<LENGTH_2>\r\n<CHUNK_2_GZIPPED_BODY>\0\r\n which when being processed we would contentate <FULL_BODY> := <CHUNK_1_GZIPPED_BODY> + <CHUNK_2_GZIPPED_BODY> + ... and then finally gunzip it <FINAL_BODY> := gunzip(<FULL_BODY>) If a "chunked" transfer-encoding is NOT applied but "gzip" is applied, we implicitly assume that they requested using "chunked" at the end. This is as per the recommendation of RFC 3.3.1. which explicitly says that for: * Request: " If any transfer coding other than chunked is applied to a request payload body, the sender MUST apply chunked as the final transfer coding to ensure that the message is properly framed." * Response: " If any transfer coding other than chunked is applied to a response payload body, the sender MUST either apply chunked as the final transfer coding or terminate the message by closing the connection." RELNOTE=yes Fixes #29162 Change-Id: Icb8b8b838cf4119705605b29725cabb1fe258491 Reviewed-on: https://go-review.googlesource.com/c/go/+/166517 Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-04-30net/http: make Server return 501 for unsupported transfer-encodingsEmmanuel T Odeke
Ensures that our HTTP/1.X Server properly responds with a 501 Unimplemented as mandated by the spec at RFC 7230 Section 3.3.1, which says: A server that receives a request message with a transfer coding it does not understand SHOULD respond with 501 (Unimplemented). Fixes #30710 Change-Id: I096904e6df053cd1e4b551774cc27523ff3d09f6 Reviewed-on: https://go-review.googlesource.com/c/go/+/167017 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-03-11all: move internal/x to vendor/golang.org/x and revendor using 'go mod vendor'Bryan C. Mills
This also updates the vendored-in versions of several packages: 'go mod vendor' selects a consistent version of each module, but we had previously vendored an ad-hoc selection of packages. Notably, x/crypto/hkdf was previously vendored in at a much newer commit than the rest of x/crypto. Bringing the rest of x/crypto up to that commit introduced an import of golang.org/x/sys/cpu, which broke the js/wasm build, requiring an upgrade of x/sys to pick up CL 165749. Updates #30228 Updates #30241 Updates #25822 Change-Id: I5b3dbc232b7e6a048a158cbd8d36137af1efb711 Reviewed-on: https://go-review.googlesource.com/c/go/+/164623 Reviewed-by: Filippo Valsorda <filippo@golang.org>
2019-03-07net/http: let Transport request body writes use sendfileChris Marchesi
net.TCPConn has the ability to send data out using system calls such as sendfile when the source data comes from an *os.File. However, the way that I/O has been laid out in the transport means that the File is actually wrapped behind two outer io.Readers, and as such the TCP stack cannot properly type-assert the reader, ensuring that it falls back to genericReadFrom. This commit does the following: * Removes transferBodyReader and moves its functionality to a new doBodyCopy helper. This is not an io.Reader implementation, but no functionality is lost this way, and it allows us to unwrap one layer from the body. * The second layer of the body is unwrapped if the original reader was wrapped with ioutil.NopCloser, which is what NewRequest wraps the body in if it's not a ReadCloser on its own. The unwrap operation passes through the existing body if there's no nopCloser. Note that this depends on change https://golang.org/cl/163737 to properly function, as the lack of ReaderFrom implementation otherwise means that this functionality is essentially walled off. Benchmarks between this commit and https://golang.org/cl/163862, incorporating https://golang.org/cl/163737: linux/amd64: name old time/op new time/op delta FileAndServer_1KB/NoTLS-4 53.2µs ± 0% 53.3µs ± 0% ~ (p=0.075 n=10+9) FileAndServer_1KB/TLS-4 61.2µs ± 0% 60.7µs ± 0% -0.77% (p=0.000 n=10+9) FileAndServer_16MB/NoTLS-4 25.3ms ± 5% 3.8ms ± 6% -84.95% (p=0.000 n=10+10) FileAndServer_16MB/TLS-4 33.2ms ± 2% 13.4ms ± 2% -59.57% (p=0.000 n=10+10) FileAndServer_64MB/NoTLS-4 106ms ± 4% 16ms ± 2% -84.45% (p=0.000 n=10+10) FileAndServer_64MB/TLS-4 129ms ± 1% 54ms ± 3% -58.32% (p=0.000 n=8+10) name old speed new speed delta FileAndServer_1KB/NoTLS-4 19.2MB/s ± 0% 19.2MB/s ± 0% ~ (p=0.095 n=10+9) FileAndServer_1KB/TLS-4 16.7MB/s ± 0% 16.9MB/s ± 0% +0.78% (p=0.000 n=10+9) FileAndServer_16MB/NoTLS-4 664MB/s ± 5% 4415MB/s ± 6% +565.27% (p=0.000 n=10+10) FileAndServer_16MB/TLS-4 505MB/s ± 2% 1250MB/s ± 2% +147.32% (p=0.000 n=10+10) FileAndServer_64MB/NoTLS-4 636MB/s ± 4% 4090MB/s ± 2% +542.81% (p=0.000 n=10+10) FileAndServer_64MB/TLS-4 522MB/s ± 1% 1251MB/s ± 3% +139.95% (p=0.000 n=8+10) darwin/amd64: name old time/op new time/op delta FileAndServer_1KB/NoTLS-8 93.0µs ± 5% 96.6µs ±11% ~ (p=0.190 n=10+10) FileAndServer_1KB/TLS-8 105µs ± 7% 100µs ± 5% -5.14% (p=0.002 n=10+9) FileAndServer_16MB/NoTLS-8 87.5ms ±19% 10.0ms ± 6% -88.57% (p=0.000 n=10+10) FileAndServer_16MB/TLS-8 52.7ms ±11% 17.4ms ± 5% -66.92% (p=0.000 n=10+10) FileAndServer_64MB/NoTLS-8 363ms ±54% 39ms ± 7% -89.24% (p=0.000 n=10+10) FileAndServer_64MB/TLS-8 209ms ±13% 73ms ± 5% -65.37% (p=0.000 n=9+10) name old speed new speed delta FileAndServer_1KB/NoTLS-8 11.0MB/s ± 5% 10.6MB/s ±10% ~ (p=0.184 n=10+10) FileAndServer_1KB/TLS-8 9.75MB/s ± 7% 10.27MB/s ± 5% +5.26% (p=0.003 n=10+9) FileAndServer_16MB/NoTLS-8 194MB/s ±16% 1680MB/s ± 6% +767.83% (p=0.000 n=10+10) FileAndServer_16MB/TLS-8 319MB/s ±10% 963MB/s ± 4% +201.36% (p=0.000 n=10+10) FileAndServer_64MB/NoTLS-8 180MB/s ±31% 1719MB/s ± 7% +853.61% (p=0.000 n=9+10) FileAndServer_64MB/TLS-8 321MB/s ±12% 926MB/s ± 5% +188.24% (p=0.000 n=9+10) Updates #30377. Change-Id: I631a73cea75371dfbb418c9cd487c4aa35e73fcd GitHub-Last-Rev: 4a77dd1b80140274bf3ed20ad7465ff3cc06febf GitHub-Pull-Request: golang/go#30378 Reviewed-on: https://go-review.googlesource.com/c/go/+/163599 Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
2018-11-29vendor/golang_org/x: move to internal/xBryan C. Mills
Packages in vendor/ directories have a "vendor/" path prefix in GOPATH mode, but intentionally do not in module mode. Since the import path is embedded in the compiled output, changing that path invalidates cache entries and causes cmd/go to try to rebuild (and reinstall) the vendored libraries, which will fail if the directory containing those libraries is read-only. If I understood correctly, this is the approach Russ suggested as an alternative to https://golang.org/cl/136138. Fixes #27285 Fixes #26988 Change-Id: I8a2507fa892b84cde0a803aaa79e460723da572b Reviewed-on: https://go-review.googlesource.com/c/147443 Run-TryBot: Bryan C. Mills <bcmills@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Russ Cox <rsc@golang.org>
2018-10-30net/http: in Transport, don't error on non-chunked response with Trailer headerBigMikes
There are cases where HTTP message specifies the Trailer header but not the Transfer-Encoding = chunked. The existing implementation would return an error in those cases, without returning also the message itself. Instead, it would be preferable to let the library user decide when the message is valid or not. This change makes the fixTrailer() function not to return an error and to keep the Trailer value in the Response.Header map but not populate Response.Trailer. Fixes #27197 Change-Id: Ic1e96791fde97f31dc5ecb8de05c8e4f49465c2c Reviewed-on: https://go-review.googlesource.com/c/145398 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-12net/http: flesh out Transport's HTTP/1 CONNECT+bidi support to match HTTP/2Brad Fitzpatrick
Fixes #17227 Change-Id: I0f8964593d69623b85d5759f6276063ee62b2915 Reviewed-on: https://go-review.googlesource.com/c/123156 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-08all: fix a bunch of misspellingsIgor Zhilianin
Change-Id: I94cebca86706e072fbe3be782d3edbe0e22b9432 GitHub-Last-Rev: 8e15a40545704fb21b41a8768079f2da19341ef3 GitHub-Pull-Request: golang/go#28067 Reviewed-on: https://go-review.googlesource.com/c/140437 Run-TryBot: Ian Lance Taylor <iant@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2018-06-27net/http/httptrace: expose request headers for http/1.1Meir Fischer
Some headers, which are set or modified by the http library, are not written to the standard http.Request.Header and are not included as part of http.Response.Request.Header. Exposing all headers alleviates this problem. This is not a complete solution to 19761 since it does not have http/2 support. Updates #19761 Change-Id: Ie8d4f702f4f671666b120b332378644f094e288b Reviewed-on: https://go-review.googlesource.com/67430 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-24net/http: conservatively flush Transport request headers by defaultBrad Fitzpatrick
This changes the http.Transport to flush the bufio.Writer between writing the request headers and the body. That wasn't done in the past to minimize the number of TCP packets on the wire, but that's just an optimization, and it causes problems when servers are waiting for the headers and the client is blocked on something before reading the body. Instead, only do the don't-flush optimization if we know we're not going to block, whitelisting a set of common in-memory Request.Body types. (the same set of types special-cased by http.NewRequest) Fixes #22088 Change-Id: I7717750aa6df32dd3eb92d181b45bc7af24b1144 Reviewed-on: https://go-review.googlesource.com/114316 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Andrew Bonventre <andybons@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-07vendor, net/http: update x/net for httplex to httpguts mergeBrad Fitzpatrick
Updates x/net to git rev cbb82b59bc for: lex/httplex, http/httpguts: merge the httplex package into httpguts https://golang.org/cl/111875 http2: set nextStreamID to 3 when AllowHTTP is set https://golang.org/cl/111835 http2: terminate await request cancel goroutine on conn close https://golang.org/cl/108415 Fixes #24776 (CL 111655 didn't actually include it) Change-Id: I0a21e169ebba2ec35219f347f1e31cd4c67bebdf Reviewed-on: https://go-review.googlesource.com/111876 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> Reviewed-by: Kunpei Sakai <namusyaka@gmail.com> Reviewed-by: Andrew Bonventre <andybons@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2018-02-20net/http: use RFC 723x as normative reference in docsDavid Url
Replace references to the obsoleted RFC 2616 with references to RFC 7230 through 7235, to avoid unnecessary confusion. Obvious inconsistencies are marked with todo comments. Updates #21974 Change-Id: I8fb4fcdd1333fc5193b93a2f09598f18c45e7a00 Reviewed-on: https://go-review.googlesource.com/94095 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-10-19net/http: ignore response body when forbidden by status codeBen Burkert
A 1XX, 204, or 304 response may not include a response body according to RFC 7230, section 3.3.3. If a buggy server returns a 204 or 304 response with a body that is chunked encoded, the invalid body is currently made readable in the Response. This can lead to data races due to the transport connection's read loop which does not wait for the body EOF when the response status is 204 or 304. The correct behavior is to ignore the body on a 204 or 304 response, and treat the body data as the beginning of the next request on the connection. Updates #22330. Change-Id: I89a457ceb783b6f66136d5bf9be0a9b0a04fa955 Reviewed-on: https://go-review.googlesource.com/71910 Reviewed-by: Tom Bergan <tombergan@google.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Tom Bergan <tombergan@google.com>
2017-08-14net/http: various small cleanupsDaniel Martí
* Remove an unnecessary type conversion * Make golint happier about consistent receiver names * Make golint happier about a foo_bar var name Change-Id: I5223808109f6f8b69ed4be76de82faf2478c6a2e Reviewed-on: https://go-review.googlesource.com/54530 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Tom Bergan <tombergan@google.com>
2017-06-08net/http: remove invalid checks of Request.Proto* for outgoing requestsBrad Fitzpatrick
The net/http package has long documented that Request.ProtoMajor and Request.ProtoMinor are ignored for outgoing requests (HTTP/1.1 or HTTP/2 is always used, never HTTP/1.0). There was one part in the code that was actually checking 1.0 vs 1.1, but it appears to have been harmless. Remove it. Fixes #18407 Change-Id: I362ed6c47ca2de7a2fbca917ed3e866273cfe41f Reviewed-on: https://go-review.googlesource.com/45155 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2017-05-24net/http: Don't write 'Connection: close' header multiple times.Filip Gruszczyński
When writing the 'Connection: close' header based on response Close attribute we also check if it is already in the headers scheduled to be written and skip if necessary. Fixes #19499 Change-Id: I92357344a37ae385454ec8006114fa4cfa585810 Reviewed-on: https://go-review.googlesource.com/38076 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2017-03-02net/http: clean up Transport.RoundTrip error handlingBrad Fitzpatrick
If I put a 10 millisecond sleep at testHookWaitResLoop, before the big select in (*persistConn).roundTrip, two flakes immediately started happening, TestTransportBodyReadError (#19231) and TestTransportPersistConnReadLoopEOF. The problem was that there are many ways for a RoundTrip call to fail (errors reading from Request.Body while writing the response, errors writing the response, errors reading the response due to server closes, errors due to servers sending malformed responses, cancelations, timeouts, etc.), and many of those failures then tear down the TCP connection, causing more failures, since there are always at least three goroutines involved (reading, writing, RoundTripping). Because the errors were communicated over buffered channels to a giant select, the error returned to the caller was a function of which random select case was called, which was why a 10ms delay before the select brought out so many bugs. (several fixed in my previous CLs the past few days). Instead, track the error explicitly in the transportRequest, guarded by a mutex. In addition, this CL now: * differentiates between the two ways writing a request can fail: the io.Copy reading from the Request.Body or the io.Copy writing to the network. A new io.Reader type notes read errors from the Request.Body. The read-from-body vs write-to-network errors are now prioritized differently. * unifies the two mapRoundTripErrorFromXXX methods into one mapRoundTripError method since their logic is now the same. * adds a (*Request).WithT(*testing.T) method in export_test.go, usable by tests, to call t.Logf at points during RoundTrip. This is disabled behind a constant except when debugging. * documents and deflakes TestClientRedirectContext I've tested this CL with high -count values, with/without -race, with/without delays before the select, etc. So far it seems robust. Fixes #19231 (TestTransportBodyReadError flake) Updates #14203 (source of errors unclear; they're now tracked more) Updates #15935 (document Transport errors more; at least understood more now) Change-Id: I3cccc3607f369724b5344763e35ad2b7ea415738 Reviewed-on: https://go-review.googlesource.com/37495 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
2016-12-22net/http: restore Transport's Request.Body byte sniff in limited casesBrad Fitzpatrick
In Go 1.8, we'd removed the Transport's Request.Body one-byte-Read-sniffing to disambiguate between non-nil Request.Body with a ContentLength of 0 or -1. Previously, we tried to see whether a ContentLength of 0 meant actually zero, or just an unset by reading a single byte of the Request.Body and then stitching any read byte back together with the original Request.Body. That historically has caused many problems due to either data races, blocking forever (#17480), or losing bytes (#17071). Thus, we removed it in both HTTP/1 and HTTP/2 in Go 1.8. Unfortunately, during the Go 1.8 beta, we've found that a few people have gotten bitten by the behavior change on requests with methods typically not containing request bodies (e.g. GET, HEAD, DELETE). The most popular example is the aws-go SDK, which always set http.Request.Body to a non-nil value, even on such request methods. That was causing Go 1.8 to send such requests with Transfer-Encoding chunked bodies, with zero bytes, confusing popular servers (including but limited to AWS). This CL partially reverts the no-byte-sniffing behavior and restores it only for GET/HEAD/DELETE/etc requests, and only when there's no Transfer-Encoding set, and the Content-Length is 0 or -1. Updates #18257 (aws-go) bug And also private bug reports about non-AWS issues. Updates #18407 also, but haven't yet audited things enough to declare it fixed. Change-Id: Ie5284d3e067c181839b31faf637eee56e5738a6a Reviewed-on: https://go-review.googlesource.com/34668 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
generated by cgit v1.3 (git 2.53.0) at 2026-04-15 19:12:31 +0000