aboutsummaryrefslogtreecommitdiff
path: root/src/internal/zstd
AgeCommit message (Collapse)Author
2025-10-17all: remove unnecessary loop variable copies in testsTobias Klauser
Copying the loop variable is no longer necessary since Go 1.22. Change-Id: Iebb21dac44a20ec200567f1d786f105a4ee4999d Reviewed-on: https://go-review.googlesource.com/c/go/+/711640 Reviewed-by: Florian Lehner <lehner.florian86@gmail.com> Auto-Submit: Damien Neil <dneil@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Damien Neil <dneil@google.com> Auto-Submit: Tobias Klauser <tobias.klauser@gmail.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2025-04-18internal,runtime: use the builtin clearapocelipes
To simplify the code. Change-Id: I023de705504c0b580718eec3c7c563b6cf2c8184 GitHub-Last-Rev: 026b32c799b13d0c7ded54f2e61429e6c5ed0aa8 GitHub-Pull-Request: golang/go#73412 Reviewed-on: https://go-review.googlesource.com/c/go/+/666118 Reviewed-by: Keith Randall <khr@golang.org> Reviewed-by: Keith Randall <khr@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Auto-Submit: Keith Randall <khr@golang.org> Reviewed-by: Michael Pratt <mpratt@google.com>
2024-09-26internal/zstd: optimize skipFrame by using io.CopyNaimuz
Replaced the manual byte skipping logic with io.CopyN to improve performance and reduce memory allocation. This change simplifies the code by directly discarding the bytes read, enhancing readability and efficiency. Change-Id: Id11496d072fb554c394947d08e63616ca48ecab4 GitHub-Last-Rev: dc5f836cc76ea1355effa49a665b9091a888541b GitHub-Pull-Request: golang/go#69619 Reviewed-on: https://go-review.googlesource.com/c/go/+/615716 Auto-Submit: Ian Lance Taylor <iant@google.com> Reviewed-by: Ian Lance Taylor <iant@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Carlos Amedee <carlos@golang.org>
2024-04-02internal/zstd: optimize window reset using make for buffer allocationaimuz
Optimize the zstd window's reset function to use `make` for buffer allocation when needed. This ensures efficient memory usage by conditionally reallocating the buffer based on its capacity, improving performance in scenarios where the existing buffer is insufficient. ``` goos: darwin goarch: amd64 pkg: internal/zstd cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz │ old.out │ new.out │ │ sec/op │ sec/op vs base │ Large-12 7.299m ± 5% 7.235m ± 1% ~ (p=0.353 n=10) │ old.out │ new.out │ │ B/s │ B/s vs base │ Large-12 35.56Mi ± 4% 35.87Mi ± 1% ~ (p=0.353 n=10) │ old.out │ new.out │ │ B/op │ B/op vs base │ Large-12 68.86Ki ± 4% 15.87Ki ± 1% -76.95% (p=0.000 n=10) │ old.out │ new.out │ │ allocs/op │ allocs/op vs base │ Large-12 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹ ¹ all samples are equal ``` Change-Id: Icd9c278b42ea149107a9861d006f40129c59831b GitHub-Last-Rev: c00344cca415342eb563c62e325f1c2f7f28327d GitHub-Pull-Request: golang/go#64221 Reviewed-on: https://go-review.googlesource.com/c/go/+/543375 Reviewed-by: qiulaidongfeng <2645477756@qq.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Ian Lance Taylor <iant@google.com> Auto-Submit: Ian Lance Taylor <iant@google.com>
2024-01-30internal/zstd: avoid panic when windowSize is negativeaimuz
This change fixes an edge case in the zstd decompressor where an int conversion could result in a negative window size. Fixes #63979 For #62513 Change-Id: Ie714bf8fb51fa509b310deb8bd2c96bd87b52852 GitHub-Last-Rev: ab0be6578247da896f16d85e102b81994b8ee5c4 GitHub-Pull-Request: golang/go#63980 Reviewed-on: https://go-review.googlesource.com/c/go/+/540415 Reviewed-by: Cherry Mui <cherryyz@google.com> Run-TryBot: M Zhuo <mengzhuo1203@gmail.com> Reviewed-by: Bryan Mills <bcmills@google.com> Reviewed-by: M Zhuo <mengzhuo1203@gmail.com> TryBot-Result: Gopher Robot <gobot@golang.org>
2023-11-17internal/zstd: s/thees/these/Jes Cok
Change-Id: I0e3dc4aadb7e8082c26653979a5e6caf5172f9df GitHub-Last-Rev: 000a903ab5d529cf434366fe6d3155221a3e250d GitHub-Pull-Request: golang/go#64117 Reviewed-on: https://go-review.googlesource.com/c/go/+/542195 Reviewed-by: Mauri de Souza Meneguzzo <mauri870@gmail.com> Reviewed-by: Than McIntosh <thanm@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Auto-Submit: Bryan Mills <bcmills@google.com> Run-TryBot: Jes Cok <xigua67damn@gmail.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Bryan Mills <bcmills@google.com>
2023-11-17internal/zstd: fix seek offset bounds check in skipFrameaimuz
This change enhances the zstd Reader's skipFrame function to validate the new offset when skipping frames in a seekable stream, preventing invalid offsets that could occur previously. A set of "bad" test strings has been added to fuzz_test.go to extend the robustness checks against potential decompression panics. Additionally, a new test named TestReaderBad is introduced in zstd_test.go to verify proper error handling with corrupted input strings. The BenchmarkLarge function has also been refactored for clarity, removing unnecessary timer stops and resets. Updates #63824 Change-Id: Iccd248756ad6348afa1395c7799350d07402868a GitHub-Last-Rev: 63055b91e9413491fe8039ea42d55b823c89ec15 GitHub-Pull-Request: golang/go#64056 Reviewed-on: https://go-review.googlesource.com/c/go/+/541220 Reviewed-by: Bryan Mills <bcmills@google.com> Reviewed-by: David Chase <drchase@google.com> Reviewed-by: Klaus Post <klauspost@gmail.com> Auto-Submit: Bryan Mills <bcmills@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2023-11-09internal/zstd: use dynamic path resolution for xxhsum in FuzzXXHashJes Cok
Updates #64000 Change-Id: I71fb80128d7e2a1f82322cbf04f74db01dcc631b GitHub-Last-Rev: 74135946664b1b65a0de0a9228de998429dc74f9 GitHub-Pull-Request: golang/go#64003 Reviewed-on: https://go-review.googlesource.com/c/go/+/539938 Run-TryBot: Jes Cok <xigua67damn@gmail.com> TryBot-Result: Gopher Robot <gobot@golang.org> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Bryan Mills <bcmills@google.com> Reviewed-by: Heschi Kreinick <heschi@google.com>
2023-11-09internal/zstd: avoid panic when the regenerated size is too smallaimuz
Description in accordance with RFC 8878 3.1.1.3.1.6. The decompressed size of each stream is equal to (Regenerated_Size+3)/4, except for the last stream, which may be up to 3 bytes smaller, to reach a total decompressed size as specified in Regenerated_Size. Fixes #63824 Change-Id: I5a8b482a995272aa2028a81a4db86c21b1770432 GitHub-Last-Rev: 76a70756bc005a8fcd33b4b6a50fd6c3bf827fb6 GitHub-Pull-Request: golang/go#63959 Reviewed-on: https://go-review.googlesource.com/c/go/+/540055 Auto-Submit: Bryan Mills <bcmills@google.com> Reviewed-by: Matthew Dempsky <mdempsky@google.com> Reviewed-by: Bryan Mills <bcmills@google.com> Reviewed-by: Klaus Post <klauspost@gmail.com> Run-TryBot: Jes Cok <xigua67damn@gmail.com> TryBot-Result: Gopher Robot <gobot@golang.org>
2023-11-08internal/zstd: use dynamic path resolution for zstd in testsaimuz
Abstract the hardcoded '/usr/bin/zstd' paths in fuzz and unit tests to support systems where zstd may be installed at different locations. The `findZstd` function uses `exec.LookPath` to locate the binary, enhancing test portability. Fixes #64000 Change-Id: I0ebe5bbcf3ddc6fccf176c13639ca9d855bcab87 GitHub-Last-Rev: c4dfe1139bdc2f4f3200f80b314a02b5df5cd995 GitHub-Pull-Request: golang/go#64002 Reviewed-on: https://go-review.googlesource.com/c/go/+/540522 Reviewed-by: Klaus Post <klauspost@gmail.com> Reviewed-by: Heschi Kreinick <heschi@google.com> Auto-Submit: Bryan Mills <bcmills@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Bryan Mills <bcmills@google.com>
2023-09-29internal/zstd: reset reader bufferAlexander Yastrebov
Reset r.buffer on Reset to avoid subsequent Read calls observing previously decoded data. For #62513 Change-Id: Icb65e76b5c5c0af32b36ec3a5999dca86407cbc8 GitHub-Last-Rev: 99c0a6fa72ad67cba5d29593fd3b28d14ddce4a4 GitHub-Pull-Request: golang/go#63288 Reviewed-on: https://go-review.googlesource.com/c/go/+/531735 Auto-Submit: Ian Lance Taylor <iant@google.com> Reviewed-by: Ian Lance Taylor <iant@google.com> Reviewed-by: Than McIntosh <thanm@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2023-09-28internal/zstd: handle match extending past windowAlexander Yastrebov
For #62513 Change-Id: I59c24b254d5073140811b41497eabb91fb0046e9 GitHub-Last-Rev: 4dd16fcfa813da2b612d5753e11c163476d44b53 GitHub-Pull-Request: golang/go#63248 Reviewed-on: https://go-review.googlesource.com/c/go/+/531255 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Auto-Submit: Ian Lance Taylor <iant@google.com> Reviewed-by: Ian Lance Taylor <iant@google.com> Reviewed-by: Than McIntosh <thanm@google.com>
2023-09-28internal/zstd: allow zero dictionary idAlexander Yastrebov
A value of 0 has same meaning as no Dictionary_ID, in which case the frame may or may not need a dictionary to be decoded, and the ID of such a dictionary is not specified. See https://github.com/facebook/zstd/issues/2172 For #62513 Change-Id: If0eafcbc5d2188576f0cb687234e30c9eb4037a6 GitHub-Last-Rev: 9cf12dcf194a90367a74b808bbe464815f71f42a GitHub-Pull-Request: golang/go#63268 Reviewed-on: https://go-review.googlesource.com/c/go/+/531515 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Ian Lance Taylor <iant@google.com> Reviewed-by: Than McIntosh <thanm@google.com> Auto-Submit: Ian Lance Taylor <iant@google.com>
2023-09-28internal/zstd: allow stream consisting of skippable framesAlexander Yastrebov
For #62513 Change-Id: I2557aed5ae106ea4684bb599cce740e9da9df780 GitHub-Last-Rev: 2b7ddc6c09a7e77874ed9aefc47fbc445d2579ec GitHub-Pull-Request: golang/go#63251 Reviewed-on: https://go-review.googlesource.com/c/go/+/531295 Reviewed-by: Ian Lance Taylor <iant@google.com> Auto-Submit: Ian Lance Taylor <iant@golang.org> Reviewed-by: Bryan Mills <bcmills@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2023-09-27internal/zstd: configure window size for single segment framesAlexander Yastrebov
Set window size to frame content size when single segment flag is set. For #62513 Change-Id: I2a60c33123aca4f6a631e6d625f4582ff31a63cb GitHub-Last-Rev: 9bafe01e45aad6a9f22abca08b25b2b8d9107040 GitHub-Pull-Request: golang/go#63224 Reviewed-on: https://go-review.googlesource.com/c/go/+/531075 Auto-Submit: Ian Lance Taylor <iant@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Ian Lance Taylor <iant@google.com> Reviewed-by: Bryan Mills <bcmills@google.com>
2023-09-27internal/zstd: allow empty compressed blocksAlexander Yastrebov
For #62513 Change-Id: I295e72f71165665b8ea999e68a5586fa785b546d GitHub-Last-Rev: 902e952d88dba505f87393fd3c97c433ae291709 GitHub-Pull-Request: golang/go#63252 Reviewed-on: https://go-review.googlesource.com/c/go/+/531217 Auto-Submit: Ian Lance Taylor <iant@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Bryan Mills <bcmills@google.com> Reviewed-by: Ian Lance Taylor <iant@google.com>
2023-09-20internal/zstd: use circular buffer for backreference windowAlexander Yastrebov
Use circular buffer to reduce data movements. The CL also increases size of bigData to make changes of benchmark results apparent. goos: linux goarch: amd64 pkg: internal/zstd │ /tmp/BenchmarkLarge.old │ /tmp/BenchmarkLarge.new │ │ sec/op │ sec/op vs base │ Large-8 12.672m ± 1% 9.521m ± 0% -24.87% (p=0.000 n=10) │ /tmp/BenchmarkLarge.old │ /tmp/BenchmarkLarge.new │ │ B/s │ B/s vs base │ Large-8 13.43Mi ± 1% 17.88Mi ± 0% +33.08% (p=0.000 n=10) │ /tmp/BenchmarkLarge.old │ /tmp/BenchmarkLarge.new │ │ B/op │ B/op vs base │ Large-8 58.23Ki ± 5% 41.72Ki ± 1% -28.35% (p=0.000 n=10) │ /tmp/BenchmarkLarge.old │ /tmp/BenchmarkLarge.new │ │ allocs/op │ allocs/op vs base │ Large-8 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) Change-Id: Ic03fabfc575c5e6d18bcd5ba1c845aa502c12497 GitHub-Last-Rev: 16cb1e13ff61f0fc8c9209cb034f31ec6c37f596 GitHub-Pull-Request: golang/go#62625 Reviewed-on: https://go-review.googlesource.com/c/go/+/528318 Reviewed-by: Ian Lance Taylor <iant@google.com> Reviewed-by: Matthew Dempsky <mdempsky@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Auto-Submit: Ian Lance Taylor <iant@google.com>
2023-09-13internal/zstd: fix window resizingAlexander Yastrebov
Incorrect window resizing led to checksum error and invalid result. To demonstrate the problem bigData must be a bit bigger, 3x is enough. This change fixes window resizing, increases bigData size and decouples TestLargeXXHash from bigData because it uses hardcoded hash value. Change-Id: I50f74315b083f42e1ccd7ab2093e084f44631bb6 GitHub-Last-Rev: dbc90ba7a5f24db198cc3eab1c38aad665e41d06 GitHub-Pull-Request: golang/go#62543 Reviewed-on: https://go-review.googlesource.com/c/go/+/527115 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Heschi Kreinick <heschi@google.com> Auto-Submit: Ian Lance Taylor <iant@google.com> Reviewed-by: Ian Lance Taylor <iant@google.com>
2023-04-18internal/zstd: new internal package for zstd decompressionIan Lance Taylor
This package only does zstd decompression, which is starting to be used for ELF debug sections. If we need zstd compression we should use github.com/klauspost/compress/zstd. But for now that is a very large package to vendor into the standard library. For #55107 Change-Id: I60ede735357d491be653477ed419cf5f2f0d3f71 Reviewed-on: https://go-review.googlesource.com/c/go/+/473356 Reviewed-by: Ian Lance Taylor <iant@google.com> Run-TryBot: Ian Lance Taylor <iant@google.com> Run-TryBot: Ian Lance Taylor <iant@golang.org> Reviewed-by: Joseph Tsai <joetsai@digital-static.net> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Bryan Mills <bcmills@google.com> Auto-Submit: Ian Lance Taylor <iant@google.com>
generated by cgit v1.3 (git 2.53.0) at 2026-04-15 23:22:05 +0000