aboutsummaryrefslogtreecommitdiff
path: root/src/unicode
AgeCommit message (Collapse)Author
2026-01-22all: update to Unicode 17Russ Cox
Process: - Disable unicode TestProperties. - Run UNICODE_VERSION=17.0.0 go -C ../src/golang.org/x/text generate - Run go generate strconv - Update unicode TestProperties by hand to add missing test cases. This will break x/text using the main repo until the corresponding x/text CL is submitted. It should not break anything else. For #77266. Change-Id: Ia3ffb5a81e212b991714eef70eebfca98a0c8aab Reviewed-on: https://go-review.googlesource.com/c/go/+/737420 Reviewed-by: Rob Pike <r@golang.org> Reviewed-by: Roland Shoemaker <roland@golang.org> TryBot-Bypass: Russ Cox <rsc@golang.org>
2025-09-03unicode/utf8: make DecodeRune{,InString} inlineableJulien Cretel
This change makes the fast path for ASCII characters inlineable in DecodeRune and DecodeRuneInString and removes most instances of manual inlining at call sites. Here are some benchmark results (no change to allocations): goos: darwin goarch: amd64 pkg: unicode/utf8 cpu: Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz │ old │ new │ │ sec/op │ sec/op vs base │ DecodeASCIIRune-8 2.4545n ± 2% 0.6253n ± 2% -74.52% (p=0.000 n=20) DecodeJapaneseRune-8 3.988n ± 1% 4.023n ± 1% +0.86% (p=0.050 n=20) DecodeASCIIRuneInString-8 2.4675n ± 1% 0.6264n ± 2% -74.61% (p=0.000 n=20) DecodeJapaneseRuneInString-8 3.992n ± 1% 4.001n ± 1% ~ (p=0.625 n=20) geomean 3.134n 1.585n -49.43% Note: when #61502 gets resolved, DecodeRune and DecodeRuneInString should be reverted to their idiomatic implementations. Fixes #31666 Updates #48195 Change-Id: I4be25c4f52417dc28b3a7bd72f1b04018470f39d GitHub-Last-Rev: 2e352a0045027e059be79cdb60241b5cf35fec71 GitHub-Pull-Request: golang/go#75181 Reviewed-on: https://go-review.googlesource.com/c/go/+/699675 Reviewed-by: Sean Liao <sean@liao.dev> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Cherry Mui <cherryyz@google.com> Reviewed-by: Michael Pratt <mpratt@google.com>
2025-07-24unicode/utf8: skip ahead during ascii runs in Valid/ValidStringKeith Randall
When we see an ASCII character, we will probably see many. Grab & check increasingly large chunks of the string for ASCII-only-ness. Also redo some of the non-ASCII code to make it more optimizer friendly. goos: linux goarch: amd64 pkg: unicode/utf8 cpu: 12th Gen Intel(R) Core(TM) i7-12700 │ base │ exp │ │ sec/op │ sec/op vs base │ ValidTenASCIIChars-20 3.596n ± 3% 2.522n ± 1% -29.86% (p=0.000 n=10) Valid100KASCIIChars-20 6.094µ ± 2% 2.115µ ± 1% -65.29% (p=0.000 n=10) ValidTenJapaneseChars-20 21.02n ± 0% 18.61n ± 2% -11.44% (p=0.000 n=10) ValidLongMostlyASCII-20 51.774µ ± 0% 3.836µ ± 1% -92.59% (p=0.000 n=10) ValidLongJapanese-20 102.40µ ± 1% 50.95µ ± 1% -50.24% (p=0.000 n=10) ValidStringTenASCIIChars-20 2.640n ± 3% 2.526n ± 1% -4.34% (p=0.000 n=10) ValidString100KASCIIChars-20 5.585µ ± 7% 2.118µ ± 1% -62.07% (p=0.000 n=10) ValidStringTenJapaneseChars-20 21.29n ± 2% 18.67n ± 1% -12.31% (p=0.000 n=10) ValidStringLongMostlyASCII-20 52.431µ ± 1% 3.841µ ± 0% -92.67% (p=0.000 n=10) ValidStringLongJapanese-20 102.66µ ± 1% 50.90µ ± 1% -50.42% (p=0.000 n=10) geomean 1.152µ 454.8n -60.53% This is an attempt to see if we can get enough performance that we don't need to consider assembly like that in CL 681695. Change-Id: I8250feb797a6b4e7d335c23929f6e3acc8b24840 Reviewed-on: https://go-review.googlesource.com/c/go/+/682778 Reviewed-by: Cuong Manh Le <cuong.manhle.vn@gmail.com> Reviewed-by: Michael Knyszek <mknyszek@google.com> Auto-Submit: Keith Randall <khr@golang.org> Reviewed-by: Keith Randall <khr@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2025-07-22all: go fmtMichael Pratt
Change-Id: I6a6a636c341d4ba3518be7f6806093bbdff11c88 Reviewed-on: https://go-review.googlesource.com/c/go/+/689535 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Auto-Submit: Michael Pratt <mpratt@google.com> Reviewed-by: Michael Knyszek <mknyszek@google.com>
2025-04-18unicode: add CategoryAliases, Cn, LCRuss Cox
CategoryAliases is for regexp to use, for things like \p{Letter} as an alias for \p{L}. Cn and LC are special-case categories that were never implemented but should have been. These changes were generated by the updated generator in CL 641395. Fixes #70780. Change-Id: Ibba20ff76191c8ae9631ac5ba19965790fe0cc81 Reviewed-on: https://go-review.googlesource.com/c/go/+/641376 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Alan Donovan <adonovan@google.com>
2025-02-25unicode/utf8: use builtin max function to simplify codeJes Cok
Change-Id: I6a73b645d074baaa4d09480bdf4192816a8c2450 GitHub-Last-Rev: 202d498eb019c18b9ba30bccc2cb169c9eb79366 GitHub-Pull-Request: golang/go#71945 Reviewed-on: https://go-review.googlesource.com/c/go/+/652177 Auto-Submit: Keith Randall <khr@golang.org> Auto-Submit: Ian Lance Taylor <iant@google.com> Reviewed-by: Keith Randall <khr@golang.org> Reviewed-by: Keith Randall <khr@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Ian Lance Taylor <iant@google.com>
2025-02-06unicode/utf8: remove init from utf8_testLuka Krmpotic
TestConstants and init test the same thing, remove init, it does not exist in utf16_test.go either. Fixes #71579 Change-Id: Ie0afd640bebde822733b6eac0bf98a17872f4e5f GitHub-Last-Rev: d7224c18376e00038261279abdfa954abc3a8303 GitHub-Pull-Request: golang/go#71582 Reviewed-on: https://go-review.googlesource.com/c/go/+/647335 Commit-Queue: Ian Lance Taylor <iant@golang.org> Reviewed-by: Keith Randall <khr@golang.org> Reviewed-by: Keith Randall <khr@google.com> Reviewed-by: qiu laidongfeng2 <2645477756@qq.com> Auto-Submit: Ian Lance Taylor <iant@golang.org> Reviewed-by: Ian Lance Taylor <iant@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-09-15unicode/utf8: speedup RuneCountCuong Manh Le
CL 612617 did speedup RuneCountInString, thus we can now use it to speedup RuneCount, too. name old time/op new time/op delta RuneCountTenASCIIChars-8 8.69ns ± 1% 3.59ns ± 2% -58.66% (p=0.000 n=9+9) RuneCountTenJapaneseChars-8 49.8ns ± 2% 40.9ns ± 0% -17.94% (p=0.000 n=10+8) Change-Id: I311750c00efc79af35fb0ca3b482a5d94e0a7977 Reviewed-on: https://go-review.googlesource.com/c/go/+/612955 Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Auto-Submit: Cuong Manh Le <cuong.manhle.vn@gmail.com> Reviewed-by: Tim King <taking@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-09-15unicode/utf8: add test that RuneCount does zero allocationsCuong Manh Le
See disccusion in CL 612955. Change-Id: I2de582321648f1798929ffb80d2f087e41146ead Reviewed-on: https://go-review.googlesource.com/c/go/+/613315 Commit-Queue: Ian Lance Taylor <iant@google.com> Reviewed-by: Tim King <taking@google.com> Reviewed-by: Ian Lance Taylor <iant@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Auto-Submit: Cuong Manh Le <cuong.manhle.vn@gmail.com>
2024-09-12unicode/utf8: use range loop in RuneCountInStringCuong Manh Le
CL 28490 speeded up non-ASCII rune decoding, and ASCII rune is also decoded faster now. Benchmark using: perflock -governor 70% go test -run=NONE -bench=BenchmarkRuneCountInString -count=10 Result: name old time/op new time/op delta RuneCountInStringTenASCIIChars-8 10.2ns ± 0% 7.1ns ± 1% -30.53% (p=0.000 n=8+9) RuneCountInStringTenJapaneseChars-8 49.3ns ± 2% 38.5ns ± 2% -21.84% (p=0.000 n=8+8) Fixes #13162 Change-Id: Ifb01f3799c5c93e7f7c7af13a95becfde85ae807 Reviewed-on: https://go-review.googlesource.com/c/go/+/612617 Reviewed-by: Tim King <taking@google.com> Reviewed-by: Ian Lance Taylor <iant@google.com> Auto-Submit: Cuong Manh Le <cuong.manhle.vn@gmail.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Auto-Submit: Tim King <taking@google.com>
2024-09-03all: omit unnecessary 0 in slice expressionnlwkobe30
All changes are related to the code, except for the comments in src/regexp/syntax/parse.go and src/slices/slices.go. Change-Id: I73c5d3c54099749b62210aa7f3182c5eb84bb6a6 GitHub-Last-Rev: 794aa9b0539811d00e1cd42be1e8d9fe9afe0281 GitHub-Pull-Request: golang/go#69170 Reviewed-on: https://go-review.googlesource.com/c/go/+/609678 Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Ian Lance Taylor <iant@google.com> Auto-Submit: Ian Lance Taylor <iant@google.com>
2024-09-03unicode: improve SimpleFold performance by 2x for non-foldable code pointsCharlie Vieth
Change SimpleFold to search the CaseRanges table only once when no folding is specified for the rune (previously up to two searches could be performed). This improves performance by 2x for runes that have no folds or are already upper case. As a side effect this improves the performance of To by roughly ~15% goos: darwin goarch: arm64 pkg: unicode cpu: Apple M1 Max │ base.10.txt │ new.10.txt │ │ sec/op │ sec/op vs base │ ToUpper-10 11.860n ± 1% 9.731n ± 1% -17.95% (p=0.000 n=10) ToLower-10 12.31n ± 1% 10.34n ± 1% -16.00% (p=0.000 n=10) SimpleFold/Upper-10 19.16n ± 0% 15.98n ± 1% -16.64% (p=0.000 n=10) SimpleFold/Lower-10 32.41n ± 1% 17.09n ± 1% -47.27% (p=0.000 n=10) SimpleFold/Fold-10 8.884n ± 4% 8.856n ± 8% ~ (p=0.700 n=10) SimpleFold/NoFold-10 30.87n ± 0% 15.49n ± 3% -49.84% (p=0.000 n=10) geomean 17.09n 12.47n -26.99% Change-Id: I6e5c7554106842955aadeef7b266c4c7944d3a97 Reviewed-on: https://go-review.googlesource.com/c/go/+/454958 Reviewed-by: Ian Lance Taylor <iant@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Auto-Submit: Ian Lance Taylor <iant@google.com>
2024-07-24bytes,strings,unicode/utf16: use slices to clean up testsapocelipes
Replace reflect.DeepEqual with slices.Equal, which is much faster. Remove some redundant helper functions. Change-Id: I51b32a3d0c3fc5ad0d3b6ff0dd03f39c507e5762 GitHub-Last-Rev: e21f46d4a026b6bf6e8d912dfb8d361a20a4e779 GitHub-Pull-Request: golang/go#67609 Reviewed-on: https://go-review.googlesource.com/c/go/+/587937 Reviewed-by: Ian Lance Taylor <iant@google.com> Auto-Submit: Ian Lance Taylor <iant@google.com> Commit-Queue: Ian Lance Taylor <iant@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Robert Griesemer <gri@google.com>
2024-07-22unicode/utf8: AppendRune and EncodeRune performance improvementDiego Augusto Molina
- Prefer the evaluation of the valid higher byte-width runes branches over the one for invalid ones - Avoid the evaluation of the bytes of the RuneError constant, and instead hard code its byte values - EncodeRune only: inline for fast handling of ASCII goos: linux goarch: amd64 pkg: unicode/utf8 cpu: 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz │ baseline.test-append.txt │ append-rune-invalid-case-last.test.txt │ │ sec/op │ sec/op vs base │ AppendASCIIRune-8 0.2135n ± 0% 0.2135n ± 0% ~ (p=0.578 n=20) AppendSpanishRune-8 1.645n ± 1% 1.509n ± 2% -8.30% (p=0.000 n=20) AppendJapaneseRune-8 2.196n ± 1% 2.004n ± 1% -8.74% (p=0.000 n=20) AppendMaxRune-8 2.670n ± 1% 2.349n ± 3% -12.01% (p=0.000 n=20) AppendInvalidRuneMaxPlusOne-8 2.214n ± 2% 1.798n ± 3% -18.77% (p=0.000 n=20) AppendInvalidRuneSurrogate-8 2.258n ± 1% 1.793n ± 2% -20.59% (p=0.000 n=20) AppendInvalidRuneNegative-8 2.171n ± 2% 1.767n ± 2% -18.61% (p=0.000 n=20) geomean 1.559n 1.361n -12.69% goos: linux goarch: amd64 pkg: unicode/utf8 cpu: 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz │ baseline.test-encode.txt │ encode-rune-invalid-last.txt │ │ sec/op │ sec/op vs base │ EncodeASCIIRune-8 1.0950n ± 1% 0.2140n ± 0% -80.46% (p=0.000 n=20) EncodeSpanishRune-8 1.499n ± 0% 1.414n ± 2% -5.64% (p=0.000 n=20) EncodeJapaneseRune-8 1.960n ± 2% 1.716n ± 4% -12.43% (p=0.000 n=20) EncodeMaxRune-8 2.145n ± 2% 2.227n ± 1% +3.78% (p=0.000 n=20) EncodeInvalidRuneMaxPlusOne-8 1.955n ± 2% 1.802n ± 2% -7.80% (p=0.000 n=20) EncodeInvalidRuneSurrogate-8 1.946n ± 3% 1.777n ± 2% -8.68% (p=0.000 n=20) EncodeInvalidRuneNegative-8 1.968n ± 2% 1.766n ± 2% -10.29% (p=0.000 n=20) geomean 1.757n 1.308n -25.57% Fixes #68131 Change-Id: Ibcafa75d63cca07a2e78cd06f6f1e382cb8c716e Reviewed-on: https://go-review.googlesource.com/c/go/+/594115 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Auto-Submit: Ian Lance Taylor <iant@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Ian Lance Taylor <iant@google.com>
2024-03-08unicode/utf8: update doc for RuneLenJes Cok
As CL 569755 did, for consistency, this CL slightly improves the documentation for RuneLen. Change-Id: Ic9776648baf2809af36cd16a94d1313938bb0e52 Reviewed-on: https://go-review.googlesource.com/c/go/+/569816 Auto-Submit: Ian Lance Taylor <iant@google.com> Reviewed-by: Michael Knyszek <mknyszek@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Ian Lance Taylor <iant@google.com> Commit-Queue: Ian Lance Taylor <iant@google.com>
2024-03-07unicode/utf16: add func RuneLenJes Cok
This CL adds func RuneLen, while here, also uses RuneLen to simplify code in Encode. Fixes #44940 Change-Id: Ifd3b537f69880dfd32a69a6733d8d3c2b5d4ecba Reviewed-on: https://go-review.googlesource.com/c/go/+/569755 Reviewed-by: Ian Lance Taylor <iant@google.com> Reviewed-by: Michael Knyszek <mknyszek@google.com> Commit-Queue: Ian Lance Taylor <iant@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Auto-Submit: Ian Lance Taylor <iant@google.com>
2023-11-06unicode: add available godoc linkcui fliter
Change-Id: I2273274249f05b0492950c27dc5a654422cefc79 Reviewed-on: https://go-review.googlesource.com/c/go/+/539856 TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Rob Pike <r@golang.org> Run-TryBot: shuang cui <imcusg@gmail.com> Reviewed-by: Cherry Mui <cherryyz@google.com> Reviewed-by: Heschi Kreinick <heschi@google.com>
2023-09-09all: calculate the median uniformlyJes Cok
Like sort.Search, use "h := int(uint(i+j) >> 1)" style code to calculate the median. Change-Id: Ifb1a19dde1c6ed6c1654bc642fc9565a8b6c5fc4 GitHub-Last-Rev: e2213b738832f1674948d6507f40e2c0b98cb972 GitHub-Pull-Request: golang/go#62503 Reviewed-on: https://go-review.googlesource.com/c/go/+/526496 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Auto-Submit: Ian Lance Taylor <iant@google.com> Reviewed-by: Ian Lance Taylor <iant@google.com> Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2023-02-06all: upgrade Unicode from 13.0.0 to 15.0.0weebney
Update unicode/tables.go to reflect changes in the Unicode Standard up to Unicode 15.0.0, released 13 Sept 2022. In order to accommodate this update, strconv/isPrint has been updated to reflect changes in printable characters. Also changed is template/exec_test.go for both text and html packages- in the test "TestJSEscaping", rune U+FDFF was used as a placeholder for an unprintable character. This codepoint was assigned and made printable in Unicode 14.0.0, breaking this test. It has been replaced with the assigned and never-printable U+FFFE to fix the test and provide resiliency in the future. This upgrade bypasses Unicode 14.0.0, but is compatible. Updates https://github.com/golang/go/issues/48621 Fixes https://github.com/golang/go/issues/55079 Change-Id: I40efd097eb746db0727ebf7437280916d1242e47 GitHub-Last-Rev: c8885cab7a0c23632e1e5a433b1e8d5634a45a30 GitHub-Pull-Request: golang/go#57265 Reviewed-on: https://go-review.googlesource.com/c/go/+/456837 Reviewed-by: Robert Griesemer <gri@google.com> Run-TryBot: Rob Pike <r@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Robert Griesemer <gri@google.com> Reviewed-by: Cherry Mui <cherryyz@google.com> Reviewed-by: Rob Pike <r@golang.org>
2023-01-23utf16: reduce utf16.Decode allocationsqmuntal
This CL avoids allocating in utf16.Decode for code point sequences with less than 64 elements. It does so by splitting the function in two, one that can be inlined that preallocates a buffer and the other that does the heavy-lifting. The mid-stack inliner will allocate the buffer in the caller stack, and in many cases this will be enough to avoid the allocation. unicode/utf16 benchmarks: name old time/op new time/op delta DecodeValidASCII-12 60.1ns ± 3% 16.0ns ±20% -73.40% (p=0.000 n=8+10) DecodeValidJapaneseChars-12 61.3ns ±10% 14.9ns ±39% -75.71% (p=0.000 n=10+10) name old alloc/op new alloc/op delta DecodeValidASCII-12 48.0B ± 0% 0.0B -100.00% (p=0.000 n=10+10) DecodeValidJapaneseChars-12 48.0B ± 0% 0.0B -100.00% (p=0.000 n=10+10) name old allocs/op new allocs/op delta DecodeValidASCII-12 1.00 ± 0% 0.00 -100.00% (p=0.000 n=10+10) DecodeValidJapaneseChars-12 1.00 ± 0% 0.00 -100.00% (p=0.000 n=10+10) I've also benchmarked os.File.ReadDir with this change applied to demonstrate that it does make a difference in the caller site, in this case via syscall.UTF16ToString: name old time/op new time/op delta ReadDir-12 592µs ± 8% 620µs ±16% ~ (p=0.280 n=10+10) name old alloc/op new alloc/op delta ReadDir-12 30.4kB ± 0% 22.4kB ± 0% -26.10% (p=0.000 n=8+10) name old allocs/op new allocs/op delta ReadDir-12 402 ± 0% 272 ± 0% -32.34% (p=0.000 n=10+10) Change-Id: I65cf5caa3fd3b3a466c0ed837a50a96e975bbe6b Reviewed-on: https://go-review.googlesource.com/c/go/+/453415 Reviewed-by: Damien Neil <dneil@google.com> Reviewed-by: Cherry Mui <cherryyz@google.com> Reviewed-by: Alex Brainman <alex.brainman@gmail.com> TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Quim Muntal <quimmuntal@gmail.com>
2022-09-07unicode/utf8: use strings.Buildercuiweixie
Change-Id: I88b55f61eccb5764cac2a9397fd99a62f8735a9a Reviewed-on: https://go-review.googlesource.com/c/go/+/428281 Reviewed-by: Ian Lance Taylor <iant@google.com> Auto-Submit: Ian Lance Taylor <iant@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Ian Lance Taylor <iant@google.com> Reviewed-by: Benny Siegert <bsiegert@gmail.com>
2022-08-19unicode/utf16: add AppendRuneqmuntal
AppendRune appends the UTF-16 encoding of a rune to a []uint16. BenchmarkEncodeValidASCII-12 24.61ns 16B 1allocs BenchmarkEncodeValidJapaneseChars-12 18.79ns 8B 1allocs BenchmarkAppendRuneValidASCII-12 6.826ns 0B 0allocs BenchmarkAppendRuneValidJapaneseChars-12 3.547ns 0B 0allocs The ASCII case is written to be inlineable. Fixes #51896 Change-Id: I593b1029f603297ef6e80e036f2fee2a0938d38d Reviewed-on: https://go-review.googlesource.com/c/go/+/409054 TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@google.com> Run-TryBot: Ian Lance Taylor <iant@google.com> Auto-Submit: Ian Lance Taylor <iant@google.com> Reviewed-by: Joedian Reid <joedian@golang.org>
2022-04-11all: gofmt main repoRuss Cox
[This CL is part of a sequence implementing the proposal #51082. The design doc is at https://go.dev/s/godocfmt-design.] Run the updated gofmt, which reformats doc comments, on the main repository. Vendored files are excluded. For #51082. Change-Id: I7332f099b60f716295fb34719c98c04eb1a85407 Reviewed-on: https://go-review.googlesource.com/c/go/+/384268 Reviewed-by: Jonathan Amsterdam <jba@google.com> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2022-04-01all: remove trailing blank doc comment linesRuss Cox
A future change to gofmt will rewrite // Doc comment. // func f() to // Doc comment. func f() Apply that change preemptively to all doc comments. For #51082. Change-Id: I4023e16cfb0729b64a8590f071cd92f17343081d Reviewed-on: https://go-review.googlesource.com/c/go/+/384259 Trust: Russ Cox <rsc@golang.org> Run-TryBot: Russ Cox <rsc@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org>
2022-03-02unicode/utf8: optimize Valid to parity with ValidStringAlan Donovan
The benchmarks added in this change revealed that ValidString runs ~17% faster than Valid([]byte) on the ASCII prefix of the input. Inspection of the assembly revealed that the code generated for p[8:] required recomputing the slice capacity to handle the cap=0 special case, which added an ADD -8 instruction. By making len=cap, the capacity becomes a common subexpression with the length, saving the ADD instruction. (Thanks to khr for the tip.) Incidentally, I tried a number of other optimizations but was unable to make consistent gains across all benchmarks. The most promising was to retain the bitmask of non-ASCII bytes from the fast loop; the slow loop would shift it, and when it becomes zero, return to the fast loop. This made the MostlyASCII benchmark 4x faster, but made the other cases slower by up to 10%. cpu: Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz benchmark old ns/op new ns/op delta BenchmarkValidTenASCIIChars-16 4.09 4.06 -0.85% BenchmarkValid100KASCIIChars-16 9325 7747 -16.92% BenchmarkValidTenJapaneseChars-16 27.0 27.2 +0.85% BenchmarkValidLongMostlyASCII-16 57277 58361 +1.89% BenchmarkValidLongJapanese-16 94002 93131 -0.93% BenchmarkValidStringTenASCIIChars-16 4.15 4.07 -1.74% BenchmarkValidString100KASCIIChars-16 7980 8019 +0.49% BenchmarkValidStringTenJapaneseChars-16 26.0 25.9 -0.38% BenchmarkValidStringLongMostlyASCII-16 58550 58006 -0.93% BenchmarkValidStringLongJapanese-16 97964 100038 +2.12% Change-Id: Ic9d585dedd9af83c27dd791ecd805150ac949f15 Reviewed-on: https://go-review.googlesource.com/c/go/+/375594 Reviewed-by: Keith Randall <khr@golang.org> Run-TryBot: Keith Randall <khr@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Trust: Alex Rakoczy <alex@golang.org>
2021-11-05unicode/utf8: add AppendRune Examplejiahua wang
Also, correct TestAppendRune error message. Change-Id: I3ca3ac7051af1ae6d449381b78efa86c2f6be8ac Reviewed-on: https://go-review.googlesource.com/c/go/+/354529 Reviewed-by: Ian Lance Taylor <iant@golang.org> Run-TryBot: Ian Lance Taylor <iant@golang.org> Trust: Robert Findley <rfindley@google.com> Trust: Cherry Mui <cherryyz@google.com> TryBot-Result: Go Bot <gobot@golang.org>
2021-10-08unicode: use IsSpace not IsUpper in IsSpace example testPedro Lopez Mareque
Change-Id: Ie3017e5507f57cbb2ae9c8b737b378cef91fefeb Reviewed-on: https://go-review.googlesource.com/c/go/+/354509 Reviewed-by: Emmanuel Odeke <emmanuel@orijtech.com> Reviewed-by: Ian Lance Taylor <iant@golang.org> Run-TryBot: Emmanuel Odeke <emmanuel@orijtech.com> TryBot-Result: Go Bot <gobot@golang.org>
2021-10-06unicode: add examples for the Is functionsPedro Lopez Mareque
Change-Id: If4afe33985dc0a758db32564244095190b82e5c4 Reviewed-on: https://go-review.googlesource.com/c/go/+/353691 Trust: Michael Knyszek <mknyszek@google.com> Reviewed-by: Rob Pike <r@golang.org>
2021-08-28unicode/utf8: add AppendRuneJoe Tsai
AppendRune appends the UTF-8 encoding of a rune to a []byte. It is a generally more user friendly than EncodeRune. EncodeASCIIRune-4 2.35ns ± 2% EncodeJapaneseRune-4 4.60ns ± 2% AppendASCIIRune-4 0.30ns ± 3% AppendJapaneseRune-4 4.70ns ± 2% The ASCII case is written to be inlineable. Fixes #47609 Change-Id: If4f71eedffd2bd4ef0d7f960cb55b41c637eec54 Reviewed-on: https://go-review.googlesource.com/c/go/+/345571 Trust: Joe Tsai <joetsai@digital-static.net> Reviewed-by: Rob Pike <r@golang.org> Run-TryBot: Rob Pike <r@golang.org> TryBot-Result: Go Bot <gobot@golang.org>
2021-02-24unicode: correctly handle negative runesDavid Benjamin
Is and isExcludingLatin did not handle negative runes when dispatching to is16. TestNegativeRune covers this along with the existing uint32 casts in IsGraphic, etc. (For tests, I picked the smallest non-Latin-1 code point in each range.) Updates #43254 Change-Id: I17261b91f0d2b5b5125d19219411b45c480df74f Reviewed-on: https://go-review.googlesource.com/c/go/+/280493 Run-TryBot: Rob Pike <r@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Rob Pike <r@golang.org> Trust: Emmanuel Odeke <emmanuel@orijtech.com>
2020-09-19unicode/utf8: document the handling of runes out of range in EncodeRuneAinar Garipov
Document the way EncodeRune currently handles runes which are out of range. Also add an example showing that behaviour. Change-Id: I0f8e7645ae053474ec319085a2bb6d7f73bc137c Reviewed-on: https://go-review.googlesource.com/c/go/+/255998 Reviewed-by: Rob Pike <r@golang.org> Reviewed-by: Giovanni Bajo <rasky@develer.com> Trust: Giovanni Bajo <rasky@develer.com> Run-TryBot: Giovanni Bajo <rasky@develer.com> TryBot-Result: Go Bot <gobot@golang.org>
2020-09-10unicode/utf8: refactor benchmarks for FullRune functioneric fang
BenchmarkFullASCIIRune tests the performance of function utf8.FullRune, which will be inlined in BenchmarkFullASCIIRune. Since the return value of FullRune is not referenced, it will be removed as dead code. This CL makes the FullRune functions return value referenced by a global variable to avoid this point. In addition, this CL adds one more benchmark to cover more code paths, and puts them together as sub benchmarks of BenchmarkFullRune. Change-Id: I6e79f4c087adf70e351498a4b58d7482dcd1ec4a Reviewed-on: https://go-review.googlesource.com/c/go/+/233979 Run-TryBot: eric fang <eric.fang@arm.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2020-08-20unicode: upgrade to Unicode 13.0.0Marcel van Lohuizen
Fixes #40755 Change-Id: I14b3977317994095db8ae1bd873c174641209356 Reviewed-on: https://go-review.googlesource.com/c/go/+/248765 Run-TryBot: Marcel van Lohuizen <mpvl@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2020-04-22unicode/utf8: optimize Valid and ValidString for ASCII checksMartin Möhrmann
Add a fastpath that uses 32bit loads and compares to check 8 ASCII characters per loop iteration. This avoids the overhead of comparing and branching for every byte individually. Combining two 32bit loads into an uint32 allows the same code to be used for 32bit and 64bit platforms. amd64 (Intel i7-3520M): name old time/op new time/op delta ValidTenASCIIChars 15.6ns ± 4% 8.5ns ±14% -45.27% (p=0.000 n=10+10) ValidTenJapaneseChars 50.0ns ± 2% 52.7ns ±15% ~ (p=0.469 n=10+10) ValidStringTenASCIIChars 13.5ns ± 1% 7.9ns ± 5% -41.56% (p=0.000 n=10+10) ValidStringTenJapaneseChars 46.3ns ± 2% 45.8ns ± 2% ~ (p=0.085 n=10+10) arm (Raspberry Pi 3): name old time/op new time/op delta ValidTenASCIIChars 87.5ns ± 0% 58.5ns ± 0% -33.11% (p=0.000 n=9+10) ValidTenJapaneseChars 359ns ± 0% 384ns ± 0% +6.96% (p=0.000 n=10+9) ValidStringTenASCIIChars 87.5ns ± 0% 57.5ns ± 0% -34.31% (p=0.000 n=10+10) ValidStringTenJapaneseChars 356ns ± 0% 377ns ± 0% +5.90% (p=0.000 n=10+10) Change-Id: I9da942bddb250ee1f0ef7aabb4a8cb48edd9053e Reviewed-on: https://go-review.googlesource.com/c/go/+/228823 Run-TryBot: Martin Möhrmann <moehrmann@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Keith Randall <khr@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2020-01-06all: fix typo in RuneSelf, runeSelf commentsTim Cooper
Fixes #36396 Change-Id: I52190f450fa9ac52fbf4ecdc814e954dc29029cd Reviewed-on: https://go-review.googlesource.com/c/go/+/213377 Reviewed-by: Daniel Martí <mvdan@mvdan.cc> Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-11-05unicode: upgrade to Unicode 12Marcel van Lohuizen
This does not include an upgrade of golang.org/x/net. This is optional and best done as a separate CL. Change-Id: Ifecc3fb6e3b7fe026b4ddefbe637186a3445b0bc Reviewed-on: https://go-review.googlesource.com/c/go/+/204658 Run-TryBot: Marcel van Lohuizen <mpvl@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Russ Cox <rsc@golang.org>
2019-10-06unicode/utf8: add link to formal UTF-8 description.Serhat Giydiren
Fixes #31590 Change-Id: I7fd6dcc5c34496776439ff0295f18b5fb5cb538a Reviewed-on: https://go-review.googlesource.com/c/go/+/199141 Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
2019-04-25all: update to Unicode 11Marcel van Lohuizen
This does *not* update the vendored tables. A commit updating these tables should follow soon, Mostly generated running UNICODE_VERSION=11.0.0 in x/text. Manually updated next.txt file. Updates golang/go#27945. Change-Id: I939a01e235aeca898ee9afc99a531e7ad8444e12 Reviewed-on: https://go-review.googlesource.com/c/go/+/154420 Run-TryBot: Marcel van Lohuizen <mpvl@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Robert Griesemer <gri@golang.org>
2019-04-25unicode/utf8: remove some bounds checks from DecodeRuneJosh Bleecher Snyder
The compiler couldn't quite see that reading p[2] and p[3] was safe. This change provides a few hints to help it. First, make sz an int throughout, rather than just when checking the input length. Second, use <= instead of == in later comparisons. name old time/op new time/op delta DecodeASCIIRune-8 2.62ns ± 3% 2.60ns ± 5% ~ (p=0.126 n=18+19) DecodeJapaneseRune-8 4.46ns ±10% 4.01ns ± 5% -10.00% (p=0.000 n=19+20) Change-Id: I2f78a17e38156fbf8b0f5dd6c07c20d6a47e9209 Reviewed-on: https://go-review.googlesource.com/c/go/+/173662 Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-04-24unicode/utf8: use binary literalsJosh Bleecher Snyder
We were using hex literals and had the binary literal in a comment. When I was working with this code, I always referred to the comment. That's an indicator that we should just use the binary literal directly. Updates #19308 Change-Id: I2279cb8efb4ae5f2e1558c15979058ab09eb4f6f Reviewed-on: https://go-review.googlesource.com/c/go/+/173663 Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-04-24unicode: update table using new generator in x/textMarcel van Lohuizen
The changes in Unicode 11 exposes a bug in maketables.go. We update the Unicode 10 tables using a new generator to minimize the changes upgrading to Unicode 11. This change switches over the generation from core to that in x/text. To properly update the tables one needs to run the generate in x/text anyway, so this makes that official. The RangeTable generator in x/text also generates slightly compacter tables. Updates golang/go#27945 See CL 154443 Change-Id: I6c59e082d5b8cd9e9332a32d8971061228581d66 Reviewed-on: https://go-review.googlesource.com/c/go/+/169617 Run-TryBot: Marcel van Lohuizen <mpvl@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-04-24unicode: remove script testMarcel van Lohuizen
The script test requires a manual update on each new Unicode release, which interupts the automated flow. The test is removed in favor of one that fits within the automated scripts. See https://go-review.googlesource.com/c/text/+/169638. Updates golang/go#27945 Change-Id: I9c9f1e2a77f7baef17040ea09eec28d7cf55b5e4 Reviewed-on: https://go-review.googlesource.com/c/go/+/169619 Reviewed-by: Russ Cox <rsc@golang.org>
2019-04-24unicode/utf8: make acceptRanges biggerJosh Bleecher Snyder
This avoids bounds checks in the calling code. The nominal increased size of the array in the binary is compensated for by the decreased size of the functions that call it. The benchmark changes are a bit scattered, but overall positive. name old time/op new time/op delta RuneCountTenASCIIChars-8 8.86ns ± 4% 7.93ns ± 5% -10.45% (p=0.000 n=45+49) RuneCountTenJapaneseChars-8 38.2ns ± 2% 37.2ns ± 1% -2.63% (p=0.000 n=44+41) RuneCountInStringTenASCIIChars-8 7.82ns ± 2% 8.70ns ± 2% +11.19% (p=0.000 n=43+43) RuneCountInStringTenJapaneseChars-8 39.3ns ± 9% 40.0ns ± 5% +1.59% (p=0.043 n=50+50) ValidTenASCIIChars-8 8.68ns ± 5% 8.74ns ± 5% ~ (p=0.070 n=50+48) ValidTenJapaneseChars-8 34.1ns ± 5% 36.8ns ± 4% +8.09% (p=0.000 n=45+50) ValidStringTenASCIIChars-8 9.76ns ± 7% 8.33ns ± 3% -14.59% (p=0.000 n=48+47) ValidStringTenJapaneseChars-8 37.7ns ± 8% 36.5ns ± 5% -3.12% (p=0.011 n=50+47) EncodeASCIIRune-8 2.60ns ± 1% 2.59ns ± 2% -0.24% (p=0.018 n=43+36) EncodeJapaneseRune-8 3.75ns ± 2% 4.56ns ± 6% +21.71% (p=0.000 n=41+50) DecodeASCIIRune-8 2.59ns ± 2% 2.59ns ± 2% ~ (p=0.350 n=44+41) DecodeJapaneseRune-8 4.29ns ± 2% 4.31ns ± 2% +0.61% (p=0.001 n=48+39) FullASCIIRune-8 0.87ns ± 6% 0.29ns ± 5% -67.31% (p=0.000 n=49+43) FullJapaneseRune-8 0.65ns ± 6% 0.65ns ± 4% ~ (p=0.375 n=50+49) [Geo mean] 7.02ns 6.51ns -7.19% Change-Id: I8d5d69c8d33ce2bff94785fba39a2203f9315cb0 Reviewed-on: https://go-review.googlesource.com/c/go/+/173537 Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-11-28unicode: improve generated comments for categoriesWil Selwood
The comments on the category range tables in the unicode package are fairly redundent and require an external source to translate into human readable category names. This adds a look up table with the category descriptions and uses it if available when generating the comments for the range tables. Fixes #28954 Change-Id: I853e2d270def6492c2c1dd2ad0ec761a74c04e5d Reviewed-on: https://go-review.googlesource.com/c/151297 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
2018-06-17unicode: fix SpecialCase to follow its docs & respect explict no-op mappingsBrad Fitzpatrick
If SpecialCase contains an explicit CaseRange with zero deltas, respect those and don't fall back to the default behavior. Fixes #25636 Change-Id: Ic554c6b3dd462b1b39c75194eec469b6ff4aa55b Reviewed-on: https://go-review.googlesource.com/117155 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Marcel van Lohuizen <mpvl@golang.org>
2018-06-01all: update comment URLs from HTTP to HTTPS, where possibleTim Cooper
Each URL was manually verified to ensure it did not serve up incorrect content. Change-Id: I4dc846227af95a73ee9a3074d0c379ff0fa955df Reviewed-on: https://go-review.googlesource.com/115798 Reviewed-by: Ian Lance Taylor <iant@golang.org> Run-TryBot: Ian Lance Taylor <iant@golang.org>
2018-05-06cmd/compile: optimize len([]rune(string))Martin Möhrmann
Adds a new runtime function to count runes in a string. Modifies the compiler to detect the pattern len([]rune(string)) and replaces it with the new rune counting runtime function. RuneCount/lenruneslice/ASCII 27.8ns ± 2% 14.5ns ± 3% -47.70% (p=0.000 n=10+10) RuneCount/lenruneslice/Japanese 126ns ± 2% 60ns ± 2% -52.03% (p=0.000 n=10+10) RuneCount/lenruneslice/MixedLength 104ns ± 2% 50ns ± 1% -51.71% (p=0.000 n=10+9) Fixes #24923 Change-Id: Ie9c7e7391a4e2cca675c5cdcc1e5ce7d523948b9 Reviewed-on: https://go-review.googlesource.com/108985 Run-TryBot: Martin Möhrmann <moehrmann@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
2017-12-01Revert "go/printer: forbid empty line before first comment in block"Joe Tsai
This reverts commit 08f19bbde1b01227fdc2fa2d326e4029bb74dd96. Reason for revert: The changed transformation takes effect on a larger set of code snippets than expected. For example, this: func foo() { // Comment bar() } becomes: func foo() { // Comment bar() } This is an unintended consequence. Change-Id: Ifca88d6267dab8a8170791f7205124712bf8ace8 Reviewed-on: https://go-review.googlesource.com/81335 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Joe Tsai <joetsai@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org>
2017-11-02go/printer: forbid empty line before first comment in blockJoe Tsai
To improve readability when exported fields are removed, forbid the printer from emitting an empty line before the first comment in a const, var, or type block. Also, when printing the "Has filtered or unexported fields." message, add an empty line before it to separate the message from the struct or interfact contents. Before the change: <<< type NamedArg struct { // Name is the name of the parameter placeholder. // // If empty, the ordinal position in the argument list will be // used. // // Name must omit any symbol prefix. Name string // Value is the value of the parameter. // It may be assigned the same value types as the query // arguments. Value interface{} // contains filtered or unexported fields } >>> After the change: <<< type NamedArg struct { // Name is the name of the parameter placeholder. // // If empty, the ordinal position in the argument list will be // used. // // Name must omit any symbol prefix. Name string // Value is the value of the parameter. // It may be assigned the same value types as the query // arguments. Value interface{} // contains filtered or unexported fields } >>> Fixes #18264 Change-Id: I9fe17ca39cf92fcdfea55064bd2eaa784ce48c88 Reviewed-on: https://go-review.googlesource.com/71990 Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Robert Griesemer <gri@golang.org>
2017-10-24unicode: update to Unicode 10.0.0Marcel van Lohuizen
Also includes all derived values as well as vendored packages. Generated by running UNICODE_VERSION=10.0.0 go generate in golang.org/x/text and modified by hand to add the tests and entries in next.txt for new script and properties. Closes Issue #21471 Change-Id: I1d10ee3887bd1fd3d5a756ee0d04bd6ec2814ba1 Reviewed-on: https://go-review.googlesource.com/63953 Run-TryBot: Marcel van Lohuizen <mpvl@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Andrew Bonventre <andybons@golang.org>
generated by cgit v1.3 (git 2.53.0) at 2026-04-15 19:36:19 +0000