aboutsummaryrefslogtreecommitdiff
path: root/src/unicode
AgeCommit message (Collapse)Author
2017-10-05all: revert "all: prefer strings.IndexByte over strings.Index"Marvin Stenger
This reverts https://golang.org/cl/65930. Fixes #22148 Change-Id: Ie0712621ed89c43bef94417fc32de9af77607760 Reviewed-on: https://go-review.googlesource.com/68430 Reviewed-by: Ian Lance Taylor <iant@golang.org>
2017-09-25all: prefer strings.IndexByte over strings.IndexMarvin Stenger
strings.IndexByte was introduced in go1.2 and it can be used effectively wherever the second argument to strings.Index is exactly one byte long. This avoids generating unnecessary string symbols and saves a few calls to strings.Index. Change-Id: I1ab5edb7c4ee9058084cfa57cbcc267c2597e793 Reviewed-on: https://go-review.googlesource.com/65930 Run-TryBot: Ian Lance Taylor <iant@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2017-09-22all: use sort.Slice in a few more placesDaniel Martí
Do the low-hanging fruit - tiny Less functions that are used exactly once. This reduces the amount of code and puts the logic in a single place. Change-Id: I9d4544cd68de5a95e55019bdad1fca0a1dbfae9c Reviewed-on: https://go-review.googlesource.com/63171 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2017-09-19unicode: allow version to be passed by env varMarcel van Lohuizen
This, in turn, to make it work with x/text’s go generate. Also eliminates need to manually update version string in maketables.go. Change-Id: Id5a8b8e27bdce5b1b5920eb9223a2d27b889149a Reviewed-on: https://go-review.googlesource.com/63952 Run-TryBot: Marcel van Lohuizen <mpvl@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Nigel Tao <nigeltao@golang.org>
2017-09-12unicode: speed-up is16/is32Ilya Tocar
Avoid division in common case. There are 5438 ranges in unicode/tables.go 4110 of them have stride 1. Stride 1 case got significantly faster. Other stride is a bit slower. Measured by import ( "testing" "unicode" ) func BenchmarkDiv1(b *testing.B) { rtb := &unicode.RangeTable{ R16: []unicode.Range16{ {0xa800, 0xdfff, 1}, // or 3 }, } for i := 0; i < b.N; i++ { unicode.Is(rtb, rune(0xc700)) } } Div1-6 15.6ns ± 1% 9.9ns ± 1% -36.54% (p=0.000 n=10+10) Div3-6 15.5ns ± 1% 16.1ns ± 1% +3.67% (p=0.000 n=10+10) Helps a bit with xml parsing from issue #21823 XMLsax-6 30.9s ± 0% 29.6s ± 0% -4.15% (p=0.000 n=10+9) Change-Id: Ibac1a91d7b9474d0c134b0add83e56caa62daa20 Reviewed-on: https://go-review.googlesource.com/63390 Run-TryBot: Ilya Tocar <ilya.tocar@intel.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2017-09-12unicode/utf8: make FullRune inlinableIlya Tocar
This has same readability and allows to inline FullRune for massive performance gain: FullASCIIRune-6 4.36ns ± 0% 1.25ns ± 0% -71.33% (p=0.000 n=8+10) FullJapaneseRune-6 4.70ns ± 0% 1.42ns ± 1% -69.68% (p=0.000 n=9+10) Change-Id: I95edd6292417a28aac244e40afb713596a087d93 Reviewed-on: https://go-review.googlesource.com/63332 Run-TryBot: Ilya Tocar <ilya.tocar@intel.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
2017-06-15unicode: move scripts from FoldCategories to FoldScriptsRuss Cox
Copy-and-paste bug was putting scripts in the categories map. Fixes #18186. Change-Id: Ife9d9bdd346fe24e578dbb2a0aac7ef6e889ae68 Reviewed-on: https://go-review.googlesource.com/45830 Run-TryBot: Russ Cox <rsc@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-06-13runtime, unicode: use consistent banner for generated codeBrad Fitzpatrick
Per golang.org/s/generatedcode Updates #nnn Change-Id: Ia7513ef6bd26c20b62b57b29f7770684a315d389 Reviewed-on: https://go-review.googlesource.com/45470 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> Reviewed-by: Matt Layher <mdlayher@gmail.com> Reviewed-by: Ian Lance Taylor <iant@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2017-06-09all: single space after periodBrad Fitzpatrick
Done with grep & interactive search & replace, to double-check replacements. Not many remained after CL 20022. Fixes #18572 Change-Id: Idbe90ba3b584f9b9661d2bbd141607daaadfa41a Reviewed-on: https://go-review.googlesource.com/45270 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2016-10-26unicode/utf8: optimize ValidRuneJoe Tsai
Re-writing the switch statement as a single boolean expression reduces the number of branches that the compiler generates. It is also arguably easier to read as a pair of numeric ranges that valid runes can exist in. No test changes since the existing test does a good job of testing all of the boundaries. This change was to gain back some performance after a correctness fix done in http://golang.org/cl/32123. The correctness fix (CL/32123) slowed down the benchmarks slightly: benchmark old ns/op new ns/op delta BenchmarkIndexRune/10-4 19.3 21.6 +11.92% BenchmarkIndexRune/32-4 33.6 35.2 +4.76% Since the fix relies on utf8.ValidRune, this CL improves benchmarks: benchmark old ns/op new ns/op delta BenchmarkIndexRune/10-4 21.6 20.0 -7.41% BenchmarkIndexRune/32-4 35.2 33.5 -4.83% Change-Id: Ib1ca10a2e29c90e879a8ef9b7221c33e85d015d8 Reviewed-on: https://go-review.googlesource.com/32122 Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-10-17runtime: speed up non-ASCII rune decodingMartin Möhrmann
Copies utf8 constants and EncodeRune implementation from unicode/utf8. Adds a new decoderune implementation that is used by the compiler in code generated for ranging over strings. It does not handle ASCII runes since these are handled directly before calls to decoderune. The DecodeRuneInString implementation from unicode/utf8 is not used since it uses a lookup table that would increase the use of cpu caches. Adds more tests that check decoding of valid and invalid utf8 sequences. name old time/op new time/op delta RuneIterate/range2/ASCII-4 7.45ns ± 2% 7.45ns ± 1% ~ (p=0.634 n=16+16) RuneIterate/range2/Japanese-4 53.5ns ± 1% 49.2ns ± 2% -8.03% (p=0.000 n=20+20) RuneIterate/range2/MixedLength-4 46.3ns ± 1% 41.0ns ± 2% -11.57% (p=0.000 n=20+20) new: "".decoderune t=1 size=423 args=0x28 locals=0x0 old: "".charntorune t=1 size=666 args=0x28 locals=0x0 Change-Id: I1df1fdb385bb9ea5e5e71b8818ea2bf5ce62de52 Reviewed-on: https://go-review.googlesource.com/28490 Run-TryBot: Martin Möhrmann <martisch@uos.de> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-10-12unicode: change SimpleFold to handle invalid runesRuss Cox
Functions like ToLower and ToUpper return the invalid rune back, so we might as well do the same here. I changed my mind about panicking when I tried to document the behavior. Fixes #16690 (again). Change-Id: If1c68bfcd66daea160fd19948e7672b0e1add106 Reviewed-on: https://go-review.googlesource.com/30935 Run-TryBot: Russ Cox <rsc@golang.org> Reviewed-by: Rob Pike <r@golang.org>
2016-10-12unicode: panic if given rune is negative in SimpleFoldHiroshi Ioka
Fixes #16690 Change-Id: I6db588c4b0f23c5ec6bc9b85a488b60fab3f2f13 Reviewed-on: https://go-review.googlesource.com/30892 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2016-09-03unicode/utf8: reduce bounds checks in EncodeRuneMartin Möhrmann
Provide bounds elim hints in EncodeRune. name old time/op new time/op delta EncodeASCIIRune-4 2.69ns ± 2% 2.69ns ± 2% ~ (p=0.193 n=47+46) EncodeJapaneseRune-4 5.97ns ± 2% 5.38ns ± 2% -9.93% (p=0.000 n=49+50) Change-Id: I1a6dcffff3bdd64ab93c2130021e3b00981de4c8 Reviewed-on: https://go-review.googlesource.com/28492 Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com> Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com> Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org>
2016-06-28unicode: upgrade to version 9.0.0Marcel van Lohuizen
Changes beyond generated tables: - Now supports aliases to handle deprecated property classes. - Some Mongolian letters are now modifiers. Other changes: - strconv: newly generated table to be in sync - regexp/syntax: updated maxFold Fixes #16191 Change-Id: I56bdf21ee2f775f2a82d0465b3772faf5c24cb61 Reviewed-on: https://go-review.googlesource.com/24496 Run-TryBot: Marcel van Lohuizen <mpvl@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2016-04-26unicode: improve SimpleFold performance for asciiEgon Elbre
This change significantly speeds up case-insensitive regexp matching. benchmark old ns/op new ns/op delta BenchmarkMatchEasy0i_32-8 2690 1473 -45.24% BenchmarkMatchEasy0i_1K-8 80404 42269 -47.43% BenchmarkMatchEasy0i_32K-8 3272187 2076118 -36.55% BenchmarkMatchEasy0i_1M-8 104805990 66503805 -36.55% BenchmarkMatchEasy0i_32M-8 3360192200 2126121600 -36.73% benchmark old MB/s new MB/s speedup BenchmarkMatchEasy0i_32-8 11.90 21.72 1.83x BenchmarkMatchEasy0i_1K-8 12.74 24.23 1.90x BenchmarkMatchEasy0i_32K-8 10.01 15.78 1.58x BenchmarkMatchEasy0i_1M-8 10.00 15.77 1.58x BenchmarkMatchEasy0i_32M-8 9.99 15.78 1.58x Issue #13288 Change-Id: I94af7bb29e75d60b4f6ee760124867ab271b9642 Reviewed-on: https://go-review.googlesource.com/16943 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2016-04-15all: remove unnecessary type conversionsMatthew Dempsky
cmd and runtime were handled separately, and I'm intentionally skipped syscall. This is the rest of the standard library. CL generated mechanically with github.com/mdempsky/unconvert. Change-Id: I9e0eff886974dedc37adb93f602064b83e469122 Reviewed-on: https://go-review.googlesource.com/22104 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Matthew Dempsky <mdempsky@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org>
2016-03-02all: single space after period.Brad Fitzpatrick
The tree's pretty inconsistent about single space vs double space after a period in documentation. Make it consistently a single space, per earlier decisions. This means contributors won't be confused by misleading precedence. This CL doesn't use go/doc to parse. It only addresses // comments. It was generated with: $ perl -i -npe 's,^(\s*// .+[a-z]\.) +([A-Z]),$1 $2,' $(git grep -l -E '^\s*//(.+\.) +([A-Z])') $ go test go/doc -update Change-Id: Iccdb99c37c797ef1f804a94b22ba5ee4b500c4f7 Reviewed-on: https://go-review.googlesource.com/20022 Reviewed-by: Rob Pike <r@golang.org> Reviewed-by: Dave Day <djd@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2016-03-01all: make copyright headers consistent with one space after periodBrad Fitzpatrick
This is a subset of https://golang.org/cl/20022 with only the copyright header lines, so the next CL will be smaller and more reviewable. Go policy has been single space after periods in comments for some time. The copyright header template at: https://golang.org/doc/contribute.html#copyright also uses a single space. Make them all consistent. Change-Id: Icc26c6b8495c3820da6b171ca96a74701b4a01b0 Reviewed-on: https://go-review.googlesource.com/20111 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org> Reviewed-by: Matthew Dempsky <mdempsky@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org>
2016-02-24unicode/utf16: speed up and clean up Encode and EncodeRuneAlberto Donizetti
name old time/op new time/op delta EncodeValidASCII-4 74.1ns ± 1% 70.1ns ± 1% -5.46% (p=0.000 n=10+10) EncodeValidJapaneseChars-4 61.3ns ± 0% 58.9ns ± 0% -3.82% (p=0.000 n=10+10) EncodeRune-4 13.1ns ± 1% 9.8ns ± 0% -25.24% (p=0.000 n=10+9) Fixes #6957 Change-Id: I9dde6d77420c34c6e2ef3e6213bb6be9b58a3074 Reviewed-on: https://go-review.googlesource.com/19891 Reviewed-by: Rob Pike <r@golang.org> Run-TryBot: Rob Pike <r@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2016-02-23unicode/utf16: speed up and clean up DecodeAlberto Donizetti
name old time/op new time/op delta DecodeValidASCII-4 94.7ns ± 1% 87.4ns ± 1% -7.71% (p=0.000 n=10+9) DecodeValidJapaneseChars-4 91.0ns ± 2% 84.8ns ± 0% -6.77% (p=0.000 n=9+10) DecodeRune-4 16.5ns ± 0% 16.6ns ± 2% ~ (p=0.108 n=9+10) For #6957 Change-Id: I618c15c2a42ef7ec6a5cd163b7c3f1a65ca4ad01 Reviewed-on: https://go-review.googlesource.com/19826 Reviewed-by: Rob Pike <r@golang.org>
2016-02-23unicode/utf16: add benchmarksAlberto Donizetti
For #6957 Change-Id: Ic497c12f33efc933e9fe81f6cd1b2a0a01abbabf Reviewed-on: https://go-review.googlesource.com/19820 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-12-01unicode/utf8: add test for FullRuneMarcel van Lohuizen
Check that it now properly handles \xC0 and \xC1. Fixes #11733. Change-Id: I66cfe0d43f9d123d4c4509a3fa18b9b6380dfc39 Reviewed-on: https://go-review.googlesource.com/17225 Reviewed-by: Russ Cox <rsc@golang.org>
2015-11-24unicode: add function level examplesCarlos Cirello
Change-Id: I9e968c03c6615a539b18701c84f830898242441d Reviewed-on: https://go-review.googlesource.com/13926 Reviewed-by: Russ Cox <rsc@golang.org>
2015-11-24unicode/utf8: don't imply that the empty string is incorrect UTF-8Aaron Jacobs
Change-Id: Idd9523949ee4f2f304b12be39f8940ba34a420be Reviewed-on: https://go-review.googlesource.com/16361 Reviewed-by: Russ Cox <rsc@golang.org>
2015-11-16unicode/utf8: table-based algorithm for decodingMarcel van Lohuizen
This simplifies covering all cases, reducing the number of branches and making unrolling for simpler functions manageable. This significantly improves performance of non-ASCII input. This change will also allow addressing Issue #11733 in an efficient manner. RuneCountTenASCIIChars-8 13.7ns ± 4% 13.5ns ± 2% ~ (p=0.116 n=7+8) RuneCountTenJapaneseChars-8 153ns ± 3% 74ns ± 2% -51.42% (p=0.000 n=8+8) RuneCountInStringTenASCIIChars-8 13.5ns ± 2% 12.5ns ± 3% -7.13% (p=0.000 n=8+7) RuneCountInStringTenJapaneseChars-8 145ns ± 2% 68ns ± 2% -53.21% (p=0.000 n=8+8) ValidTenASCIIChars-8 14.1ns ± 3% 12.5ns ± 5% -11.38% (p=0.000 n=8+8) ValidTenJapaneseChars-8 147ns ± 3% 71ns ± 4% -51.72% (p=0.000 n=8+8) ValidStringTenASCIIChars-8 12.5ns ± 3% 12.3ns ± 3% ~ (p=0.095 n=8+8) ValidStringTenJapaneseChars-8 146ns ± 4% 70ns ± 2% -51.62% (p=0.000 n=8+7) DecodeASCIIRune-8 5.91ns ± 2% 4.83ns ± 3% -18.28% (p=0.001 n=7+7) DecodeJapaneseRune-8 12.2ns ± 7% 8.5ns ± 3% -29.79% (p=0.000 n=8+7) FullASCIIRune-8 5.95ns ± 3% 4.27ns ± 1% -28.23% (p=0.000 n=8+7) FullJapaneseRune-8 12.0ns ± 6% 4.3ns ± 3% -64.39% (p=0.000 n=8+8) Change-Id: Iea1d6b0180cbbee1739659a0a38038126beecaca Reviewed-on: https://go-review.googlesource.com/16940 Reviewed-by: Russ Cox <rsc@golang.org>
2015-11-16unicode/utf8: removed uses of ranging over stringMarcel van Lohuizen
Ranging over string is much slower than using DecodeRuneInString. See golang.org/issue/13162. Replacing ranging over a string with the implementation of the Bytes counterpart results in the following performance improvements: RuneCountInStringTenASCIIChars-8 43.0ns ± 1% 16.4ns ± 2% -61.80% (p=0.000 n=7+8) RuneCountInStringTenJapaneseChars-8 161ns ± 2% 154ns ± 2% -4.58% (p=0.000 n=8+8) ValidStringTenASCIIChars-8 52.2ns ± 1% 13.2ns ± 1% -74.62% (p=0.001 n=7+7) ValidStringTenJapaneseChars-8 173ns ± 2% 153ns ± 2% -11.78% (p=0.000 n=7+8) Update golang/go#13162 Change-Id: Ifc40a6a94bb3317f1f2d929d310bd2694645e9f6 Reviewed-on: https://go-review.googlesource.com/16695 Reviewed-by: Russ Cox <rsc@golang.org>
2015-10-26unicode/utf8: added benchmarksMarcel van Lohuizen
Cover some functions that weren't benched before and add InString variants if the underlying implementation is different. Note: compare (Valid|RuneCount)InString* to their (Valid|RuneCount)* counterparts. It shows, somewhat unexpectedly, that ranging over a string is *much* slower than using calls to DecodeRune. Results: In order to avoid a discrepancy in measuring the performance of core we could leave the names of the string-based measurements unchanged and suffix the added alternatives with Bytes. Compared to old: BenchmarkRuneCountTenASCIIChars-8 44.3 12.4 -72.01% BenchmarkRuneCountTenJapaneseChars-8 167 67.1 -59.82% BenchmarkEncodeASCIIRune-8 3.37 3.44 +2.08% BenchmarkEncodeJapaneseRune-8 7.19 7.24 +0.70% BenchmarkDecodeASCIIRune-8 5.41 5.53 +2.22% BenchmarkDecodeJapaneseRune-8 8.17 8.41 +2.94% All benchmarks: BenchmarkRuneCountTenASCIIChars-8 100000000 12.4 ns/op BenchmarkRuneCountTenJapaneseChars-8 20000000 67.1 ns/op BenchmarkRuneCountInStringTenASCIIChars-8 30000000 44.5 ns/op BenchmarkRuneCountInStringTenJapaneseChars-8 10000000 165 ns/op BenchmarkValidTenASCIIChars-8 100000000 12.5 ns/op BenchmarkValidTenJapaneseChars-8 20000000 71.1 ns/op BenchmarkValidStringTenASCIIChars-8 30000000 50.0 ns/op BenchmarkValidStringTenJapaneseChars-8 10000000 161 ns/op BenchmarkEncodeASCIIRune-8 500000000 3.44 ns/op BenchmarkEncodeJapaneseRune-8 200000000 7.24 ns/op BenchmarkDecodeASCIIRune-8 300000000 5.53 ns/op BenchmarkDecodeJapaneseRune-8 200000000 8.41 ns/op BenchmarkFullASCIIRune-8 500000000 3.91 ns/op BenchmarkFullJapaneseRune-8 300000000 4.22 ns/op Change-Id: I674d2ee4917b975a37717bbfa1082cc84dcd275e Reviewed-on: https://go-review.googlesource.com/14431 Reviewed-by: Russ Cox <rsc@golang.org>
2015-08-24unicode: include rune 0 in RangeTables.Marcel van Lohuizen
All of Go passes. No changes for the text repo. Fixes #10153 Change-Id: I313369bf471c8974390a6d42075e5c54f6a81750 Reviewed-on: https://go-review.googlesource.com/13667 Reviewed-by: Russ Cox <rsc@golang.org> Reviewed-by: Rob Pike <r@golang.org>
2015-06-26unicode: upgrade to 8.0.0Marcel van Lohuizen
Not sure if I'm on time for 1.5; Unicode 8 just got released. Straighforward upgrade. Only changed maketables.go to prevent it from adding the Cherokee upper and lower case mappings. This change causes the caseOrbit table to NOT change. Added tests to verify that the relevant functions still produce the correct result, even for Cherokee. Fixes #11309 Change-Id: I42850f5b3399bde125b002efc78eff96dbd86a08 Reviewed-on: https://go-review.googlesource.com/11286 Reviewed-by: Russ Cox <rsc@golang.org>
2015-03-18all: use "reports whether" in place of "returns true if(f)"Josh Bleecher Snyder
Comment changes only. Change-Id: I56848814564c4aa0988b451df18bebdfc88d6d94 Reviewed-on: https://go-review.googlesource.com/7721 Reviewed-by: Rob Pike <r@golang.org>
2015-02-16unicode: Fixed an out of date comment (MaxLatin1, not Latin1Max).Aaron Jacobs
Change-Id: I3ca878e9685f650a9ff02aaac0e2e3cca89634c6 Reviewed-on: https://go-review.googlesource.com/4970 Reviewed-by: Minux Ma <minux@golang.org>
2014-10-16unicode/utf8: fix docs for DecodeRune(empty) and friends.Nigel Tao
LGTM=r R=r CC=golang-codereviews https://golang.org/cl/157080043
2014-09-08build: move package sources from src/pkg to srcRuss Cox
Preparation was in CL 134570043. This CL contains only the effect of 'hg mv src/pkg/* src'. For more about the move, see golang.org/s/go14nopkg.
generated by cgit v1.3 (git 2.53.0) at 2026-04-15 16:35:35 +0000