aboutsummaryrefslogtreecommitdiff
path: root/src/cmd/internal/obj
AgeCommit message (Collapse)Author
2025-05-01cmd/internal/obj/riscv: add support for vector floating-point instructionsJoel Sing
Add support for vector floating-point instructions to the RISC-V assembler. This includes single-width and widening addition and subtraction, multiplication and division, fused multiply-addition, comparison, min/max, sign-injection, classification and type conversion instructions. Change-Id: I8bceb1c5d7eead0561ba5407ace00805a6144f51 Reviewed-on: https://go-review.googlesource.com/c/go/+/646777 Reviewed-by: Carlos Amedee <carlos@golang.org> Reviewed-by: Junyang Shao <shaojunyang@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Meng Zhuo <mengzhuo1203@gmail.com> Reviewed-by: Mark Ryan <markdryan@rivosinc.com>
2025-04-28cmd/internal/obj/loong64: fix the error parameters when calling ↵Guoqi Chen
UnspillRegisterArgs This bug was introduced in CL 648518. Fixes #73518. Change-Id: I4988dd0b636c6a6a48d2aa2e2ae868e43f69995a Reviewed-on: https://go-review.googlesource.com/c/go/+/668475 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Meidan Li <limeidan@loongson.cn> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Keith Randall <khr@google.com>
2025-04-25cmd/compile: allow all of the preamble to be preemptibleKeith Randall
We currently make some parts of the preamble unpreemptible because it confuses morestack. See comments in the code. Instead, have morestack handle those weird cases so we can remove unpreemptible marks from most places. This CL makes user functions preemptible everywhere if they have no write barriers (at least, on x86). In cmd/go the fraction of functions that need preemptible markings drops from 82% to 36%. Makes the cmd/go binary 0.3% smaller. Update #35470 Change-Id: Ic83d5eabfd0f6d239a92e65684bcce7e67ff30bb Reviewed-on: https://go-review.googlesource.com/c/go/+/648518 Auto-Submit: Keith Randall <khr@google.com> Reviewed-by: Keith Randall <khr@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Cherry Mui <cherryyz@google.com>
2025-04-24cmd/internal/obj: add new assembly format for BFPT and BFPF on loong64Guoqi Chen
On loong64, BFPT and BFPF are mapped to the platform assembly as follows: Go asm syntax: BFPT FCCx, offs21 BFPF FCCx, offs21 Equivalent platform assembler syntax: bcnez cj, offs21 bceqz cj, offs21 If the condition register is not specified, it defaults to FCC0. Change-Id: I2cc3df62a9c55d4b5eb124789358983c6737319c Reviewed-on: https://go-review.googlesource.com/c/go/+/667456 Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Junyang Shao <shaojunyang@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Meidan Li <limeidan@loongson.cn>
2025-04-15cmd/internal/obj/loong64: add support for {V,XV}SET{EQ,NE}Z.V series ↵limeidan
instructions Change-Id: If3794dfde3ff461662c8a493ff51d0c779e81bca Reviewed-on: https://go-review.googlesource.com/c/go/+/664795 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: abner chenc <chenguoqi@loongson.cn> Reviewed-by: Michael Pratt <mpratt@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
2025-04-09cmd/internal/obj/wasm: use i64 for large return addrZxilly
Use i64 to avoid overflow when getting PC_F from the return addr. Fixes #73246 Change-Id: I5683dccf7eada4b8536edf53e2e83116a2f6d943 GitHub-Last-Rev: 267d9a1a031868430d0af530de14229ee1ae8609 GitHub-Pull-Request: golang/go#73277 Reviewed-on: https://go-review.googlesource.com/c/go/+/663995 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Cherry Mui <cherryyz@google.com>
2025-04-03cmd/internal/obj/arm64: return a bit shift from movconJoel Sing
Return the shift in bits from movcon, rather than returning an index. This allows a number of multiplications to be removed, making the code more readable. Scale down to an index only when encoding. Change-Id: I1be91eb526ad95d389e2f8ce97212311551790df Reviewed-on: https://go-review.googlesource.com/c/go/+/650939 Auto-Submit: Joel Sing <joel@sing.id.au> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Cherry Mui <cherryyz@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2025-04-03cmd/internal/obj/arm64: deduplicate con32classJoel Sing
Teach conclass how to handle 32 bit values and deduplicate the code between con32class and conclass. Change-Id: I9c5eea31d443fd4c2ce700c6ea21e1d0bef665b0 Reviewed-on: https://go-review.googlesource.com/c/go/+/650938 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Cherry Mui <cherryyz@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Auto-Submit: Joel Sing <joel@sing.id.au>
2025-04-03cmd/internal/obj/arm64: simplify conclassJoel Sing
Reduce repetition by pulling some common conversions into variables. Change-Id: I8c1cc806236b5ecdadf90f4507923718fa5de9b6 Reviewed-on: https://go-review.googlesource.com/c/go/+/650937 Reviewed-by: Cherry Mui <cherryyz@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2025-04-01cmd/internal/obj/riscv: add support for vector fixed-point arithmetic ↵Joel Sing
instructions Add support for vector fixed-point arithmetic instructions to the RISC-V assembler. This includes single width saturating addition and subtraction, averaging addition and subtraction and scaling shift instructions. Change-Id: I9aa27e9565ad016ba5bb2b479e1ba70db24e4ff5 Reviewed-on: https://go-review.googlesource.com/c/go/+/646776 Reviewed-by: Mark Ryan <markdryan@rivosinc.com> Reviewed-by: Carlos Amedee <carlos@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2025-03-29cmd/internal/obj/arm64: factor out constant classification codeJoel Sing
This will allow for further improvements and deduplication. Change-Id: I9374fc2d16168ced06f3fcc9e558a9c85e24fd01 Reviewed-on: https://go-review.googlesource.com/c/go/+/650936 Reviewed-by: Fannie Zhang <Fannie.Zhang@arm.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Cherry Mui <cherryyz@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
2025-03-29cmd/internal/obj/riscv: add support for vector integer arithmetic instructionsJoel Sing
Add support for vector integer arithmetic instructions to the RISC-V assembler. This includes vector addition, subtraction, integer extension, add-with-carry, subtract-with-borrow, bitwise logical operations, comparison, min/max, integer division and multiplication instructions. Change-Id: I8c191ef8e31291e13743732903e4f12356133a46 Reviewed-on: https://go-review.googlesource.com/c/go/+/646775 Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Cherry Mui <cherryyz@google.com> Reviewed-by: Meng Zhuo <mengzhuo1203@gmail.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Mark Ryan <markdryan@rivosinc.com>
2025-03-27cmd/internal/obj/riscv,internal/bytealg: synthesize MIN/MAX/MINU/MAXU ↵Joel Sing
instructions Provide a synthesized version of the MIN/MAX/MINU/MAXU instructions if they're not natively available. This allows these instructions to be used in assembly unconditionally. Use MIN in internal/bytealg.compare. Cq-Include-Trybots: luci.golang.try:gotip-linux-riscv64 Change-Id: I8a5a3a59f0a9205e136fc3d673b23eaf3ca469f8 Reviewed-on: https://go-review.googlesource.com/c/go/+/653295 Reviewed-by: Mark Ryan <markdryan@rivosinc.com> Reviewed-by: Cherry Mui <cherryyz@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Meng Zhuo <mengzhuo1203@gmail.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2025-03-27cmd/internal/obj/riscv: improve constant constructionJoel Sing
Attempt to construct large constants that have a consecutive sequence of ones from a small negative constant, with a logical right and/or left shift. This allows for a large range of mask like constants to be constructed with only two or three instructions, avoiding the need to load from memory. Change-Id: I35a77fecdd2df0ed3f33b772d518f85119d4ff66 Reviewed-on: https://go-review.googlesource.com/c/go/+/652778 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Cherry Mui <cherryyz@google.com> Reviewed-by: Mark Ryan <markdryan@rivosinc.com> Reviewed-by: Meng Zhuo <mengzhuo1203@gmail.com>
2025-03-26cmd/internal/obj/arm64: add support for BTI instructionJoel Sing
Add support for the `BTI' instruction to the arm64 assembler. This instruction provides Branch Target Identification for targets of indirect branches. A BTI can be marked with a target type of 'C' (call), 'J' (jump) or 'JC' (jump or call). Updates #66054 Change-Id: I1cf31a0382207bb75b9b2deb49ac298a59c00d8a Reviewed-on: https://go-review.googlesource.com/c/go/+/646781 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Marvin Drees <marvin.drees@9elements.com> Reviewed-by: Cherry Mui <cherryyz@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
2025-03-25cmd/internal/obj/loong64: add [X]VSHUF4I.{B/H/W/D} instructions supportXiaolin Zhao
Go asm syntax: VSHUF4I{B/H/W/V} $1, V1, V2 XVSHUF4I{B/H/W/V} $2, X1, X2 Equivalent platform assembler syntax: vshuf4i.{b/h/w/d} v2, v1, $1 xvshuf4i.{b/h/w/d} x2, x1, $2 Change-Id: I6a847ccbd2c93432d87bd1390b5cf1508da06496 Reviewed-on: https://go-review.googlesource.com/c/go/+/658376 Reviewed-by: Cherry Mui <cherryyz@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: abner chenc <chenguoqi@loongson.cn> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2025-03-25cmd/internal/obj/arm64: move register encoding into opldrr/opstrrJoel Sing
Rather than having register encoding knowledge in each caller of opldrr/opstrr (and in a separate olsxrr function), pass the registers into opldrr/opstrr and let them handle the encoding. This reduces duplication and improves readability. Change-Id: I50a25263f305d01454f3ff95e8b6e7c76e760ab0 Reviewed-on: https://go-review.googlesource.com/c/go/+/471521 Reviewed-by: Cherry Mui <cherryyz@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2025-03-25cmd/internal/obj/loong64: add [X]VMULW{EV/OD} series instructions supportXiaolin Zhao
Go asm syntax: VMULW{EV/OD}{HB/WH/VW/QV}[U] VK, VJ, VD XVMULW{EV/OD}{HB/WH/VW/QV}[U] XK, XJ, XD VMULW{EV/OD}{HBUB/WHUH/VWUW/QVUV} VK, VJ, VD XVMULW{EV/OD}{HBUB/WHUH/VWUW/QVUV} XK, XJ, XD Equivalent platform assembler syntax: vmulw{ev/od}.{h.b/w.h/d.w/q.d}[u] vd, vj, vk xvmulw{ev/od}.{h.b/w.h/d.w/q.d}[u] xd, xj, xk vmulw{ev/od}.{h.bu.b/w.hu.h/d.wu.w/q.du.d} vd, vj, vk xvmulw{ev/od}.{h.bu.b/w.hu.h/d.wu.w/q.du.d} xd, xj, xk Change-Id: Ib1b5fb9605417a2b81841deae40e0e2beb90d03c Reviewed-on: https://go-review.googlesource.com/c/go/+/658375 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: abner chenc <chenguoqi@loongson.cn> Reviewed-by: Cherry Mui <cherryyz@google.com> Reviewed-by: David Chase <drchase@google.com>
2025-03-25cmd/internal/obj/arm64: provide and use an oprrrr functionJoel Sing
Provide a four register version of oprrr, which takes an additional 'ra' register. Use this instead of oprrr where appropriate. Change-Id: I8882957a83c2b08e407f37a37c61864cd920bbc9 Reviewed-on: https://go-review.googlesource.com/c/go/+/471519 Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Cherry Mui <cherryyz@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2025-03-25cmd/internal/obj/arm64: move register encoding into oprrrJoel Sing
Rather than having register encoding knowledge in each caller of oprrr, pass the registers into oprrr and let it handle the encoding. This reduces duplication and improves readability. Change-Id: Iab6c70f7796b7a8c071419654b8a5686aeee8c1b Reviewed-on: https://go-review.googlesource.com/c/go/+/471518 Reviewed-by: Fannie Zhang <Fannie.Zhang@arm.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Cherry Mui <cherryyz@google.com>
2025-03-25cmd/internal/obj/arm64: replace range checks with isaddcon2Joel Sing
isaddcon2 tests for the range 0 <= v <= 0xffffff - replace duplicated range checks with calls to isaddcon2. Change-Id: Ia6f331852ed3d77715b265cb4fcc500579eac711 Reviewed-on: https://go-review.googlesource.com/c/go/+/650935 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Fannie Zhang <Fannie.Zhang@arm.com> Reviewed-by: Cherry Mui <cherryyz@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
2025-03-24cmd/asm: add LCDBR instruction on s390xVishwanatha HD
This CL is to add LCDBR assembly instruction mnemonics, mainly used in math package. The LCDBR instruction has the same effect as the FNEG pseudo-instructions, just that it sets the flag. Change-Id: I3f00f1ed19148d074c3b6c5f64af0772289f2802 Reviewed-on: https://go-review.googlesource.com/c/go/+/648036 Reviewed-by: Srinivas Pokala <Pokala.Srinivas@ibm.com> Reviewed-by: Cherry Mui <cherryyz@google.com> Reviewed-by: Michael Munday <mike.munday@lowrisc.org> Reviewed-by: Michael Pratt <mpratt@google.com> Run-TryBot: Michael Munday <mike.munday@lowrisc.org> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> TryBot-Result: Gopher Robot <gobot@golang.org>
2025-03-21cmd/internal/obj/riscv: factor out shift constant codeJoel Sing
Move the code that tests to see if a constant can be represented by a 32 bit signed integer and a logical left shift. This reduces duplication and increases readability. Also add test coverage now that this is an independent function. Change-Id: Id25395b1380b00cf5b69ca201b7715ef84f7ade6 Reviewed-on: https://go-review.googlesource.com/c/go/+/652777 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Meng Zhuo <mengzhuo1203@gmail.com> Reviewed-by: David Chase <drchase@google.com> Reviewed-by: Cherry Mui <cherryyz@google.com>
2025-03-20cmd/internal/obj/riscv: fix the encoding for REV8 and ORCBMark Ryan
The instructions are currently encoded and validated using an iIIEncoding which is incorrect as these instructions do not take an immediate operand. Encode them instead using an rIIEncoding as is done for the other two register argument bitmanip instructions. Change-Id: Ia4d9c6f6ebd2dfc381935ebc11afa8fc3664232b Reviewed-on: https://go-review.googlesource.com/c/go/+/637317 Reviewed-by: David Chase <drchase@google.com> Reviewed-by: Joel Sing <joel@sing.id.au> Reviewed-by: Meng Zhuo <mengzhuo1203@gmail.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Cherry Mui <cherryyz@google.com>
2025-03-19cmd/internal/obj/riscv: prevent panics on bad branchesMark Ryan
Syntactically incorrect branches, such as BEQ X5, X6, $1 BEQ X5, X6, 31(X10) cause the assembler to panic, which they shouldn't really do. It's better for the user to see a normal error, as reported for other syntax errors in riscv64 assembly. The panics also prevent us from writing negative tests for these sorts of errors. Here we fix the issue by ensuring we generate a normal error instead of panicking when the user provides an invalid branch target. We also add a couple of negative tests. Change-Id: I1da568999a75097484b61a01d418f5d4be3e04fa Reviewed-on: https://go-review.googlesource.com/c/go/+/637316 Reviewed-by: Cherry Mui <cherryyz@google.com> Reviewed-by: Joel Sing <joel@sing.id.au> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: David Chase <drchase@google.com> Reviewed-by: Meng Zhuo <mengzhuo1203@gmail.com>
2025-03-18cmd/internal/obj/riscv: prevent duplicate error reportsMark Ryan
The riscv64 Go assembler can output certain errors, ones produced by instructionsForProg, multiple times. These errors are guaranteed to be output at least twice and can appear three or more times if a rescan is needed to recompute branch addresses. For example, the syntactically incorrect instruction MOV (X10), $1 will generate at least two identical errors asm: 86076 (asm.s:21524) MOV (X10), $1: unsupported MOV asm: 86076 (asm.s:21524) MOV (X10), $1: unsupported MOV asm: assembly failed In addition to confusing the user, these duplicate errors make it difficult to write negative tests for certain types of instructions, e.g., branches, whose duplicate errors are not always identical, and so not ignored by endtoend_test.go. We fix the issue by returning from preprocess if any errors have been generated by the time we reach the end of the rescan loop. One implication of this change is that validation errors will no longer be reported if an error is generated earlier in the preprocess stage. Negative test cases for validation errors are therefore moved to their own file as the existing riscv64error.s file contains errors generated by instructionsForProg that will now suppress the validation errors. Change-Id: Iffacdbefce28f44970dd5dda44990b822b8a23d4 Reviewed-on: https://go-review.googlesource.com/c/go/+/637315 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Joel Sing <joel@sing.id.au> Reviewed-by: David Chase <drchase@google.com> Reviewed-by: Cherry Mui <cherryyz@google.com>
2025-03-16cmd/internal/obj/loong64: add {V,XV}NEG{B/H/W/V} instructions supportXiaolin Zhao
Go asm syntax: VNEG{B/H/W/V} VJ, VD XVNEG{B/H/W/V} XJ, XD Equivalent platform assembler syntax: vneg.{b/h/w/d} vd, vj xvneg.{b/h/w/d} xd, xj Change-Id: Ie0a82a434b0ffbcb77425a65b96eff56e030028c Reviewed-on: https://go-review.googlesource.com/c/go/+/635935 Reviewed-by: abner chenc <chenguoqi@loongson.cn> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Cherry Mui <cherryyz@google.com> Reviewed-by: David Chase <drchase@google.com>
2025-03-13cmd/internal/obj/loong64: add {V,XV}{FSQRT/FRECIP/FRSQRT}.{S/D} instructions ↵Xiaolin Zhao
support Go asm syntax: V{FSQRT/FRECIP/FRSQRT}{F/D} VJ, VD XV{FSQRT/FRECIP/FRSQRT}{F/D} XJ, XD Equivalent platform assembler syntax: v{fsqrt/frecip/frsqrt}.{s/d} vd, vj xv{fsqrt/frecip/frsqrt}.{s/d} xd, xj Change-Id: I3fdbe3193659d7532164451b087ccf725053172f Reviewed-on: https://go-review.googlesource.com/c/go/+/636395 Reviewed-by: David Chase <drchase@google.com> Reviewed-by: abner chenc <chenguoqi@loongson.cn> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Junyang Shao <shaojunyang@google.com>
2025-03-12cmd/internal/obj/loong64: add {V,XV}DIV{B/H/W/V}[U] and ↵Xiaolin Zhao
{V,XV}MOD{B/H/W/V}[U] instructions support Go asm syntax: VDIV{B/H/W/V}[U] VK, VJ, VD XVDIV{B/H/W/V}[U] XK, XJ, XD VMOD{B/H/W/V}[U] VK, VJ, VD XVMOD{B/H/W/V}[U] XK, XJ, XD Equivalent platform assembler syntax: vdiv.{b/h/w/d}[u] vd, vj, vk xvdiv.{b/h/w/d}[u] xd, xj, xk vmod.{b/h/w/d}[u] vd, vj, vk xvmod.{b/h/w/d}[u] xd, xj, xk Change-Id: I3676721c3c415de0f2ebbd480ecd1b2400a28dba Reviewed-on: https://go-review.googlesource.com/c/go/+/636376 Reviewed-by: abner chenc <chenguoqi@loongson.cn> Reviewed-by: Junyang Shao <shaojunyang@google.com> Reviewed-by: David Chase <drchase@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2025-03-11cmd/internal/obj/loong64: add {V,XV}MUL{B/H/W/V} and {V,XV}MUH{B/H/W/V}[U] ↵Xiaolin Zhao
instructions support Go asm syntax: VMUL{B/H/W/V} VK, VJ, VD VMUH{B/H/W/V}[U] VK, VJ, VD XVMUL{B/H/W/V} XK, XJ, XD XVMUH{B/H/W/V}[U] XK, XJ, XD Equivalent platform assembler syntax: vmul.{b/h/w/d} vd, vj, vk vmuh.{b/h/w/d}[u] vd, vj, vk xvmul.{b/h/w/d} xd, xj, xk xvmuh.{b/h/w/d}[u] xd, xj, xk Change-Id: I2f15a5b4b6303a0f82cb85114477f58e1b5fd950 Reviewed-on: https://go-review.googlesource.com/c/go/+/636375 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Auto-Submit: abner chenc <chenguoqi@loongson.cn> Reviewed-by: David Chase <drchase@google.com> Reviewed-by: Junyang Shao <shaojunyang@google.com> Reviewed-by: abner chenc <chenguoqi@loongson.cn>
2025-03-10cmd/internal/obj/loong64: add {V,XV}SEQI, {V,XV}.{AND,OR,XOR,NOR} ↵Guoqi Chen
instructions support Go asm syntax: VSEQB $1, V2, V3 XVSEQB $2, X2, X3 V{AND,OR,XOR,NOR}B $1, V2, V3 XV{AND,OR,XOR,NOR}B $1, V2, V3 V{AND,OR,XOR,NOR,ANDN,ORN}V V1, V2, V3 XV{AND,OR,XOR,NOR,ANDN,ORN}V V1, V2, V3 Equivalent platform assembler syntax: vseqi.b v3, v2, $1 xvseqi.b x3, x2 ,$2 v{and,or,xor,nor}.b v3, v2, $1 xv{and,or,xor,nor}.b x3, x2, $1 v{and,or,xor,nor,andn,orn}v v3, v2, v1 xv{and,or,xor,nor,andn,orn}v x3, x2, x1 Change-Id: I56ae0db72c7f473755cbdc7f7171c1058a9def97 Reviewed-on: https://go-review.googlesource.com/c/go/+/635515 Reviewed-by: Meidan Li <limeidan@loongson.cn> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Junyang Shao <shaojunyang@google.com> Reviewed-by: David Chase <drchase@google.com>
2025-03-06cmd/internal/obj/loong64: add {V,XV}ILV{L/H}.{B/H/W/D} instructions supportXiaolin Zhao
Go asm syntax: VILV{L/H}{B/H/W/V} VK, VJ, VD XVILV{L/H}{B/H/W/V} XK, XJ, XD Equivalent platform assembler syntax: vilv{l/h}.{b/h/w/d} vd, vj, vk xvilv{l/h}.{b/h/w/d} xd, xj, xk Change-Id: I40e21737649d9fdbbc9a423e859f4c0a56d069fb Reviewed-on: https://go-review.googlesource.com/c/go/+/635936 Reviewed-by: Junyang Shao <shaojunyang@google.com> Reviewed-by: Michael Pratt <mpratt@google.com> Reviewed-by: abner chenc <chenguoqi@loongson.cn> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Auto-Submit: Junyang Shao <shaojunyang@google.com>
2025-03-05cmd/internal/obj/loong64: add {V,XV}{ADD/SUB}.{B,H,W,D,Q} and ↵Xiaolin Zhao
{V,XV}{ADD/SUB}.{B,H,W,D}Uinstructions support Go asm syntax: V{ADD/SUB}{B,H,W,V,Q} VK, VJ, VD XV{ADD/SUB}{B,H,W,V,Q} XK, XJ, XD V{ADD/SUB}{B,H,W,V}U $1, VJ, VD XV{ADD/SUB}{B,H,W,V}U $1, XJ, XD Equivalent platform assembler syntax: v{add/sub}.{b,h,w,d,q} vd, vj, vk xv{add/sub}.{b,h,w,d,q} xd, xj, xk v{add/sub}i.{b,h,w,d}u vd, vj, $1 xv{add/sub}i.{b,h,w,d}u xd, xj, $1 Change-Id: Ia1ef0bc062f4403bb0b1514c2cf1c0264f5d22ee Reviewed-on: https://go-review.googlesource.com/c/go/+/637795 Reviewed-by: Meidan Li <limeidan@loongson.cn> Reviewed-by: Junyang Shao <shaojunyang@google.com> Reviewed-by: Michael Pratt <mpratt@google.com> Reviewed-by: abner chenc <chenguoqi@loongson.cn> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2025-03-03cmd/internal/obj/loong64: add F{MAXA/MINA}.{S/D} instructionsXiaolin Zhao
Go asm syntax: F{MAXA/MINA}{F/D} FK, FJ, FD Equivalent platform assembler syntax: f{maxa/mina}.{s/d} fd, fj, fk Ref: https://loongson.github.io/LoongArch-Documentation/LoongArch-Vol1-EN.html Change-Id: I6790657d2f36bdf5e6818b6c0aaa48117e782b8d Reviewed-on: https://go-review.googlesource.com/c/go/+/653915 Reviewed-by: abner chenc <chenguoqi@loongson.cn> Reviewed-by: Michael Pratt <mpratt@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Junyang Shao <shaojunyang@google.com>
2025-03-03cmd/internal/obj/loong64: add {V,XV}{SLL/SRL/SRA/ROTR}[I].{B/H/W/D} ↵Xiaolin Zhao
instructions support Go asm syntax: V{SLL/SRL/SRA/ROTR}{B/H/W/V} $1, V2, V3 XV{SLL/SRL/SRA/ROTR}{B/H/W/V} $1, X2, X3 V{SLL/SRL/SRA/ROTR}{B/H/W/V} VK, VJ, VD XV{SLL/SRL/SRA/ROTR}{B/H/W/V} XK, XJ, XD Equivalent platform assembler syntax: v{sll/srl/sra/rotr}i.{b/h/w/d} v3, v2, $1 xv{sll/srl/sra/rotr}i.{b/h/w/d} x3, x2, $1 v{sll/srl/sra/rotr}.{b/h/w/d} vd, vj, vk xv{sll/srl/sra/rotr}.{b/h/w/d} xd, xj, xk Change-Id: Ie4f04de1c77491a71688d226f7d91cd1a699ab47 Reviewed-on: https://go-review.googlesource.com/c/go/+/637775 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Meidan Li <limeidan@loongson.cn> Reviewed-by: abner chenc <chenguoqi@loongson.cn> Reviewed-by: Junyang Shao <shaojunyang@google.com> Reviewed-by: Michael Pratt <mpratt@google.com>
2025-03-01cmd/internal/obj/riscv: add riscv64 CSR mapMark Ryan
The map is automatically generated by running the latest version of parse.py from github.com/riscv/riscv-opcodes. Change-Id: I05e00ab27ec583750752c25e1835c2578b339fbf Reviewed-on: https://go-review.googlesource.com/c/go/+/630518 Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Pengcheng Wang <wangpengcheng.pp@bytedance.com> Reviewed-by: Meng Zhuo <mengzhuo1203@gmail.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Joel Sing <joel@sing.id.au> Reviewed-by: Michael Pratt <mpratt@google.com>
2025-02-28cmd/internal/obj, cmd/asm: reclassify 32-bit immediate value of loong64limeidan
Change-Id: If9fd257ca0837a8c8597889c4f5ed3d4edc602c1 Reviewed-on: https://go-review.googlesource.com/c/go/+/636995 Reviewed-by: abner chenc <chenguoqi@loongson.cn> Reviewed-by: Junyang Shao <shaojunyang@google.com> Reviewed-by: sophie zhao <zhaoxiaolin@loongson.cn> Reviewed-by: Michael Pratt <mpratt@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2025-02-27cmd/internal/obj/riscv: implement vector load/store instructionsJoel Sing
Implement vector unit stride, vector strided, vector indexed and vector whole register load and store instructions. The vector unit stride instructions take an optional vector mask register, which if specified must be register V0. If only two operands are given, the instruction is encoded as unmasked. The vector strided and vector indexed instructions also take an optional vector mask register, which if specified must be register V0. If only three operands are given, the instruction is encoded as unmasked. Cq-Include-Trybots: luci.golang.try:gotip-linux-riscv64 Change-Id: I35e43bb8f1cf6ae8826fbeec384b95ac945da50f Reviewed-on: https://go-review.googlesource.com/c/go/+/631937 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Mark Ryan <markdryan@rivosinc.com> Reviewed-by: Michael Knyszek <mknyszek@google.com> Reviewed-by: Meng Zhuo <mengzhuo1203@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Pengcheng Wang <wangpengcheng.pp@bytedance.com>
2025-02-26runtime: remove ret field from gobufKeith Randall
It's not used for anything. Change-Id: I031b3cdfe52b6b1cff4b3cb6713ffe588084542f Reviewed-on: https://go-review.googlesource.com/c/go/+/652276 Reviewed-by: Keith Randall <khr@google.com> Reviewed-by: David Chase <drchase@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2025-02-25cmd/compile, runtime: use PC of deferreturn for panic transferDavid Chase
this removes the old conditional-on-register-value handshake from the deferproc/deferprocstack logic. The "line" for the recovery-exit frame itself (not the defers that it runs) is the closing brace of the function. Reduces code size slightly (e.g. go command is 0.2% smaller) Sample output showing effect of this change, also what sort of code it requires to observe the effect: ``` package main import "os" func main() { g(len(os.Args) - 1) // stack[0] } var gi int var pi *int = &gi //go:noinline func g(i int) { switch i { case 0: defer func() { println("g0", i) q() // stack[2] if i == 0 }() for j := *pi; j < 1; j++ { defer func() { println("recover0", recover().(string)) }() } default: for j := *pi; j < 1; j++ { defer func() { println("g1", i) q() // stack[2] if i == 1 }() } defer func() { println("recover1", recover().(string)) }() } p() } // stack[1] (deferreturn) //go:noinline func p() { panic("p()") } //go:noinline func q() { panic("q()") // stack[3] } /* Sample output for "./foo foo": recover1 p() g1 1 panic: q() goroutine 1 [running]: main.q() .../main.go:46 +0x2c main.g.func3() .../main.go:29 +0x48 main.g(0x1?) .../main.go:37 +0x68 main.main() .../main.go:6 +0x28 */ ``` Change-Id: Ie39ea62ecc244213500380ea06d44024cadc2317 Reviewed-on: https://go-review.googlesource.com/c/go/+/650795 Reviewed-by: Cherry Mui <cherryyz@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2025-02-22cmd: initial compiler+linker support for DWARF5 .debug_addrThan McIntosh
This patch rolls the main .debug_info DWARF section from version 4 to version 5, and also introduces machinery in the Go compiler and linker for taking advantage of the DWARF5 ".debug_addr" section for subprogram DIE "high" and "low" PC attributes. All functionality is gated by GOEXPERIMENT=dwarf5. For the compiler portion of this patch, we add a new DIE attribute form "DW_FORM_addrx", which accepts as an argument a function (text) symbol. The dwarf "putattr" function is enhanced to handle this format by invoking a new dwarf context method "AddIndirectTextRef". Under the hood, this method invokes the Lsym method WriteDwTxtAddrx, which emits a new objabi.R_DWTXTADDR_* relocation. The size of the relocation is dependent on the number of functions in the package; we pick a size that is just big enough for the largest func index. In the linker portion of this patch, we now switch over to writing out a version number of 5 (instead of 4) in the compile unit header (this is required if we want to use addrx attributes). In the parallel portion of DWARF gen, within each compilation unit we scan subprogram DIEs to look for R_DWTXTADDR_* relocations, and when we find such a reloc, we assign a slot in the .debug_addr section for the func targeted. After the parallel portion is complete, we then walk through all of the compilation units to assign a value to their DW_AT_addr_base attribute, which points to the portion of the single .debug_addr section containing the text addrs for that compilation unit. Note that once this patch is in, programs built with GOEXPERIMENT=dwarf5 will have broken/damaged DWARF info; in particular, since we've changed only the CU and subprogram DIEs and haven't incorported the other changes mandated by DWARF5 (ex: .debug_ranges => .debug_rnglists) a lot of the variable location info will be missing/incorrect. This will obviously change in subsequent patches. Note also that R_DWTXTADDR_* can't be used effectively for lexical scope DIE hi/lo PC attrs, since there isn't a viable way to encode "addrx + constant" in the attribute value (you would need a new entry for each attr endpoint in .debug_addr, which would defeat the point). Updates #26379. Change-Id: I2dfc45c9a8333e7b2a58f8e3b88fc8701fefd006 Reviewed-on: https://go-review.googlesource.com/c/go/+/635337 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: David Chase <drchase@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
2025-02-20cmd/internal/obj/loong64: optimize immediate loadinglimeidan
| old | new | | sec/op | sec/op vs base | BinaryTree17 11.08 ± 2% 11.16 ± 1% ~ (p=0.529 n=10) Fannkuch11 2.716 ± 0% 2.737 ± 0% +0.79% (p=0.000 n=10) FmtFprintfEmpty 67.37n ± 0% 66.42n ± 0% -1.41% (p=0.000 n=10) FmtFprintfString 95.28n ± 0% 90.85n ± 0% -4.64% (p=0.000 n=10) FmtFprintfInt 97.69n ± 0% 98.06n ± 0% +0.38% (p=0.000 n=10) FmtFprintfIntInt 149.1n ± 0% 147.4n ± 0% -1.14% (p=0.000 n=10) FmtFprintfPrefixedInt 223.6n ± 0% 196.5n ± 0% -12.10% (p=0.000 n=10) FmtFprintfFloat 290.9n ± 0% 281.6n ± 1% -3.21% (p=0.000 n=10) FmtManyArgs 670.6n ± 0% 642.6n ± 0% -4.18% (p=0.000 n=10) GobDecode 10.26m ± 1% 10.23m ± 1% ~ (p=0.105 n=10) GobEncode 12.09m ± 1% 11.94m ± 1% -1.24% (p=0.000 n=10) Gzip 316.9m ± 0% 315.9m ± 0% -0.32% (p=0.001 n=10) Gunzip 65.48m ± 0% 59.77m ± 0% -8.72% (p=0.000 n=10) HTTPClientServer 70.36µ ± 0% 68.72µ ± 0% -2.34% (p=0.000 n=10) JSONEncode 13.61m ± 1% 13.19m ± 1% -3.13% (p=0.000 n=10) JSONDecode 57.52m ± 1% 54.15m ± 1% -5.86% (p=0.000 n=10) Mandelbrot200 4.577m ± 0% 4.572m ± 0% -0.10% (p=0.002 n=10) GoParse 6.466m ± 0% 6.363m ± 0% -1.58% (p=0.000 n=10) RegexpMatchEasy0_32 89.20n ± 0% 87.72n ± 0% -1.65% (p=0.000 n=10) RegexpMatchEasy0_1K 748.6n ± 0% 907.6n ± 0% +21.22% (p=0.000 n=10) RegexpMatchEasy1_32 94.14n ± 0% 93.81n ± 0% -0.35% (p=0.000 n=10) RegexpMatchEasy1_1K 832.1n ± 0% 953.6n ± 0% +14.59% (p=0.000 n=10) RegexpMatchMedium_32 982.7n ± 0% 1018.0n ± 0% +3.59% (p=0.000 n=10) RegexpMatchMedium_1K 30.51µ ± 0% 30.00µ ± 0% -1.65% (p=0.000 n=10) RegexpMatchHard_32 1.721µ ± 0% 1.664µ ± 0% -3.34% (p=0.000 n=10) RegexpMatchHard_1K 50.76µ ± 0% 50.92µ ± 0% +0.32% (p=0.000 n=10) Revcomp 870.5m ± 0% 710.5m ± 0% -18.38% (p=0.000 n=10) Template 93.18m ± 1% 93.67m ± 1% ~ (p=0.123 n=10) TimeParse 309.2n ± 0% 307.8n ± 0% -0.45% (p=0.000 n=10) TimeFormat 401.5n ± 0% 394.2n ± 0% -1.82% (p=0.000 n=10) geomean 72.73µ 71.70µ -1.41% Change-Id: Id8d342ef3bb82a420434b2b841674683efef67be Reviewed-on: https://go-review.googlesource.com/c/go/+/620737 Reviewed-by: Cherry Mui <cherryyz@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: abner chenc <chenguoqi@loongson.cn> Reviewed-by: sophie zhao <zhaoxiaolin@loongson.cn>
2025-02-20cmd/internal/obj/x86: add support for the endbr64 instruction on amd64Joel Sing
Add support for endbr64, which terminates an indirect branch in 64-bit mode. This is effectively used to mark locations where an indirect branch is permitted to land, when Indirect Branch Tracking (IBT) is enforced on Intel CPUs. Updates #66054 Change-Id: Ib898031711cfaaa6e05c197bfe727ded0bce6f52 Reviewed-on: https://go-review.googlesource.com/c/go/+/649215 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Michael Knyszek <mknyszek@google.com> Reviewed-by: Keith Randall <khr@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
2025-02-19all: implement plugin build mode for riscv64Meng Zhuo
Change-Id: I8d7bbeebbf4a46f2fd8d630b1edbaf79b8ffccc5 Reviewed-on: https://go-review.googlesource.com/c/go/+/420114 Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Michael Knyszek <mknyszek@google.com> Reviewed-by: Joel Sing <joel@sing.id.au> TryBot-Bypass: Joel Sing <joel@sing.id.au>
2025-02-14cmd/internal/obj/riscv: support MOVD with floating point constantsJoel Sing
Currently, we only support loading of values from memory (or other registers). Add floating point constant support to MOVD. This is implemented by storing the floating point constant to a symbol, which is then loaded into the floating point register. Change-Id: I6db242d27f606f0d5d084a3ab93538698d3a4f8c Reviewed-on: https://go-review.googlesource.com/c/go/+/631876 Reviewed-by: Meng Zhuo <mengzhuo1203@gmail.com> Reviewed-by: Mark Ryan <markdryan@rivosinc.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Cherry Mui <cherryyz@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2025-02-14cmd/asm,cmd/internal/obj/riscv: implement vector configuration setting ↵Joel Sing
instructions Implement vector configuration setting instructions (VSETVLI, VSETIVLI, VSETL). These allow the vector length (vl) and vector type (vtype) CSRs to be configured via a single instruction. Unfortunately each instruction has its own dedicated encoding. In the case of VSETVLI/VSETIVLI, the vector type is specified via a series of special operands, which specify the selected element width (E8, E16, E32, E64), the vector register group multiplier (M1, M2, M4, M8, MF2, MF4, MF8), the vector tail policy (TU, TA) and vector mask policy (MU, MA). Note that the order of these special operands matches non-Go assemblers. Partially based on work by Pengcheng Wang <wangpengcheng.pp@bytedance.com>. Cq-Include-Trybots: luci.golang.try:gotip-linux-riscv64 Change-Id: I431f59c1e048a3e84754f0643a963da473a741fe Reviewed-on: https://go-review.googlesource.com/c/go/+/631936 Reviewed-by: Mark Ryan <markdryan@rivosinc.com> Reviewed-by: Meng Zhuo <mengzhuo1203@gmail.com> Reviewed-by: Cherry Mui <cherryyz@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
2025-02-13cmd/link,cmd/internal: add R_GOT_PCREL_ITYPE_RELOC for riscv64Meng Zhuo
This CL adds new relocation type for riscv64: R_GOT_PCREL_ITYPE_RELOC which generate an AUIPC + I-type pair with relocation type of GOT_HI20 and PCREL_LO12_I. According to RISCV elf psabi doc, medium position independent code model, the GNU as example is: ``` # Calculate address of non-local symbol .Ltmp3: aupipc a0, %got_pcrel_hi(symbol) ld a0, %pcrel_lo(.Ltmp3)(a0) ``` Change-Id: I719dd05e009ca2d9291f0689b346c059f9c56918 Reviewed-on: https://go-review.googlesource.com/c/go/+/612635 Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Joel Sing <joel@sing.id.au> Reviewed-by: Cherry Mui <cherryyz@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2025-02-13cmd: use cmd/internal/hash.New32 and Sum32 onlyRuss Cox
Do not use New16, New20, Sum16, Sum20 anymore. As of CL 641096, these are just wrappers around New32 and Sum32. Change call sites to use them directly. Change-Id: Icea91a77449f6839b903894997057ba404bd04e0 Reviewed-on: https://go-review.googlesource.com/c/go/+/641076 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Keith Randall <khr@google.com> Reviewed-by: Cherry Mui <cherryyz@google.com> Auto-Submit: Russ Cox <rsc@golang.org> Reviewed-by: Keith Randall <khr@golang.org>
2025-02-10cmd/compile: avoid ifaceeq call if we know the interface is directKeith Randall
We can just use == if the interface is direct. Fixes #70738 Change-Id: Ia9a644791a370fec969c04c42d28a9b58f16911f Reviewed-on: https://go-review.googlesource.com/c/go/+/635435 Auto-Submit: Keith Randall <khr@golang.org> Reviewed-by: Cuong Manh Le <cuong.manhle.vn@gmail.com> Reviewed-by: Cherry Mui <cherryyz@google.com> Reviewed-by: David Chase <drchase@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2025-02-07cmd/internal/obj/riscv: update references to RISC-V specificationJoel Sing
Update references to version 20240411 of the RISC-V specifications. Reorder and regroup instructions to maintain ordering. Also be consistent with formatting. The instruction encodings table was seemingly missed in CL 616115. Change-Id: I47b7c8538383ff3b0503ba59db570c3d4f0d5653 Reviewed-on: https://go-review.googlesource.com/c/go/+/631935 Reviewed-by: Cherry Mui <cherryyz@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Ian Lance Taylor <iant@google.com> Reviewed-by: Meng Zhuo <mengzhuo1203@gmail.com> Reviewed-by: Pengcheng Wang <wangpengcheng.pp@bytedance.com>