aboutsummaryrefslogtreecommitdiff
path: root/devtools/cmd/seeddb/main.go
AgeCommit message (Collapse)Author
2026-03-24all: pgx/v4 to v5 and upgrade pqEthan Lee
- In preparation for upgrade to PostgreSQL 17, upgrade stale dependencies within pkgsite. Change-Id: Id54cbb46182711b9808ee3bbbe2f0d5d9a04583d Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/758121 Auto-Submit: Ethan Lee <ethanalee@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Jonathan Amsterdam <jba@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2026-03-13all: fix forvarHana Kim
Change-Id: I02d2be535d9e3ac1788306adcc78d997d0caf951 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/753422 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Ethan Lee <ethanalee@google.com>
2026-03-12devtools/cmd/seeddb, internal/database: reduce contention and increase retriesHana Kim
Lower the concurrency limit of seeddb from 10 to 5 to reduce database contention. Increase the maximum number of transaction retries on serialization failure from 10 to 20 to improve reliability for large module insertions. Change-Id: Id57eea3bf7b7cf5554d4e99fbe68ac89bb355904 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/754540 kokoro-CI: kokoro <noreply+kokoro@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Ethan Lee <ethanalee@google.com>
2025-01-15devtools/cmd/seeddb: apply -keep_going to versions checkJonathan Amsterdam
Some module proxies may return invalid responses for versions for some modules. This change helps skip through those, when the majority of other modules would succeed. Concrete example: given a GitHub repo with no tagged releases, when querying the official Go module proxy for versions at /v@/list, it returns 200 OK and no content (since there are no versions). The jFrog Artifactory go module proxy's behaviour, in contrast, is to return 404 NOT FOUND if there are no versions. That means that if you're using seeddb and any of your seed.txt modules does not have tagged versions, we fail-fast when we try to list that module's versions, despite using -keep_going. This change applies -keep_going to that section of code, so that we skip over the 404 errors. Original author: Jean Barkhuysen <jean.barkhuysen@gmail.com> Fixes #71140 Change-Id: I2ccbcc356c322deed81860ee92274fba04a079b2 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/641675 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Michael Knyszek <mknyszek@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Jean Barkhuysen <jean.barkhuysen@gmail.com>
2024-11-13devtools/cmd/seeddb: improve loggingJonathan Amsterdam
- Use the stdlib log package, to distinguish seeddb logs from pkgsite logs. - Log events as they happen, instead of at the end. Change-Id: Ia3877e821def72d89144ad50d2b6706bb531eb1c Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/626857 Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com> Reviewed-by: Robert Findley <rfindley@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2024-08-22devtools/cmd/seeddb: limit concurrencyJonathan Amsterdam
Limit the number of goroutines that are fetching modules at once, to reduce the chance of transactions timing out. We've been noticing kokoro failures due to that recently. I chose a limit of 10 arbitrarily. We'll see if it helps. I tried testing locally, but I don't get the same behavior as on kokoro. Also, use an errgroup with a context to clean up seeddb failures faster. Also, remove the seeddb flag permanently. Change-Id: Ib0244c76572c9f9e34064f99a05b26d70cc55aea Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/607755 Reviewed-by: Robert Findley <rfindley@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
2024-03-04tests/screentest: fix bug in scriptJonathan Amsterdam
Missing backslash was causing a panic. Change-Id: I630acd72a56a2d6242f64b17e6a4f944cb5b1c75 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/568239 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Michael Matloob <matloob@golang.org>
2023-08-29internal/proxy: inject transport in proxy.NewMichael Matloob
This removes the dependency from internal/proxy on ochttp. For golang/go#61399 Change-Id: Ib2a350151752dc18c9a6e03b38b2d423ed55f4a1 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/523513 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Run-TryBot: Michael Matloob <matloob@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Robert Findley <rfindley@google.com>
2023-08-29internal/source: inject *http.Client into source.NewClientMichael Matloob
This removes the dependency from package source onto ochttp. The users of source.NewClient that want an ochttp.Transport can set the transport on the *http.Client. For golang/go#61399 Change-Id: Ifb7126c482f664ee5a359f594d9324f0fd90f8b2 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/523510 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Robert Findley <rfindley@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Michael Matloob <matloob@golang.org> kokoro-CI: kokoro <noreply+kokoro@google.com>
2023-08-25internal/config: separate config initialization into serverconfigMichael Matloob
This change creates a new package that does config initialization and other GCP-specific operations that were previously done in package config, so that config can have no cloud dependencies. For golang/go#61399 Change-Id: I8d78294834e325b47d838892a1cef87003a4b90a Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/522516 Run-TryBot: Michael Matloob <matloob@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Robert Findley <rfindley@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2022-04-11all: gofmtRuss Cox
Gofmt to update doc comments to the new formatting. For golang/go#51082. Change-Id: Ia9e71e7ecac75822ff43d6c7e60f512442a5fa50 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/399617 Run-TryBot: Russ Cox <rsc@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Auto-Submit: Russ Cox <rsc@golang.org> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-09-29devtools/cmd/seeddb: support bypassing the license checkJonathan Amsterdam
For golang/go#48566 Change-Id: Ia775d7a37924ec509a782227c099055fe5ebfc0c Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/352900 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com>
2021-09-27devtools/cmd/seeddb: add -keep_goingJonathan Amsterdam
Add a flag that continues fetching modules even if there is an error. For golang/go#48587 Change-Id: I9df1070ee9bb0e206ddd569228239bde5bcb05cd Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/352489 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-08-11devtools/cmd/seeddb: process versions of the same module sequentiallyJonathan Amsterdam
This time for real. Before we were only processing the versions expanded from "@all" sequentially, so if there were two lines in the seed file M@v1.1.0 M@v1.2.0 they would still be processed concurrently. Now we handle that case as well. Change-Id: I6051ff622d68c316386e7c09de6ddf603c8ab273 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/341349 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-08-10internal: add ParseModverJonathan Amsterdam
Change-Id: I301e4b13b13dcc04afd9282ba4e4e9237097a458 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/340709 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-08-06devtools/cmd/seeddb: process same module sequentiallyJonathan Amsterdam
Process multiple versions of the same module sequentially. Doing so in parallel causes DB contention because of the lock on the module path and other ordinary DB locks. This can cause a fetch to reach the limit on the number of transaction retries. Change-Id: Ibb58e583441e0b471a628edd1b64ec7f8010e174 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/340118 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-08-05internal, etc.: factor out (module path, version) pairJonathan Amsterdam
A lot of code reinvents the type that is a pair of module path and version. Factor out to a single type. Change-Id: I729214ebc45c0d496b6b03a5f8a8c17410836572 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/340109 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-08-02devtools/cmd/seeddb: do not refetch existing modulesJulie Qiu
By default, the seeddb script no longer refetches modules if it already exists in the test database. This makes it faster to rerun the e2e and search tests locally, so that modules do not need to be refetched after each test run. The -refetch flag can be used to refetch existing modules. Change-Id: If503111202c987a56ccb888e89501730207b8168 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/338933 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-07-26internal: add ReadFileLinesJonathan Amsterdam
Factor out this commonly used function to the internal package. Also, update some comments in devtools/cmd/seeddb. Change-Id: I0276ce51d0dd1f4aafa2f6c562d1b83ce21a0d14 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/337330 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Reviewed-by: Julie Qiu <julie@golang.org>
2021-07-20devtools,tests: move seed filesJulie Qiu
Seed files are moved to the relevant tests/* directories, so that they are in the same location as other config files. Change-Id: I18771d15f5ef0d7cd7f3c271749a202962736f71 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/335392 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-07-19cmd,devtools,internal/config: set log level in config.InitJulie Qiu
log.SetLevel is now called in config.Init, so that it also applies to logs in the Init function. Change-Id: Id552923b4d81bb1678b8ffa72f1b1d858de559c1 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/335257 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-07-19devtools/cmd/seeddb: filter stdlib versionsJulie Qiu
It's relatively expensive to fetch a stdlib module version, so when std@all is specified, only fetch the minor versions (ones with a patch version of 0). Change-Id: Iabf31977ca068a61f7fbbcd38f6da25c14b65082 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/335133 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-07-12devtools/cmd/seeddb: support fetching all version of a moduleJulie Qiu
The seeddb script now supports syntax for fetching all versions of a module, by adding <module>@all to a seed file. Change-Id: Ia609df5033ac5522d0eb1d17d024453bd3b0e1bc Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/333156 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-07-12devtools: add GO_DISCOVERY_SEED_DB_FILE to customize seed fileJulie Qiu
The GO_DISCOVERY_SEED_DB_FILE variable is now used to determine which seeddb file should be read by the seedb container in docker-compose. This allows us to use different files for development and the e2e tests. The seedfile for e2e tests is renamed to e2e_test_modules.txt, to make it clear what the file was used for. In future CLs we will also have additional seed files. Change-Id: I5c3c64386f419b582b043d933482bcc84f186495 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/333153 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-07-12devtools/cmd/seeddb: support experimentsJulie Qiu
The seeddb script now supports experiments, which can be set using GO_DISCOVERY_CONFIG_DYNAMIC. Change-Id: I491776b3a4aca5646b613bdee393794d065e38a1 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/333154 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-07-12devtools/cmd/seeddb: remove extra seeddb directoryJulie Qiu
devtools/cmd/seeddb/seedb had an extra level of nesting that wasn't necessary. Change-Id: I997d97fba0339ca08645fa1a29ba7c3a40caf30c Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/333151 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
generated by cgit v1.3 (git 2.53.0) at 2026-04-15 10:27:24 +0000