aboutsummaryrefslogtreecommitdiff
path: root/internal/database/database_test.go
AgeCommit message (Collapse)Author
2026-03-26all: fix rangeintHana Kim
Change-Id: I58ed5e7d2a3bbc13bc389ac7239737e18856dbfe Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/753427 Reviewed-by: Jonathan Amsterdam <jba@google.com> Reviewed-by: Ethan Lee <ethanalee@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2026-03-24all: pgx/v4 to v5 and upgrade pqEthan Lee
- In preparation for upgrade to PostgreSQL 17, upgrade stale dependencies within pkgsite. Change-Id: Id54cbb46182711b9808ee3bbbe2f0d5d9a04583d Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/758121 Auto-Submit: Ethan Lee <ethanalee@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Jonathan Amsterdam <jba@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2026-03-05internal/database: remove sawRetries logic from TestTransactSerializableEthan Lee
- This test has a history of flakiness. PostgreSQL 14 includes significant improvements to its Serializable Snapshot Isolation which means it is much efficient in reducing false positive serialization conflicts. To amend this, this test now focuses on data integrity rather than an arbitary retry counter. Change-Id: I7d9a46303a6cc4eb011d9d3764ebf0c61e2d6287 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/751680 Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com> kokoro-CI: kokoro <noreply+kokoro@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Auto-Submit: Ethan Lee <ethanalee@google.com>
2023-08-23all: remove arbitrary hard-coded timeouts in testsBryan C. Mills
If a test times out, that implies that it got stuck on something. By default, the Go testing package dumps goroutines when its own timeout is passed, which prints a goroutine dump, helping to reveal what was stuck. Adding an arbitrary timeout on top of the testing package's own timeout is, in my experience, almost always counterproductive. If the arbitrary timeout catches a real hang, it causes the test to fail instead of dumping goroutines, making it much harder to see what was stuck. On the other hand, if the timeouts are set aggressively enough to make the test fail early, they are often too aggressive for CI testing, causing flakes that then have to be triaged on an ongoing basis. On balance, the value of saving a minute or two for developers who have introduced a hang is not worth the cost of suppressing debugging information and causing flakes that have to be triaged. Fixes #61556. For #59347. Change-Id: I0263d0d9b18283470f100e5a0155818b87b5312f Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/521837 TryBot-Result: Gopher Robot <gobot@golang.org> Auto-Submit: Bryan Mills <bcmills@google.com> Run-TryBot: Bryan Mills <bcmills@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Michael Matloob <matloob@golang.org>
2022-11-15all: convert interface{} to anyHana (Hyang-Ah) Kim
Change-Id: I1f3b7cc8899c7707abb01e3d14807c37c3451382 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/449695 TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Run-TryBot: Hyang-Ah Hana Kim <hyangah@gmail.com>
2022-04-21internal: remove experimentsJonathan Amsterdam
Remove the deprecated-doc and deps-dev-link experiments, which have been fully rolled out. I found I also had to increase some parameters on the serializablity test in internal/database to make it pass on my machine. Reasons unknown. Change-Id: Ibcb1f8167474884d6c029da55f2d31b3c5adcfc6 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/401237 Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2022-03-07internal/database: use generics for Collect functionsJonathan Amsterdam
Reimplement some reflection-based collection functions to use generics. In some cases we still need reflection, but at least we can provide a type-safe wrapper with generics. Change-Id: Id95949a7a22ee687166ecdfc1191150d79568889 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/389657 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-07-21internal/{database,postgres}: export CollectStringsJonathan Amsterdam
Export the CollectStrings function and move to the database package so it can be used elsewhere. Change-Id: I1032e4f5042209e42e8f0de9cebfb48653d56f69 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/336209 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-07-16internal/database: merge dbtest into packageJulie Qiu
There doesn't seem to be any reason that dbtest needs to be its own package, and the functions there are useful for scripts beyond use in Go test files. db_test.go is merged into the internal/datbase package and the file is renamed to dbutil.go. References are fixed. Change-Id: I5ed295f147c5b9e891c243e2c5028708b12f5c75 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/335089 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-06-18internal/database: add ability to run queries incrementallyJonathan Amsterdam
Postgres lets you define a cursor and fetch rows from it as top-level statements, without needing to write any PS-SQL. See https://www.postgresql.org/docs/11/sql-declare.html and https://www.postgresql.org/docs/11/sql-fetch.html for details. Use this feature to define RunQueryIncrementally, which repeatedly fetches query rows in batches until it runs out or the passed function says it's done. If grouping by module paths is enabled, use RunQueryIncrementally with a very large limit to read rows until we've seen a page's worth of module paths. This CL doesn't handle pagination correctly. Any page after the first is going to be wrong. Change-Id: Idf8233160b0cf74412a688e1a6b95f4f2b720008 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/329469 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-06-10internal/database: fix flaky test (again)Jonathan Amsterdam
The TransactSerializable test flaked about one time in fifty. Retry it to deflake. Now it doesn't fail even with -count 1000 on my machine. Change-Id: I61e1f115dfe479d12f848dc4596572804cbbc19c Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/326451 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-06-08internal/database: fix flaky testJonathan Amsterdam
Reduce the number of concurrent transactions so that there is always a conflict, but they don't reach the max number of retries. Change-Id: I15bb6c710473cbddcf070517f0eb907d708a46a7 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/325699 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-06-01internal/database: reduce max retries on serialization failureJonathan Amsterdam
If a save-module transaction fails due to serialization, then most likely another version of the same module is being processed at the same time. The right thing to do is to retry the transaction, and we do, up to 30 times. But if many versions are being reprocessed at once--as seems to be frequently the case now--we are just wasting time and compute resources, because we retry immediately, without any backoff. It would be better to fail sooner, and let the natural exponential backoff of the task queue do the retrying for us. We don't want to eliminate the immediate retry completely, but reduce the count to 10. Change-Id: I89f1cb8a9c021083f838c6659d3e5b39226cb144 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/323411 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2020-12-02internal/database: add a test showing that pgx copy only insertsJonathan Amsterdam
The fast pgx CopyFrom function can only insert new rows; it does not upsert (that is, replace existing rows). A new test demonstrates that by copying a row with an existing primary key, and getting a constraint violation. This means, unfortunately, that we can't use copy for anything where we could really use the extra speed. However, it's still worth switching to pgx because it's still a bit faster than lib/pq, and it's better maintained. Change-Id: Ib11c6a80fc04cdc88d59f48f51b175b0722c4e65 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/274215 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2020-11-24internal/database: support pgx driverJonathan Amsterdam
Test with both the postgres and pgx drivers. Recognize the pgx error type as well as the pq error type. Change-Id: If290e3e16013fd40ce2899b1cbae9fc0c9416ebd Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/272786 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2020-11-24internal/database: benchmark pgx copy against bulk insertJonathan Amsterdam
Add a benchmark which demonstrates that pgx's CopyFrom method is significantly faster than doing a bulk insert with pq. In this simple example, it is 55 times faster (!). For golang/go#42707 Change-Id: I8bb37ba95b6207ead320b9720fbcc5a2fb8c8514 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/272506 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2020-11-20internal: rename tc to testJulie Qiu
The variable tc is renamed to test when used to refer to a test case, for consistency throughout the rest of the codebase, which uses the variable test in most cases. Change-Id: I88a87aa020928af98e8579a299f3111718dfa044 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/271809 Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com> Trust: Julie Qiu <julie@golang.org>
2020-06-22all.bash,internal: force DB testing in CIJonathan Amsterdam
If the environment variable GO_DISCOVERY_TESTDB is "true" wherever we would skip DB tests, fail instead. Set it to "true" in all.bash. Fixes golang/go#39713. Change-Id: I80fe27578a05610e29cc3ad9f6d477c9768fd51a Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/239257 Reviewed-by: Julie Qiu <julie@golang.org>
2020-06-17internal/config: remove globalJonathan Amsterdam
Remove the `cfg` global and the functions that accessed it. Fixes b/145301722. Change-Id: I58ab9fbd4fc29f66dbc5b120f04c88ee0703ee57 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/238437 Reviewed-by: Julie Qiu <julie@golang.org>
2020-06-16internal: link to postgres setup doc if no DBJonathan Amsterdam
Change-Id: I2d8bef8f8961f148321d0f87cadf734441bc396b Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/238178 Reviewed-by: Julie Qiu <julie@golang.org>
2020-06-16internal: skip tests if no DBJonathan Amsterdam
Fixes golang/go#39598. Change-Id: Iae5a4174ca392009a9c3290c096eb8711e6ad522 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/238177 Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
2020-06-09internal/database: support bulk upsertJonathan Amsterdam
Add DB.BulkUpsert, which adds an ON CONFLICT clause to the INSERT that replaces existing column values. Change-Id: I59f36be0bcb0c0854f42da489e265f2a1396c439 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/766360 Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-06-09internal/database: BulkInsert: only prepare when necessaryJonathan Amsterdam
We were preparing the full statement (of size stride) even if we never used it, because the number of values to insert was less than stride. Also, tweak TestLargeBulkInsert: - the logging is no longer voluminous, so we don't have to turn it off; - make the number of values not divisible by 1000, so we test the case where rightBound > len(values). Change-Id: I169130ab370b192a629025886173b1e33f5d158b Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/766240 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-06-04internal/database: use prepared statement for BulkInsertJonathan Amsterdam
Using a prepared statement for the query seems to speed up an insert significantly. Change-Id: I32a1455b376e08fb435b2e756aa8df4b0c2bc1b3 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/761661 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-06-02internal/database: Transact supports any isolation levelJonathan Amsterdam
- Add an arg to Transact for the isolation level - Remove TransactSerializable This makes it possible to use other levels, and makes it easier to see which level is being used for each transaction. Change-Id: Iba5e2920b4139e5e2f0f8c6b331a658d7c84f60f Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/758942 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-05-21internal/database: support serializable transactionsJonathan Amsterdam
Add DB.TransactSerializable, which executes a transaction with serializable isolation. Although serializable transactions are more expensive, they reduce the risk for anomalies, like the constraint violations we sometimes see. This CL does not change any existing behavior. In particular, the DB.Transact method, despite calling sql.DB.BeginTx instead of sql.DB.Begin, uses the background context exactly as Begin does (source: https://github.com/golang/go/blob/go1.14.3/src/database/sql/sql.go#L1689). Change-Id: Iab9c99ca35de9ef884b149a3b39ecac3300ecac0 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/750952 Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-04-23all: rename module to golang.org/x/pkgsiteJulie Qiu
golang.org/x/discovery is renamed to golang.org/x/pkgsite. When the repository is open sourced, it will be hosted at go.googlesource.com/pkgsite. Change-Id: Ifc3b45b771a385b99179e785447f2a87afcacf87 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/724273 Reviewed-by: Jonathan Amsterdam <jba@google.com>
2020-04-16internal/database: support returning values from a bulk insertJonathan Amsterdam
Add DB.BulkInsertReturning, which supports the INSERT ... RETURNING feature. Change-Id: I8b7ca21295addde1ef29331d6f3d587bd848b4fd Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/719741 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-04-16internal/database: log transaction start and endJonathan Amsterdam
This requires adding a context parameter to the Transact method. Change-Id: I98e8b9cbd8ce724cbed693d7549e82cf8ee75990 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/719480 Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-04-16internal/database: support bulk updatesJonathan Amsterdam
Implement BulkUpdate. See the function's doc comment for details. Change-Id: I050227a16ebe2f93cfc535fe95d1cd310fd6d00b Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/716478 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-04-14internal/database: use the DB for inside a transaction as well as outJonathan Amsterdam
The database.DB type now can represent a DB connection in the middle of a transaction. Such a DB is created only by calling DB.Transact. The resulting API is much simpler, since all the ...Tx methods disappear. Change-Id: I41afada87738e1eacdec2fcf115902edddeff867 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/716719 Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-03-27internal/database, internal/testing/dbtest: site-agnostic DB functionalityJonathan Amsterdam
Extract into a separate package the core functionality from internal/postgres that doesn't depend on our particular schema. This makes it available for other uses, like devtools commands and etl autocomplete. Do the same for testing functionality. We now have three packages where before we had only one: - internal/postgres: discovery-specific DB operations and test support - internal/database: discovery-agnostic DB operations - internal/testing/dbtest: discovery-agnostic DB test support Change-Id: I54c59aee328dae71ba6c77170a72e7a83da7c785 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/602327 Reviewed-by: Robert Findley <rfindley@google.com>
generated by cgit v1.3 (git 2.53.0) at 2026-04-15 08:36:41 +0000