aboutsummaryrefslogtreecommitdiff
path: root/internal/database
AgeCommit message (Collapse)Author
2026-03-26all: fix rangeintHana Kim
Change-Id: I58ed5e7d2a3bbc13bc389ac7239737e18856dbfe Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/753427 Reviewed-by: Jonathan Amsterdam <jba@google.com> Reviewed-by: Ethan Lee <ethanalee@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2026-03-24all: pgx/v4 to v5 and upgrade pqEthan Lee
- In preparation for upgrade to PostgreSQL 17, upgrade stale dependencies within pkgsite. Change-Id: Id54cbb46182711b9808ee3bbbe2f0d5d9a04583d Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/758121 Auto-Submit: Ethan Lee <ethanalee@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Jonathan Amsterdam <jba@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2026-03-17all: fix minmaxHana Kim
Change-Id: I1eacfe303e724fc4dfe371b10a3762627162285d Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/753425 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Ethan Lee <ethanalee@google.com>
2026-03-13all: fix inlineHana Kim
Change-Id: I12a964f825257f416724f4a2712c7f2332d3bcf1 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/753423 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Ethan Lee <ethanalee@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2026-03-12devtools/cmd/seeddb, internal/database: reduce contention and increase retriesHana Kim
Lower the concurrency limit of seeddb from 10 to 5 to reduce database contention. Increase the maximum number of transaction retries on serialization failure from 10 to 20 to improve reliability for large module insertions. Change-Id: Id57eea3bf7b7cf5554d4e99fbe68ac89bb355904 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/754540 kokoro-CI: kokoro <noreply+kokoro@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Ethan Lee <ethanalee@google.com>
2026-03-05internal/database: remove sawRetries logic from TestTransactSerializableEthan Lee
- This test has a history of flakiness. PostgreSQL 14 includes significant improvements to its Serializable Snapshot Isolation which means it is much efficient in reducing false positive serialization conflicts. To amend this, this test now focuses on data integrity rather than an arbitary retry counter. Change-Id: I7d9a46303a6cc4eb011d9d3764ebf0c61e2d6287 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/751680 Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com> kokoro-CI: kokoro <noreply+kokoro@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Auto-Submit: Ethan Lee <ethanalee@google.com>
2026-02-23internal/database: set db connection pool limitsHana Kim
Under high-load, the frontend can issue many queries and trigger many connections. This can exceed the Cloud SQL connection limit of 100 per instance enforced by the Cloud Run environment, even if the database itself has remaining global capacity. To prevent this, we introduce limits on the database connection pool. We also include settings for idle connections and connection lifetimes to ensure better resource management and connection rotation. Introduces new env vars: GO_DISCOVERY_DATABASE_MAX_OPEN_CONNS GO_DISCOVERY_DATABASE_MAX_IDLE_CONNS GO_DISCOVERY_DATABASE_CONN_MAX_LIFETIME GO_DISCOVERY_DATABASE_CONN_MAX_IDLE_TIME Updates SetPoolSettings to validate that MaxIdleConns does not exceed MaxOpenConns, providing a warning if it does and capping it. Change-Id: I74edac05c4a23102d64e74a180c661c223e1b757 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/747620 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Jonathan Amsterdam <jba@google.com> Reviewed-by: Nicholas Husin <nsh@golang.org> Reviewed-by: Nicholas Husin <husin@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2025-07-21internal/database: allow running db tests under different db nameJean Barkhuysen
On a mac, brew install postgresql installs postgres without the postgres user: it uses $USER instead. The current tests don't take that into account. This CL changes that by: - Amending the connecting URI to explicitly specify the db name (the current "" approach makes use of the postgres user, which by coincidence is the same name as the database we want: now instead of we specify the postgres user). - One user test is amended to use the env var provided user if necessary. Also adds some logging to make it more clear why failures around the URI are happening. Change-Id: I03d5b40a62d7034e00f45ba8151292688ab61de4 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/685995 Reviewed-by: Michael Knyszek <mknyszek@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Jonathan Amsterdam <jba@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Michael Pratt <mpratt@google.com>
2023-08-25internal/config: separate config initialization into serverconfigMichael Matloob
This change creates a new package that does config initialization and other GCP-specific operations that were previously done in package config, so that config can have no cloud dependencies. For golang/go#61399 Change-Id: I8d78294834e325b47d838892a1cef87003a4b90a Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/522516 Run-TryBot: Michael Matloob <matloob@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Robert Findley <rfindley@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2023-08-23all: remove arbitrary hard-coded timeouts in testsBryan C. Mills
If a test times out, that implies that it got stuck on something. By default, the Go testing package dumps goroutines when its own timeout is passed, which prints a goroutine dump, helping to reveal what was stuck. Adding an arbitrary timeout on top of the testing package's own timeout is, in my experience, almost always counterproductive. If the arbitrary timeout catches a real hang, it causes the test to fail instead of dumping goroutines, making it much harder to see what was stuck. On the other hand, if the timeouts are set aggressively enough to make the test fail early, they are often too aggressive for CI testing, causing flakes that then have to be triaged on an ongoing basis. On balance, the value of saving a minute or two for developers who have introduced a hang is not worth the cost of suppressing debugging information and causing flakes that have to be triaged. Fixes #61556. For #59347. Change-Id: I0263d0d9b18283470f100e5a0155818b87b5312f Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/521837 TryBot-Result: Gopher Robot <gobot@golang.org> Auto-Submit: Bryan Mills <bcmills@google.com> Run-TryBot: Bryan Mills <bcmills@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Michael Matloob <matloob@golang.org>
2023-07-18internal: remove repetitive wordscui fliter
Change-Id: Ie5a688c08ba46018f0fa8f5a1af495e9c29fd684 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/509695 TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org> Reviewed-by: Robert Findley <rfindley@google.com> Run-TryBot: Ian Lance Taylor <iant@golang.org> Auto-Submit: Ian Lance Taylor <iant@golang.org> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Bryan Mills <bcmills@google.com> Run-TryBot: Robert Findley <rfindley@google.com>
2023-04-27all: remove repeated definite articlescui fliter
Change-Id: I5c0291156ec8b1fa0a207c680d121f4b76a63b61 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/489616 Run-TryBot: Ian Lance Taylor <iant@google.com> Run-TryBot: shuang cui <imcusg@gmail.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Auto-Submit: Ian Lance Taylor <iant@google.com> Reviewed-by: Ian Lance Taylor <iant@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
2022-11-15all: convert interface{} to anyHana (Hyang-Ah) Kim
Change-Id: I1f3b7cc8899c7707abb01e3d14807c37c3451382 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/449695 TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Run-TryBot: Hyang-Ah Hana Kim <hyangah@gmail.com>
2022-04-21internal: remove experimentsJonathan Amsterdam
Remove the deprecated-doc and deps-dev-link experiments, which have been fully rolled out. I found I also had to increase some parameters on the serializablity test in internal/database to make it pass on my machine. Reasons unknown. Change-Id: Ibcb1f8167474884d6c029da55f2d31b3c5adcfc6 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/401237 Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2022-04-11all: gofmtRuss Cox
Gofmt to update doc comments to the new formatting. For golang/go#51082. Change-Id: Ia9e71e7ecac75822ff43d6c7e60f512442a5fa50 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/399617 Run-TryBot: Russ Cox <rsc@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Auto-Submit: Russ Cox <rsc@golang.org> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2022-03-07internal/database: use generics for Collect functionsJonathan Amsterdam
Reimplement some reflection-based collection functions to use generics. In some cases we still need reflection, but at least we can provide a type-safe wrapper with generics. Change-Id: Id95949a7a22ee687166ecdfc1191150d79568889 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/389657 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-09-20internal/database: add CollectIntsJulie Qiu
A CollectInts function is added as a shortcut for fetching IDs from a single column. Change-Id: Ide61b936382545a73dd9e7c623b6d26364cc8608 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/349895 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-09-15internal/database: back off serialization failure exponentiallyJonathan Amsterdam
When a transaction encounters a serialization failure, retry after waiting some time, instead of retrying immediately. Double the wait time on each failure. Change-Id: I2ca3252182115e78e00048aa229fd16dd427f61c Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/350111 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-08-06devtools/cmd/seeddb: process same module sequentiallyJonathan Amsterdam
Process multiple versions of the same module sequentially. Doing so in parallel causes DB contention because of the lock on the module path and other ordinary DB locks. This can cause a fetch to reach the limit on the number of transaction retries. Change-Id: Ibb58e583441e0b471a628edd1b64ec7f8010e174 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/340118 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-08-03internal/database: CASCADE search_documents DELETEsJulie Qiu
CASACDE deletes on search_documents to symbol_search_documents. Change-Id: I6b7ccfdc17fe58bb0f411c6e7a3f054eced51580 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/339150 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-08-02internal/database: change log to debug levelJulie Qiu
Since serialization failures are not real errors, downgrade these from warning to debug to reduce noise in the seeddb logs. Change-Id: I800e2a1588d57156b0ebe0f39c7c7ef677f3583b Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/338593 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-07-21internal/{database,postgres}: export CollectStringsJonathan Amsterdam
Export the CollectStrings function and move to the database package so it can be used elsewhere. Change-Id: I1032e4f5042209e42e8f0de9cebfb48653d56f69 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/336209 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-07-20tests/search: add setup for testing searchJulie Qiu
A method is added for testing symbol search. See tests/README.md for set up details and tests/search/scripts/symbolsearch.txt for an example. These tests run in the searchtests docker container and can be run using tests/search/run.sh test/search/run.sh do not pass yet, since the symbol search query needs to be fixed, but in the future they will run in CI. For golang/go#44142 Change-Id: I0a2d6818e9874385a0bd79d668cce63495a363c3 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/333934 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-07-19devtools/cmd/db: clean upJulie Qiu
* Add migrate commond to flag.Usage * Move each switch statement to a helper function for readability * Only try to connect to the database for truncating; the database won't exist if we are trying to create it * Don't fail we if try to create a database that exists * Don't fail we if try to drop a database that does not exist Change-Id: I8e06c76edaa6eaeb42fd1e95b8f084358306c283 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/335256 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-07-16devtools/cmd/db: setup db scriptsJulie Qiu
We currently have two different setups for our databases with local development, one using the CLI tools and one with Go scripts. Instead, a script is now added at devtools/cmd/db to consolidate places where logic is implemented. Existing scripts are still maintained for ergonomics, but they just invoke the relevant devtools/cmd/db command. The devtools/cmd/db script will also be used in a later CL to simplify the docker setup. We no longer need the seeddb and migrate containers, since they can be reduced to a Go binary. Change-Id: I6d9fcdd452ab024bcffd91bf3f6f15598f4e502b Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/333937 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-07-16internal/database,postgres: move TryToMigrateJulie Qiu
TryToMigrate is useful in db scripts and doesn't depend on anything in the postgres package, so it is moved to internal/database to group with the other dbutil functions. Change-Id: I03df86238723689a2c5c697c5cfe296dde8b5b0a Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/335090 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-07-16internal/database: merge dbtest into packageJulie Qiu
There doesn't seem to be any reason that dbtest needs to be its own package, and the functions there are useful for scripts beyond use in Go test files. db_test.go is merged into the internal/datbase package and the file is renamed to dbutil.go. References are fixed. Change-Id: I5ed295f147c5b9e891c243e2c5028708b12f5c75 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/335089 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-06-18internal/database: add ability to run queries incrementallyJonathan Amsterdam
Postgres lets you define a cursor and fetch rows from it as top-level statements, without needing to write any PS-SQL. See https://www.postgresql.org/docs/11/sql-declare.html and https://www.postgresql.org/docs/11/sql-fetch.html for details. Use this feature to define RunQueryIncrementally, which repeatedly fetches query rows in batches until it runs out or the passed function says it's done. If grouping by module paths is enabled, use RunQueryIncrementally with a very large limit to read rows until we've seen a page's worth of module paths. This CL doesn't handle pagination correctly. Any page after the first is going to be wrong. Change-Id: Idf8233160b0cf74412a688e1a6b95f4f2b720008 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/329469 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-06-14internal/database: add CopyInsertJonathan Amsterdam
Change-Id: I6a9619f9cfd9b86862a5638994ec399c5a01a35f Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/327670 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-06-10internal/database: fix flaky test (again)Jonathan Amsterdam
The TransactSerializable test flaked about one time in fifty. Retry it to deflake. Now it doesn't fail even with -count 1000 on my machine. Change-Id: I61e1f115dfe479d12f848dc4596572804cbbc19c Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/326451 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-06-08internal/database: fix flaky testJonathan Amsterdam
Reduce the number of concurrent transactions so that there is always a conflict, but they don't reach the max number of retries. Change-Id: I15bb6c710473cbddcf070517f0eb907d708a46a7 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/325699 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-06-01internal/database: reduce max retries on serialization failureJonathan Amsterdam
If a save-module transaction fails due to serialization, then most likely another version of the same module is being processed at the same time. The right thing to do is to retry the transaction, and we do, up to 30 times. But if many versions are being reprocessed at once--as seems to be frequently the case now--we are just wasting time and compute resources, because we retry immediately, without any backoff. It would be better to fail sooner, and let the natural exponential backoff of the task queue do the retrying for us. We don't want to eliminate the immediate retry completely, but reduce the count to 10. Change-Id: I89f1cb8a9c021083f838c6659d3e5b39226cb144 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/323411 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-05-14internal/database: remove duplicate space from commentsryoya
Change-Id: I9685f55616dec14aa8f329c577266f77107fb5d7 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/320009 Reviewed-by: Jamal Carvalho <jamal@golang.org> Reviewed-by: Julie Qiu <julie@golang.org>
2021-04-07internal/database: add DB.WithPGXConnJonathan Amsterdam
This function lets code outside the package use pgx-specific features like CopyFrom. Change-Id: I40d2c8b5d9a92f61f535fc25059fb976baf5d178 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/307870 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-04-06internal/database: CopyUpsert honors QueryLoggingDisabledJonathan Amsterdam
Don't log from CopyUpsert if query logging is disabled. Change-Id: Iee6d1998b6330294043d8f80ab5b5507651750ee Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/307471 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Reviewed-by: Julie Qiu <julie@golang.org>
2021-03-30internal/database: CopyUpsert: support dropping a columnJonathan Amsterdam
You can't do a CopyFrom on a table with a generated column: postgres complains about the column value being null. To fix, drop the column on the temporary table. Change-Id: Ia52f59af6d026b3fcdaafe3c7865a2eb85deb179 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/305830 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-03-29internal/database: improve temp table creationJonathan Amsterdam
- Use LIKE to copy a table's scheme, instead of a SELECT. - Drop the table on commit. Change-Id: I82f5bb64e99f7983498a7e81bdeba51025a49b2b Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/305529 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-03-25internal/database: add CopyUpsertJonathan Amsterdam
Add the CopyUpsert method, which uses an efficient Postgres protocol to insert rows. For this to work, we need the connection underlying a sql.Tx value. Since sql.Tx doesn't expose its connection, we create one explicitly in DB.transact. Change-Id: Ie48ce7a4318f4531d4756f779943188a6f0fb6cd Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/304631 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-03-25internal/database: wrap OpenCensus sql driverJonathan Amsterdam
The datbase/sql package provides a way to get at the underlying driver's connection. We will need that feature to access pgx's fast CopyFrom method. Unfortunately, we do not use the pgx driver directly. Instead, we wrap it in an OpenCensus driver for tracing and metrics. And that OpenCensus driver's connection does not allow us to dig into _it_ to retrieve the pgx connection that it wraps. Hence this dance: we must write our own driver and connection implementations. The driver return the connection, and the connection holds both the underlying (pgx) connection and the OpenCensus connection, to which it delegates. This CL is a no-op; everything behaves as before. But it lays the groundwork for a subsequent CL that will use pgx's CopyFrom. Change-Id: I5bf308aa23f07f20d1f6410ebb04cd6b9a5e0922 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/304630 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-03-25internal/database: fix benchmarkJonathan Amsterdam
Drop the table only if it exists. Change-Id: Id2d772fade7ab2624c6a155d0985f09284c734e4 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/304629 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com>
2021-03-16internal/database: downgrade retryable errors to warningsJonathan Amsterdam
If a DB statement that is inside a retryable transaction fails because of a serialization error, the the transaction will be retried. That is a normal condition, not an error, so don't log it as an error. Instead, log it as a warning so that if we ever exceed the max number of retries, we can easily find the problem. Change-Id: I481a0c900ccf12fc7d1c897a14b45ae5d2169529 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/302190 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-03-04internal/database: don't report errors in logQueryJonathan Amsterdam
Remove the call to the error reporting service in logQuery. It is at too low level: in transactions that are retried, it can report an error even though the error is a serialization failure that is normal and will cause a retry. Change-Id: Ia4d4ab354f68bff237adc93a5cf61ff7448a05af Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/298509 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-02-22internal/database: gather more info about serialization retryJonathan Amsterdam
A serialization failure isn't being retried for reasons I don't understand, so added logging to gather more information. Change-Id: Ifa2bc12d4caccde3d48520706692ad8f7bc75cd5 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/294951 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-02-12internal/database: catch missing retriable errorsJonathan Amsterdam
We've noticed that we aren't retrying errors that have the right serialization code. The likely reason is that the error type is not what we expect. Get information about the type so we can recognize it in the code. For golang/go#43899 Change-Id: I72aceb12002eecbf580654b77c68d845759e8182 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/291609 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-01-20internal/database: fix StructScanner for byte slicesJonathan Amsterdam
StructScanner was wrapping `[]byte` fields in pq.Array. It shouldn't. Change-Id: Id7267178d2be3c805c00f176a85a35217bd268ed Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/284237 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2020-12-17internal/derrors,etc.: report errors lower in the stackJonathan Amsterdam
The data we get from the errorreporting API isn't too useful because the call to Report happens high on the stack. Try to call Report closer to where the error happens. Change-Id: I101fb4de0892c5d9967f6eb46d7c9cbd72fb567e Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/278952 Trust: Jonathan Amsterdam <jba@google.com> Trust: Julie Qiu <julie@golang.org> Run-TryBot: Jonathan Amsterdam <jba@google.com> Run-TryBot: Julie Qiu <julie@golang.org> Reviewed-by: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com>
2020-12-02internal/database: add a test showing that pgx copy only insertsJonathan Amsterdam
The fast pgx CopyFrom function can only insert new rows; it does not upsert (that is, replace existing rows). A new test demonstrates that by copying a row with an existing primary key, and getting a constraint violation. This means, unfortunately, that we can't use copy for anything where we could really use the extra speed. However, it's still worth switching to pgx because it's still a bit faster than lib/pq, and it's better maintained. Change-Id: Ib11c6a80fc04cdc88d59f48f51b175b0722c4e65 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/274215 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2020-11-24internal/database: support pgx driverJonathan Amsterdam
Test with both the postgres and pgx drivers. Recognize the pgx error type as well as the pq error type. Change-Id: If290e3e16013fd40ce2899b1cbae9fc0c9416ebd Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/272786 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2020-11-24internal/database: benchmark pgx copy against bulk insertJonathan Amsterdam
Add a benchmark which demonstrates that pgx's CopyFrom method is significantly faster than doing a bulk insert with pq. In this simple example, it is 55 times faster (!). For golang/go#42707 Change-Id: I8bb37ba95b6207ead320b9720fbcc5a2fb8c8514 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/272506 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2020-11-20internal: rename tc to testJulie Qiu
The variable tc is renamed to test when used to refer to a test case, for consistency throughout the rest of the codebase, which uses the variable test in most cases. Change-Id: I88a87aa020928af98e8579a299f3111718dfa044 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/271809 Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com> Trust: Julie Qiu <julie@golang.org>
generated by cgit v1.3 (git 2.53.0) at 2026-04-15 10:52:00 +0000