aboutsummaryrefslogtreecommitdiff
path: root/internal/database/database.go
AgeCommit message (Collapse)Author
2026-03-24all: pgx/v4 to v5 and upgrade pqEthan Lee
- In preparation for upgrade to PostgreSQL 17, upgrade stale dependencies within pkgsite. Change-Id: Id54cbb46182711b9808ee3bbbe2f0d5d9a04583d Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/758121 Auto-Submit: Ethan Lee <ethanalee@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Jonathan Amsterdam <jba@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2026-03-17all: fix minmaxHana Kim
Change-Id: I1eacfe303e724fc4dfe371b10a3762627162285d Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/753425 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Ethan Lee <ethanalee@google.com>
2026-03-12devtools/cmd/seeddb, internal/database: reduce contention and increase retriesHana Kim
Lower the concurrency limit of seeddb from 10 to 5 to reduce database contention. Increase the maximum number of transaction retries on serialization failure from 10 to 20 to improve reliability for large module insertions. Change-Id: Id57eea3bf7b7cf5554d4e99fbe68ac89bb355904 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/754540 kokoro-CI: kokoro <noreply+kokoro@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Ethan Lee <ethanalee@google.com>
2026-02-23internal/database: set db connection pool limitsHana Kim
Under high-load, the frontend can issue many queries and trigger many connections. This can exceed the Cloud SQL connection limit of 100 per instance enforced by the Cloud Run environment, even if the database itself has remaining global capacity. To prevent this, we introduce limits on the database connection pool. We also include settings for idle connections and connection lifetimes to ensure better resource management and connection rotation. Introduces new env vars: GO_DISCOVERY_DATABASE_MAX_OPEN_CONNS GO_DISCOVERY_DATABASE_MAX_IDLE_CONNS GO_DISCOVERY_DATABASE_CONN_MAX_LIFETIME GO_DISCOVERY_DATABASE_CONN_MAX_IDLE_TIME Updates SetPoolSettings to validate that MaxIdleConns does not exceed MaxOpenConns, providing a warning if it does and capping it. Change-Id: I74edac05c4a23102d64e74a180c661c223e1b757 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/747620 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Jonathan Amsterdam <jba@google.com> Reviewed-by: Nicholas Husin <nsh@golang.org> Reviewed-by: Nicholas Husin <husin@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2022-11-15all: convert interface{} to anyHana (Hyang-Ah) Kim
Change-Id: I1f3b7cc8899c7707abb01e3d14807c37c3451382 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/449695 TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Run-TryBot: Hyang-Ah Hana Kim <hyangah@gmail.com>
2022-04-11all: gofmtRuss Cox
Gofmt to update doc comments to the new formatting. For golang/go#51082. Change-Id: Ia9e71e7ecac75822ff43d6c7e60f512442a5fa50 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/399617 Run-TryBot: Russ Cox <rsc@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Auto-Submit: Russ Cox <rsc@golang.org> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2022-03-07internal/database: use generics for Collect functionsJonathan Amsterdam
Reimplement some reflection-based collection functions to use generics. In some cases we still need reflection, but at least we can provide a type-safe wrapper with generics. Change-Id: Id95949a7a22ee687166ecdfc1191150d79568889 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/389657 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-09-20internal/database: add CollectIntsJulie Qiu
A CollectInts function is added as a shortcut for fetching IDs from a single column. Change-Id: Ide61b936382545a73dd9e7c623b6d26364cc8608 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/349895 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-09-15internal/database: back off serialization failure exponentiallyJonathan Amsterdam
When a transaction encounters a serialization failure, retry after waiting some time, instead of retrying immediately. Double the wait time on each failure. Change-Id: I2ca3252182115e78e00048aa229fd16dd427f61c Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/350111 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-07-21internal/{database,postgres}: export CollectStringsJonathan Amsterdam
Export the CollectStrings function and move to the database package so it can be used elsewhere. Change-Id: I1032e4f5042209e42e8f0de9cebfb48653d56f69 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/336209 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-06-18internal/database: add ability to run queries incrementallyJonathan Amsterdam
Postgres lets you define a cursor and fetch rows from it as top-level statements, without needing to write any PS-SQL. See https://www.postgresql.org/docs/11/sql-declare.html and https://www.postgresql.org/docs/11/sql-fetch.html for details. Use this feature to define RunQueryIncrementally, which repeatedly fetches query rows in batches until it runs out or the passed function says it's done. If grouping by module paths is enabled, use RunQueryIncrementally with a very large limit to read rows until we've seen a page's worth of module paths. This CL doesn't handle pagination correctly. Any page after the first is going to be wrong. Change-Id: Idf8233160b0cf74412a688e1a6b95f4f2b720008 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/329469 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-06-01internal/database: reduce max retries on serialization failureJonathan Amsterdam
If a save-module transaction fails due to serialization, then most likely another version of the same module is being processed at the same time. The right thing to do is to retry the transaction, and we do, up to 30 times. But if many versions are being reprocessed at once--as seems to be frequently the case now--we are just wasting time and compute resources, because we retry immediately, without any backoff. It would be better to fail sooner, and let the natural exponential backoff of the task queue do the retrying for us. We don't want to eliminate the immediate retry completely, but reduce the count to 10. Change-Id: I89f1cb8a9c021083f838c6659d3e5b39226cb144 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/323411 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-03-25internal/database: add CopyUpsertJonathan Amsterdam
Add the CopyUpsert method, which uses an efficient Postgres protocol to insert rows. For this to work, we need the connection underlying a sql.Tx value. Since sql.Tx doesn't expose its connection, we create one explicitly in DB.transact. Change-Id: Ie48ce7a4318f4531d4756f779943188a6f0fb6cd Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/304631 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-03-16internal/database: downgrade retryable errors to warningsJonathan Amsterdam
If a DB statement that is inside a retryable transaction fails because of a serialization error, the the transaction will be retried. That is a normal condition, not an error, so don't log it as an error. Instead, log it as a warning so that if we ever exceed the max number of retries, we can easily find the problem. Change-Id: I481a0c900ccf12fc7d1c897a14b45ae5d2169529 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/302190 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-02-22internal/database: gather more info about serialization retryJonathan Amsterdam
A serialization failure isn't being retried for reasons I don't understand, so added logging to gather more information. Change-Id: Ifa2bc12d4caccde3d48520706692ad8f7bc75cd5 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/294951 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-02-12internal/database: catch missing retriable errorsJonathan Amsterdam
We've noticed that we aren't retrying errors that have the right serialization code. The likely reason is that the error type is not what we expect. Get information about the type so we can recognize it in the code. For golang/go#43899 Change-Id: I72aceb12002eecbf580654b77c68d845759e8182 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/291609 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2020-11-24internal/database: support pgx driverJonathan Amsterdam
Test with both the postgres and pgx drivers. Recognize the pgx error type as well as the pq error type. Change-Id: If290e3e16013fd40ce2899b1cbae9fc0c9416ebd Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/272786 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2020-11-10internal/database: set an explicit timeout for opening a DB.Jonathan Amsterdam
When opening a database, use the PingContext method with an explicit timeout, so we fail fast if we can't open the DB. (The pq driver we use seems to do nothing on the Open call, and all the actual connection work on Ping.) We want to know quickly if there's a problem connecting to the DB, and we want a clear error message in the logs. Change-Id: I55e8126e14db14a90dd85f784c4ffe6da743613c Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/268639 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Robert Findley <rfindley@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2020-10-27internal/database: fix logQuery call in QueryRowJonathan Amsterdam
The logQuery call was inside the defer, so instead of happening at the start of the query, it happened at the end. Also, tweak the error logging so information is only computed when needed. Change-Id: I2bf4b6e93880a2c44c9fb5732676935b1cdd418a Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/265477 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2020-10-16internal/database: only log QueryRow debug info if errorJulie Qiu
QueryRow debug info is only logged now if there is an error. Logging on every call resulted in too much noise. For golang/go#39629 Change-Id: I5e5be0b1b4bdeda0ebe2618a7036fbf521c225a7 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/262702 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2020-10-05internal/database: log additional info for QueryRow errorsJulie Qiu
We're seeing several 500s from queries that should be cheap, such as db.IsExcluded, db.GetUnitMeta, and getPathID in db.GetUnit. These queries all call QueryRow. Additional logging is added to debug this issue. Change-Id: I286b5b2115d757ef6bcc2a3f4d8530d04037bbae Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/259619 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> Reviewed-by: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com>
2020-09-11internal/worker: add health check endpointJonathan Amsterdam
This will be used with GKE health checks. The health checks hit "/" by default, which causes unnecessary work. Change-Id: I6093e5985102153bbabeb19feebbae1a44d55a9f Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/254378 Reviewed-by: Julie Qiu <julie@golang.org>
2020-09-09internal/database: change Exec to return rows affectedJonathan Amsterdam
Change Exec so it return the number of rows affected by the statement, rather than a sql.Result. - We never use the other methods of sql.Result. - It's annoying to get the number of affected rows from a sql.Result because of the error return value. - Examination of github.com/lib/pq shows that there is no extra cost to calling sql.Result.RowsAffected after an Exec. Change-Id: If16a15cbabf38755518732c4489109e0b01f2cd1 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/253079 Reviewed-by: Julie Qiu <julie@golang.org>
2020-09-09internal/database: add ExecRowsAffectedJonathan Amsterdam
Add a method that simplifies a common pattern: exec a statment and return the number of rows affected. Change-Id: Ie8583627a753a98ab6a90e3e03fef867b1b420b1 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/253078 Reviewed-by: Julie Qiu <julie@golang.org>
2020-09-09internal/database: move logging code to a separate fileJonathan Amsterdam
Pure code motion. Change-Id: Id9a019e114018bbe04ef5bea6eed0f563aeb886c Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/253077 Reviewed-by: Julie Qiu <julie@golang.org>
2020-08-07internal/database: downgrade context canceled errors to debug levelJulie Qiu
There are many places in our logs when a query will be canceled, because all unfinished search queries for a given request are canceled. We don't want to log these as errors, because it makes the logs very noisy. These are downgraded to a debug level log. Change-Id: Iaddce240394cfb4eb79b03f384e5dabbd8e84ea5 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/247280 Reviewed-by: Robert Findley <rfindley@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com> Run-TryBot: Robert Findley <rfindley@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com>
2020-06-22all.bash,internal: force DB testing in CIJonathan Amsterdam
If the environment variable GO_DISCOVERY_TESTDB is "true" wherever we would skip DB tests, fail instead. Set it to "true" in all.bash. Fixes golang/go#39713. Change-Id: I80fe27578a05610e29cc3ad9f6d477c9768fd51a Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/239257 Reviewed-by: Julie Qiu <julie@golang.org>
2020-06-17internal/config: remove globalJonathan Amsterdam
Remove the `cfg` global and the functions that accessed it. Fixes b/145301722. Change-Id: I58ab9fbd4fc29f66dbc5b120f04c88ee0703ee57 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/238437 Reviewed-by: Julie Qiu <julie@golang.org>
2020-06-16internal/database: fix bug with error handling in transactJulie Qiu
A bug is fixed with error handling in transact, due to a change in commit 02f4e9876d3489f62e5fb8d28e6f4cc037271dfc. The change resulted in the local error to be modified, instead of the return value which was the desired behavior. Change-Id: I59f778b2516fea72a40d90baad1d81b5c61d13db Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/238246 Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
2020-06-16internal: avoid "if err = f(); err != nil" patternDaniel Martí
Use shadowing via ":=", which clearly signals we're not storing the error value in a variable to be used at a later time. This way, one knows that the error value is scoped to the if statement, so there is less to keep in one's head. Change-Id: Ibfb5efc92e9500a525ca58ad341d59ced0581fd9 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/238043 Reviewed-by: Julie Qiu <julie@golang.org>
2020-06-09internal/database: support bulk upsertJonathan Amsterdam
Add DB.BulkUpsert, which adds an ON CONFLICT clause to the INSERT that replaces existing column values. Change-Id: I59f36be0bcb0c0854f42da489e265f2a1396c439 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/766360 Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-06-09internal/database: BulkInsert: only prepare when necessaryJonathan Amsterdam
We were preparing the full statement (of size stride) even if we never used it, because the number of values to insert was less than stride. Also, tweak TestLargeBulkInsert: - the logging is no longer voluminous, so we don't have to turn it off; - make the number of values not divisible by 1000, so we test the case where rightBound > len(values). Change-Id: I169130ab370b192a629025886173b1e33f5d158b Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/766240 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-06-08internal/database: assign to right err variableJonathan Amsterdam
We were assigning an error to a dead variable (the `err` defined in the `QueryContext` call shadowed the one above the `if`). Caught by staticcheck. Change-Id: Ifbf8a78c336ab1918fdf5714548b286b73199ce9 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/765142 Reviewed-by: Julie Qiu <julieqiu@google.com> CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
2020-06-04internal/database: use prepared statement for BulkInsertJonathan Amsterdam
Using a prepared statement for the query seems to speed up an insert significantly. Change-Id: I32a1455b376e08fb435b2e756aa8df4b0c2bc1b3 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/761661 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-06-02internal/database: Transact supports any isolation levelJonathan Amsterdam
- Add an arg to Transact for the isolation level - Remove TransactSerializable This makes it possible to use other levels, and makes it easier to see which level is being used for each transaction. Change-Id: Iba5e2920b4139e5e2f0f8c6b331a658d7c84f60f Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/758942 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-05-26internal/database: correctly wrap bulk statement errorsJonathan Amsterdam
Wrap DB errors in BulkInsert and BulkUpdate, instead of hiding them. This will allow TransactSerializable to retry the transaction; now it is just failing. Also, log the number of retries so we can get a sense of how congested we are. Change-Id: Ice538c5c4de56d88c89ec7a290767c0d5c0b9e39 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/752351 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-05-21internal/database: log isolation levelJonathan Amsterdam
Change-Id: Ibd8d29e91b47e37419e5f76c122d6ac73f6989f9 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/750957 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-05-21internal/database: DB.Transact uses current contextJonathan Amsterdam
Use the context passed to Transact for the transaction. Previously, we used the background context, because that is what sql.DB.Begin does. But we should use the argument context. This is a behavior change, but it should be unnoticeable. All our transaction functions use the same context we pass to Transact, so the only difference this CL makes is that the DB calls in Transact itself, like BeginTx, Commit and Rollback, now have the potential to fail because the context is done. In practice that shouldn't matter. Change-Id: I5eeff51bcd28cb877e5288b83d9e08fa3d249482 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/750953 Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-05-21internal/database: support serializable transactionsJonathan Amsterdam
Add DB.TransactSerializable, which executes a transaction with serializable isolation. Although serializable transactions are more expensive, they reduce the risk for anomalies, like the constraint violations we sometimes see. This CL does not change any existing behavior. In particular, the DB.Transact method, despite calling sql.DB.BeginTx instead of sql.DB.Begin, uses the background context exactly as Begin does (source: https://github.com/golang/go/blob/go1.14.3/src/database/sql/sql.go#L1689). Change-Id: Iab9c99ca35de9ef884b149a3b39ecac3300ecac0 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/750952 Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-05-20many: use nil instead of empty sliceJonathan Amsterdam
It's more idiomatic in Go to use nil rather than an empty slice, in situations where either works. Change-Id: I1f85ba39772b873b7b35ca35837fecddd3c33ba0 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/750950 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-04-23all: rename module to golang.org/x/pkgsiteJulie Qiu
golang.org/x/discovery is renamed to golang.org/x/pkgsite. When the repository is open sourced, it will be hosted at go.googlesource.com/pkgsite. Change-Id: Ifc3b45b771a385b99179e785447f2a87afcacf87 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/724273 Reviewed-by: Jonathan Amsterdam <jba@google.com>
2020-04-16internal/database: support returning values from a bulk insertJonathan Amsterdam
Add DB.BulkInsertReturning, which supports the INSERT ... RETURNING feature. Change-Id: I8b7ca21295addde1ef29331d6f3d587bd848b4fd Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/719741 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-04-16internal/database: log transaction start and endJonathan Amsterdam
This requires adding a context parameter to the Transact method. Change-Id: I98e8b9cbd8ce724cbed693d7549e82cf8ee75990 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/719480 Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-04-16internal/database: support bulk updatesJonathan Amsterdam
Implement BulkUpdate. See the function's doc comment for details. Change-Id: I050227a16ebe2f93cfc535fe95d1cd310fd6d00b Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/716478 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-04-14internal/database: use the DB for inside a transaction as well as outJonathan Amsterdam
The database.DB type now can represent a DB connection in the middle of a transaction. Such a DB is created only by calling DB.Transact. The resulting API is much simpler, since all the ...Tx methods disappear. Change-Id: I41afada87738e1eacdec2fcf115902edddeff867 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/716719 Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-04-06internal/database: add RunQueryTxJonathan Amsterdam
Add a function to run a query inside a transaction. Use it in the couple of places it can be used. Change-Id: If9230f434633ff1b5c60087c8281039b533354c6 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/696538 Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-04-06discovery: fix issues highlighted by mvdan.cc/unparamJonathan Amsterdam
I ran unparam (`go get mvdan.cc/unparam`) on our codebase, and it found a few issues that I fixed. - etl/main.go:getRedis: Use the context to set the DialTimout option when dialing redis. - config.go:readOverrideFile: Use the bucketName argument, not a global constant. - various: Remove unused parameters and return values. Change-Id: I98a1e5d728d41033b5f632ccc6c22442d91c2b6d Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/678587 Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-03-27internal/{log,middleware}: add trace ID to context, use in loggingJonathan Amsterdam
- The requestlog middleware gets the trace ID from the request and adds it to the request's context. - The internal/log package retrieves the traceID and adds it to log messages. It also creates two logs instead of one, a "parent" log used in requestlog, and a "child" log used for all other logging. Together, these two changes will cause the Stackdriver log viewer to group all log messages for a request with the request start and end log messages. - These changes require that all log functions take a context, so I plumbed one through everywhere. In a handful of cases it didn't seem worth doing the plumbing. I used context.TODO() for those so we can easily find and re-evalaute them. Change-Id: I663588463520187d0549a8f802ba9cb44a893592 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/622940 Reviewed-by: Robert Findley <rfindley@google.com>
2020-03-27devtools/cmd/dbpopulate: set sort_versionJonathan Amsterdam
Change dbpopulate to update sort_version. Change-Id: Icb0e7504501d99487048633a6c06b33a0990225e Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/605782 Reviewed-by: Robert Findley <rfindley@google.com>
2020-03-27internal/etl: use database.DB instead of sql.DBJonathan Amsterdam
Replace the direct use of sql.DB with the new database.DB. This will give us logging of autocomplete queries. Change-Id: I656e07b42359915a69dfc567c4c6f56d41900aab Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/604404 Reviewed-by: Julie Qiu <julieqiu@google.com> Reviewed-by: Robert Findley <rfindley@google.com>
generated by cgit v1.3 (git 2.53.0) at 2026-04-15 09:01:27 +0000