aboutsummaryrefslogtreecommitdiff
path: root/internal/postgres/versionstate.go
AgeCommit message (Collapse)Author
2026-03-24all: pgx/v4 to v5 and upgrade pqEthan Lee
- In preparation for upgrade to PostgreSQL 17, upgrade stale dependencies within pkgsite. Change-Id: Id54cbb46182711b9808ee3bbbe2f0d5d9a04583d Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/758121 Auto-Submit: Ethan Lee <ethanalee@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Jonathan Amsterdam <jba@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2024-03-05worker: display request state on main pageJonathan Amsterdam
Add a RequestInfo.State field to record the current state of a request for display. Add a function internal.RequestState to for updating the state, and use it for long-running requests. Display the state on the worker main page. Change-Id: I1a04036982aba23cf27776fac75248972a1336f3 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/568515 Reviewed-by: Michael Matloob <matloob@golang.org> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2022-11-15all: convert interface{} to anyHana (Hyang-Ah) Kim
Change-Id: I1f3b7cc8899c7707abb01e3d14807c37c3451382 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/449695 TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Run-TryBot: Hyang-Ah Hana Kim <hyangah@gmail.com>
2021-09-14internal/postgres: change UpsertModuleVersionStates to updateJulie Qiu
UpsertModuleVersionStates is changed to UpdateModuleVersionStates. There should never be a situation where UpsertModuleVersionStates is called and a row does not already exist for that module. If that happens, an error is now returned. For golang/go#46985 Fixes golang/go#39628 Change-Id: I357396cee6eb743513ae249609f76f4cd4c19e9b Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/341860 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> Reviewed-by: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com>
2021-09-14internal/worker: insert module if not in module_version_statesJulie Qiu
If a module is introduced for the first time to pkgsite through frontend fetch, and not the index, it won't have a row in module_version_states. A row is now inserted before the fetch process begins if the proxy's info endpoint tells us that this is a valid module. This fixes a bug in the fetch flow where module_version_states.last_processed_at is not updated in upsertModuleVersionState if a row did not already exist. For golang/go#46985 Change-Id: I2a166228df20b1fb935bbabebb5a651da0e2c1ba Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/341892 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-06-17internal/postgres: carefully change status on index insertJonathan Amsterdam
We've been updating the status to 0 on every module that comes from the index, even if it's already in the DB with a different status. While we do want to change the status when it's reasonable to reprocess (see the linked issue below), we don't in most other cases. So only update the status when we see a 404. For golang/go#46117 Change-Id: I2774a0e5637873754cf69805bf154aaab632b0b8 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/328929 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com>
2021-05-12internal/postgres: mark processed modules from index with status 0Julie Qiu
When a module is added for the first time using InsertIndexVersions, set the status to 0, even if the module already exists in module_version_states. That way we will prioritize reprocessing this module, in case its previous status was a non-2xx code. For golang/go#46117 Change-Id: Ie2c01a84449a0bb2e446a0c1388b315b580d3b12 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/319049 Trust: Julie Qiu <julie@golang.org> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-04-29internal/postgres: add function to clean modulesJonathan Amsterdam
Add CleanModuleVersions, which deletes module versions and gives them Cleaned status in module_version_states. For golang/go#45852 Change-Id: I26af0981f75e735aebbf3ca2606ce8261330ee7c Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/315090 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-03-25internal/postgres: include less in error messageJonathan Amsterdam
The error messages from UpsertModuleVersionState included a large data structure. Just include the module path and version. Change-Id: I1e68f387c7cf2f3aacf959b443e738e6852fb02d Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/304929 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-03-17internal/postgres: check for go.mod includes module_version_statesJonathan Amsterdam
We sometimes need to know whether a a particular module version's zip has a go.mod file in order to determine the latest version of the module. As an optimization, we check the DB first before getting the zip from the proxy. Up until now, we checked only the modules table, which includes only "good" modules. This CL extends that check to the module_version_states table, which includes every version we've seen. For golang/go#44710 Change-Id: If3381615ac30c0dfc0706f9e4b14ed387fad25b2 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/302529 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com>
2021-03-02internal/postgres: read/write module_version_states.has_go_modJonathan Amsterdam
Change-Id: I797da3b0199611ac511d7f9460279998be9927ad Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/296811 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-03-02internal/postgres: group UpsertModuleVersionState argsJonathan Amsterdam
Organize the many arguments to UpsertModuleVersionState into a struct. Change-Id: Ia0266af623ca2be3961fde6ba8e15ba5540cf36b Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/296810 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-02-12content,internal: update versions pageJulie Qiu
The versions page is updated to: - Display incompatible versions in a separate section - Remove minor versions of other modules A version.IsIncompatible function is added to the internal/version package for determining if a version is an incompatible version. Mock: https://photos.app.goo.gl/wWEFsMrmtaxggTAk7 Change-Id: I9a1c533066c3ea8b3f9f9b65f9bd171290655b5e Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/288762 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-02-12internal/postgres: add a stack to all wrapped errorsJonathan Amsterdam
Add a stack trace when we wrap an error from the DB. These traces can be sent to the error reporting service. For golang/go#44231 Change-Id: I096cdec4e97a6dcb0b7eb2ccdb4c955e1a0f4ccd Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/291492 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2020-09-21internal/postgres: rename variable versionJulie Qiu
Various uses of the variable version is renamed to requestedVersion and resolvedVersion for clarity. Change-Id: I9dc4987d605631549497261b18f165fc9c157508 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/256178 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> Reviewed-by: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com>
2020-09-09internal/postgres: set modules.status when updating module_version_statesJonathan Amsterdam
Populate the new modules.status column whenever we insert or update module_version_states.status, in the same transaction. After this code is deployed, we should manually update NULL modules.status values to match module_version_states.status. This CL implements part of https://golang.org/cl/c/pkgsite/+/249447. For golang/go#40807 Change-Id: I8623fd71207b6578ff03de09ee12d292b36465b5 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/253080 Reviewed-by: Julie Qiu <julie@golang.org>
2020-09-09internal/database: change Exec to return rows affectedJonathan Amsterdam
Change Exec so it return the number of rows affected by the statement, rather than a sql.Result. - We never use the other methods of sql.Result. - It's annoying to get the number of affected rows from a sql.Result because of the error return value. - Examination of github.com/lib/pq shows that there is no extra cost to calling sql.Result.RowsAffected after an Exec. Change-Id: If16a15cbabf38755518732c4489109e0b01f2cd1 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/253079 Reviewed-by: Julie Qiu <julie@golang.org>
2020-08-19internal/postgres: don't read module_version_states.incompatibleJonathan Amsterdam
We don't need to read the 'incompatible' column from module_version_states. This also fixes a bug where there was a mismatch between the number of columns in the query and the number of arguments to rows.Scan in queryModuleVersionStates. Change-Id: I5a8e0d41666ce7b5e721ba78342cb18ee21ead2f Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/249297 Reviewed-by: Julie Qiu <julie@golang.org>
2020-08-17internal/postgres: populate module_version_states.incompatible fieldMiguel Acero
This change adds support for inserting, sorting and updating module_version_states.incompatible. Updates golang/go#37714 Change-Id: I40b0831adef6e78beafee259e972cb0e4d4b90c4 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/248182 Run-TryBot: Julie Qiu <julie@golang.org> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2020-06-26internal/worker: add /poll and /enqueue endpoints.Jonathan Amsterdam
Add a /poll endpoint that polls the index, but just writes to module_version_states without enqueuing. Add an /enqueue endpoint that is identical to /requeue. The name more accurately reflects that it is for new modules as well as reprocessing. Update the worker status page to use the new endpoints. We'll delete the old endpoints after deploying and changing the scheduler jobs. Updates b/158866584. Change-Id: Id116bf9fd99fa55aaacd71bb4ca6b60770ca8812 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/239480 Reviewed-by: Julie Qiu <julie@golang.org>
2020-06-02internal/database: Transact supports any isolation levelJonathan Amsterdam
- Add an arg to Transact for the isolation level - Remove TransactSerializable This makes it possible to use other levels, and makes it easier to see which level is being used for each transaction. Change-Id: Iba5e2920b4139e5e2f0f8c6b331a658d7c84f60f Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/758942 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-04-30internal/postgres: change reprocessing logicJulie Qiu
At the moment, we reprocess and requeue modules using the following logic: 1. Set all modules to be reprocessed = 505. 2. Requeue modules with status=0 or status >= 500. Prioritize the following: - IsLatest: sorted by release vs prerelease modules - IsBig: hardcoded list of modules we know are big This poses the following problems: 1. Requeue order is not idempotent: priority is given to categories of modules, but within each category, the order of modules being queued can change each time requeue is called. This leads to many modules sitting in the task queue, and a lack of clarity as to how much progress we have made when looking at the logs. 2. Modules missing from isBig list: there are several modules missing from the isBig list, but these aren't being accounted for. We deproritize large modules because they take a really long time to process and can timeout if too many are being processed at once, so we want to process them at a slower rate than other modules. 3. Alternative modules have the same priority as non-alternative modules: we usually don't care about alternative modules, and they will be deleted from search_documents once identified. These should be processed after the lastest version of non-alternative modules are processed to prevent unnecessary deletes. To address these issues, reprocessing / requeue now follows the following logic: 1. All modules are reprocess with a 50x status code based on their last fetch status in module version states. 2. Modules are requeued in the following order (with the exception of large modules): - Latest version of modules previously with 20x status - Latest version of bad modules and alternative modules - Any version of modules previously with 20x status - Any version of bad modules and alternative modules - Any module with a status=0 or status=500 (we expect these to already be in the queue) 3. All large modules are queued last, since these take up a lot of time and need to be processed at a slower rate. Within each category, modules are sorted as follows: 1. num_packages 2. version DESC 3. module_path This keeps the order idempotent, and prioritizes smaller and newer modules. It also allows modules of similar sizes to be processed together. Change-Id: I49580ed75bf60cc2698b756882bfdc906f72d935 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/725873 Reviewed-by: Jonathan Amsterdam <jba@google.com>
2020-04-29internal/postgres: refactor upsert version stateJulie Qiu
Split UpsertModuleVersionState into upsertModuleVersionState and upsertPackageVersionStates to improve readability and allow us to only upsert module_version_states in tests. Change-Id: I59b2675bb007dfbb62c9906d64d6a74dbc159c9e Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/729692 Reviewed-by: Jonathan Amsterdam <jba@google.com>
2020-04-29internal/postgres: read/write module_version_states.num_packagesJulie Qiu
Change-Id: Ia0e4e59e81b9db3e5a439a58caf1b7627fa52f52 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/729485 Reviewed-by: Jonathan Amsterdam <jba@google.com>
2020-04-29internal/postgres: sort packageVersionStatesJulie Qiu
db.UpsertModuleVersionState now sorts packageVersionStates by package path before sorting to prevent deadlocks. Change-Id: Ic83f08db343dbd8e8790bddddb66b7bb20ae6d26 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/729481 Reviewed-by: Jonathan Amsterdam <jba@google.com>
2020-04-23all: rename module to golang.org/x/pkgsiteJulie Qiu
golang.org/x/discovery is renamed to golang.org/x/pkgsite. When the repository is open sourced, it will be hosted at go.googlesource.com/pkgsite. Change-Id: Ifc3b45b771a385b99179e785447f2a87afcacf87 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/724273 Reviewed-by: Jonathan Amsterdam <jba@google.com>
2020-04-23internal/postgres: do not delete package version statesJulie Qiu
At the moment when we upsert a row into module_version_states, we delete and repopulate existing rows in package_version_states. This adds an additional query that is costly and isn't necessary, since we wouldn't expect there to be packages missing in a future round of processing. Also, if for some reason that did happen, it would be useful to have this data captured in package_version_states anyways. Change-Id: Iceb64235957ade115293bb7c1ff62cdcb2e2073f Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/725870 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2020-04-17internal/postgres: log errors from deleting package_version_statesJulie Qiu
Change-Id: Ib6c36be1c493277e6581fabae464c0854075a0a7 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/720897 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2020-04-16internal/database: log transaction start and endJonathan Amsterdam
This requires adding a context parameter to the Transact method. Change-Id: I98e8b9cbd8ce724cbed693d7549e82cf8ee75990 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/719480 Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-04-14internal/database: use the DB for inside a transaction as well as outJonathan Amsterdam
The database.DB type now can represent a DB connection in the middle of a transaction. Such a DB is created only by calling DB.Transact. The resulting API is much simpler, since all the ...Tx methods disappear. Change-Id: I41afada87738e1eacdec2fcf115902edddeff867 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/716719 Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-04-06internal/postgres: improve GetNextVersionsToFetchJonathan Amsterdam
One of the queries in GetNextVersionsToFetch was taking 30s to run. I couldn't find a way to speed it up. Finally I gave up and rewrote the method to do something much more straightforward from the DB's point of view. First I run a query to get the latest versions of everything in module_version_states. Then I scan the entire table for rows eligible for reprocessing, and group them by priority. Despite the apparent slowness of these queries, the whole thing actually runs much faster than the previous version. Change-Id: Ia88bc26849f0df131d8c00d07bf6a7d160c6b364 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/672575 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-04-06internal/{etl,postgres}: change ModuleVersionState.GoModPath to string typeJulie Qiu
Change-Id: I38209749ed8e82b4065188033d829f169d272e10 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/672058 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2020-04-06internal/{etl,postgres}: change type for module_version_states status and errorJulie Qiu
Previously, module_version_states.status and module_version_states.error were pointers. These are changed to type int and string, respectively, and will be set to not null field with default values of 0 and empty string in the next CL. This change makes it more ergonomic to work with these fields, while maintaining its behavior. Updates b/135954292 Change-Id: I3783c7a05259a787f57b5cb94299cc16174ffcee Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/669973 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2020-04-06internal/{etl,postgres}: insert package version state to postgresJulie Qiu
The following methods are added to internal/postgres: - GetPackageVersionStatesForModule - GetPackageVersionState UpsertModuleVersionState is modified, so that it will also insert the provided package version states into the database. The ETL is modified to use these methods to insert data into package_version_states. A bug is also fixed in the ETL, where the requestVersion was being inserted into module_version_states and used to find a search_document row, instead of the resolved version. Updates b/135954292 Change-Id: I01e74ba26e95538069a4d34137f99a5e8be074dc Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/663069 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2020-03-27internal/postgres: improve reprocessing orderJonathan Amsterdam
It takes several days to reprocess all our data. Most of that is taken up with numerous forks of kubernetes and other large modules. Many of these are quickly marked as alternatives, but many older versions lack a go.mod file, so we fully process each one, and they take a long time. Process the most recent versions of all modules first. Then process all non-kubernetes, etc. modules. Leave non-latest versions of the large modules for last. Change-Id: Ibdfec3c5c3ac996d3cdcfd8988421db498e5e553 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/653246 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-03-27internal: rename VersionState to ModuleVersionStateJulie Qiu
VersionState is renamed to ModuleVersionState because a PackageVersionState will be introduced in a future CL. Updates b/133187024 Change-Id: If21c0feee9bd1c996b43cd16c7a5d0e79aa8c7ce Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/632662 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2020-03-27internal/postgres: read and write module_version_states.go_mod_pathJonathan Amsterdam
This CL adds go_mod_path to the low-level code that reads and writes the module_version_states table. A later CL will plumb the path from fetch through to the insert call. Change-Id: I5822d4faae0b1e2aec22ea9071ad6f0908c25a6e Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/632523 Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-03-27internal/{log,middleware}: add trace ID to context, use in loggingJonathan Amsterdam
- The requestlog middleware gets the trace ID from the request and adds it to the request's context. - The internal/log package retrieves the traceID and adds it to log messages. It also creates two logs instead of one, a "parent" log used in requestlog, and a "child" log used for all other logging. Together, these two changes will cause the Stackdriver log viewer to group all log messages for a request with the request start and end log messages. - These changes require that all log functions take a context, so I plumbed one through everywhere. In a handful of cases it didn't seem worth doing the plumbing. I used context.TODO() for those so we can easily find and re-evalaute them. Change-Id: I663588463520187d0549a8f802ba9cb44a893592 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/622940 Reviewed-by: Robert Findley <rfindley@google.com>
2020-03-27internal/etl: use module_version_states.sort_versionJonathan Amsterdam
- Insert/upsert it. - Sort by it when getting versions to fetch. Updates b/139482696. Change-Id: If4c477d57a5751e8c355903df4f6964df2c381f5 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/605781 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Robert Findley <rfindley@google.com>
2020-03-27internal/database, internal/testing/dbtest: site-agnostic DB functionalityJonathan Amsterdam
Extract into a separate package the core functionality from internal/postgres that doesn't depend on our particular schema. This makes it available for other uses, like devtools commands and etl autocomplete. Do the same for testing functionality. We now have three packages where before we had only one: - internal/postgres: discovery-specific DB operations and test support - internal/database: discovery-agnostic DB operations - internal/testing/dbtest: discovery-agnostic DB test support Change-Id: I54c59aee328dae71ba6c77170a72e7a83da7c785 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/602327 Reviewed-by: Robert Findley <rfindley@google.com>
2020-03-27various: use internal/log packageJonathan Amsterdam
Take advantage of log levels when logging DB queries: use Debug, except for errors. Change-Id: Icd99d1474799b89d2d8149e5f5bb6f417c6448b5 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/556725 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-03-27internal/postgres: use DB.runQuery in some functionsJonathan Amsterdam
Convert the following DB methods to use runQuery: - GetImports - Search - GetVersionStats Also, add missing rows.Err check to collectLicenses. Change-Id: Icf5bf26d9aae54e767b2a4678512ac6f27963c63 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/529193 Reviewed-by: Robert Findley <rfindley@google.com>
2020-03-27internal/postgres: annotate errors in remaining functionsJonathan Amsterdam
Updates b/139152543 Change-Id: Ibb92f32106acb1c16f4413e0f374475a4203884c Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/528829 Reviewed-by: Robert Findley <rfindley@google.com>
2020-03-27internal/postgres: have db functions return error contextJonathan Amsterdam
Add context info to the DB.exec and DB.query methods, and remove it from call sites. Updates b/139152543. Change-Id: Id4896f60cb3ab07b358b6db62d561e7101c5091b Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/526008 Reviewed-by: Robert Findley <rfindley@google.com>
2020-03-27internal/postgres: rename XXXContext to XXXJonathan Amsterdam
Since we now control the methods of DB, we can remove the unnecessary Context suffix. You have to remember to use lower-case "exec" etc. anyway, so this change doesn't add any additional confusion. Change-Id: I57bde0dcf359fb37e57457accd170cdddfd2ef3e Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/525153 Reviewed-by: Robert Findley <rfindley@google.com>
2020-03-27internal/postgres: make sql.DB a regular fieldJonathan Amsterdam
By not embedding it, we don't expose all its methods. Add selected methods back. Fixes b/139178399 Change-Id: Ic146dc01c8531e1fd5b56e085f53a7735d44c146 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/525149 Reviewed-by: Robert Findley <rfindley@google.com>
2020-03-27etl/fetch: remove from DB on 410 GoneJonathan Amsterdam
When the proxy's .info endpoint returns a 410 for a module@version, that means it has been removed permanently. So we delete it from the database. We still record the 410 in the module_version_states table for reference. Fixes b/137743165. Change-Id: I0a705e80cf458c51a2ecabcc93c85eb8efdc7cf9 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/524046 Reviewed-by: Robert Findley <rfindley@google.com> Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-03-27internal/derrors: switch to using xerrorsRob Findley
The derrors package is reduced to a set of error values and the helper method FromHTTPStatus, by using the xerrors package. Change-Id: Ifdcec6abe7e14f127c20afa1f7e9ab1521272806 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/521329 Reviewed-by: Jonathan Amsterdam <jba@google.com>
2020-03-27internal/postgres: fix a few places where rows were not closedRob Findley
Updates b/137689600 Change-Id: Iaacee7a0b9ca50c4b665459f3f88bf503ff37bd4 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/509283 Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-03-27internal: add reprocess handler for outdated versionsJulie Qiu
A repocess handler is added to allow for reprocessing outdated module versions. The endpoint /reprocess?app_version=<app_version> will mark versions last processed before the specified app_version with HTTP Status 505. These versions will then be requeued using the existing /requeue handler. Existing references to "cron" are also updated to "etl". Update b/133518045 Change-Id: I12f0822340d0a3a30c2769a52fea2879409a669b Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/505099 Reviewed-by: Robert Findley <rfindley@google.com>
generated by cgit v1.3 (git 2.53.0) at 2026-04-15 07:11:00 +0000