| Age | Commit message (Collapse) | Author |
|---|
|
Instead of serializable transactions, we will use an advisory
lock on the module path. See http://tg/760100.
Change-Id: Iaae7777bab8efa21347480b6f3929a1feed98e44
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/762240
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
|
|
Use Postgres advisory locks (see
https://www.postgresql.org/docs/11/functions-admin.html#FUNCTIONS-ADVISORY-LOCKS)
to prevent more than one transaction from thinking it is the latest
version of a particular module.
Change-Id: Ie928dac240b6e8679b41434b2433f89fcb2f49ae
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/760100
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
|
|
If an experiment flag is set, use a serializable transaction
to insert a module.
Change-Id: Ieca2d5ce68f456d48e69b9de6fc3714a130cbca3
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/750955
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
|
|
Use the results of postgres.GetDirectoryNew to build
package pages.
Change-Id: Iff812416f4fb879baa16536639a80810fc1248b2
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/747780
Reviewed-by: Julie Qiu <julieqiu@google.com>
|
|
Replace many calls to sample.DefaultModule, and all calls to
sample.DefaultPackage, with calls to the new Module and Package
functions.
Change-Id: I76921e14502585f8ca9a4dba5de01d7055522f3f
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/733604
Reviewed-by: Julie Qiu <julieqiu@google.com>
|
|
This is the first step towards making sample clearer and less
error-prone.
- Rename Module and Package to DefaultModule and DefaultPackage, so
existing code continues to work. In later CLs, I'll replace most or
all calls to these functions.
- Define ModuleInfo, Module and Package to take as arguments the most
common fields. The Module function does _not_ create a package; it
returns a module with no packages.
- Most important: define AddPackage, which adds a Package to a Module
and correctly updates Module.Directories.
- Remove VersionedPackage, since it's clearer to create one directly.
- Update a couple of call sites, just to get tests to pass. More
updates in a later CL.
Change-Id: I46eb94ba897d4f122483b58435107b8782c6044f
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/733619
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
|
|
When we delete the last version of a module, remove it
from imports_unique.
Updates b/154616892.
Change-Id: I988c439ae117cca4b79ea33752f730fb48c2496f
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/728094
Reviewed-by: Julie Qiu <julieqiu@google.com>
|
|
fetchTask.timings is added to time FetchModule and each DB insert. The
results are logged as a single log line, which will be exported to
BigQuery for analysis as needed.
postgres.DeleteModule is rewritten to db.DeleteModule, for consistency
with other exported functions from internal/postgres.
Change-Id: Id6ecd7fccd238fb17a45554ec033ebf404c1020c
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/729940
Reviewed-by: Jonathan Amsterdam <jba@google.com>
|
|
When a module is inserted using postgres.InsertModule, it was first
deleted, then inserted. This allowed us to guarantee a clean module
insert, but also created a lot of load on the database. This was
particularly evident when reprocessing our dataset; during these
periods, running the following query showed that most active connections
were due to DeleteModule, which could take several minutes:
SELECT FROM pg_stat_activity WHERE state = 'active' ORDER BY query_start;
Many DeleteModule queries have also failed due to a deadlock.
This is because running DeleteModule results not only in a delete on
module, but also cascaded deletes to several tables:
1. packages
2. paths
3. licenses
4. documentation
5. readmes
6. search_documents
7. imports
8. package_imports
In general when we are inserting a module, it can be can upserted,
instead of deleted then inserted. For a given module version, we never
expect to lose rows in any of the tables above when reprocessing it.
To confirm this, postgres.validateModule has been updated to compare the
existing data in the database, and data from the new module that was
just fetched. If data went missing, we mark the module with a status
derrors.DBInvalidModuleInsert. It is assumed that comparing data only on
the packages, paths, and licenses table is a good enough proxy that data
in the other tables will also be valid.
If an error occurs, the module will be deleted inside
worker.FetchAndUpdateState (as with all fetches with status > 400).
A new derrors.DBExecErr is also added to capture insert/update/delete
errors.
Change-Id: I100ac7e4e14409065517a3268e3e215d0a03bb1c
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/728043
Reviewed-by: Jonathan Amsterdam <jba@google.com>
|
|
When inserting a new module, upsert the search documents in the same
transaction as the other inserts into modules, packages and so on.
Hopefully this will remove the foreign-key violation we've been seeing.
Updates b/154318694.
Change-Id: Iaaa0c6ff9b51286d29b10e8a72970baa814a2cff
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/727144
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
|
|
golang.org/x/discovery is renamed to golang.org/x/pkgsite.
When the repository is open sourced, it will be hosted at
go.googlesource.com/pkgsite.
Change-Id: Ifc3b45b771a385b99179e785447f2a87afcacf87
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/724273
Reviewed-by: Jonathan Amsterdam <jba@google.com>
|
|
Previously experiments were always inactive for the worker, because:
(1) X-Forwarded-For was empty since the request was coming from
Cloud Tasks. This is fixed by setting all experiments with rollout=100
to always be on. (There shouldn't be any case where worker flags
would only be partially on).
(2) Experiments were not being set on the new context produced by
trace.StartSpan and when a contextWithCancel was created and passed to
fetch.FetchModule. These are now set with the new
experiment.WithExperiment function.
Change-Id: Ie14b699bf435fecc8791c3ad435f73afe579812f
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/720661
Reviewed-by: Jonathan Amsterdam <jba@google.com>
|
|
InsertVersion now populates the following tables:
- paths
- readmes
- documentation
- package_imports
Change-Id: Ib02220101d148a261cf55bd564cb9f9a46e60544
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/704882
Reviewed-by: Jonathan Amsterdam <jba@google.com>
|
