aboutsummaryrefslogtreecommitdiff
path: root/internal/postgres/path.go
AgeCommit message (Collapse)Author
2025-11-10internal/postgres: remove the exclusive lock in upsertPathRob Findley
As described in golang/go#75959, the lock in upsertPath causes a significant amount of contention during module inserts. Meanwhile, the upsertPaths method does not acquire a lock, and does not seem to cause deadlocks. We still don't understand the deadlocks avoided by this lock, but it seems that a low level of failed transactions due to deadlock may be preferable to the contention we're seeing currently. Let's experiment with removing the lock. For golang/go#75959 Change-Id: I2d577b426724010e96ffc8129ae469825fdcb938 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/718720 kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Ethan Lee <ethanalee@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2022-11-15all: convert interface{} to anyHana (Hyang-Ah) Kim
Change-Id: I1f3b7cc8899c7707abb01e3d14807c37c3451382 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/449695 TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Run-TryBot: Hyang-Ah Hana Kim <hyangah@gmail.com>
2021-06-10internal/postgres: write to imports tableJonathan Amsterdam
Write a package's import paths to the new imports table as well as the existing package_imports table. Change-Id: Ie0afe5aa90e79e8371a51cc6fc5f68de3dc62cd8 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/326449 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Reviewed-by: Julie Qiu <julie@golang.org>
2021-05-18internal/postgres: sort paths before insertion to avoid deadlockJonathan Amsterdam
Deadlock is rare but we have observed it when multiple versions of the same module are inserted concurrently. Change-Id: Idd28bd558009ff053ac198bdd310fa780af66f83 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/320750 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-05-06internal/{postgres,frontend}: group search resultsJonathan Amsterdam
Group search results by module. Show the highest-ranking package of each module, and display the other packages as sub-results, in the same snippet as the winning package but smaller. Also, display packages from later major module versions before earlier ones, even if they have a lower score. All this is protected by an experiment. Change-Id: Iabc7fcf4e916289617b6c1c755904a27163ad554 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/316471 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-05-05internal/postgres: export GetSymbolHistoryForBuildContext and GetPathIDJulie Qiu
GetSymbolHistoryForBuildContext and GetPathID are exported for use in test scripts. Change-Id: Ibb31e22d2a99b9a7c2be8b728f2038cf363d8a30 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/317189 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-05-04internal/postgres: optimize getSymbolHistoryForBuildContext queryJulie Qiu
getSymbolHistoryForBuildContext now accepts a pathID instead of packagePath as an argument, to avoid a JOIN on the paths table. For golang/go#37102 Change-Id: I4e6ac004eb56f2c7744c2ebd7b0e8dcd995f0f4f Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/316230 Trust: Julie Qiu <julie@golang.org> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-04-06internal/postgres: fix GetLatestMajorPathForV1Path for gopkg.inJonathan Amsterdam
GetLatestMajorPathForV1Path was not handling the special case of gopkg.in modules correctly. Also, split the logic for extracting the major version of a module path to a separate function. Change-Id: Ie7758330a14ebaab6360fafcc4bf2d3ae46f6479 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/307470 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Reviewed-by: Julie Qiu <julie@golang.org>
2021-04-06internal/postgres: clarify TestGetLatestMajorPathForV1PathJonathan Amsterdam
I found this test hard to understand because it was doing so much to the test inputs. I brought the test inputs and expected values closer to the actual arguments and return values of the function under test. Change-Id: I6d72033e21eddfcc78f5803563c4bbf9e2e19356 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/307392 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Reviewed-by: Julie Qiu <julie@golang.org>
2021-03-09internal/postgres: lock paths table on insertJonathan Amsterdam
Lock the paths table when we insert into it. That should address deadlocks we see with this table, presumably because concurrent transactions are inserting rows in different orders. The lock mode is EXCLUSIVE, allowing readers but no other writers. See https://www.postgresql.org/docs/11/explicit-locking.html. The lock persists until the end of the transaction. There is no way to unlock sooner. See https://www.postgresql.org/docs/11/sql-lock.html. Change-Id: I2767f40c265b6c40581bca7512ea5cde7b2d3d42 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/300109 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-02-26internal/postgres: refactor path insertionJonathan Amsterdam
Move the module-independent part of insertPaths to a separate function, upsertPaths, and put it in the paths.go file so it is near it's single-path cousin. Write a simple test for it. Change-Id: I3f4a6d3e21836b2840153354c880f8be823b466b Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/296551 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-02-26internal/postgres: rewrite upsertPath and test concurrencyJonathan Amsterdam
Originally upsertPath selected a path first, then inserted it if it wasn't present. This led to uniqueness constraint violations even with fully serializable transactions (see https://www.postgresql.org/message-id/CAOqyxwL4E_JmUScYrnwd0_sOtm3bt4c7G%2B%2BUiD2PnmdGJFiqyQ%40mail.gmail.com). If the upsert is done first and then the select, then everything works fine, as the test verifies. Change-Id: I182458b6a7cc788a4adc4e72c90ef605c035e9ed Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/296550 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-02-24internal/postgres: upsert path correctlyJonathan Amsterdam
Check for a row in the paths table before inserting. For golang/go#44437 Change-Id: I763867c27b1d729311b7ec545c5cdab4e9b64036 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/295894 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-02-24internal/postgres: get and update raw_latest_versionsJonathan Amsterdam
Add functions to get a row from the raw_latest_versions table, and to update a row if the new version is later. For golang/go#44437 Change-Id: I9f44b815b4eb42c6cd286f10f2e946e753c38989 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/295450 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-02-12internal/postgres: add a stack to all wrapped errorsJonathan Amsterdam
Add a stack trace when we wrap an error from the DB. These traces can be sent to the error reporting service. For golang/go#44231 Change-Id: I096cdec4e97a6dcb0b7eb2ccdb4c955e1a0f4ccd Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/291492 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-01-19internal: delete experiment not-at-v1Julie Qiu
Additionally, only redirect if the non-v1 path does not match the full path, and is actually a non-v1 path. Otherwise, this will redirect requests to <path>@<missing-version> to <path>@latest, or <path>/v3 to <path>, instead of return a 404 that allows the user to fetch the missing version. Change-Id: I66946062d341477fdc19bfc2a0f18b848943807d Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/284580 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-01-12internal/postgres: GetLatestMajorPathForV1PathJulie Qiu
GetLatestMajorPathForV1Path returns the path that is the latest major for the v1path. Change-Id: Iec02b0771427b535817b5bd3794f6aa1a807280d Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/282617 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2020-11-19internal/postgres: move unit queries to unit.goJulie Qiu
GetUnitMeta and getPathsInModule are moved to unit.go. path.go is renamed to stdlib.go Pure code in motion. Change-Id: If34cd8f3a88104637085994d846ed3ca823c639f Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/271098 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2020-11-02internal/postgres: rename paths to unitsJonathan Amsterdam
Rename all occurrences of the paths table to "units". Don't change the table alias "p" to "u" yet -- just make the minimal change for now. Change-Id: I9b8144788a28ea61259e036f51d7c5a7b7e3d8ef Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/266599 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2020-10-20x/pkgsite: mark unit meta as redistributable in license bypass modeSean Hildebrand
Without this, the frontend is unable to display doc contents for non-redistributable packages even when in license bypass mode. Change-Id: I8e83929934bbdf5a6d1ec0adb556e901bdfee473 GitHub-Last-Rev: 4722ee39cb954f34cec0dff820ba3dae7a52f42f GitHub-Pull-Request: golang/pkgsite#11 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/263419 Trust: Rebecca Stambler <rstambler@golang.org> Reviewed-by: Julie Qiu <julie@golang.org>
2020-10-16internal/{frontend,postgres}: gather timing statsJonathan Amsterdam
Collect the elapsed time for various functions on the serving path of the unit page. Change-Id: Ib4d16f9ad84532f4870f360566e6ed30554a1357 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/262578 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2020-10-07internal/postgres: fix misspelling of preferredJulie Qiu
Change-Id: I50871bf562b13ecca2bc683a43fca708ae1722b1 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/260457 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2020-10-07internal/postgres: update definition of latestJulie Qiu
The definition of latest is updated to prefer incompatible release/prerelease versions over pseudoversion. This matches the definition by cmd/go. Change-Id: Ic3c79c32e14dc4df3cf11d52df6548f0fd781a6b Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/259997 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2020-09-21internal/postgres: rename variable versionJulie Qiu
Various uses of the variable version is renamed to requestedVersion and resolvedVersion for clarity. Change-Id: I9dc4987d605631549497261b18f165fc9c157508 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/256178 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> Reviewed-by: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com>
2020-09-14internal/postgres: prefer prerelease over pseudo as latest versionJulie Qiu
The definition of @latest is changed to prefer pre-release versions over pseudo-versions. This matches the definition of @latest for the go command as of go1.13. For golang/go#41379 Change-Id: Ia5e4cc42822245636d216467aee1e088a40569c5 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/254577 Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com> Trust: Julie Qiu <julie@golang.org>
2020-09-01internal/postgres: rename variable pi to umJulie Qiu
The variable pi is renamed to um, to match PathInfo being renamed to UnitMeta. For golang/go#39629 Change-Id: Iaff62089c0e4dcd511910a77f3e7d7117b3d0fc2 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/252018 Reviewed-by: Jonathan Amsterdam <jba@google.com>
2020-09-01internal: rename PathInfo to UnitMetaJulie Qiu
PathInfo is renamed to UnitMeta, and GetPathInfo is renamed to GetUnitMeta. For golang/go#39629 Change-Id: Ia91cf7d2752988653efcc0e46a4d6b8b63017ef3 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/251978 Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2020-09-01internal/postgres: add data to PathInfoJulie Qiu
The following fields are added to GetPathInfo: - CommitTime - License metadata - RepositoryURL - PackageSourceURL For golang/go#39629 Change-Id: I664a0e96647ae2ad2404631e27f714b64db5c79c Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/251919 Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2020-08-28internal: update GetPathInfo to return PathInfoJulie Qiu
For golang/go#39629 Change-Id: Iedefbc03b24c6ce2c1b99e3148c0f757a726b687 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/251162 Reviewed-by: Jonathan Amsterdam <jba@google.com>
2020-08-19internal/postgres: update GetPathInfo to prefer non-incompatible modulesMiguel Acero
Updates GetPathInfo to use orderByLatest in the query to prefer compatible modules. Updates golang/go#37714 Change-Id: I89fd654c414f860a2f8a327ff9a8fde07fd35120 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/249119 Run-TryBot: Miguel Acero <acero@google.com> Reviewed-by: Julie Qiu <julie@golang.org> Reviewed-by: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com>
2020-08-19internal/postgres: move orderByLatest to path.goMiguel Acero
This change moves orderByLatest to path.go as we transition into a path based data model. Additionally "incompatible" is added so that we would prefer non-incompatible modules. Since there was a lot of duplicated logic for the ORDER BY keyword for the queries for modules, this change modifies orderByLatest to be a helper function that returns an ORDER BY sql string used in many functions to sort modules. Updates golang/go#37714 Change-Id: I5c8e5264e90442a960328439dcc445397fc4cdbd Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/248497 Reviewed-by: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2020-07-01internal/postgres: support requests for master version in GetPathInfoJulie Qiu
At the moment, GetPathInfo only supports requests for the latest version or semantic versions. GetPathInfo is updated to support requests for the master version of a path, by joining with version_map for these requests. Updates golang/go#36811 Change-Id: I05f5df5df9cf95253b137827065ee56e6830cbc2 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/240613 Reviewed-by: Jonathan Amsterdam <jba@google.com>
2020-05-18internal/postgres: add a method for stdlib shortcutsJonathan Amsterdam
Add GetStdlibPaths, which finds all paths in the standard library that end in a given suffix. The goal is to redirect stdlib paths like, e.g. "http" to "net/http". Updates b/149933479. Change-Id: Ib79d55ebb52f1203d8e68f52f6ed8c05db786dfa Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/747784 Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-04-29internal/postgres: support for getting path infoJonathan Amsterdam
Add GetPathInfo, a function that gets information about a path. The main handler would call either GetPathInfo(path, "unknown", "latest") if the incoming URL path had no "@", or GetPathInfo(path, "unknown", version) if the incoming path looked like "foo@v". Based on the returned path (or absence of one), it would decide whether to call GetDirectory, GetPackage or GetModule. Change-Id: Ifc09202d8ec110960e2c3bbaf04b79fc7a7c72df Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/728098 Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-04-29internal/postgres: remove DeleteModule from InsertModuleJulie Qiu
When a module is inserted using postgres.InsertModule, it was first deleted, then inserted. This allowed us to guarantee a clean module insert, but also created a lot of load on the database. This was particularly evident when reprocessing our dataset; during these periods, running the following query showed that most active connections were due to DeleteModule, which could take several minutes: SELECT FROM pg_stat_activity WHERE state = 'active' ORDER BY query_start; Many DeleteModule queries have also failed due to a deadlock. This is because running DeleteModule results not only in a delete on module, but also cascaded deletes to several tables: 1. packages 2. paths 3. licenses 4. documentation 5. readmes 6. search_documents 7. imports 8. package_imports In general when we are inserting a module, it can be can upserted, instead of deleted then inserted. For a given module version, we never expect to lose rows in any of the tables above when reprocessing it. To confirm this, postgres.validateModule has been updated to compare the existing data in the database, and data from the new module that was just fetched. If data went missing, we mark the module with a status derrors.DBInvalidModuleInsert. It is assumed that comparing data only on the packages, paths, and licenses table is a good enough proxy that data in the other tables will also be valid. If an error occurs, the module will be deleted inside worker.FetchAndUpdateState (as with all fetches with status > 400). A new derrors.DBExecErr is also added to capture insert/update/delete errors. Change-Id: I100ac7e4e14409065517a3268e3e215d0a03bb1c Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/728043 Reviewed-by: Jonathan Amsterdam <jba@google.com>
generated by cgit v1.3 (git 2.53.0) at 2026-04-15 10:44:25 +0000