aboutsummaryrefslogtreecommitdiff
path: root/internal/postgres/insert_module.go
AgeCommit message (Collapse)Author
2025-11-17Revert "internal/postgres: utilize num_imports column in insert_module"Ethan Lee
This reverts commit 28eed86815232e48f5e757569fcd061f858142d4. Reason for revert: migration takes too long Change-Id: I279e62342c79d2604cda4e7c9819bdaa8aac752c Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/721101 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Robert Findley <rfindley@google.com>
2025-11-13internal/postgres: utilize num_imports column in insert_moduleEthan Lee
- Avoid recomputing number of imports for every query in getUnitWithAllFields. For golang/go#76284 Change-Id: I7f7d683efb1b2ff49e33681802896a65b1354d25 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/720261 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Robert Findley <rfindley@google.com> Auto-Submit: Ethan Lee <ethanalee@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2025-10-29add github.com/cdktf/cdktf-provider-google-go/google/v16 to skip-symbolsEthan Lee
Change-Id: Ieac78460d008f79068804d6e9cbbbfb85fc2da24 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/716100 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2025-10-23internal/postgres: add skip-symbols list to saveModuleEthan Lee
- Modules in the skip-symbols list will not have symbol information inserted into postgres. Change-Id: Ia52b9eb2d7cab662b967fa4a74c40cced0e44be4 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/713060 Reviewed-by: Jonathan Amsterdam <jba@google.com> Reviewed-by: Peter Weinberger <pjw@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2025-08-14internal/postgres: apply bypassLicenseCheck to IsRedistributable columnJean Barkhuysen
This enables symbol search for non-LICENSE'd repos when -bypass_license_check is supplied. I think the way this works is that when these get insert into units, we don't consider bypassLicenseCheck, and the IsRedistributable column gets set to false. That all seems reasonable by itself. But later, in upsertSymbolSearchDocuments, we only upsert symbol docs for units that are redistributable. That check happens in SQL, so it's hard to factor in the bypassLicenseCheck flag. It's not really clear to me whether we should "pollute" the database with this column set to true under the flag. If you run the program again with the flag turned off, your database keeps the values set under the old run. Anyways, I think this CL is probably the pragmatic take: it's a small code change that achieves the desired result without added complexity, and it's unlikely that anyone will run into the undesired outcome I describe above (who is running this program sometimes with the flag on, sometimes with it off?). Change-Id: Ie15bcf0605b945e4af6574abf689827ea946ddae Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/685457 Reviewed-by: Jonathan Amsterdam <jba@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: David Chase <drchase@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Commit-Queue: Jonathan Amsterdam <jba@google.com>
2024-09-04internal/postgres: insert module path even if not a packageJonathan Amsterdam
When inserting the paths of a module into the paths table, we were inserting all the package paths and the v1 module path (the module path without "/vN" suffix). That is almost always sufficient. But if a module has a version suffix and is not itself a package, then we were not inserting its path. The module path is needed by insertSymbols, so this was causing that function to fail. Add the module path, refactor the code to simplify it and allow testing, and add a test. (For a vN module the code also adds the v1 package path for every package. I think this is unnecessary, but it's too risky to remove it.) Change-Id: I8caa3dd945692a4ec0a1128199237de6b77aa719 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/609117 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Robert Findley <rfindley@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2024-09-04several: use stdlib functionsJonathan Amsterdam
Replace some functions with simpler code that uses recent functions added to the standard library. Change-Id: Ifcd570679a897fb165a4307c7ce9f0f53fccea05 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/609116 kokoro-CI: kokoro <noreply+kokoro@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Robert Findley <rfindley@google.com>
2024-03-05worker: display request state on main pageJonathan Amsterdam
Add a RequestInfo.State field to record the current state of a request for display. Add a function internal.RequestState to for updating the state, and use it for long-running requests. Display the state on the worker main page. Change-Id: I1a04036982aba23cf27776fac75248972a1336f3 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/568515 Reviewed-by: Michael Matloob <matloob@golang.org> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2022-11-15all: convert interface{} to anyHana (Hyang-Ah) Kim
Change-Id: I1f3b7cc8899c7707abb01e3d14807c37c3451382 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/449695 TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Run-TryBot: Hyang-Ah Hana Kim <hyangah@gmail.com>
2021-11-15internal/licenses: remove Metadata.OldCoverageJonathan Amsterdam
We don't need the OldCoverage field anymore. All DBs have only the new Coverage information. Change-Id: Ib2d8e0cdacdc2f7b6514a8a5310d6905be15136c Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/363954 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-10-15internal/{postgres,frontend}: support modules with zero commit timesJonathan Amsterdam
It's valid to have a module with a zero commit time (as reported by the proxy .info endpoint). That can happen if the module is served from a vanity URL with a go-import meta tag of type `mod`. Process those modules and display them with a publish date of "unknown." Fixes golang/go#48952 Change-Id: I8efdebfa27c8b6ed68c183bac77d87ee715f8262 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/356129 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-09-10internal/postgres: remove some error loggingJonathan Amsterdam
We were logging some information to debug a deadlock, but it was cluttering the logs and the messages had no context viewed in isolation. Instead just log one row. Change-Id: I0c7c261af0371f12edf51102a0b6ec20daf4bc9b Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/348932 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-08-18internal/postgres: delete other packages from search during reconciliationJonathan Amsterdam
When reconciling search to match the latest good version, delete from search_documents any packages in the module that are not in the good version. Change-Id: I654eda3d3066bb82963fc98ca537d4b3761ad6eb Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/343189 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-08-16internal/postgres: split deleteModuleOrPackagesInModuleFromSearchDocumentsJulie Qiu
deleteModuleOrPackagesInModuleFromSearchDocuments is split into deleteModuleFromSearchDocuments and deletePackagesInModuleFromSearchDocuments to reduce confusion as to which query is being executed. Change-Id: I4559d812f54401857a153aa52166c12f228f6354 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/342631 Trust: Julie Qiu <julie@golang.org> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-08-16internal/postgres: don't insert into search if a prefixJonathan Amsterdam
In ReconcileSearch, don't insert a module into search_documents if there is already a module path that is a suffix of the current one. This avoids cases like the following: 1. github.com/araalinetworks/api, which has a golang package, is fetched and added to search_documents. 2. github.com/araalinetworks/api/golang is fetched and added to search_documents, correctly replacing github.com/araalinetworks/api because it is a longer module path. 3. Another version of github.com/araalinetworks/api is fetched. It is bad, so it isn't inserted in the usual way, but ReconcileSearch re-inserts the latest good version, which overwrites the longer module path inserted in step 2. Change-Id: If8ae59acfde7838c0ea48882d6beb0a457e125db Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/342469 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Reviewed-by: Julie Qiu <julie@golang.org>
2021-08-16internal/postgres: rename ReInsertLatestVersionJonathan Amsterdam
Rename to ReconcileSearch, which better describes its purpose. Change-Id: Ib7562e6bf511ff1ad88129619feec110c81168d3 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/342169 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-08-16internal/{worker,postgres}: check current module for alternativeJonathan Amsterdam
When reconciling the contents of search_documents when we insert a module, consider whether the version currently being processed is alternative. Previously, we didn't consider the current module because we only checked the DB, and the status code for the current module hasn't been inserted yet. Change-Id: Ibeb02d7f56341119e2bf4f20f148b345b2b82828 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/341868 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-08-13internal/{worker,postgres}: remove alternative modules from searchJonathan Amsterdam
When processing a module, if it is an alternative module, remove it from search_documents and imports_unique. Change-Id: I3fd12e8522331b8a4f060c9576d97c0ee710785d Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/341854 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com>
2021-08-13internal/postgres: factor out delete from imports_uniqueJonathan Amsterdam
Add a function that deletes a module from imports_unique, and call it where needed. Change-Id: I0209d09855bf939e644cac646f1b8918bc48e990 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/341852 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-08-12internal/postgres: remove a fully retracted module from searchJonathan Amsterdam
If all the versions of a module have been retracted, remove it from the search_documents table so it doesn't appear in search. Also remove it from the imports_unique table so it doesn't contribute to the imported-by counts of other modules. For golang/go#47590 Change-Id: I76a6e33a1291abe4a19f1b3521c9dc8ebed4f110 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/341591 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-08-11internal/postgres: on insert, delete only old packages from searchJonathan Amsterdam
Instead of deleting the entire module from search_documents on insert, just delete packages that are not in the current version. That will avoid resetting imported-by counts to zero when we reprocess the latest version of a module. For golang/go#47555 Change-Id: Ie2aedc920f0e840c20f4487ed10bf7801b43a3f8 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/341269 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-08-02internal/postgres: move DeleteModuleJulie Qiu
DeleteModule is moved to delete.go. Pure code in motion. Change-Id: I2a9ab37ce4b5dd584d4163f6bf2cb2e81565c69c Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/339151 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-07-27internal/postgres: delete old versions from search_documents on insertJonathan Amsterdam
When inserting the latest version of a module, delete all versions from search_documents before inserting the latest one. That will remove from search all packages that are not in the latest version. Change-Id: I58fe9a1e8974d7f3fbd1fa64212c4b0381895d09 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/337751 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-07-16internal/postgres: delete alternative module from search_documentsJonathan Amsterdam
When inserting a module whose latest version alternative, delete all versions of the module from search_documents. Before, we didn't insert the module into search_documents, but we did not delete other versions of that module. Change-Id: I87f14b883faf226be151e78605ec44cc216888b6 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/334950 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-07-13internal/postgres: fix bug in isAlternativeModuleJulie Qiu
At the moment, isAlternativeModulePath does not filter on the modulePath argument, which means that it reports whether the latest version of a random module has status 491, and not the specified module. This bug is fixed and a test is added. Change-Id: I7a0feb960e1f1e6b7de2fb7a82ec0ea2fdcce738 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/332378 Trust: Julie Qiu <julie@golang.org> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-07-13internal/postgres: change upsertSymbolSearchDocuments queryJulie Qiu
The upsertSymbolSearchDocuments query is changed to filter on module path and version, instead of passing an array of unit_ids. Additional tests are added to TestSymbolSearch. For golang/go#44142 Change-Id: Ic0c493711c2c8fcc32e207ea466c509e6f99209b Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/329924 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-07-13internal/postgres: do not insert symbols for commandsJulie Qiu
Symbols for commands don't appear in the documentation, so don't insert them for search. For golang/go#44142 Change-Id: I5178bfac19686245d5b5055aa38ad73c04ab9dce Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/333377 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-06-21internal/postgres: change logic for upsertSearchDocumentSymbolsJulie Qiu
upsertSearchDocumentSymbols is changed based on the new schema. For golang/go#44142 Change-Id: I1857b04cceeb9b702e4c54fc3a8d39efa811a3ce Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/329493 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-06-21internal/postgres: move insertSymbols to saveModuleJulie Qiu
Logic for insertSymbols is moved out of insertUnits to saveModule. This will allows us to run that code after the check for isLatest, which is necessary to insert symbols where the latest unit is not a release version. For golang/go#44142 Change-Id: Id917b6b74020957549ad1c6594f21d3f8d27e464 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/329349 Trust: Julie Qiu <julie@golang.org> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-06-18many: remove read-imports experimentJonathan Amsterdam
Remove the read-imports experiment and all references to the package_imports table. Change-Id: If926c47e1d01b3ab0f31bb6efb9c2234418fca1b Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/329229 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-06-14internal/postgres: don't touch package_importsJonathan Amsterdam
If the read-imports experiment is on, replace all references to the package_imports table with the imports table: don't write into or read from package_imports. Change-Id: I02631aba574b9938a44b06bce855997606984939 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/327749 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-06-10internal/postgres: write to imports tableJonathan Amsterdam
Write a package's import paths to the new imports table as well as the existing package_imports table. Change-Id: Ie0afe5aa90e79e8371a51cc6fc5f68de3dc62cd8 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/326449 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Reviewed-by: Julie Qiu <julie@golang.org>
2021-06-03internal/postgres: only skip documentation_symbolsJonathan Amsterdam
When skip-insert-symbols is on, skip only the documentation_symbols table, not the others. Change-Id: Ia1f6bf975f02a2cc5aad15cd220b8a6c54585e97 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/324790 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-06-03internal/postgres: add experiment to disable symbol insertionJonathan Amsterdam
Inserting symbols is causing errors because the generated sequence values overflow the integer type. Add an experiment that will let us turn off writing to the symbols tables until we can change the type. Change-Id: Ied3484a8db07df0f0795ff28de96760a3595ee67 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/324631 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-05-26internal/postgres: change symbol lock from table to pathJonathan Amsterdam
I noticed some contention around the table lock that upsertSymbolHistory was acquiring. (I looked at the logs of tasks that had failed, and noticed that their last statement was the LOCK, suggesting that they blocked on acquiring the lock and eventually timed out.) Instead of locking the table, we lock the module path, as we do elsewhere for latest-version inserts. That should avoid conflicts in upsertSymbolHistory and reduce contention, since two transactions will only block each other if they are working on different versions of the same module. We acquire the lock earlier than we need to, to avoid passing additional information down the call stack. That shouldn't matter much. Change-Id: I9c86e1ff6c638baade70fdee25e0ee31b30a03c4 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/321790 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-05-13internal/postgres: remove delete from documentationJonathan Amsterdam
Don't delete old rows from the documentation table. Change-Id: Idd2e7d5fc02887f0fdcea4c2297b4bd499bd8ad4 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/319909 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-05-06internal/postgres: use moduleID in getPackagesInUnitjunjunjunk
Add argument moduleID to getPackagesInUnit for more efficiency. Fixes golang/go#45854 Change-Id: Ib8c7721ae5991704c28f966c6750f17289442dc8 GitHub-Last-Rev: 4986eff55818bfacfeddcc8300d563aa12594187 GitHub-Pull-Request: golang/pkgsite#27 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/315489 Reviewed-by: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org> Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org>
2021-04-21internal/postgres: remove do-not-insert-new-documentationJulie Qiu
Change-Id: I4b5e4b4e6556f6f724dd01c08b7014c30b10a2b3 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/310313 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-04-16internal/postgres: add exp to getDocIDsForPathJulie Qiu
ExperimentDoNotInsertNewDocumentation is added to getDocIDsForPath, so that we can deprecate new_documentation in a future CL. Change-Id: I53b35d4e629974c70aec43ed0b4f124d46085dd6 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/310312 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-04-16internal/postgres: delete insert-symbolsJulie Qiu
Change-Id: I654b4b6186d49a9a8a83874747ab5ee9d909fe9a Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/310380 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-04-16internal: add experiment do-not-insert-new-documentationJulie Qiu
We want to rename new_documentation to documentation, and drop the current documentation table. INSERTs into new_documentation are added as a first step. Change-Id: I2f4038890436946b2c140c108faa4740e16abe4e Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/310375 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-04-13internal/postgres: remove uses of modules.deprecated_commentJonathan Amsterdam
The deprecated_column in the modules table has been superseded by the information in the latest_module_versions table. For golang/go#43265 Change-Id: Ib53e0b295a3edf8e807ff825b36baa6701b927b1 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/309610 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-03-30Revert "internal/postgres: remove documentation delete"Julie Qiu
This reverts commit 101343b2dd9dbde2dcbdf934e05d636cba8b7a26. Reason for revert: We still need to delete rows when an "all" row is inserted to prevent panics in https://go.googlesource.com/pkgsite/+/master/internal/build_context.go#51. Change-Id: Ic2dec11978774cde4bffeba040ef2e212385e994 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/305949 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> Reviewed-by: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com>
2021-03-30internal/postgres: use CopyUpsert to insert documentationJonathan Amsterdam
Insert into the documentation and new_documentation columns using the new CopyUpsert method. This saves some time. It saves even more space, because we can generate the rows one at a time instead of making a big slice of all the values. The difference can be quite significant on modules with many documentation rows. Change-Id: I859bdfaf90a204d755eb7a067da000014e0259a2 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/305831 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-03-25internal/postgres: remove documentation deleteJulie Qiu
The documentation rows for a given module no longer need to be deleted. Instead, we are planning to recreate the documentation table by inserting into new_documentation. Change-Id: Ia32957fb7947e793d8f8972091d16995d70db04a Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/304749 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-03-22internal/worker: call ReInsertLatestVersionJonathan Amsterdam
After processing a module, call ReInsertLatestVersion. Skip the call if this is the cooked latest version of an alternative path. The same check is made inside ReInsertLatestVersion, but using the module_version_states table, and the module being processed has not yet been inserted into that table. For golang/go#44710 Change-Id: Icd0d0c0045ccadf3e97d32c63146262a3b442577 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/303649 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com>
2021-03-22internal/postgres: add imports_unique to ReInsertLatestVersionJonathan Amsterdam
The ReInsertLatestVersion method now changes imports_unique as well as search_documents. For golang/go#44710 Change-Id: Ie5dcfe8e362313be76f0ad4958882e37fc725e44 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/303313 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-03-22internal/postgres: add ReInsertLatestVersionJonathan Amsterdam
New latest-version info can retract the current latest good version, resulting in an older version becoming the new latest good version. Add a function that updates the search_documents table when that happens. A later CL will also update imports_unique, and will call this function from the worker. For golang/go#44710 Change-Id: I09569e27d384ed7defb06a4e81facdb96c22caeb Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/303312 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Reviewed-by: Julie Qiu <julie@golang.org>
2021-03-19internal/postgres: insert into new_documentationJulie Qiu
Data is now inserted into the new_documentation table, which will replace the documentation table once reprocessing is done. The documentation_id rows for documentation_symbols now reference new_documentation. Tests for symbols which were deleted in CL 303116 are added back. TestFrontendMainPage/main_page_hello_-_multi_GOOS_JS_page is flaky and will be added back in a future CL. Change-Id: I8d7aed2092b3d19c45c14be5f9a0986dfe224aeb Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/303115 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-03-19internal/postgres: reimplement alternative-module checkJonathan Amsterdam
When inserting a module, we skip inserting into search_documents if the module path is an alternative one. This CL factors out that test, and also reimplements it in terms of the latest cooked version. Previously, any later version with a 491 status would classify the path as alternative, but this version uses the latest cooked (non-retracted) version's status to decide, allowing module authors to fix mistakes. Change-Id: I3588f3df6a1a4dd4db59333314c34167eda51df1 For golang/go#44710 Change-Id: I670d9085800fed198e6dd6469f1a8d5728c5f039 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/303309 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
generated by cgit v1.3 (git 2.53.0) at 2026-04-15 18:05:09 +0000