aboutsummaryrefslogtreecommitdiff
path: root/internal/postgres
AgeCommit message (Collapse)Author
7 daysinternal/postgres: allow matching exact prefix with exact:Madeline Kalil
Currently, the package exclusion process is case insensitive, because certain hosts such as GitHub are case insensitive. The only way to exclude a case sensitive version without also excluding all case insensitive matches is to add the prefix to caseSensitiveExcludedPaths in internal/frontend/details.go. This CL implements a new approach to excluding case sensitive paths: it enables specifying an "exact:" prefix in excluded.txt that will force matching the pattern exactly, instead of with case insensitivity. For golang/go#77822 Change-Id: I123ad2e13ff6673084006c6d51377308d95230ce Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/761521 kokoro-CI: kokoro <noreply+kokoro@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Ethan Lee <ethanalee@google.com>
2026-03-27all: run go fix -stringsseqHana Kim
Change-Id: Id22b6fbc926c516043a2a9869ce9042dedebc725 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/753433 Reviewed-by: Jonathan Amsterdam <jba@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2026-03-26all: run go fix -slicescontainsHana Kim
Change-Id: I14479d9e612dfa7eed9188206746af5b51c10201 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/753428 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Jonathan Amsterdam <jba@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2026-03-26internal: consolidate build context and unit resolution logicEthan Lee
- Centralize logic for resolving units and selecting the best matching build context. - New helpers MatchingBuildContext, SortedBuildContexts provide a consistent, efficient approach to build context selection. - DocumentationForBuildContext now finds the best match based on precdence rules rather than just returning the first match. - A new getUnitContext helper resolves unit metadata and the best build context in a single query. - GetSymbols and GetUnit are now both optimized to use pre-resolved IDs, simplifying their queries and removing redundant joins. - Replaced redundant matchingDoc functions in fetchdatasource and fakedatasource. Change-Id: I9207d3bfe03404483c816090a0b99666f14a36e3 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/759162 Reviewed-by: Jonathan Amsterdam <jba@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Auto-Submit: Ethan Lee <ethanalee@google.com>
2026-03-26all: fix rangeintHana Kim
Change-Id: I58ed5e7d2a3bbc13bc389ac7239737e18856dbfe Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/753427 Reviewed-by: Jonathan Amsterdam <jba@google.com> Reviewed-by: Ethan Lee <ethanalee@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2026-03-26internal/api: implement package symbols endpointEthan Lee
- Introduce a new ServePackageSymbols handler that utilizes the new db.GetSymbols method. - Consolidated module resolution logic shared by ServePackages into resolveModulePath. - Updated Datasource with GetSymbols, which provides a more efficient way to retrieve all symbols for a package at a given version and build context. This differs from the existing getPackageSymbols, since that will return symbols for the latest release/compatible version. - Refactored packageSymbolQueryJoin into a generic helper by removing hardcoded version filters. This allows GetSymbols to query for any specific version, while getPackageSymbols was updated to explicitly include release-only filters to preserve its existing behavior. - GetSymbols still utilizes the same underlying packageSymbolQueryJoin, but it will also attempt to match the most relevant BuildContext for the query. It will also provide a mapping of parent and child symbols. Change-Id: Ib18d2511d24ac6bc5b75c7b3809c4ce126245036 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/754864 Auto-Submit: Ethan Lee <ethanalee@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2026-03-25all: add LatestVersion to ModuleInfoEthan Lee
- Add LatestVersion field to internal.ModuleInfo and update tests accordingly. - Changed some test formatting to improve readability. Change-Id: I1238e54614ef276d219b31d125cd2cad6b5a66f7 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/756901 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Auto-Submit: Ethan Lee <ethanalee@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2026-03-24all: pgx/v4 to v5 and upgrade pqEthan Lee
- In preparation for upgrade to PostgreSQL 17, upgrade stale dependencies within pkgsite. Change-Id: Id54cbb46182711b9808ee3bbbe2f0d5d9a04583d Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/758121 Auto-Submit: Ethan Lee <ethanalee@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Jonathan Amsterdam <jba@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2026-03-24internal/api: implement module packages endpointEthan Lee
- Implement the ServeModulePackages handler and introduce GetModulePackages to return the list of packages for a given module version. Change-Id: I20c1618e2fdbf0126cb913665a1d7457b8951177 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/754863 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Auto-Submit: Ethan Lee <ethanalee@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2026-03-20internal: conditionally return documentation in GetUnitEthan Lee
- Large documentations blobs should avoid being returned unless explicitly requested. Add a WithDocsSource to enable retrieval of unit information without the large documentation source information. - This does not functionally change existing GetUnit callsites. Change-Id: Ie983b7eaeb61bb92f2716ae64a3d1e1ea738dbe2 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/756480 Reviewed-by: Jonathan Amsterdam <jba@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Auto-Submit: Ethan Lee <ethanalee@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2026-03-17all: fix minmaxHana Kim
Change-Id: I1eacfe303e724fc4dfe371b10a3762627162285d Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/753425 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Ethan Lee <ethanalee@google.com>
2026-03-13all: fix forvarHana Kim
Change-Id: I02d2be535d9e3ac1788306adcc78d997d0caf951 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/753422 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Ethan Lee <ethanalee@google.com>
2026-03-12internal/postgres: clean up duplicated comments in upsertSymbolSearchDocumentsEthan Lee
Change-Id: I463a3a7eaf00119c931743e6841b08339760dea3 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/754800 kokoro-CI: kokoro <noreply+kokoro@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Auto-Submit: Ethan Lee <ethanalee@google.com> Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
2026-02-03internal/postgres: optimize unit metadata lookup for unknown module pathsEthan Lee
When a user requests a package with a symbolic version (e.g., @master) and the module boundary is not yet known, pkgsite currently triggers a "greedy" index scan on the modules table's series_path index to find the best match. This change optimizes the lookup by explicitly providing candidate module paths using internal.CandidateModulePaths. This allows the Postgres optimizer to perform efficient point lookups on the module_path index rather than a linear scan. Fixes golang/go#77367 Change-Id: I072a72dc043ff7fb704977f07ba9fbd4ec821815 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/741300 kokoro-CI: kokoro <noreply+kokoro@google.com> Auto-Submit: Ethan Lee <ethanalee@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2025-11-28internal/postgres: fix %q verb uses with wrong typeDmitri Shuralyov
Caught early by the improved vet check gated behind the 1.26 language version combined with a tiplang builder that tests with 1.26 language version. Change-Id: I25d2375d694af39ad1b2d8f97f3920990f852a15 Cq-Include-Trybots: luci.golang.try:x_pkgsite-gotip-linux-amd64-tiplang Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/725300 Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Auto-Submit: Alan Donovan <adonovan@google.com> Reviewed-by: Alan Donovan <adonovan@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Commit-Queue: Alan Donovan <adonovan@google.com>
2025-11-17Revert "internal/postgres: utilize num_imports column in insert_module"Ethan Lee
This reverts commit 28eed86815232e48f5e757569fcd061f858142d4. Reason for revert: migration takes too long Change-Id: I279e62342c79d2604cda4e7c9819bdaa8aac752c Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/721101 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Robert Findley <rfindley@google.com>
2025-11-13internal/postgres: utilize num_imports column in insert_moduleEthan Lee
- Avoid recomputing number of imports for every query in getUnitWithAllFields. For golang/go#76284 Change-Id: I7f7d683efb1b2ff49e33681802896a65b1354d25 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/720261 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Robert Findley <rfindley@google.com> Auto-Submit: Ethan Lee <ethanalee@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2025-11-13internal/postgres: use JOINs in GetModuleVersionsToClean queryEthan Lee
- Previously, the query used EXCEPT clauses to filter module versions to clean from the database. - Now, using JOINs in the query avoids expensive sequential scans. For golang/go#76284 Change-Id: I103b99b633d57ab00f26d0b24c4d4a8993de4872 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/719800 kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Robert Findley <rfindley@google.com> Auto-Submit: Ethan Lee <ethanalee@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2025-11-13internal/testing: avoid data race to NonRedistributableLicenseRob Findley
During insertion, pkgsite may mutate non-redistributable licenses to purge their contents. This leads to data races and path dependence in tests that were reusing the sample.NonRedistributableLicense value. Turn this variable into a function, so that we return a new value each time. This revealed a logic error in TestGetLicensesBypass: the test which was nominally checking that the non-redistributable contents were returned, was in fact also emptying out the expected contents, probably to make the test pass given the bug fixed above. Change-Id: I957014d009ae41638599d61592766b7ae1c175e7 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/720260 Reviewed-by: Ethan Lee <ethanalee@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2025-11-10internal/postgres: remove the exclusive lock in upsertPathRob Findley
As described in golang/go#75959, the lock in upsertPath causes a significant amount of contention during module inserts. Meanwhile, the upsertPaths method does not acquire a lock, and does not seem to cause deadlocks. We still don't understand the deadlocks avoided by this lock, but it seems that a low level of failed transactions due to deadlock may be preferable to the contention we're seeing currently. Let's experiment with removing the lock. For golang/go#75959 Change-Id: I2d577b426724010e96ffc8129ae469825fdcb938 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/718720 kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Ethan Lee <ethanalee@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2025-11-05internal/postgres: exclude internal packages when doing shortcut resolutionNicholas S. Husin
pkgsite allows users to use a shortcut when looking up std packages. For example, pkg.go.dev/http will resolve to pkg.go.dev/net/http. Currently, this shortcut resolution also works for internal packages (e.g. pkg.go.dev/boring will resolve to pkg.go.dev/crypto/internal/boring), even though Go users would likely not care about internal packages. An unfortunate side-effect of this is that our shortcut resolution does not work for packages which has a similarly-named internal package: pkg.go.dev/synctest does not resolve to pkg.go.dev/testing/synctest as the existence of pkg.go.dev/internal/synctest makes the resolution ambiguous. This change updates our shortcut resolution logic to exclude all internal packages. For golang/go#76136 Change-Id: I83fcf32861079d051bc8dc3be29092480d53f667 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/717840 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Jonathan Amsterdam <jba@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Nicholas Husin <husin@google.com>
2025-10-29add github.com/cdktf/cdktf-provider-google-go/google/v16 to skip-symbolsEthan Lee
Change-Id: Ieac78460d008f79068804d6e9cbbbfb85fc2da24 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/716100 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2025-10-23internal/postgres: add skip-symbols list to saveModuleEthan Lee
- Modules in the skip-symbols list will not have symbol information inserted into postgres. Change-Id: Ia52b9eb2d7cab662b967fa4a74c40cced0e44be4 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/713060 Reviewed-by: Jonathan Amsterdam <jba@google.com> Reviewed-by: Peter Weinberger <pjw@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2025-08-14internal/postgres: apply bypassLicenseCheck to IsRedistributable columnJean Barkhuysen
This enables symbol search for non-LICENSE'd repos when -bypass_license_check is supplied. I think the way this works is that when these get insert into units, we don't consider bypassLicenseCheck, and the IsRedistributable column gets set to false. That all seems reasonable by itself. But later, in upsertSymbolSearchDocuments, we only upsert symbol docs for units that are redistributable. That check happens in SQL, so it's hard to factor in the bypassLicenseCheck flag. It's not really clear to me whether we should "pollute" the database with this column set to true under the flag. If you run the program again with the flag turned off, your database keeps the values set under the old run. Anyways, I think this CL is probably the pragmatic take: it's a small code change that achieves the desired result without added complexity, and it's unlikely that anyone will run into the undesired outcome I describe above (who is running this program sometimes with the flag on, sometimes with it off?). Change-Id: Ie15bcf0605b945e4af6574abf689827ea946ddae Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/685457 Reviewed-by: Jonathan Amsterdam <jba@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: David Chase <drchase@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Commit-Queue: Jonathan Amsterdam <jba@google.com>
2025-07-21internal/database: allow running db tests under different db nameJean Barkhuysen
On a mac, brew install postgresql installs postgres without the postgres user: it uses $USER instead. The current tests don't take that into account. This CL changes that by: - Amending the connecting URI to explicitly specify the db name (the current "" approach makes use of the postgres user, which by coincidence is the same name as the database we want: now instead of we specify the postgres user). - One user test is amended to use the env var provided user if necessary. Also adds some logging to make it more clear why failures around the URI are happening. Change-Id: I03d5b40a62d7034e00f45ba8151292688ab61de4 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/685995 Reviewed-by: Michael Knyszek <mknyszek@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Jonathan Amsterdam <jba@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Michael Pratt <mpratt@google.com>
2025-04-01internal/postgres: use a more straightforward return valuecuishuang
Change-Id: I954ebf3d4908839e7c7e746b142127887fd936c5 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/653216 Auto-Submit: Carlos Amedee <carlos@golang.org> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Robert Findley <rfindley@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Carlos Amedee <carlos@golang.org>
2025-02-18all: upgrade go directive to at least 1.23.0 [generated]Gopher Robot
By now Go 1.24.0 has been released, and Go 1.22 is no longer supported per the Go Release Policy (https://go.dev/doc/devel/release#policy). For golang/go#69095. [git-generate] (cd . && go get go@1.23.0 && go mod tidy && go fix ./... && go mod edit -toolchain=none) Change-Id: Iffce79b33bf11a871952a64d3dea922c50ebf412 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/649380 kokoro-CI: kokoro <noreply+kokoro@google.com> Auto-Submit: Gopher Robot <gobot@golang.org> Reviewed-by: Cherry Mui <cherryyz@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2025-02-06vuln: fix missed usage of Wrap after removing legacy vulndb supportKha Nguyen
When removing legacy vulndb support, some uses of derrors.Wrap were missed. This commit fixes those to ensure errors are wrapped properly. Change-Id: Iddf5bc9e72d0d21e329dcf1e4fb097cb1f449f0f Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/643475 Reviewed-by: Jonathan Amsterdam <jba@google.com> Reviewed-by: Robert Findley <rfindley@google.com> Auto-Submit: Jonathan Amsterdam <jba@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2025-01-27internal/postgres: add search knobsJonathan Amsterdam
This CL does not change the behavior of search. It adds two parameters to search, for experimentation. - Whether to perform the post-search grouping. This is exported for experiments. - The normalization argument to ts_rank. This is set to zero, which is the default. I added it to remind maintainers that this provides another option for adjusting search results. Change-Id: I90724b1499e7d3595e8cb0a8f7b8fa841a726de6 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/644675 kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Robert Findley <rfindley@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2025-01-21internal/postgres: search doesn't group stdlibJonathan Amsterdam
Search will display all standard library packages at top level. Previously it grouped them based on the top-level directory, meaning that some could be hidden. For example, searching for "path" appears to omit the standard library package "path". In fact, it is nested under "path/filepath", which is more popular. With this change, "path" would appear second. Updates golang/go#64358. Change-Id: I2f35862ac514df0891b6f5b9055b6bf0a7ae37c7 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/643317 Reviewed-by: Robert Findley <rfindley@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2025-01-03all: make function and struct comments match the namescuishuang
Change-Id: Ic939b5b002e881f8041ccb95114010fe7e751f9a Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/639558 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Ian Lance Taylor <iant@google.com> Auto-Submit: Ian Lance Taylor <iant@google.com>
2024-12-26internal/frontend/search: don't return 500 on timeoutJonathan Amsterdam
If searches time out, return 408 (request timed out) instead of 500. We previously did this for symbol searches, but sometimes package searches can also time out. Fixes #64422. Change-Id: I423845e0663866f9743e19ed768f68e6abde6250 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/638596 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2024-11-08internal/godoc: increase max doc HTML limitJonathan Amsterdam
Double the maximum allowed size of rendered documentation. Increase the maximum file size as well. Also, add information about the documentation size to logs and error messages, so we can see the amount by which the limit was exceeded next time. For golang/go#40576. Change-Id: Ia2aa11915d583e8e0c5ca0e273202a57679bd5f7 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/622556 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Cody Oss <codyoss@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Robert Findley <rfindley@google.com>
2024-11-08test/screentest: remove unneeded seedsJonathan Amsterdam
Remove modules that don't matter from the list of modules used to seed the database in ci mode. Also combine two identical tests. Change-Id: I979ed07ce1f3bdf0fa164e46643cffefcdce45e3 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/626003 Reviewed-by: Robert Findley <rfindley@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2024-09-04internal/postgres: insert module path even if not a packageJonathan Amsterdam
When inserting the paths of a module into the paths table, we were inserting all the package paths and the v1 module path (the module path without "/vN" suffix). That is almost always sufficient. But if a module has a version suffix and is not itself a package, then we were not inserting its path. The module path is needed by insertSymbols, so this was causing that function to fail. Add the module path, refactor the code to simplify it and allow testing, and add a test. (For a vN module the code also adds the v1 package path for every package. I think this is unnecessary, but it's too risky to remove it.) Change-Id: I8caa3dd945692a4ec0a1128199237de6b77aa719 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/609117 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Robert Findley <rfindley@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2024-09-04several: use stdlib functionsJonathan Amsterdam
Replace some functions with simpler code that uses recent functions added to the standard library. Change-Id: Ifcd570679a897fb165a4307c7ce9f0f53fccea05 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/609116 kokoro-CI: kokoro <noreply+kokoro@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Robert Findley <rfindley@google.com>
2024-04-05internal/postgres: exclusions should be case insensitiveRobert Findley
Update exclusion logic to be case insensitive. When there is no go.mod file, GitHub module paths may be spelled with any casing. As a result of this change, we will no longer be able to exclude only certain capitalizations of a module. Change-Id: I67342cf3543e795e7795aa74081541d7e6fed18f Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/576595 Reviewed-by: Jonathan Amsterdam <jba@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Auto-Submit: Robert Findley <rfindley@google.com>
2024-03-05worker: display request state on main pageJonathan Amsterdam
Add a RequestInfo.State field to record the current state of a request for display. Add a function internal.RequestState to for updating the state, and use it for long-running requests. Display the state on the worker main page. Change-Id: I1a04036982aba23cf27776fac75248972a1336f3 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/568515 Reviewed-by: Michael Matloob <matloob@golang.org> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2024-02-27internal/postgres: speed up imported-by countsJonathan Amsterdam
The SQL statement that updated imported-by counts was using an unindexed column (package_path) in search_documents, so it ran very slowly (5 minutes to update 1000 rows, for example.) Switching to an indexed column (package_path_id) should make it faster. Change-Id: I8a32aff6ed43870af4508c382e3d9dabcec3155d Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/567137 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Michael Matloob <matloob@golang.org>
2024-02-22internal/worker: add batch param to update-imported-by-count routeJonathan Amsterdam
Inserting 5000 imported-by counts into search_documents timed out. Make that size configurable, so we can experiment with something smaller. Change-Id: Ie0c5f42a28c0468c600af6d836edc345529b769f Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/565682 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Michael Matloob <matloob@golang.org> kokoro-CI: kokoro <noreply+kokoro@google.com>
2024-02-21internal/postgres: support module@version exclusionsJonathan Amsterdam
Allow a specific module@version to be excluded. Previously, only entire module paths or path prefixes (at every version) could be excluded. This will allow us to exclude specific versions that we can't process, because they are too large for instance. A necessary part of the change is adding a version argument to the IsExcluded method. While we are there, we drop the error return value, since it's always nil. Change-Id: I68fd1c8f66ee654e56bd7dcf230a88959b32bbc8 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/565815 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Michael Matloob <matloob@golang.org> Run-TryBot: Jonathan Amsterdam <jba@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2024-01-23internal: move UnitMeta.IsRedistributable to UnitMichael Matloob
Change-Id: Ia0a18df32d752f20b0400b47860638c7d0910eb7 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/556435 Reviewed-by: Jonathan Amsterdam <jba@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Robert Findley <rfindley@google.com>
2024-01-22internal: move UnitMeta.Licenses to UnitMichael Matloob
Change-Id: If2b3c1acd4830948e2d134517608f89771a7abcb Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/555735 kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Robert Findley <rfindley@google.com>
2023-12-18internal/postgres: delete dead code depending on goldmarkMichael Matloob
For golang/go#61399 Change-Id: Iac6e41368d2bfbc0ff1c4c02aa51671ff5a24b82 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/550037 kokoro-CI: kokoro <noreply+kokoro@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2023-08-25internal/config: separate config initialization into serverconfigMichael Matloob
This change creates a new package that does config initialization and other GCP-specific operations that were previously done in package config, so that config can have no cloud dependencies. For golang/go#61399 Change-Id: I8d78294834e325b47d838892a1cef87003a4b90a Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/522516 Run-TryBot: Michael Matloob <matloob@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Robert Findley <rfindley@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2023-08-23all: remove arbitrary hard-coded timeouts in testsBryan C. Mills
If a test times out, that implies that it got stuck on something. By default, the Go testing package dumps goroutines when its own timeout is passed, which prints a goroutine dump, helping to reveal what was stuck. Adding an arbitrary timeout on top of the testing package's own timeout is, in my experience, almost always counterproductive. If the arbitrary timeout catches a real hang, it causes the test to fail instead of dumping goroutines, making it much harder to see what was stuck. On the other hand, if the timeouts are set aggressively enough to make the test fail early, they are often too aggressive for CI testing, causing flakes that then have to be triaged on an ongoing basis. On balance, the value of saving a minute or two for developers who have introduced a hang is not worth the cost of suppressing debugging information and causing flakes that have to be triaged. Fixes #61556. For #59347. Change-Id: I0263d0d9b18283470f100e5a0155818b87b5312f Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/521837 TryBot-Result: Gopher Robot <gobot@golang.org> Auto-Submit: Bryan Mills <bcmills@google.com> Run-TryBot: Bryan Mills <bcmills@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Michael Matloob <matloob@golang.org>
2023-08-22internal/testing/fakedatasource: implement PostgresDB interfaceMichael Matloob
This change adds the rest of the methods from the PostgresDB interface to the fake data source so it can be used to test some functions that expect a postgres db. Most of the methods are left unimplemented. I removed the IsPostgresDB that I originally put on the PostgresDB interface since now we will have other types try to be PostgresDBs. In an upcoming CL, I will extend the DataSource interface so that it's the same as PostgresDB, and just have the frontend handle errors that are errors.ErrUnsupported. The CL then updates tests on some of the methods on the frontend that need a PostgresDB to use the FakeDataSource instead. This CL then removes the code that sets up and runs the internal/frontend tests with a PostgresDB. For golang/go#61399 Change-Id: I3bae5adc5ec03536b2074eee32b4e5558dfa7478 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/520878 Run-TryBot: Michael Matloob <matloob@golang.org> Reviewed-by: Robert Findley <rfindley@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>
2023-08-04internal/middleware: move stats to its own packageMichael Matloob
Make a package internal/middleware/stats for middleware.Stats and middleware.ElapsedStat. This is part of removing the dependency from internal/frontend on internal/middleware. For golang/go#61399 Change-Id: I44afbfc9b9e28e1caabab8fe700376ec026c863d Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/514521 TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Michael Matloob <matloob@golang.org> Reviewed-by: Jamal Carvalho <jamal@golang.org> kokoro-CI: kokoro <noreply+kokoro@google.com>
2023-07-18internal: add an interface for internal/postgres.DBMichael Matloob
And use that interface to remove the dependency from internal/frontend to internal/postgres. The dependency still exists in the test code. That will be taken care of in another CL. For #61399 Change-Id: Ic1b3c257e6eb34858cf31b7596251aaff71d3482 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/510417 kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Reviewed-by: Michael Matloob <matloob@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Michael Matloob <matloob@golang.org>
2023-07-18internal: remove repetitive wordscui fliter
Change-Id: Ie5a688c08ba46018f0fa8f5a1af495e9c29fd684 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/509695 TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org> Reviewed-by: Robert Findley <rfindley@google.com> Run-TryBot: Ian Lance Taylor <iant@golang.org> Auto-Submit: Ian Lance Taylor <iant@golang.org> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Bryan Mills <bcmills@google.com> Run-TryBot: Robert Findley <rfindley@google.com>
generated by cgit v1.3 (git 2.53.0) at 2026-04-15 07:32:16 +0000