aboutsummaryrefslogtreecommitdiff
path: root/internal/postgres/search_test.go
AgeCommit message (Collapse)Author
2026-03-27all: run go fix -stringsseqHana Kim
Change-Id: Id22b6fbc926c516043a2a9869ce9042dedebc725 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/753433 Reviewed-by: Jonathan Amsterdam <jba@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2026-03-26all: fix rangeintHana Kim
Change-Id: I58ed5e7d2a3bbc13bc389ac7239737e18856dbfe Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/753427 Reviewed-by: Jonathan Amsterdam <jba@google.com> Reviewed-by: Ethan Lee <ethanalee@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2026-03-24all: pgx/v4 to v5 and upgrade pqEthan Lee
- In preparation for upgrade to PostgreSQL 17, upgrade stale dependencies within pkgsite. Change-Id: Id54cbb46182711b9808ee3bbbe2f0d5d9a04583d Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/758121 Auto-Submit: Ethan Lee <ethanalee@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Jonathan Amsterdam <jba@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2025-08-14internal/postgres: apply bypassLicenseCheck to IsRedistributable columnJean Barkhuysen
This enables symbol search for non-LICENSE'd repos when -bypass_license_check is supplied. I think the way this works is that when these get insert into units, we don't consider bypassLicenseCheck, and the IsRedistributable column gets set to false. That all seems reasonable by itself. But later, in upsertSymbolSearchDocuments, we only upsert symbol docs for units that are redistributable. That check happens in SQL, so it's hard to factor in the bypassLicenseCheck flag. It's not really clear to me whether we should "pollute" the database with this column set to true under the flag. If you run the program again with the flag turned off, your database keeps the values set under the old run. Anyways, I think this CL is probably the pragmatic take: it's a small code change that achieves the desired result without added complexity, and it's unlikely that anyone will run into the undesired outcome I describe above (who is running this program sometimes with the flag on, sometimes with it off?). Change-Id: Ie15bcf0605b945e4af6574abf689827ea946ddae Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/685457 Reviewed-by: Jonathan Amsterdam <jba@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: David Chase <drchase@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Commit-Queue: Jonathan Amsterdam <jba@google.com>
2025-01-21internal/postgres: search doesn't group stdlibJonathan Amsterdam
Search will display all standard library packages at top level. Previously it grouped them based on the top-level directory, meaning that some could be hidden. For example, searching for "path" appears to omit the standard library package "path". In fact, it is nested under "path/filepath", which is more popular. With this change, "path" would appear second. Updates golang/go#64358. Change-Id: I2f35862ac514df0891b6f5b9055b6bf0a7ae37c7 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/643317 Reviewed-by: Robert Findley <rfindley@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2024-02-22internal/worker: add batch param to update-imported-by-count routeJonathan Amsterdam
Inserting 5000 imported-by counts into search_documents timed out. Make that size configurable, so we can experiment with something smaller. Change-Id: Ie0c5f42a28c0468c600af6d836edc345529b769f Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/565682 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Michael Matloob <matloob@golang.org> kokoro-CI: kokoro <noreply+kokoro@google.com>
2024-02-21internal/postgres: support module@version exclusionsJonathan Amsterdam
Allow a specific module@version to be excluded. Previously, only entire module paths or path prefixes (at every version) could be excluded. This will allow us to exclude specific versions that we can't process, because they are too large for instance. A necessary part of the change is adding a version argument to the IsExcluded method. While we are there, we drop the error return value, since it's always nil. Change-Id: I68fd1c8f66ee654e56bd7dcf230a88959b32bbc8 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/565815 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Michael Matloob <matloob@golang.org> Run-TryBot: Jonathan Amsterdam <jba@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2024-01-23internal: move UnitMeta.IsRedistributable to UnitMichael Matloob
Change-Id: Ia0a18df32d752f20b0400b47860638c7d0910eb7 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/556435 Reviewed-by: Jonathan Amsterdam <jba@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Robert Findley <rfindley@google.com>
2023-08-23all: remove arbitrary hard-coded timeouts in testsBryan C. Mills
If a test times out, that implies that it got stuck on something. By default, the Go testing package dumps goroutines when its own timeout is passed, which prints a goroutine dump, helping to reveal what was stuck. Adding an arbitrary timeout on top of the testing package's own timeout is, in my experience, almost always counterproductive. If the arbitrary timeout catches a real hang, it causes the test to fail instead of dumping goroutines, making it much harder to see what was stuck. On the other hand, if the timeouts are set aggressively enough to make the test fail early, they are often too aggressive for CI testing, causing flakes that then have to be triaged on an ongoing basis. On balance, the value of saving a minute or two for developers who have introduced a hang is not worth the cost of suppressing debugging information and causing flakes that have to be triaged. Fixes #61556. For #59347. Change-Id: I0263d0d9b18283470f100e5a0155818b87b5312f Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/521837 TryBot-Result: Gopher Robot <gobot@golang.org> Auto-Submit: Bryan Mills <bcmills@google.com> Run-TryBot: Bryan Mills <bcmills@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Michael Matloob <matloob@golang.org>
2021-11-13internal, static, tests: remove search-grouping experimentJonathan Amsterdam
We now do search grouping by default. Change-Id: I698690f26784e9b668832e870a854bb5c95f449b Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/363737 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-10-25internal/postgres: record other module versions in SearchResultJonathan Amsterdam
Previously, a SearchResult stored only the paths of modules at other major versions. Now it remembers the major versions as well. This CL doesn't change any UI, but it is preparation for displaying the major versions of other modules instead of their paths. For golang/go#49069 Change-Id: I510eeb27aaab1b8764c72434760265d4c6ee9359 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/357290 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-09-14internal/postgres: change UpsertModuleVersionStates to updateJulie Qiu
UpsertModuleVersionStates is changed to UpdateModuleVersionStates. There should never be a situation where UpsertModuleVersionStates is called and a row does not already exist for that module. If that happens, an error is now returned. For golang/go#46985 Fixes golang/go#39628 Change-Id: I357396cee6eb743513ae249609f76f4cd4c19e9b Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/341860 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> Reviewed-by: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com>
2021-09-08internal,static: remove ApproximateJulie Qiu
Since we limit search results to 100, we never display the approximate number (even in the current search UI). Logic for approximate is deleted. Change-Id: I7ff123ea4e1da20d3a5a97a785e54f17e99bf03e Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/348111 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-09-07internal/postgres: change searcher signatureJulie Qiu
The searcher type is changed from the type: func(db *DB, ctx context.Context, q string, limit, offset, maxResultCount int) searchResponse to: func(db *DB, ctx context.Context, q string, limit int, opts SearchOptions) searchResponse More fields will be added to SearchOptions in a later CL. For golang/go#44142 Change-Id: I96e8b81e5fe10fcf3637d420eaf723c061699225 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/347609 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-09-03internal/search: fix stdlib groupingJonathan Amsterdam
Fix a bug that resulted in net/http/prof not being grouped with net/http. The bug arose from a mistake in extracting the top-level directory from a stdlib package path. After coming across the proposal for `strings.Cut` (https://golang.org/issue/46336), I was on the fence about adding a copy to this repo and using it here. Ultimately I decided that the job was simple enough to do without it. I was wrong. So this CL adds `Cut` and uses it here. We can incremently use it in other places where it simplifies things. Change-Id: I804aea93f3850bba52d9e0edde8ea136746093f8 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/347555 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-09-02internal/postgres: in search, group stdlib packages by top-level dirJonathan Amsterdam
When grouping search results, group packages from the standard library by their top-level directory, instead of treating them all as part of a single module. So "net", "net/http" and "net/url" will get grouped together. Change-Id: Ib638c39de56ecde104d569a607255d2b6a74ce55 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/346969 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-09-01internal/postgres: keep only one module major version in search groupingJonathan Amsterdam
When grouping search results by module, keep only the highest-scoring package from the highest tagged major version. Previously, we kept all major versions, and only ranked the higher one first, so as not to lose information (like import counts) for higher-scoring packages in a lower major version. But the results are unexpected. Fixes golang/go#47839 Change-Id: I66dedec470dd1588f02dcfed5604a6e63f020d62 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/345611 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-08-16internal/postgres: update imported-by counts in chunksJonathan Amsterdam
If too many rows change imported-by counts, then the /update-imported-by-count endpoint times out after 30 minutes, and nothing gets inserted because it runs as one transaction. Instead, break it into multiple smaller transactions. Since we only update changed counts, then as long as one transaction completes we have made some counts identical, so we will make progress. For golang/go#47555 Change-Id: I546aaabcc5e0f99d71efe38748475274871382c4 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/341249 Reviewed-by: Jamal Carvalho <jamal@golang.org> Reviewed-by: Julie Qiu <julie@golang.org> Trust: Julie Qiu <julie@golang.org> Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Julie Qiu <julie@golang.org>
2021-08-13internl/postgres: move delete functions into delete.goJonathan Amsterdam
Pure code motion. Change-Id: I80055dbb95f2341c72d3716b74e08335bdaa6fd4 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/341851 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-08-11internal/postgres: rename functionJonathan Amsterdam
For golang/go#47555 Change-Id: I9e90de83d441748e70b0992f3a31a02f46f1bce9 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/341429 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com>
2021-08-11internal/postgres: on insert, delete only old packages from searchJonathan Amsterdam
Instead of deleting the entire module from search_documents on insert, just delete packages that are not in the current version. That will avoid resetting imported-by counts to zero when we reprocess the latest version of a module. For golang/go#47555 Change-Id: Ie2aedc920f0e840c20f4487ed10bf7801b43a3f8 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/341269 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-08-06internal/postgres: prefer longer module paths in searchJonathan Amsterdam
If two packages have the same import path in different modules, then search_documents should store the one with the longer module path. Change-Id: I0c7afbe4f3d4e15152cf8383e973da175db90cf7 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/340122 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-07-27internal/postgres: delete old versions from search_documents on insertJonathan Amsterdam
When inserting the latest version of a module, delete all versions from search_documents before inserting the latest one. That will remove from search all packages that are not in the latest version. Change-Id: I58fe9a1e8974d7f3fbd1fa64212c4b0381895d09 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/337751 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-07-16internal/postgres: delete alternative module from search_documentsJonathan Amsterdam
When inserting a module whose latest version alternative, delete all versions of the module from search_documents. Before, we didn't insert the module into search_documents, but we did not delete other versions of that module. Change-Id: I87f14b883faf226be151e78605ec44cc216888b6 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/334950 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-07-14internal: move SearchResult to postgres packageJonathan Amsterdam
Change-Id: I73f0c7c35a5221302c27012c9118fa3bed5fdd62 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/334529 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-07-14internal/postgres: add test for path tokensJulie Qiu
A test is added which tests that output of TO_TSVECTOR('symbols', <path tokens>). For golang/go#44142 Change-Id: If1e105ad88fc7a79f83d33ead0db346af081015d Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/333335 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-07-14internal/postgres: put Search args in a structJonathan Amsterdam
There are enough arguments to Search that it is clearer to put them in an options struct. Also, add an option for DB query limit that is distinct from the MaxResults option (but defaults to it, as currently.) Change-Id: I191f2cb8d5e1eb49ec96af109ef1f1ee51fc3bdd Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/334251 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-07-13internal/postgres: move TestSymbolSearch to symbolsearch_test.goJulie Qiu
Pure code in motion. For golang/go#44142 Change-Id: I8a072ec35db33bfb573cd838a95626941a3d8dc7 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/330129 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-06-22internal/postgres: report row offsets from searchJonathan Amsterdam
Each SearchResult holds the offset of the row in search_documents that it came from. This will enable offset-based pagination. Change-Id: I2a8551a6eff972f795048c2b0207824275dc3d9b Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/330029 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-06-21internal: return symbol search metadataJulie Qiu
In addition to the symbol name, the symbol synopsis, kind, goos and goarch are also returned. For golang/go#44142 Change-Id: I954328faaef8e08734de5c77ef8976d24b6c7023 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/329495 Reviewed-by: Jonathan Amsterdam <jba@google.com> Trust: Julie Qiu <julie@golang.org>
2021-06-21internal/postgres: change logic for upsertSearchDocumentSymbolsJulie Qiu
upsertSearchDocumentSymbols is changed based on the new schema. For golang/go#44142 Change-Id: I1857b04cceeb9b702e4c54fc3a8d39efa811a3ce Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/329493 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-06-18internal/database: add ability to run queries incrementallyJonathan Amsterdam
Postgres lets you define a cursor and fetch rows from it as top-level statements, without needing to write any PS-SQL. See https://www.postgresql.org/docs/11/sql-declare.html and https://www.postgresql.org/docs/11/sql-fetch.html for details. Use this feature to define RunQueryIncrementally, which repeatedly fetches query rows in batches until it runs out or the passed function says it's done. If grouping by module paths is enabled, use RunQueryIncrementally with a very large limit to read rows until we've seen a page's worth of module paths. This CL doesn't handle pagination correctly. Any page after the first is going to be wrong. Change-Id: Idf8233160b0cf74412a688e1a6b95f4f2b720008 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/329469 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-06-17internal/{postgres,frontend}: fix search groupingJonathan Amsterdam
Grouping of search results was broken (the sort predicate was not transitive). This CL fixes it. Raises the score of the first package in the module with the highest major version of its series so that it appears first in the list of search results. Also, replace the list of search results of lower-major-version modules with a set of module paths at _all_ other major versions. Make corresponding changes to the frontend code and templates. Change-Id: I2236baa00d4edacf65246e2f65809a1f74b231c1 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/328335 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-06-15internal/postgres: skip TestSymbolSearchJulie Qiu
Temporarily skip TestSymbolSearch while the schema is changed. Change-Id: I2bd37f48512d6a85faf8d21d987886b08b5e2246 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/327950 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-05-17internal: add experiment InsertSymbolSearchDocumentsJulie Qiu
Change-Id: Ia10ca6eed9f0c21f967e590aa76d050aa2353a5b Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/320449 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-05-17internal/postgres: add symbol searchJulie Qiu
An initial version of symbol search is added. This runs separately from regular search and is not connected to the frontend yet. For golang/go#44142 Change-Id: I34a21a4a743a07dcfcdfeb64b52a86bc2254f62a Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/318870 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-05-12internal/postgres: ensure good version not later than cookedJonathan Amsterdam
The good latest version of a module should never be later than the cooked latest version. If it is, then pkgsite will show a different version than the go command will download. One way this can happen is if a module retracts all tagged versions, and pseudo-versions were built on top of some of them. For example, initially it could have versions v0.1.0 v0.1.0-DATE-HASH (a pseudo-version) Then v0.1.0 is retracted and a new pseudo-version appears at head: v0.1.0 (retracted) v0.1.0-DATE-HASH v0.0.0-DATE-HASH (head of default branch) The go command will get the v0.0.0 version, but technically the v0.1.0 is later. For golang/go#43265 Change-Id: I8ff30de4eb2dcdf108205de99af93d2f31772cff Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/318069 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-05-06internal/{postgres,frontend}: group search resultsJonathan Amsterdam
Group search results by module. Show the highest-ranking package of each module, and display the other packages as sub-results, in the same snippet as the winning package but smaller. Also, display packages from later major module versions before earlier ones, even if they have a lower score. All this is protected by an experiment. Change-Id: Iabc7fcf4e916289617b6c1c755904a27163ad554 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/316471 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-04-08internal/postgres,etc: finish MustInsertModule cleanupJonathan Amsterdam
Now that there are no callers of MustInsertModule, rename MustInsertModuleLatest to MustInsertModule. We can also remove MustInsertModuleLMV. Change-Id: Ieb554ee32696c168be4cc0a14ecece9f4c6b91b4 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/308271 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-04-07internal/postgres: use MustInsertModuleLatestJonathan Amsterdam
For most internal/postgres tests, update latest-version information when we insert a module. That is more like what actually happens on the worker, and some features now rely on it, so we should do it everywhere. Some tests still use a custom go.mod file, so we can't switch to MustInsertModuleLatest for those, because it uses a minimal go.mod file. Change-Id: Ie5030aee4ed9c837931fddb8757e37bf6373a8c7 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/307871 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com>
2021-03-16internal/worker,etc.: improvements to latest-version infoJonathan Amsterdam
- Fetch and update the latest-version info before processing a module rather than after, so the information in the DB matches what InsertModule is using. - Pass latest-version info to InsertModule so it can use it to decide whether the version being inserted is the latest version. - InsertModule writes the good latest version to the latest_module_versions table while holding the advisory lock, avoiding a race condition where two different versions both think they're the latest. For golang/go#44710 Change-Id: Id5e8fa7a384ec6d4d86257362fd33a8fcd5215f7 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/301929 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-03-15internal/postgres: upsert search with module path and version parametersJonathan Amsterdam
When we upsert a row in search_documents, we know the package, module and version. Use that information to select the information to upsert, instead of finding the latest version with an ORDER BY clause. Besides being unnecessarily inefficient, the ORDER BY is not accurate in the presence of retractions and other adjustments. Change-Id: I49a96add12ede3e909d70b5e55094c1b704a1822 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/301510 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-03-03internal/postgres: Search dedups license typesJonathan Amsterdam
When search retrieves a document, deduplicate its license types. Fixes golang/go#44655 Change-Id: I6cf010cf3efc14bcc7e948394aa3a3c73aea8267 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/297893 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-03-02internal/postgres: group UpsertModuleVersionState argsJonathan Amsterdam
Organize the many arguments to UpsertModuleVersionState into a struct. Change-Id: Ia0266af623ca2be3961fde6ba8e15ba5540cf36b Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/296810 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-02-17internal/sample: improve treatment of sample licensesJonathan Amsterdam
To avoid sharing between tests, which can lead to race conditions, create new sample license slices and structs each time we need them. This uncovered some problems in the postgres tests. - The UpsertModule test was changing the top-level module license, but didn't propagate that change through all the units. We sample.ReplaceLicense to address that. - The checkModule was filling in missing license with defaults, instead of the correct data. Change-Id: I6074914155eb33d6ed6882f81a9d57b20135e422 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/292929 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-02-12internal/postgres: parallelize testsJonathan Amsterdam
Run tests in parallel on separate databases. With four DBs, the time to run all the package's tests is cut in about half, from 50s to 25s (the time varies from run to run). Instead of a global variable holding a single DB, we set up a channel populated with DBs. Each test receives from the channel to get a DB to use, and sends it to the channel when done. Because a test may have to wait to get a DB, we can't set an individual timeout for some tests because acquiring the DB can take some time. Individual test timeouts aren't really important, because `go test` will time out the entire run after 10 minutes anyway. The only test that can't be run in parallel is TestSearch, because it reads metrics before and after each sub-test to compute a delta, thereby implicitly assuming that no other Search calls are running in parallel. Change-Id: Icb35dbd2f146e27d82d4eef81343cf9725252155 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/291449 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-02-11internal/postgres: add MustInsertModule for testingJonathan Amsterdam
Simplifies frequent calls to InsertModule. Change-Id: Iafcf5ec8d71eb9ab541ceecb2d85eb0d9b5119b1 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/291269 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-02-09internal/postgres: fix importGraph to insert fewer modulesJonathan Amsterdam
The importGraph function was supposed to insert two modules, but it was inserting many. This change speeds up TestSearch from about 40 seconds to about 5 seconds. Change-Id: Ie5a3d8fe20056fe0b63ad8cc8b5f86cc6811b156 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/290670 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-02-09internal/sample: generate sample doc dynamicallyJonathan Amsterdam
Generate all internal.Documentation values for testing during the test run, removing the need for running `go generate`. - Move godoc.DocumentationForTesting to sample.Documentation, now that the import cycle is broken. - Add sample.Doc for tests that don't care about the Documentation contents. - Delete gen_documentation.go, the program that previously generated the documentation, as well as its output, documentation.gen.go. Change-Id: I81116aa133837a80047031c890b00f30cdd2e825 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/290372 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-02-02content,internal/frontend: update count on importedby pageJulie Qiu
When loading the imported by page, fetch the imported by count from search_documents.num_imported_by, instead of limiting at 20,000. This will use the same logic for the imported by count with the main page and search page. For golang/go#39138 Change-Id: Iae188d01129b6e35cfd82b79e51a0ef12938c459 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/288849 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
generated by cgit v1.3 (git 2.53.0) at 2026-04-15 08:08:58 +0000