aboutsummaryrefslogtreecommitdiff
path: root/internal/fetchdatasource
AgeCommit message (Collapse)Author
2026-03-30internal/api: implement package imported-by endpointEthan Lee
- Implement ServePackageImportedBy handler and GetImportedBy and GetImportedByCount datasource methods. Change-Id: I8c4cc65fbff7172eaf48e5426e4f3f41c82bd38e Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/754865 Reviewed-by: Jonathan Amsterdam <jba@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Auto-Submit: Ethan Lee <ethanalee@google.com>
2026-03-26internal: consolidate build context and unit resolution logicEthan Lee
- Centralize logic for resolving units and selecting the best matching build context. - New helpers MatchingBuildContext, SortedBuildContexts provide a consistent, efficient approach to build context selection. - DocumentationForBuildContext now finds the best match based on precdence rules rather than just returning the first match. - A new getUnitContext helper resolves unit metadata and the best build context in a single query. - GetSymbols and GetUnit are now both optimized to use pre-resolved IDs, simplifying their queries and removing redundant joins. - Replaced redundant matchingDoc functions in fetchdatasource and fakedatasource. Change-Id: I9207d3bfe03404483c816090a0b99666f14a36e3 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/759162 Reviewed-by: Jonathan Amsterdam <jba@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Auto-Submit: Ethan Lee <ethanalee@google.com>
2026-03-26internal/api: implement package symbols endpointEthan Lee
- Introduce a new ServePackageSymbols handler that utilizes the new db.GetSymbols method. - Consolidated module resolution logic shared by ServePackages into resolveModulePath. - Updated Datasource with GetSymbols, which provides a more efficient way to retrieve all symbols for a package at a given version and build context. This differs from the existing getPackageSymbols, since that will return symbols for the latest release/compatible version. - Refactored packageSymbolQueryJoin into a generic helper by removing hardcoded version filters. This allows GetSymbols to query for any specific version, while getPackageSymbols was updated to explicitly include release-only filters to preserve its existing behavior. - GetSymbols still utilizes the same underlying packageSymbolQueryJoin, but it will also attempt to match the most relevant BuildContext for the query. It will also provide a mapping of parent and child symbols. Change-Id: Ib18d2511d24ac6bc5b75c7b3809c4ce126245036 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/754864 Auto-Submit: Ethan Lee <ethanalee@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2026-03-24internal/api: implement module packages endpointEthan Lee
- Implement the ServeModulePackages handler and introduce GetModulePackages to return the list of packages for a given module version. Change-Id: I20c1618e2fdbf0126cb913665a1d7457b8951177 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/754863 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Auto-Submit: Ethan Lee <ethanalee@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2026-03-23internal/api: implement module versions endpointEthan Lee
- Implement module versions endpoint using ds.GetVersionsForPath - Introduce paginate helper to generalize paginaton logic - Update fakedatasource to correctly use V1Path to return all versions Change-Id: Icc028bf8ca9c13978bb6eba84afe9736ccd6bcee Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/754862 Reviewed-by: Jonathan Amsterdam <jba@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Auto-Submit: Ethan Lee <ethanalee@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2026-03-20internal: conditionally return documentation in GetUnitEthan Lee
- Large documentations blobs should avoid being returned unless explicitly requested. Add a WithDocsSource to enable retrieval of unit information without the large documentation source information. - This does not functionally change existing GetUnit callsites. Change-Id: Ie983b7eaeb61bb92f2716ae64a3d1e1ea738dbe2 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/756480 Reviewed-by: Jonathan Amsterdam <jba@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Auto-Submit: Ethan Lee <ethanalee@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2026-02-23frontend: fix loading stdlib in direct_proxy modeEthan Lee
- internal/fetch: fix stdlib path extraction by removing invalid "std/" prefix. - internal/fetchdatasource: fix "std" proxy errors in GetLatestInfo. - cmd/frontend: prioritize stdlib getter and add local GOROOT optimization. Reproduction: - `go run ./cmd/frontend -dev -direct_proxy` should utilize the local GOROOT as the source for stdlib. - `go run ./cmd/frontend -direct_proxy` should enable one to via a stdlib library at `localhost:8080/std` or any other stdlib pkg. Fixes golang/go#77112 Change-Id: I0686922804178129550d9bb9edc87b2adff51aa6 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/746700 Auto-Submit: Ethan Lee <ethanalee@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Nicholas Husin <husin@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Nicholas Husin <nsh@golang.org>
2025-04-17internal/fetchdatasource: check for error earlierJonathan Amsterdam
Check for an error before using the other return value. Fixes golang/go#73377. Change-Id: Idfe6b53a179aac7f4b49ccf009150fa09bef8956 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/665638 Reviewed-by: Michael Matloob <matloob@golang.org> kokoro-CI: kokoro <noreply+kokoro@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-04-16all: fix function names in commentcuishuang
Change-Id: I25ba419d2fc5878e14cf3b5026724e17efaaa39d Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/579177 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: qiu laidongfeng2 <2645477756@qq.com> Reviewed-by: Cherry Mui <cherryyz@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Ian Lance Taylor <iant@google.com> Auto-Submit: Ian Lance Taylor <iant@golang.org>
2024-02-06internal/fetch: split FetchModule in twoMichael Matloob
First we get a LazyModule that essentially just contains the UnitMetas for each of the modules and then we process each Unit as needed so that FetchDataSource doesn't need to process all the Units when it just needs one. To construct at FetchResult, FetchModule will now get a LazyModule and then compute all the Units. A compromise to get this to work is that FetchDataSource will no longer populate Synopsis, IsRedistributable and Licenses on Subdirectories to avoid computing all those fields for all units when returning a single unit. MainVersion and MasterVersion are removed from FetchResult because they were only set on the internal/worker.FetchTask struct that embeds a FetchResult. Change-Id: Ia0db850ae570d421712ec484ee8b7815a779128e Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/557818 Run-TryBot: Michael Matloob <matloob@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> kokoro-CI: kokoro <noreply+kokoro@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Jonathan Amsterdam <jba@google.com> Reviewed-by: Robert Findley <rfindley@google.com>
2024-01-23internal: move UnitMeta.IsRedistributable to UnitMichael Matloob
Change-Id: Ia0a18df32d752f20b0400b47860638c7d0910eb7 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/556435 Reviewed-by: Jonathan Amsterdam <jba@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Robert Findley <rfindley@google.com>
2023-08-31all: skip tests that can't run on wasm platformsMichael Matloob
Some because the network support isn't enough for httpreplay, and some because we need the Go command. For golang/go#61209 Change-Id: I02c6b7d10553d931c746723ce9608f39834c131c Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/524797 Reviewed-by: Robert Findley <rfindley@google.com> Run-TryBot: Michael Matloob <matloob@golang.org> kokoro-CI: kokoro <noreply+kokoro@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> TryBot-Result: Gopher Robot <gobot@golang.org>
2023-08-30internal/lru: add a really simple LRU cache implementationMichael Matloob
To replace github.com/hashicorp/golang-lru. The size used with the cache in fetchdatasource is 100, so the efficiency of the cache is not super important. For golang/go#61399 Change-Id: I48383a4d1c00c4d153c0bad7b0a1a44c026b2314 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/524458 TryBot-Result: Gopher Robot <gobot@golang.org> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Run-TryBot: Michael Matloob <matloob@golang.org> Reviewed-by: Robert Findley <rfindley@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2023-08-10cmd/pkgsite: more changes to use local stdlibMichael Matloob
This cl makes more changes to enable using the local stdlib. It always uses the local stdlib from cmd/pkgsite. It also changes code that processes and shows versions to expect and pass through v0.0.0, the fake version we use when we show local packages. For #57742 Change-Id: Ic407e58e9658b0703dbc6df47f40bbe1f1bd2b2a Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/517915 kokoro-CI: kokoro <noreply+kokoro@google.com> Run-TryBot: Michael Matloob <matloob@golang.org> Reviewed-by: Robert Findley <rfindley@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>
2023-07-06internal/testenv: add and use MustHaveExternalNetwork functionMichael Matloob
This CL adds a copy of the MustHaveExternalNetwork function from the testenv package in the Go repo for use for filtering out tests that need the network access from running on builders that don't guarantee it. (External network access is only enabled on longtest builders). It also calls the function from two tests that are failing because they make external network calls. For #61209 Change-Id: I68ed1d3f8bb65697b1025bca0ba40aa0ab7423a3 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/508295 Reviewed-by: Robert Findley <rfindley@google.com> Run-TryBot: Michael Matloob <matloob@google.com> TryBot-Bypass: Michael Matloob <matloob@google.com> Reviewed-by: Michael Matloob <matloob@google.com>
2023-04-14internal/fetchdatasource: remove arbitrary test timeoutRob Findley
Kokoro tests are failing with this arbitrary timeout. Remove it. Updates golang/go#59608 Change-Id: Ifd3d8cf02d5c14f54e505fa5812e8de7dbc6c5d5 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/484738 TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com> Run-TryBot: Robert Findley <rfindley@google.com>
2023-03-15cmd/pkgsite: hide irrelevant content in local mode; add quick linksRobert Findley
This CL contains several UI improvements when running pkgsite in local mode via cmd/pkgsite: - hide irrelevant or inaccurate content - fix a bug using -gopath_mode (trim the `go env GOPATH` output) - fix panics navigating to vuln pages - link the GO button to the root page, rather than go.dev - add quick-links to the homepage to browse local modules Also: - add -dev and -static flags, to facilitate development - replace TestBuildGetters with more cases in TestServer - fix some redundancy in setting the <title> element for pages; consolidate on using the basePage.HTMLTitle value Updates golang/go#40371 Change-Id: I459a0d0fd39897dd4f902dd023aec653a3fb12cd Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/475755 TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Run-TryBot: Robert Findley <rfindley@google.com>
2023-03-14cmd/pkgsite: add multi-module support, invalidation, and searchRobert Findley
This change adds a new goPackagesModuleGetter, which uses x/tools/go/packages to query package information for modules requested by the pkgsite command. Additionally, add new extension interfaces that allow getters to add support for search and content invalidation. The go/packages getter uses these extensions to implement search (via a simple fuzzy-matching algorithm copied from x/tools), and invalidation via statting package files. Along the way, refactor slightly for testing ergonomics. Updates golang/go#40371 Updates golang/go#50229 Fixes golang/go#54479 Change-Id: Iea91a4d6327707733cbbc4f74a9d93052f33e295 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/474295 TryBot-Result: kokoro <noreply+kokoro@google.com> Run-TryBot: Robert Findley <rfindley@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2022-01-05internal/fetchdatasource: add directory informationJonathan Amsterdam
When fetching a module, populate information about unit subdirectories. Normally (when using a DB) we don't need to do this because a DB query synthesizes it from the units table, but FetchDataSource returns information for immediate consumption so it has to do it on its own. One difference between this and the DB path: this won't have submodules, since FetchDataSource processes one module at a time. Fixes golang/go#49847 Change-Id: I7a215e03dd9cebe31a14c3c118c42978f3f7c8a3 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/375714 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-10-28many: cmd/pkgsite embeds static assetsJonathan Amsterdam
The `pkgsite` command now embeds all the static assets it needs, so it need not be told the location of the static/ directory with a flag. The binary is self-contained and can be placed and invoked from anywhere. This required embedding the static/ and third_party/ directories. Since //go:embed cannot reference files outside the containing package's tree, we had to add trivial Go packages in static/ and third_party/ to construct `embed.FS`s for those directories. Also, the frontend needed to accept `fs.FS` values where it previously took paths, and `template.TrustedFS` values where it previously used `template.TrustedSources`. We ended up clumsily requiring four separate config values: - A TrustedFS to load templates. - Two fs.FSs, one for static and one for third_party, to load other assets. - The path to the static directory as a string, solely to support dynamic loading in dev mode. For golang/go#48410 Change-Id: I9eeb351b1c6f23444b9e65b60f2a1d3905d59ef9 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/359395 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-09-09internal/fetchdatasource: fix test to work on macsJonathan Amsterdam
Accept a wider range of URLs to accommodate Mac temp dirs. Change-Id: I2933b71ab1956a4fbf504e842d3ec2fe5c56e127 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/348795 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Reviewed-by: Julie Qiu <julie@golang.org>
2021-09-07cmd/pkgsite,internal: serve sourceJonathan Amsterdam
- Set up a /files endpoint on the server that can serve files from fs.FS implementations. - Add source.FilesInfo, which returns a source.Info that links to /files paths. Use it to implement the SourceInfo method of the local ModuleGetters. - Add a SourceFS method to the local MethodGetters so they can tell the server how to serve their files. - Improve the tests in cmd/pkgsite to verify the source links. Fixes golang/go#47982 Change-Id: Iff42bad15c4abaf408a364e58b7c0f0dad60b40d Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/347932 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-09-07internal/fetch: get source info from module getterJonathan Amsterdam
Make it the ModuleGetter's responsibility to provide links to source files, instead of passing around a separate source.Client. This will allow getters that fetch from disk to serve those local files. For golang/go#47982 Change-Id: I6c7cd7890b03ad61d9410850a510c1c9a63f1c9a Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/347749 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-09-03internal/fetchdatasource: cache resolved latest versionJonathan Amsterdam
After successfully fetching a module at latest, cache it at its resolved version. This avoids a second fetch when the frontend redirects "latest". Change-Id: I127029e7818205f7ef3b74e8723aac753f53a896 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/347550 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-09-01internal/{worker/fetch}: include DB activity in load-sheddingJonathan Amsterdam
Move the load-shedding logic to the worker and have it span both the fetch and processing of the module (as previously) as well as inserting it into the database. This is a more accurate estimation of load, since running a lot of concurrent queries definitely slows down processing. Most of the time this won't make much difference, but under high load, such as when processing multiple large modules, it will reduce DB contention and should result in greater throughput. For golang/go#48010 Change-Id: I7d0922e02d00182e867fd3b29fc284c32ecab5ee Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/346749 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-08-26cmd/pkgsite: support the module cacheJonathan Amsterdam
With the -cache flag, fetch modules from the module cache. For golang/go#47780 Change-Id: I2aa6467955cd90a80ccae559f27928cca6e06079 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/345272 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Reviewed-by: Julie Qiu <julie@golang.org>
2021-08-26internal/fetchdatasource: handle build contexts properlyJonathan Amsterdam
GetUnit returns a Unit with its Documentation set to a matching BuildContext. This is the same behavior as postgres.DB.GetUnit. For golang/go#47780 Change-Id: I2bc23b7bc5a006e78bec54f6f3229e59ab5a03ef Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/345269 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Reviewed-by: Julie Qiu <julie@golang.org>
2021-08-26internal/fetchdatasource: GetUnitMeta returns NotFound on missing packageJonathan Amsterdam
If a module exists but the package path is not in it, return NotFound. For golang/go#47780 Change-Id: If3a6602df4b99c8470020e8538e01c685880d86d Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/345251 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Reviewed-by: Julie Qiu <julie@golang.org>
2021-08-26internal/fetchdatasource: ignore latest-version errorsJonathan Amsterdam
A FetchDataSource may be configured with a proxy for latest-version information while still being used for local modules. So ignore latest-version errors. For golang/go#47780 Change-Id: I30448e133faec80002194df59e6f2db449f80625 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/345249 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Reviewed-by: Julie Qiu <julie@golang.org>
2021-08-25internal/fetchdatasource: rename from datasourceJonathan Amsterdam
Rename the package and type by including the word "fetch," to describe what it does and distinguish it from other DataSources. For golang/go#47780 Change-Id: I7c674d8ba3dc16084a857039b0cc2b3147f27a29 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/344953 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
generated by cgit v1.3 (git 2.53.0) at 2026-04-15 07:44:21 +0000