aboutsummaryrefslogtreecommitdiff
path: root/internal/fetchdatasource/fetchdatasource.go
AgeCommit message (Collapse)Author
2026-03-30internal/api: implement package imported-by endpointEthan Lee
- Implement ServePackageImportedBy handler and GetImportedBy and GetImportedByCount datasource methods. Change-Id: I8c4cc65fbff7172eaf48e5426e4f3f41c82bd38e Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/754865 Reviewed-by: Jonathan Amsterdam <jba@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Auto-Submit: Ethan Lee <ethanalee@google.com>
2026-03-26internal: consolidate build context and unit resolution logicEthan Lee
- Centralize logic for resolving units and selecting the best matching build context. - New helpers MatchingBuildContext, SortedBuildContexts provide a consistent, efficient approach to build context selection. - DocumentationForBuildContext now finds the best match based on precdence rules rather than just returning the first match. - A new getUnitContext helper resolves unit metadata and the best build context in a single query. - GetSymbols and GetUnit are now both optimized to use pre-resolved IDs, simplifying their queries and removing redundant joins. - Replaced redundant matchingDoc functions in fetchdatasource and fakedatasource. Change-Id: I9207d3bfe03404483c816090a0b99666f14a36e3 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/759162 Reviewed-by: Jonathan Amsterdam <jba@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Auto-Submit: Ethan Lee <ethanalee@google.com>
2026-03-26internal/api: implement package symbols endpointEthan Lee
- Introduce a new ServePackageSymbols handler that utilizes the new db.GetSymbols method. - Consolidated module resolution logic shared by ServePackages into resolveModulePath. - Updated Datasource with GetSymbols, which provides a more efficient way to retrieve all symbols for a package at a given version and build context. This differs from the existing getPackageSymbols, since that will return symbols for the latest release/compatible version. - Refactored packageSymbolQueryJoin into a generic helper by removing hardcoded version filters. This allows GetSymbols to query for any specific version, while getPackageSymbols was updated to explicitly include release-only filters to preserve its existing behavior. - GetSymbols still utilizes the same underlying packageSymbolQueryJoin, but it will also attempt to match the most relevant BuildContext for the query. It will also provide a mapping of parent and child symbols. Change-Id: Ib18d2511d24ac6bc5b75c7b3809c4ce126245036 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/754864 Auto-Submit: Ethan Lee <ethanalee@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2026-03-24internal/api: implement module packages endpointEthan Lee
- Implement the ServeModulePackages handler and introduce GetModulePackages to return the list of packages for a given module version. Change-Id: I20c1618e2fdbf0126cb913665a1d7457b8951177 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/754863 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Auto-Submit: Ethan Lee <ethanalee@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2026-03-23internal/api: implement module versions endpointEthan Lee
- Implement module versions endpoint using ds.GetVersionsForPath - Introduce paginate helper to generalize paginaton logic - Update fakedatasource to correctly use V1Path to return all versions Change-Id: Icc028bf8ca9c13978bb6eba84afe9736ccd6bcee Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/754862 Reviewed-by: Jonathan Amsterdam <jba@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Auto-Submit: Ethan Lee <ethanalee@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2026-02-23frontend: fix loading stdlib in direct_proxy modeEthan Lee
- internal/fetch: fix stdlib path extraction by removing invalid "std/" prefix. - internal/fetchdatasource: fix "std" proxy errors in GetLatestInfo. - cmd/frontend: prioritize stdlib getter and add local GOROOT optimization. Reproduction: - `go run ./cmd/frontend -dev -direct_proxy` should utilize the local GOROOT as the source for stdlib. - `go run ./cmd/frontend -direct_proxy` should enable one to via a stdlib library at `localhost:8080/std` or any other stdlib pkg. Fixes golang/go#77112 Change-Id: I0686922804178129550d9bb9edc87b2adff51aa6 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/746700 Auto-Submit: Ethan Lee <ethanalee@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Nicholas Husin <husin@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Nicholas Husin <nsh@golang.org>
2025-04-17internal/fetchdatasource: check for error earlierJonathan Amsterdam
Check for an error before using the other return value. Fixes golang/go#73377. Change-Id: Idfe6b53a179aac7f4b49ccf009150fa09bef8956 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/665638 Reviewed-by: Michael Matloob <matloob@golang.org> kokoro-CI: kokoro <noreply+kokoro@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-04-16all: fix function names in commentcuishuang
Change-Id: I25ba419d2fc5878e14cf3b5026724e17efaaa39d Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/579177 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: qiu laidongfeng2 <2645477756@qq.com> Reviewed-by: Cherry Mui <cherryyz@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Ian Lance Taylor <iant@google.com> Auto-Submit: Ian Lance Taylor <iant@golang.org>
2024-02-06internal/fetch: split FetchModule in twoMichael Matloob
First we get a LazyModule that essentially just contains the UnitMetas for each of the modules and then we process each Unit as needed so that FetchDataSource doesn't need to process all the Units when it just needs one. To construct at FetchResult, FetchModule will now get a LazyModule and then compute all the Units. A compromise to get this to work is that FetchDataSource will no longer populate Synopsis, IsRedistributable and Licenses on Subdirectories to avoid computing all those fields for all units when returning a single unit. MainVersion and MasterVersion are removed from FetchResult because they were only set on the internal/worker.FetchTask struct that embeds a FetchResult. Change-Id: Ia0db850ae570d421712ec484ee8b7815a779128e Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/557818 Run-TryBot: Michael Matloob <matloob@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> kokoro-CI: kokoro <noreply+kokoro@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Jonathan Amsterdam <jba@google.com> Reviewed-by: Robert Findley <rfindley@google.com>
2024-01-23internal: move UnitMeta.IsRedistributable to UnitMichael Matloob
Change-Id: Ia0a18df32d752f20b0400b47860638c7d0910eb7 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/556435 Reviewed-by: Jonathan Amsterdam <jba@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Robert Findley <rfindley@google.com>
2023-08-30internal/lru: add a really simple LRU cache implementationMichael Matloob
To replace github.com/hashicorp/golang-lru. The size used with the cache in fetchdatasource is 100, so the efficiency of the cache is not super important. For golang/go#61399 Change-Id: I48383a4d1c00c4d153c0bad7b0a1a44c026b2314 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/524458 TryBot-Result: Gopher Robot <gobot@golang.org> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Run-TryBot: Michael Matloob <matloob@golang.org> Reviewed-by: Robert Findley <rfindley@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2023-08-10cmd/pkgsite: more changes to use local stdlibMichael Matloob
This cl makes more changes to enable using the local stdlib. It always uses the local stdlib from cmd/pkgsite. It also changes code that processes and shows versions to expect and pass through v0.0.0, the fake version we use when we show local packages. For #57742 Change-Id: Ic407e58e9658b0703dbc6df47f40bbe1f1bd2b2a Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/517915 kokoro-CI: kokoro <noreply+kokoro@google.com> Run-TryBot: Michael Matloob <matloob@golang.org> Reviewed-by: Robert Findley <rfindley@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>
2023-03-15cmd/pkgsite: hide irrelevant content in local mode; add quick linksRobert Findley
This CL contains several UI improvements when running pkgsite in local mode via cmd/pkgsite: - hide irrelevant or inaccurate content - fix a bug using -gopath_mode (trim the `go env GOPATH` output) - fix panics navigating to vuln pages - link the GO button to the root page, rather than go.dev - add quick-links to the homepage to browse local modules Also: - add -dev and -static flags, to facilitate development - replace TestBuildGetters with more cases in TestServer - fix some redundancy in setting the <title> element for pages; consolidate on using the basePage.HTMLTitle value Updates golang/go#40371 Change-Id: I459a0d0fd39897dd4f902dd023aec653a3fb12cd Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/475755 TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Run-TryBot: Robert Findley <rfindley@google.com>
2023-03-14cmd/pkgsite: add multi-module support, invalidation, and searchRobert Findley
This change adds a new goPackagesModuleGetter, which uses x/tools/go/packages to query package information for modules requested by the pkgsite command. Additionally, add new extension interfaces that allow getters to add support for search and content invalidation. The go/packages getter uses these extensions to implement search (via a simple fuzzy-matching algorithm copied from x/tools), and invalidation via statting package files. Along the way, refactor slightly for testing ergonomics. Updates golang/go#40371 Updates golang/go#50229 Fixes golang/go#54479 Change-Id: Iea91a4d6327707733cbbc4f74a9d93052f33e295 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/474295 TryBot-Result: kokoro <noreply+kokoro@google.com> Run-TryBot: Robert Findley <rfindley@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2022-01-05internal/fetchdatasource: add directory informationJonathan Amsterdam
When fetching a module, populate information about unit subdirectories. Normally (when using a DB) we don't need to do this because a DB query synthesizes it from the units table, but FetchDataSource returns information for immediate consumption so it has to do it on its own. One difference between this and the DB path: this won't have submodules, since FetchDataSource processes one module at a time. Fixes golang/go#49847 Change-Id: I7a215e03dd9cebe31a14c3c118c42978f3f7c8a3 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/375714 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-09-07internal/fetch: get source info from module getterJonathan Amsterdam
Make it the ModuleGetter's responsibility to provide links to source files, instead of passing around a separate source.Client. This will allow getters that fetch from disk to serve those local files. For golang/go#47982 Change-Id: I6c7cd7890b03ad61d9410850a510c1c9a63f1c9a Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/347749 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-09-03internal/fetchdatasource: cache resolved latest versionJonathan Amsterdam
After successfully fetching a module at latest, cache it at its resolved version. This avoids a second fetch when the frontend redirects "latest". Change-Id: I127029e7818205f7ef3b74e8723aac753f53a896 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/347550 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-09-01internal/{worker/fetch}: include DB activity in load-sheddingJonathan Amsterdam
Move the load-shedding logic to the worker and have it span both the fetch and processing of the module (as previously) as well as inserting it into the database. This is a more accurate estimation of load, since running a lot of concurrent queries definitely slows down processing. Most of the time this won't make much difference, but under high load, such as when processing multiple large modules, it will reduce DB contention and should result in greater throughput. For golang/go#48010 Change-Id: I7d0922e02d00182e867fd3b29fc284c32ecab5ee Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/346749 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-08-26cmd/pkgsite: support the module cacheJonathan Amsterdam
With the -cache flag, fetch modules from the module cache. For golang/go#47780 Change-Id: I2aa6467955cd90a80ccae559f27928cca6e06079 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/345272 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Reviewed-by: Julie Qiu <julie@golang.org>
2021-08-26internal/fetchdatasource: handle build contexts properlyJonathan Amsterdam
GetUnit returns a Unit with its Documentation set to a matching BuildContext. This is the same behavior as postgres.DB.GetUnit. For golang/go#47780 Change-Id: I2bc23b7bc5a006e78bec54f6f3229e59ab5a03ef Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/345269 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Reviewed-by: Julie Qiu <julie@golang.org>
2021-08-26internal/fetchdatasource: GetUnitMeta returns NotFound on missing packageJonathan Amsterdam
If a module exists but the package path is not in it, return NotFound. For golang/go#47780 Change-Id: If3a6602df4b99c8470020e8538e01c685880d86d Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/345251 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Reviewed-by: Julie Qiu <julie@golang.org>
2021-08-26internal/fetchdatasource: ignore latest-version errorsJonathan Amsterdam
A FetchDataSource may be configured with a proxy for latest-version information while still being used for local modules. So ignore latest-version errors. For golang/go#47780 Change-Id: I30448e133faec80002194df59e6f2db449f80625 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/345249 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Reviewed-by: Julie Qiu <julie@golang.org>
2021-08-25internal/fetchdatasource: rename from datasourceJonathan Amsterdam
Rename the package and type by including the word "fetch," to describe what it does and distinguish it from other DataSources. For golang/go#47780 Change-Id: I7c674d8ba3dc16084a857039b0cc2b3147f27a29 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/344953 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
generated by cgit v1.3 (git 2.53.0) at 2026-04-15 07:44:09 +0000