aboutsummaryrefslogtreecommitdiff
path: root/internal/worker
AgeCommit message (Collapse)Author
2026-03-30all.bash: re-enable unparamJonathan Amsterdam
It's working now. Also fix, issues that it detected. Change-Id: Ifd3fe5baf9a56d31bafe099bf4b38e02356bde21 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/760582 Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Ethan Lee <ethanalee@google.com>
2026-03-27all: run go fix -waitgroupHana Kim
Change-Id: Iefc4a999101eb1fc5ea2ccd2dc6981f1967c7222 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/753435 kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2026-03-27all: run go fix -stringscutprefixHana Kim
Change-Id: I3c0a7e155ead790300ebe53e090ef9a42e655d3e Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/753432 kokoro-CI: kokoro <noreply+kokoro@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2026-03-26all: fix rangeintHana Kim
Change-Id: I58ed5e7d2a3bbc13bc389ac7239737e18856dbfe Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/753427 Reviewed-by: Jonathan Amsterdam <jba@google.com> Reviewed-by: Ethan Lee <ethanalee@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2026-03-25all: add LatestVersion to ModuleInfoEthan Lee
- Add LatestVersion field to internal.ModuleInfo and update tests accordingly. - Changed some test formatting to improve readability. Change-Id: I1238e54614ef276d219b31d125cd2cad6b5a66f7 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/756901 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Auto-Submit: Ethan Lee <ethanalee@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2026-03-20internal: conditionally return documentation in GetUnitEthan Lee
- Large documentations blobs should avoid being returned unless explicitly requested. Add a WithDocsSource to enable retrieval of unit information without the large documentation source information. - This does not functionally change existing GetUnit callsites. Change-Id: Ie983b7eaeb61bb92f2716ae64a3d1e1ea738dbe2 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/756480 Reviewed-by: Jonathan Amsterdam <jba@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Auto-Submit: Ethan Lee <ethanalee@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2026-03-13all: fix forvarHana Kim
Change-Id: I02d2be535d9e3ac1788306adcc78d997d0caf951 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/753422 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Ethan Lee <ethanalee@google.com>
2026-02-25internal/queue: move InMemory queue to its own packageJean Barkhuysen
Currently, InMemory queue sits in queue, and is instantiated in gcpqueue for convenience. In preparation for a third queue type (Postgres), this CL separates the two more cleanly, making it more ergonomic for the new queue type to slot in next to the existing two. This CL doesn't change any logic: it just exists to make the next CL smaller and easier to review. I also took the liberty of adding some tests specific to the InMemory queue, since I didn't find any. Let me know if it's tested in another place, though, and if you'd prefer me to remove it. Updates golang/go#74027. Change-Id: I44bd92129f33bc7975fcd138c905e0b7ab49d257 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/747881 kokoro-CI: kokoro <noreply+kokoro@google.com> Auto-Submit: Jonathan Amsterdam <jba@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Ethan Lee <ethanalee@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2026-02-24internal/worker: add an option for in-process periodic poll/enqueue-ingJean Barkhuysen
Fixes golang/go#77689. Change-Id: Id754eeafdc8b680acb1aa137225292c808d0bf04 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/747880 Reviewed-by: Jonathan Amsterdam <jba@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: David Chase <drchase@google.com>
2025-10-28all: remove code for the decommissioned beta environmentNicholas S. Husin
The beta environment is no longer in use. This change removes related logic and references, such as the middleware redirect to beta.pkg.go.dev. Change-Id: I0c6ebbe11d5689ccf2abf8c722399d8dc339db89 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/715321 kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Michael Matloob <matloob@google.com> Reviewed-by: Michael Matloob <matloob@golang.org>
2025-10-23all: fix some commentscuishuang
Change-Id: Ia70b35ab9aef270692a3892629c3c81208771089 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/713360 Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Sean Liao <sean@liao.dev> kokoro-CI: kokoro <noreply+kokoro@google.com> Auto-Submit: Sean Liao <sean@liao.dev> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: David Chase <drchase@google.com>
2025-07-04internal/worker: Add since form param to /poll.Jean Barkhuysen
If the worker's latest timestamp is t, it's hard to get it to back and re-poll the index for older work. This CL adds a very simple utility that lets you do so. Change-Id: If12d385e245b6e64880dd7aa704b749603c7fecc Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/685455 kokoro-CI: kokoro <noreply+kokoro@google.com> Auto-Submit: Jonathan Amsterdam <jba@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2024-03-05worker: display request state on main pageJonathan Amsterdam
Add a RequestInfo.State field to record the current state of a request for display. Add a function internal.RequestState to for updating the state, and use it for long-running requests. Display the state on the worker main page. Change-Id: I1a04036982aba23cf27776fac75248972a1336f3 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/568515 Reviewed-by: Michael Matloob <matloob@golang.org> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2024-03-01internal/middleware,etc.: store requestsJonathan Amsterdam
This CL provides several improvements to the worker home page: - Display all requests, not just fetches. - Link to the logs for each request. - Link that will cancel a request. At the heart of these changes is a new piece of middleware that tracks all active requests, along with their trace ID and a function that can be used to cancel them. This change also affects logging, because the logger doesn't need to maintain its own trace ID. Change-Id: Id022170073d2d7ca4e45aaa1d78b216d8a512f35 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/568236 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Michael Matloob <matloob@golang.org> kokoro-CI: kokoro <noreply+kokoro@google.com>
2024-02-29internal/worker: improve template-parsing codeJonathan Amsterdam
We can turn some repeated code into a loop by using a function that can create a TrustedSource from a dynamic string. Change-Id: Id1bed179bfe3cf22585e44f03482f385a0163f06 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/568235 kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Michael Matloob <matloob@golang.org> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-02-27internal/worker: improve home and debug pagesJonathan Amsterdam
- Move excluded prefix list to a separate page. - Move debug pages under "/debug". (Previously the dcensus ones weren't even exposed.) - Add links to debug pages at top of home page. Change-Id: If26e139984b34cb6b2ef78af8ef154d96436a26a Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/566418 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Michael Matloob <matloob@golang.org>
2024-02-22internal/worker: add batch param to update-imported-by-count routeJonathan Amsterdam
Inserting 5000 imported-by counts into search_documents timed out. Make that size configurable, so we can experiment with something smaller. Change-Id: Ie0c5f42a28c0468c600af6d836edc345529b769f Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/565682 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Michael Matloob <matloob@golang.org> kokoro-CI: kokoro <noreply+kokoro@google.com>
2024-02-21internal/postgres: support module@version exclusionsJonathan Amsterdam
Allow a specific module@version to be excluded. Previously, only entire module paths or path prefixes (at every version) could be excluded. This will allow us to exclude specific versions that we can't process, because they are too large for instance. A necessary part of the change is adding a version argument to the IsExcluded method. While we are there, we drop the error return value, since it's always nil. Change-Id: I68fd1c8f66ee654e56bd7dcf230a88959b32bbc8 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/565815 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Michael Matloob <matloob@golang.org> Run-TryBot: Jonathan Amsterdam <jba@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2024-02-21internal/worker: fetch returns 404 on m@vJonathan Amsterdam
We were parsing a fetch path of module@v1.2.3, but returning a version of "1.2.3", not "v1.2.3". But we shouldn't accept this syntax at all, only the proxy syntax "module/@v/v1.2.3". Make that behavior change, but also improve both parseModulePathAndVersion and its test. Change-Id: I5e085b4845a939458838ca6f6da6600c854ed9d4 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/565676 kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Michael Matloob <matloob@golang.org> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-02-14internal/fetch: handle error in root packageJonathan Amsterdam
If a module had a problem with the package at its root, we would end up with two packageVersionStates. That caused a DB error in upsertPackageVersionStates. Distinguish between the module unit and package units in a couple of places. That addresses the root cause of the problem. Also, fix an unrelated bug that was causing a panic when we tried to send the error to the errorreporting service. Change-Id: Ia02faea50f4477999c6c6eca0674b8dde0f15bb3 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/564076 Reviewed-by: Michael Matloob <matloob@golang.org> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2024-02-06internal/fetch: split FetchModule in twoMichael Matloob
First we get a LazyModule that essentially just contains the UnitMetas for each of the modules and then we process each Unit as needed so that FetchDataSource doesn't need to process all the Units when it just needs one. To construct at FetchResult, FetchModule will now get a LazyModule and then compute all the Units. A compromise to get this to work is that FetchDataSource will no longer populate Synopsis, IsRedistributable and Licenses on Subdirectories to avoid computing all those fields for all units when returning a single unit. MainVersion and MasterVersion are removed from FetchResult because they were only set on the internal/worker.FetchTask struct that embeds a FetchResult. Change-Id: Ia0db850ae570d421712ec484ee8b7815a779128e Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/557818 Run-TryBot: Michael Matloob <matloob@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> kokoro-CI: kokoro <noreply+kokoro@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Jonathan Amsterdam <jba@google.com> Reviewed-by: Robert Findley <rfindley@google.com>
2024-01-23internal: move UnitMeta.IsRedistributable to UnitMichael Matloob
Change-Id: Ia0a18df32d752f20b0400b47860638c7d0910eb7 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/556435 Reviewed-by: Jonathan Amsterdam <jba@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Robert Findley <rfindley@google.com>
2024-01-22internal: move UnitMeta.Licenses to UnitMichael Matloob
Change-Id: If2b3c1acd4830948e2d134517608f89771a7abcb Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/555735 kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Robert Findley <rfindley@google.com>
2023-08-29internal/source: inject *http.Client into source.NewClientMichael Matloob
This removes the dependency from package source onto ochttp. The users of source.NewClient that want an ochttp.Transport can set the transport on the *http.Client. For golang/go#61399 Change-Id: Ifb7126c482f664ee5a359f594d9324f0fd90f8b2 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/523510 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Robert Findley <rfindley@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Michael Matloob <matloob@golang.org> kokoro-CI: kokoro <noreply+kokoro@google.com>
2023-08-25internal/config: separate config initialization into serverconfigMichael Matloob
This change creates a new package that does config initialization and other GCP-specific operations that were previously done in package config, so that config can have no cloud dependencies. For golang/go#61399 Change-Id: I8d78294834e325b47d838892a1cef87003a4b90a Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/522516 Run-TryBot: Michael Matloob <matloob@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Robert Findley <rfindley@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2023-08-24internal/worker: filter index entries for invalid versionsJamal Carvalho
A bug in the module proxy caused an entries in the index to contain empty version information breaking the pkgsite worker polling job. This change filters out index entries with invalid version information and logs the entry as a warning. Change-Id: I221100d7b26fcd7a17cb87968bb9d7c1c843a0bb Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/522183 TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Robert Findley <rfindley@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Run-TryBot: Jamal Carvalho <jamal@golang.org>
2023-08-24internal/worker: add verbose logging to /poll to diagnose failuresRob Findley
Change-Id: I89c4c20ec23d2de4e46734e0eb70d70eae5dc3cc Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/522241 Run-TryBot: Robert Findley <rfindley@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org>
2023-08-23all: remove arbitrary hard-coded timeouts in testsBryan C. Mills
If a test times out, that implies that it got stuck on something. By default, the Go testing package dumps goroutines when its own timeout is passed, which prints a goroutine dump, helping to reveal what was stuck. Adding an arbitrary timeout on top of the testing package's own timeout is, in my experience, almost always counterproductive. If the arbitrary timeout catches a real hang, it causes the test to fail instead of dumping goroutines, making it much harder to see what was stuck. On the other hand, if the timeouts are set aggressively enough to make the test fail early, they are often too aggressive for CI testing, causing flakes that then have to be triaged on an ongoing basis. On balance, the value of saving a minute or two for developers who have introduced a hang is not worth the cost of suppressing debugging information and causing flakes that have to be triaged. Fixes #61556. For #59347. Change-Id: I0263d0d9b18283470f100e5a0155818b87b5312f Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/521837 TryBot-Result: Gopher Robot <gobot@golang.org> Auto-Submit: Bryan Mills <bcmills@google.com> Run-TryBot: Bryan Mills <bcmills@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Michael Matloob <matloob@golang.org>
2023-08-22internal/worker: upgrade x/mod to allow fetching 1.21.0 modulesRob Findley
Upgrade x/mod so that modfile.Parse accepts the newly relaxed go directive syntax. Add a test. For golang/go#62031 Change-Id: I63a8eb73ff7428e67d80446b493c624a52ad2e96 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/521125 TryBot-Result: Gopher Robot <gobot@golang.org> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Run-TryBot: Robert Findley <rfindley@google.com>
2023-08-07cmd/log: move stackdriver logging to its own packageMichael Matloob
This change breaks out the stackdriver logging to its own package. It also only does logging of trace ids and labels to the stackdriver logger, to keep the logging interface simple. For golang/go#61399 Change-Id: I9a25d0c6391d1667fe476e5fdc30fc057f07c40f Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/515375 TryBot-Result: Gopher Robot <gobot@golang.org> kokoro-CI: kokoro <noreply+kokoro@google.com> Run-TryBot: Michael Matloob <matloob@golang.org> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2023-08-04internal/frontend: remove dependency on cloud error reporting clientMichael Matloob
This removes the dependency of internal/frontend on the cloud error reporting client, both directly, and through the derrors package by introducing a new interface Reporter that is used both to set the reporting client for internal/derrors, and on the Server. For golang/go#61399 Change-Id: Id4d4def522cda9b4e49f53cff6708019dec2693c Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/514676 Reviewed-by: Jamal Carvalho <jamal@golang.org> kokoro-CI: kokoro <noreply+kokoro@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Michael Matloob <matloob@golang.org>
2023-07-17internal/fetch: add two stdlib gettersMichael Matloob
This breaks out the behavior of getting the stdlib into its own getter, stdlibZipModuleGetter. Most (but not all) of the special cases in the fetch code for the stdlib are in the stdlibZipModuleGetter, which is installed on the frontend. For the local pkgsite command, we add a new builder for the goPackagesModuleGetter that loads the packages from the stdlib given a GOROOT. The stdlibZipModuleGetter is also installed for pkgsite in as a fallback if the goPackagesModuleGetter doesn't successfully fetch the stdlib module (I'm not sure if that's possible). Fixes #60114 Change-Id: Ida6a5367343643cc337c6d05e0d40095f79ca9e5 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/507883 Reviewed-by: Jamal Carvalho <jamal@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Michael Matloob <matloob@golang.org> kokoro-CI: kokoro <noreply+kokoro@google.com>
2023-05-04internal/worker: allow worker to process dynamic excluded prefix filesJamal Carvalho
For golang/go#59622. Change-Id: I4059df405e248824a3521fcc084568b052fe4380 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/486458 Reviewed-by: Suzy Mueller <suzmue@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Robert Findley <rfindley@google.com> Run-TryBot: Jamal Carvalho <jamal@golang.org>
2023-04-14internal/worker: proxy timeouts should be retryableRob Findley
When fetching e.g. the master branch, it is common for the proxy client to timeout as it waits for the branch to be resolved. This should be a retryable error, as otherwise we end up in a state where an asynchronous refresh of master actually breaks documentation at master. There is still a problem that documentation is broken after the first failure, but this at least puts us in a recoverable state. Again, as with CL 482162 this is unfortunately not possible to test in the current setup without significant refactoring. Such refactoring may be warranted, but I do not have time to do this now. Updates golang/go#59464 Change-Id: I321cc2eaabac1d7e052d07efcaadefcac24208ac Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/484736 Reviewed-by: Jamal Carvalho <jamal@golang.org> Run-TryBot: Robert Findley <rfindley@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com>
2022-11-15all: convert interface{} to anyHana (Hyang-Ah) Kim
Change-Id: I1f3b7cc8899c7707abb01e3d14807c37c3451382 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/449695 TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Run-TryBot: Hyang-Ah Hana Kim <hyangah@gmail.com>
2022-04-11all: gofmtRuss Cox
Gofmt to update doc comments to the new formatting. For golang/go#51082. Change-Id: Ia9e71e7ecac75822ff43d6c7e60f512442a5fa50 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/399617 Run-TryBot: Russ Cox <rsc@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Auto-Submit: Russ Cox <rsc@golang.org> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2022-04-04all: fix typosDan Kortschak
Change-Id: I71373b98c4bf80b176f0f01c3ce50335813678b1 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/391974 Reviewed-by: Ian Lance Taylor <iant@golang.org> Run-TryBot: Ian Lance Taylor <iant@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Trust: Cherry Mui <cherryyz@google.com>
2022-01-13internal/stdlib: refactor go repo logicJonathan Amsterdam
Add a goRepo interface, implemented for remote, local and test repos. Functions of stdlib use a global instance that interface instead of if statements. Also, improve and rewrite some tests. For golang/go#50229 Change-Id: I73641813e6c7a6bb9667f44a672e37dce964b17a Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/378094 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2022-01-11internal: GOOS setting preserved across doc linksJamal Carvalho
When a doc page is rendered with a GOOS value set this value is preserved in links within the doc page. Fixes golang/go#49424 Change-Id: I26d1a3466365d2ee940f9afe0801af114ee03c47 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/377396 Trust: Jamal Carvalho <jamal@golang.org> Run-TryBot: Jamal Carvalho <jamal@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2022-01-06cmd,internal: add worker endpoint to clear beta cacheJamal Carvalho
Added an endpoint to clear the cache during cloudbuild deployments of the beta frontend. Change-Id: I3fb19c3f1eef6b1b15eda2601b27a403b39b0f8d Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/375956 Trust: Jamal Carvalho <jamal@golang.org> Run-TryBot: Jamal Carvalho <jamal@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-11-19internal/memory: memory statisticsJonathan Amsterdam
Move the memory stats in the worker to their own package, so we can use them elsewhere. Change-Id: I31d7b891d88a9a623be081805852ea0ad3427230 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/365655 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-11-15internal/licenses: remove Metadata.OldCoverageJonathan Amsterdam
We don't need the OldCoverage field anymore. All DBs have only the new Coverage information. Change-Id: Ib2d8e0cdacdc2f7b6514a8a5310d6905be15136c Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/363954 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-11-01internal/worker: update static worker pathJamal Carvalho
CL 359395 updated dochtml.LoadTemplates to expect a static path relative to /doc. This CL updates the path to fix the worker. Change-Id: If1562e786112f5d00daa6d083ed24709499d1bce Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/360434 Trust: Jamal Carvalho <jamal@golang.org> Run-TryBot: Jamal Carvalho <jamal@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-10-28many: cmd/pkgsite embeds static assetsJonathan Amsterdam
The `pkgsite` command now embeds all the static assets it needs, so it need not be told the location of the static/ directory with a flag. The binary is self-contained and can be placed and invoked from anywhere. This required embedding the static/ and third_party/ directories. Since //go:embed cannot reference files outside the containing package's tree, we had to add trivial Go packages in static/ and third_party/ to construct `embed.FS`s for those directories. Also, the frontend needed to accept `fs.FS` values where it previously took paths, and `template.TrustedFS` values where it previously used `template.TrustedSources`. We ended up clumsily requiring four separate config values: - A TrustedFS to load templates. - Two fs.FSs, one for static and one for third_party, to load other assets. - The path to the static directory as a string, solely to support dynamic loading in dev mode. For golang/go#48410 Change-Id: I9eeb351b1c6f23444b9e65b60f2a1d3905d59ef9 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/359395 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-09-14internal/postgres: change UpsertModuleVersionStates to updateJulie Qiu
UpsertModuleVersionStates is changed to UpdateModuleVersionStates. There should never be a situation where UpsertModuleVersionStates is called and a row does not already exist for that module. If that happens, an error is now returned. For golang/go#46985 Fixes golang/go#39628 Change-Id: I357396cee6eb743513ae249609f76f4cd4c19e9b Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/341860 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> Reviewed-by: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com>
2021-09-14internal/worker: log some InsertModule errors at InfoJonathan Amsterdam
If the error's status is less than 500, log as Info rather than Error. Change-Id: Id8cfeceac4063f77a3461f3c27786f34424c920e Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/349754 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-09-14internal/{worker,stdlib}: /fetch-std-master: avoid cloning go repoJonathan Amsterdam
On each call of the /fetch-std-master endpoint, we cloned the Go repo for each supported branch to determine its current commit. Although not a big deal, that is a bit wasteful and it also would occasionally fail obscurely. Instead we use git's remote list feature to get all the repo's refs in one call. We then compare the hashes with the stored pseudo-versions to see if the branch has moved. Change-Id: I1d70e63668fcf6d35b90b9c231255faec614a04e Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/349753 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-09-14internal/worker: avoid redirect for /fetch-std-masterJonathan Amsterdam
The final slash in the registered URL results in an unnecessary 301 on every request. Change-Id: I4eee8ba087615f8e20546f4afc31d182c1db6cb8 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/349750 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-09-14internal/{queue,worker}: add worker source paramJulie Qiu
When a request is enqueued via the /enqueue endpoint, add a source=worker query param so that it is clear from the logs where this request came from. Change-Id: I396bb8a952640c244e0b217e48356e8088ed5eb0 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/349530 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-09-14internal/worker: insert module if not in module_version_statesJulie Qiu
If a module is introduced for the first time to pkgsite through frontend fetch, and not the index, it won't have a row in module_version_states. A row is now inserted before the fetch process begins if the proxy's info endpoint tells us that this is a valid module. This fixes a bug in the fetch flow where module_version_states.last_processed_at is not updated in upsertModuleVersionState if a row did not already exist. For golang/go#46985 Change-Id: I2a166228df20b1fb935bbabebb5a651da0e2c1ba Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/341892 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
generated by cgit v1.3 (git 2.53.0) at 2026-04-15 07:54:03 +0000