aboutsummaryrefslogtreecommitdiff
path: root/cmd/worker
AgeCommit message (Collapse)Author
7 daysinternal/queue: add postgres queue implementation, and use it in worker|frontendJean Barkhuysen
Fixes golang/go#74027. Change-Id: I916ac81093e782d4eda21fe11ef47eeff4f5f0b1 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/751480 Reviewed-by: Jonathan Amsterdam <jba@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Ethan Lee <ethanalee@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2026-03-24all: pgx/v4 to v5 and upgrade pqEthan Lee
- In preparation for upgrade to PostgreSQL 17, upgrade stale dependencies within pkgsite. Change-Id: Id54cbb46182711b9808ee3bbbe2f0d5d9a04583d Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/758121 Auto-Submit: Ethan Lee <ethanalee@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Jonathan Amsterdam <jba@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2026-02-25internal/queue: move InMemory queue to its own packageJean Barkhuysen
Currently, InMemory queue sits in queue, and is instantiated in gcpqueue for convenience. In preparation for a third queue type (Postgres), this CL separates the two more cleanly, making it more ergonomic for the new queue type to slot in next to the existing two. This CL doesn't change any logic: it just exists to make the next CL smaller and easier to review. I also took the liberty of adding some tests specific to the InMemory queue, since I didn't find any. Let me know if it's tested in another place, though, and if you'd prefer me to remove it. Updates golang/go#74027. Change-Id: I44bd92129f33bc7975fcd138c905e0b7ab49d257 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/747881 kokoro-CI: kokoro <noreply+kokoro@google.com> Auto-Submit: Jonathan Amsterdam <jba@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Ethan Lee <ethanalee@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2026-02-24internal/worker: add an option for in-process periodic poll/enqueue-ingJean Barkhuysen
Fixes golang/go#77689. Change-Id: Id754eeafdc8b680acb1aa137225292c808d0bf04 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/747880 Reviewed-by: Jonathan Amsterdam <jba@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: David Chase <drchase@google.com>
2025-10-28all: remove code for the decommissioned beta environmentNicholas S. Husin
The beta environment is no longer in use. This change removes related logic and references, such as the middleware redirect to beta.pkg.go.dev. Change-Id: I0c6ebbe11d5689ccf2abf8c722399d8dc339db89 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/715321 kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Michael Matloob <matloob@google.com> Reviewed-by: Michael Matloob <matloob@golang.org>
2024-03-01internal/middleware,etc.: store requestsJonathan Amsterdam
This CL provides several improvements to the worker home page: - Display all requests, not just fetches. - Link to the logs for each request. - Link that will cancel a request. At the heart of these changes is a new piece of middleware that tracks all active requests, along with their trace ID and a function that can be used to cancel them. This change also affects logging, because the logger doesn't need to maintain its own trace ID. Change-Id: Id022170073d2d7ca4e45aaa1d78b216d8a512f35 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/568236 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Michael Matloob <matloob@golang.org> kokoro-CI: kokoro <noreply+kokoro@google.com>
2024-02-27internal/worker: improve home and debug pagesJonathan Amsterdam
- Move excluded prefix list to a separate page. - Move debug pages under "/debug". (Previously the dcensus ones weren't even exposed.) - Add links to debug pages at top of home page. Change-Id: If26e139984b34cb6b2ef78af8ef154d96436a26a Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/566418 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Michael Matloob <matloob@golang.org>
2023-08-30internal/trace: add a wrapper for tracingMichael Matloob
This package adds a StartSpan function that does nothing unless an implementation for the StartSpan function is injected in. That allows us to have the frontend and worker mains inject the implementation while internal/fetch and internal/source can use the StartSpan function without depending on opencensus's tracing library. For golang/go#61399 Change-Id: I2c650bdbb56c8fcc47ffabab8353cef74e0d1c4e Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/523739 Reviewed-by: Robert Findley <rfindley@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Run-TryBot: Michael Matloob <matloob@golang.org> kokoro-CI: kokoro <noreply+kokoro@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>
2023-08-29internal/proxy: inject transport in proxy.NewMichael Matloob
This removes the dependency from internal/proxy on ochttp. For golang/go#61399 Change-Id: Ib2a350151752dc18c9a6e03b38b2d423ed55f4a1 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/523513 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Run-TryBot: Michael Matloob <matloob@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Robert Findley <rfindley@google.com>
2023-08-29internal/source: inject *http.Client into source.NewClientMichael Matloob
This removes the dependency from package source onto ochttp. The users of source.NewClient that want an ochttp.Transport can set the transport on the *http.Client. For golang/go#61399 Change-Id: Ifb7126c482f664ee5a359f594d9324f0fd90f8b2 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/523510 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Robert Findley <rfindley@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Michael Matloob <matloob@golang.org> kokoro-CI: kokoro <noreply+kokoro@google.com>
2023-08-25internal/config: separate config initialization into serverconfigMichael Matloob
This change creates a new package that does config initialization and other GCP-specific operations that were previously done in package config, so that config can have no cloud dependencies. For golang/go#61399 Change-Id: I8d78294834e325b47d838892a1cef87003a4b90a Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/522516 Run-TryBot: Michael Matloob <matloob@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Robert Findley <rfindley@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2023-08-04internal/frontend: remove dependency on cloud error reporting clientMichael Matloob
This removes the dependency of internal/frontend on the cloud error reporting client, both directly, and through the derrors package by introducing a new interface Reporter that is used both to set the reporting client for internal/derrors, and on the Server. For golang/go#61399 Change-Id: Id4d4def522cda9b4e49f53cff6708019dec2693c Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/514676 Reviewed-by: Jamal Carvalho <jamal@golang.org> kokoro-CI: kokoro <noreply+kokoro@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Michael Matloob <matloob@golang.org>
2023-07-31internal/middleware: move the timeout middleware to its own packageMichael Matloob
This removes the direct dependency from cmd/pkgsite to the middleware package. The middleware package has a lot of dependencies but most middlewares are not used by cmd/pkgsite. There are still some middlewares used by internal/frontend so the indirect dependency still exists. For #61399 Change-Id: I09eee46b2ff13e112d24df62073074eb5aa3e901 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/511315 Run-TryBot: Michael Matloob <matloob@golang.org> Reviewed-by: Jamal Carvalho <jamal@golang.org> kokoro-CI: kokoro <noreply+kokoro@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>
2023-07-19internal/queue: split out gcp queue into different packageMichael Matloob
internal/frontend depends on internal/queue. Move the parts with gcp dependencies into its own package so internal/frontend no longer has those gcp queue dependencies. For #61399 Change-Id: I0083c31da1b367f135e53686e4c994d7c4d6420f Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/510815 TryBot-Result: Gopher Robot <gobot@golang.org> kokoro-CI: kokoro <noreply+kokoro@google.com> Run-TryBot: Michael Matloob <matloob@golang.org> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2023-05-04internal/worker: allow worker to process dynamic excluded prefix filesJamal Carvalho
For golang/go#59622. Change-Id: I4059df405e248824a3521fcc084568b052fe4380 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/486458 Reviewed-by: Suzy Mueller <suzmue@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Robert Findley <rfindley@google.com> Run-TryBot: Jamal Carvalho <jamal@golang.org>
2022-01-06cmd,internal: add worker endpoint to clear beta cacheJamal Carvalho
Added an endpoint to clear the cache during cloudbuild deployments of the beta frontend. Change-Id: I3fb19c3f1eef6b1b15eda2601b27a403b39b0f8d Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/375956 Trust: Jamal Carvalho <jamal@golang.org> Run-TryBot: Jamal Carvalho <jamal@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-09-13{cmd,internal}/worker: export DB info metricsJonathan Amsterdam
Add metrics for the numbers of active and waiting DB processes. For golang/go#48010 Change-Id: Ia3c14e492b29c07371ee903182c7ba55f04c584a Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/349310 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Reviewed-by: Julie Qiu <julie@golang.org>
2021-09-01internal/{fetch,worker}: include DB insertion in fetch metricsJonathan Amsterdam
Fetch latency and related metrics include the time spent inserting into the DB, as well as the time to fetch and process the module. For golang/go#48010 Change-Id: I1d685bd25f1b632b0b20de5b1bfac5003bff0caa Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/346750 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-09-01internal/{worker/fetch}: include DB activity in load-sheddingJonathan Amsterdam
Move the load-shedding logic to the worker and have it span both the fetch and processing of the module (as previously) as well as inserting it into the database. This is a more accurate estimation of load, since running a lot of concurrent queries definitely slows down processing. Most of the time this won't make much difference, but under high load, such as when processing multiple large modules, it will reduce DB contention and should result in greater throughput. For golang/go#48010 Change-Id: I7d0922e02d00182e867fd3b29fc284c32ecab5ee Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/346749 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-08-17cmd/worker: log timeout valueJonathan Amsterdam
Change-Id: Ie5d3e056fb2507d2e0e4b093f4a437031841ed55 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/342672 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Reviewed-by: Julie Qiu <julie@golang.org>
2021-08-12cmd/worker: remove the proxy-removed listJonathan Amsterdam
We no longer need a special list of removed modules, because the disable-fetch header serves the same purpose. Change-Id: Ia52a5cfc1036127b78d36f668917809952c57e81 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/341532 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Reviewed-by: Julie Qiu <julie@golang.org>
2021-08-06all: delete redis HA clientsJulie Qiu
The redis HA was used by search autocomplete and is no longer used by anything. Change-Id: I39a2c81c47da53823d7b668a980159c4726c6b13 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/340393 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-07-26internal: add ReadFileLinesJonathan Amsterdam
Factor out this commonly used function to the internal package. Also, update some comments in devtools/cmd/seeddb. Change-Id: I0276ce51d0dd1f4aafa2f6c562d1b83ce21a0d14 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/337330 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Reviewed-by: Julie Qiu <julie@golang.org>
2021-07-19cmd,devtools,internal/config: set log level in config.InitJulie Qiu
log.SetLevel is now called in config.Init, so that it also applies to logs in the Init function. Change-Id: Id552923b4d81bb1678b8ffa72f1b1d858de559c1 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/335257 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-07-14internal/config: use pkgsite internal/log packageJulie Qiu
The pkgsite logger is now used instead of the log package from the standard library. Change-Id: I5083d8cd6bf7a96a9245f848633b8803d0d80483 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/333931 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-06-14cmd/worker: add unprocessed-new-modules metricJonathan Amsterdam
Change-Id: I41990fe0fe344ee8038e5841a242b087e3065540 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/327751 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-06-12internal: switch static reads from content/static to staticJamal Carvalho
Change-Id: Icec167d645b9178375b959491206277e13b455cf Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/327283 Trust: Jamal Carvalho <jamal@golang.org> Run-TryBot: Jamal Carvalho <jamal@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-05-27internal/{postgres,worker}: add metric for number unprocessedJonathan Amsterdam
Add a metric that records how many modules need to be processed (those with status codes of 0 or 5xx). That will tell us how much the worker is backed up. Change-Id: I68c47255e3720b5f978037c1251b0e502961d089 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/323109 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-03-05cmd: add e2e to .gcloudignoreJamal Carvalho
Change-Id: Id211af73d327d92d3cf361b5909619ac14856860 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/298751 Trust: Jamal Carvalho <jamal@golang.org> Run-TryBot: Jamal Carvalho <jamal@golang.org> Reviewed-by: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com>
2021-02-24internal/worker: call WithFetchDisabled soonerJonathan Amsterdam
Convert the proxy client to one which doesn't fetch at the beginning of the request, so we don't have to pass a boolean argument through several layers. For golang/go#44437 Change-Id: I0495b5673cc399a1cdc16349d53ba4a9a4ecc380 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/295889 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-01-26many: add node_modules directory to .gcloudignoreJonathan Amsterdam
This directory is created by npm and is large. In fact, including it results in too many files for `gcloud app deploy`. Also, put .gcloudignore files under version control. Change-Id: I6cbf1c42b312911b9b2b16215469cb027bedebe1 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/286932 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2020-12-21internal/worker: honor proxyfetch query paramJonathan Amsterdam
If the fetch handler sees the "proxyfetch" query param set to "off", it tells internal/fetch to disable fetching by the proxy. Change-Id: I1c9303f96f9249a5b420a73451551aba21d9d417 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/278953 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2020-12-21internal/worker: factor out FetchAndUpdateState argsJonathan Amsterdam
Move some common args to a struct. Change-Id: I4e500bc72f0cc622ec7641568beb5c7dbde1e69a Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/278713 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2020-12-18internal/worker: record processing lag metricJonathan Amsterdam
Create a metric for the age of the oldest unprocessed module, and record it periodically. We choose to record it whenever the /poll endpoint is hit. Since that endpoint is hit periodically (once a minute) by Cloud Scheduler, we can be sure that we will get exactly one data point per minute. The downside is that we won't have finer granularity than that, but since the main purpose of this metric is to detect large lags, that doesn't matter. Change-Id: I7ce3f043cceaf567fb0c871676063eda91bb2420 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/278614 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org> Reviewed-by: Robert Findley <rfindley@google.com>
2020-12-03internal/worker: fetch response shows resolved versionJonathan Amsterdam
The worker's /fetch endpoint now replies on success with a message that uses the resolved version of the module, not the requested one. Change-Id: Id48e624bb9d34a1a1fd2541c1b11956850faf14c Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/274955 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2020-11-30internal/middleware: use redis to store quota informationJonathan Amsterdam
Our previous quota middleware stored data in RAM. That meant that it was ineffective when there were multiple instances (processes) running, because the totals were separate. This CL uses the same redis instance we use for caching to hold the data, so it's shared. Also, previously the only way to set some quota parameters was via the override file, which is intended only for emergencies. Allow the important ones to be set by environment variables, like most of our other config. Also, the rate-limiter package we use requires v8 of the redis client, so move all other uses of it to v8. Change-Id: I1804014030db834a30511786f0a7edfd02e279a5 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/269737 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2020-11-24cmd/internal/cmdconfig: factor out DB openingJonathan Amsterdam
Change-Id: If9be542ae7b50789c548f19dbab419a2db74e8ca Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/272793 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2020-11-24internal/config: add config for database driverJonathan Amsterdam
The worker and frontend get the driver name from the config struct, via an environment variable. The default is "postgres", which was the previous hardcoded value, so this CL on its own has no effect. Change-Id: Ia1ecd39e15ea75f21dc7c94aa5567b870a646bf7 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/272787 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2020-11-16internal/queue: use time-independent task IDsJonathan Amsterdam
The task names we generated for Cloud Tasks included time information, so that we could control how freqeuently tasks were de-duped. We've been using a 3-hour window, meaning that if a module's task finished or was deleted, another task for that module couldn't be added to the queue (would be de-duped) for 3 hours. This scheme has the unintended consequence that if a module is still on the queue after 3 hours, then another task for the same module could be enqueued (since its task ID would be different). That means that if the queue ever gets 3 hours behind, it could get filled with duplicate modules. For example, this morning the queue had about 25,000 tasks, of which about 7,000 were duplicates. This CL sets a task's name to a function of the module path and version. That will prevent a task on the queue from being duplicated. A repeat task can be added about 1 hour after the previous task finishes or is deleted (see the Task De-duplication section of https://cloud.google.com/tasks/docs/reference/rpc/google.cloud.tasks.v2#task), which is fine. It is still possible to add a suffix to the task name to requeue sooner. Change-Id: I34ffce5ea67b9e00b88ca4cf38182e34a6ba8657 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/270337 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2020-10-13cmd/worker: extend redis cache write timeoutJonathan Amsterdam
Double the timeout for writing to the redis cache instance, from 3 to 6 seconds. We saw an error on /clear-cache that was due to hitting the write timeout: https://pantheon.corp.google.com/errors/COmQ7oS0hITl3gE?time=P30D&project=go-discovery https://pantheon.corp.google.com/logs/query;query=resource.type%3D%22gae_app%22%20resource.labels.module_id%3D%22etl%22%0A%0Atrace%3D%22d3d2cbb054cf3b8e4627e049d3c5cc9e%2F8803268222527035810%3Bo%3D1%22;timeRange=2020-10-12T17:57:33.989Z%2F2020-10-13T12:57:33.989Z?project=go-discovery Change-Id: Ib9aae6ccece58ecccf78b2627c65a176295199ee Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/261817 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Robert Findley <rfindley@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2020-10-07cmd: remove DB experiments from worker and frontendJonathan Amsterdam
All code paths get experiments from a config file. Change-Id: I53e3c22e443573546d8ebdf65303012f845b980c Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/259840 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2020-09-29internal/fetch: add a metric for number of packages successfully fetchedJonathan Amsterdam
This should be a smoother metric than the number of fetched modules, since processing time per package should have a much smaller range than time per module. Change-Id: I00f5ba83fcc4fe3c606883bd644e570b502c9090 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/257243 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2020-09-23internal/fetch: add fetch metricsJonathan Amsterdam
Add metrics that count the number of shedded and actual fetches, along with the time of actual fetches. Also factor out some common code to internal/dcensus. Change-Id: Icb52fb0d0456ac760ac9d846aea4a77dd0d16c47 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/256764 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com>
2020-09-23internal/config,others: GetEnvInt dies on parse errorJonathan Amsterdam
Have config.GetEnvInt terminate the program if the environment variable's value is non-empty but not an integer. That's clearly a bug. Also, use it in more places. Change-Id: Ibcc192853e4b5fc01dad709de4de2e5db6fff709 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/256760 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2020-09-23internal/worker,middleware: validate IAP headerJonathan Amsterdam
Validate the header that comes from the IAP. This ensures that no one is accessing the server in a way that bypasses the IAP. The worker will perform this validation, but only if an env var is set, so it won't affect local testing and we can turn it on or off easily. Change-Id: I5932548d8a4ef0502f0c936b394f711ac310bcd3 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/256759 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2020-09-22internal/queue: use an experiment getter instead of the DBJonathan Amsterdam
To get the list of experiments, pass in an ExperimentGetter instead of a DB client. Change-Id: I89ccd402a178d195c85efcdaf363e59c45358b2f Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/256381 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2020-09-21internal/cmdconfig: use dynamic config for experiments on env varJonathan Amsterdam
If the GO_DISCOVERY_EXPERIMENTS_FROM_CONFIG env var is "true", read experiments from dynamic config instead of the database. Temporary, until we switch over completely to dynamic config. For golang/go#41483 Change-Id: Icf6038fb9f2322eba24bedf60450cd9fce84bd6c Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/256380 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2020-09-21internal/middleware: remove ExperimentSourceJonathan Amsterdam
All our ExperimentSources had the same GetExperiment behavior over time, so the ExperimentSource was an unnecessary level of indirection. Remove it. Change-Id: I975d3095d87d074736e48487bc9f8d8b1c9a14f1 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/256379 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2020-09-21cmd/internal/cmdconfig: add experimenter configJonathan Amsterdam
Change-Id: I0d4bb1c4aafa7ba7c00f5f15cedb475022c1cf86 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/256378 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2020-09-21cmd/internal/cmdconfig: common config functions for commandsJonathan Amsterdam
Factor out some common configuration code from the worker and frontend commands. Change-Id: I640a90ad56f0866f2bdbb621f64de93ec75dfd47 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/256377 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com>
generated by cgit v1.3 (git 2.53.0) at 2026-04-15 09:39:02 +0000