aboutsummaryrefslogtreecommitdiff
path: root/internal/queue
AgeCommit message (Collapse)Author
7 daysinternal/queue: add postgres queue implementation, and use it in worker|frontendJean Barkhuysen
Fixes golang/go#74027. Change-Id: I916ac81093e782d4eda21fe11ef47eeff4f5f0b1 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/751480 Reviewed-by: Jonathan Amsterdam <jba@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Ethan Lee <ethanalee@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2026-02-25internal/queue: move InMemory queue to its own packageJean Barkhuysen
Currently, InMemory queue sits in queue, and is instantiated in gcpqueue for convenience. In preparation for a third queue type (Postgres), this CL separates the two more cleanly, making it more ergonomic for the new queue type to slot in next to the existing two. This CL doesn't change any logic: it just exists to make the next CL smaller and easier to review. I also took the liberty of adding some tests specific to the InMemory queue, since I didn't find any. Let me know if it's tested in another place, though, and if you'd prefer me to remove it. Updates golang/go#74027. Change-Id: I44bd92129f33bc7975fcd138c905e0b7ab49d257 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/747881 kokoro-CI: kokoro <noreply+kokoro@google.com> Auto-Submit: Jonathan Amsterdam <jba@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Ethan Lee <ethanalee@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2023-09-06all: migrate to cloud.google.com message typesDmitri Shuralyov
Done with: go run cloud.google.com/go/internal/aliasfix/cmd/aliasfix@latest . go mod tidy As suggested in the package deprecation note¹. ¹ https://pkg.go.dev/google.golang.org/genproto/googleapis/cloud/tasks/v2#pkg-overview Change-Id: I78df9037198e5460a2e00339727a06282cf8a701 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/526315 Reviewed-by: Bryan Mills <bcmills@google.com> Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2023-08-25internal/config: separate config initialization into serverconfigMichael Matloob
This change creates a new package that does config initialization and other GCP-specific operations that were previously done in package config, so that config can have no cloud dependencies. For golang/go#61399 Change-Id: I8d78294834e325b47d838892a1cef87003a4b90a Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/522516 Run-TryBot: Michael Matloob <matloob@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Robert Findley <rfindley@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2023-07-19internal/queue: split out gcp queue into different packageMichael Matloob
internal/frontend depends on internal/queue. Move the parts with gcp dependencies into its own package so internal/frontend no longer has those gcp queue dependencies. For #61399 Change-Id: I0083c31da1b367f135e53686e4c994d7c4d6420f Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/510815 TryBot-Result: Gopher Robot <gobot@golang.org> kokoro-CI: kokoro <noreply+kokoro@google.com> Run-TryBot: Michael Matloob <matloob@golang.org> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2023-02-09internal: fix some commentscui fliter
Change-Id: I837c46bba9cd5c012670b432274c6e3318a3275b Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/465997 Reviewed-by: Suzy Mueller <suzmue@golang.org> Run-TryBot: Suzy Mueller <suzmue@golang.org> Reviewed-by: David Chase <drchase@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Auto-Submit: Suzy Mueller <suzmue@golang.org>
2022-01-04go.mod: update cloud.google.com/go imports to latest versionsJonathan Amsterdam
Change-Id: I44ac70e194eb6731c6886b7616954cb7d77b9a44 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/375274 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-09-14internal/{queue,worker}: add worker source paramJulie Qiu
When a request is enqueued via the /enqueue endpoint, add a source=worker query param so that it is clear from the logs where this request came from. Change-Id: I396bb8a952640c244e0b217e48356e8088ed5eb0 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/349530 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-09-13internal: add frontend fetch sourceJulie Qiu
Frontend fetch requests now have a source=fetch query param. This query param will be used in the next CL to update module_version_states correctly. Change-Id: I658ccea9fba7583838f8ccf733e2862966aef86d Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/348816 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> Reviewed-by: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com>
2021-09-13internal/queue: add OptionsJulie Qiu
queue.Options is now used to pass optional arguments to ScheduleFetch. Change-Id: I4e00a91ad7d2cfa2ccf669d643a21fda6b3b3fb8 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/348815 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-08-05internal, etc.: factor out (module path, version) pairJonathan Amsterdam
A lot of code reinvents the type that is a pair of module path and version. Factor out to a single type. Change-Id: I729214ebc45c0d496b6b03a5f8a8c17410836572 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/340109 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-06-11internal/queue: fix race conditionJonathan Amsterdam
InMemory.WaitForTesting was waiting for the set of workers to be empty, but that's not quite the same as the queue being empty: all the workers can finish momentarily before a new one starts up. Wait for the queue itself to be empty. Change-Id: Icf0e970fdb2f45aa30c358c9d4af1083442a0293 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/327109 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-03-25internal/queue: check for unknown module pathJonathan Amsterdam
ScheduleFetch checks that it is not given "unknownModulePath". Change-Id: Ie2ae0bde75ad10103b9ea8fb526aa7f09c285733 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/304889 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-02-19internal/queue: restore CreateTask timeoutJonathan Amsterdam
It turns out that the comment in ScheduleFetch was right: CreateTask does require a timeout of at most 30s. Change-Id: I2cc8086247bf606b054132a2908b705f20ecd490 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/294089 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Robert Findley <rfindley@google.com>
2021-02-18internal/queue: remove CreateTask timeoutJonathan Amsterdam
Remove the a timeout on the CreateTask API call. It was causing failures on the production worker. A comment said this was a requirement of the Tasks API, but I couldn't find that in the docs. In any case, if it is, then presumably CreateTask will return an error. Also, add a stack to the returned error, for better error reporting. Change-Id: Id40e596cafade6d673c1787fe446bfdff1a8a62a Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/293609 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2020-12-21internal/worker: honor proxyfetch query paramJonathan Amsterdam
If the fetch handler sees the "proxyfetch" query param set to "off", it tells internal/fetch to disable fetching by the proxy. Change-Id: I1c9303f96f9249a5b420a73451551aba21d9d417 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/278953 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2020-12-21internal/queue,frontend,worker: enqueue tasks disabling proxy fetchJonathan Amsterdam
If the status of a task to be enqueued indicates reprocessing, then enqueue the task with a URL whose query param indicates that proxy fetching should be disabled. Change-Id: Id7255e861c1dea87d131d83151eb1a46df0ea4ff Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/278712 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2020-12-01internal/{config,queue}: remove queue serviceJonathan Amsterdam
Remove support for the QueueService config and for queuing tasks for an AppEngine service. Change-Id: I05dcdeaef2e378a5261490badd8b11cd68f64429 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/274494 Trust: Jonathan Amsterdam <jba@google.com> Trust: Julie Qiu <julie@golang.org> Run-TryBot: Jonathan Amsterdam <jba@google.com> Run-TryBot: Julie Qiu <julie@golang.org> Reviewed-by: Julie Qiu <julie@golang.org>
2020-11-30internal/queue: allow GAE frontend/GKE worker setupJonathan Amsterdam
Change-Id: I42036c0b3941e6b50825e04adfe6b27a61576de7 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/274115 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2020-11-16internal/queue: use time-independent task IDsJonathan Amsterdam
The task names we generated for Cloud Tasks included time information, so that we could control how freqeuently tasks were de-duped. We've been using a 3-hour window, meaning that if a module's task finished or was deleted, another task for that module couldn't be added to the queue (would be de-duped) for 3 hours. This scheme has the unintended consequence that if a module is still on the queue after 3 hours, then another task for the same module could be enqueued (since its task ID would be different). That means that if the queue ever gets 3 hours behind, it could get filled with duplicate modules. For example, this morning the queue had about 25,000 tasks, of which about 7,000 were duplicates. This CL sets a task's name to a function of the module path and version. That will prevent a task on the queue from being duplicated. A repeat task can be added about 1 hour after the previous task finishes or is deleted (see the Task De-duplication section of https://cloud.google.com/tasks/docs/reference/rpc/google.cloud.tasks.v2#task), which is fine. It is still possible to add a suffix to the task name to requeue sooner. Change-Id: I34ffce5ea67b9e00b88ca4cf38182e34a6ba8657 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/270337 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2020-09-29internal/queue: increase task timeoutJonathan Amsterdam
HTTP tasks (but not AppEngine tasks) have a default timeout of 10 minutes. Extend that to the maximum of 30 minutes. Change-Id: I75efc0523ff83471a0a2a8d453e1284483feefe1 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/258197 Reviewed-by: Julie Qiu <julie@golang.org> Trust: Jonathan Amsterdam <jba@google.com>
2020-09-23internal/queue: downgrade ignore message to debugJonathan Amsterdam
Change the log message about ignoring a duplicate task from Info to Debug to avoid cluttering the logs. Change-Id: I08baadf10e731150cfd5a8be26d28f2598e02bda Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/256763 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2020-09-22internal/queue: use an experiment getter instead of the DBJonathan Amsterdam
To get the list of experiments, pass in an ExperimentGetter instead of a DB client. Change-Id: I89ccd402a178d195c85efcdaf363e59c45358b2f Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/256381 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2020-09-14internal/queue: task queue authenticationJonathan Amsterdam
Arrange for the task queue to authenticate itself to the worker, which is behind the IAP. This requires adding information to the task that allows the queue to construct an identity token. Change-Id: I7e10a2d2d3151852a220f4ad7009df6dda0dd98c Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/254517 Reviewed-by: Julie Qiu <julie@golang.org>
2020-09-11internal/worker: improve logging of scheduled tasksJonathan Amsterdam
Report how many tasks were actually scheduled; take duplicate tasks into account. Change-Id: I855b7ca3a2c72da7cc104d6b5960e5f3a39c2f54 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/254381 Reviewed-by: Julie Qiu <julie@golang.org>
2020-09-03internal/queue: use new proto packageJonathan Amsterdam
As staticcheck tells us, the right proto package to use is google.golang.org/protobuf.proto. Change-Id: I6e0f987bdd9217c3d26397547a7df250d3e6fd11 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/252838 Reviewed-by: Julie Qiu <julie@golang.org>
2020-09-02internal/queue: compare protos with proto.EqualJonathan Amsterdam
We were calling cmp.Diff on protobuf messages directly, which is not a good idea. Use proto.Equal instead. Since it doesn't play nicely with the IgnoreFields option, ignore the Name field in a different way. Change-Id: Ie0dfa82b2b78c40402d2a05fed9f4dceff2fc2bf Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/252777 Reviewed-by: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com>
2020-08-28internal/queue: fix TestNewTaskRequestJonathan Amsterdam
Instead of setting environment variables and calling config.Init, which can trigger code paths that can't work on some dev machines, just create a config in the test. Change-Id: Ie85dd86540f9bc11481f647a1856d4e4d828f150 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/251497 Reviewed-by: Julie Qiu <julie@golang.org> Reviewed-by: Miguel Acero <acero@google.com> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com>
2020-08-28internal/queue: support arbitrary URLs for tasksJonathan Amsterdam
Allow GCP queues to create a task destined for any URL, not just an AppEngine service. This lets the worker run somewhere other than AppEngine, like Kubernetes. Also, reorganize the GCP type so any bad config is detected at creation time. Change-Id: Ic3ff988999517f917d24f07f7fe49df0e4d8bc31 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/251061 Reviewed-by: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com>
2020-08-27many: use OnGCP instead of OnAppEngineJonathan Amsterdam
Where it makes sense, call the new Config.OnGCP function to determine if we're running anywhere in GCP (AppEngine or GKE). Change-Id: I1efee29b0c32f9d4818d1a7514b44ae65d7aa252 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/251059 Reviewed-by: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com>
2020-07-08internal/queue: refactor existing queue creators into queue.NewAnze Kolar
Add a unified function for creating new queues so that separate implementations of newQueue in cmd/frontend and cmd/worker can be removed. Since queue.New contains the necessary logic for deciding between GCP and InMemory, the two could be made unexported after updating the corresponding tests. Fixes golang/go#40097. Change-Id: Ie509ba39ef293cca3ff95f2ce12833339c0542ea Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/241477 Reviewed-by: Jonathan Amsterdam <jba@google.com>
2020-07-07internal/config: have queue service default to GAE_SERVICEJonathan Amsterdam
If no GO_DISCOVERY_QUEUE_SERVICE is specified, default to the value of GAE_SERVICE. Without this, the QueueService config field is empty, so scheduled tasks are not delivered to the worker. Change-Id: Ia3d46f122208562c11c76dd0b857a0a5f6e5a822 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/241066 Reviewed-by: Robert Findley <rfindley@google.com>
2020-07-06internal/experiment: improve ergonomics for NewSet and NewContextJulie Qiu
Previously, NewSet and NewContext required the caller to construct a set that was passed in. The functions now accept experiment names, to make them easier to use. Change-Id: Id306902910d51483c48a6a3bc8205cf484694d6a Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/240857 Reviewed-by: Jonathan Amsterdam <jba@google.com>
2020-06-27internal/queue: reorganize InMemoryJonathan Amsterdam
InMemory now expects the given function to close over things like the proxy and source clients. Change-Id: I7b3a2793a824ca29453b19b47b96bdedb2a91010 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/238441 Reviewed-by: Julie Qiu <julie@golang.org>
2020-06-25internal/config: add GO_DISCOVERY_QUEUE_SERVICEJulie Qiu
GO_DISCOVERY_QUEUE_SERVICE is added, which is used to determine which service the tasks queue should send requests to. Change-Id: I27d29ef9ffcd26a5c45412d5a4bb7832348f219d Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/239917 Reviewed-by: Jonathan Amsterdam <jba@google.com>
2020-06-17internal/config: remove globalJonathan Amsterdam
Remove the `cfg` global and the functions that accessed it. Fixes b/145301722. Change-Id: I58ab9fbd4fc29f66dbc5b120f04c88ee0703ee57 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/238437 Reviewed-by: Julie Qiu <julie@golang.org>
2020-06-02internal/queue: pass experiments explictly to InMemoryQueueJulie Qiu
Rather than reading experiments from the context, the experiment set for an InMemoryQueue is now pass in as an argument. This makes it more explicit what experiments are being set. Change-Id: Ib68f567ea5b7ff0fc2157ad2713c76d33827c442 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/759927 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2020-06-02cmd,internal: move taskIDChangeInterval to configJulie Qiu
At the moment, taskIDChangeInterval is a hardcoded value in internal/queue. However, we will soon have two task queues running, which require different change intervals, so this value is now set in internal/config. Additionally, the taskIDChangeInterval for the worker is changed to 3 hours. Change-Id: I498abefce6543005463be7da99a5a778f3a6e973 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/758919 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2020-05-26cmd,internal/queue: pass experiments to InMemoryQueueJulie Qiu
At the moment, experiment flags won't be set after they are passed from a service to the InMemoryQueue. Experiments with a rollout > 0 are now set when initiating the InMemoryQueue for local development with cmd/frontend and cmd/worker. Change-Id: I209024e8281b735d37b4e76d81b4b05ff500e7c3 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/753601 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2020-04-30internal/worker: improve error messages and logsJulie Qiu
Error messages returned by requests to the worker are cleaned up. Change-Id: I79354eafc8785bbffd4bc1fb6db93cee7b517371 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/729941 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2020-04-23all: rename module to golang.org/x/pkgsiteJulie Qiu
golang.org/x/discovery is renamed to golang.org/x/pkgsite. When the repository is open sourced, it will be hosted at go.googlesource.com/pkgsite. Change-Id: Ifc3b45b771a385b99179e785447f2a87afcacf87 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/724273 Reviewed-by: Jonathan Amsterdam <jba@google.com>
2020-04-17internal/queue: change task IDs more frequentlyJonathan Amsterdam
Have the task IDs change every hour, instead of every 3 hours. We want to be able to retry a task more frequently. Fixes #154277084. Change-Id: Ib687775705e019ac62ea75efd29eff540e6fa2d8 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/720902 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-04-06internal/source: add ClientJulie Qiu
source.Client is added, which replaces the used of http.DefaultClient when fetching source code. This also allows us to set a custom timeout for tests vs when running the worker. Change-Id: I5b7b0fd32fa7a2cf836b951af1ad471751fade00 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/709198 Reviewed-by: Jonathan Amsterdam <jba@google.com>
2020-04-06internal/queue: bugfix: plumb configJonathan Amsterdam
The NewGCP queue constructor was not setting the cfg field, resulting in panics like Change-Id: If44809543f0c0ccfb548720351a2bf6687bf7850 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/678585 Reviewed-by: Robert Findley <rfindley@google.com>
2020-04-06internal/config: make more statelessJonathan Amsterdam
As per the bug listed below, we are trying to remove the global cfg variable from the config package. This CL brings us close to that goal. There are still two uses of the global that are harder to remove. Updates b/145301722. Change-Id: Ia6c350ef5d817ff7fb3125cd207343d10911d64f Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/672580 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Robert Findley <rfindley@google.com>
2020-04-06internal/queue: create packageJulie Qiu
The code for creating a new package is moved to internal/queue. This allows for adding items to a task queue from the frontend, without importing the entire internal/etl package. No logic is changed, except the signature of queue.InMemoryQueue.process, so that etl.FetchAndUpdateState can be passed in, and the entire internal/etl package does not need to be imported by internal/queue, which would lead to a circular import. Updates b/135954292 Change-Id: I33a6418c73d85e15c4aa5593d633e19eb7e4eb1b Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/671014 Reviewed-by: Jonathan Amsterdam <jba@google.com>
generated by cgit v1.3 (git 2.53.0) at 2026-04-15 09:45:59 +0000