| Age | Commit message (Collapse) | Author |
|---|
|
internal/frontend depends on internal/queue. Move the parts with gcp
dependencies into its own package so internal/frontend no longer has
those gcp queue dependencies.
For #61399
Change-Id: I0083c31da1b367f135e53686e4c994d7c4d6420f
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/510815
TryBot-Result: Gopher Robot <gobot@golang.org>
kokoro-CI: kokoro <noreply+kokoro@google.com>
Run-TryBot: Michael Matloob <matloob@golang.org>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
|
|
Change-Id: I44ac70e194eb6731c6886b7616954cb7d77b9a44
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/375274
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
|
|
queue.Options is now used to pass optional arguments to ScheduleFetch.
Change-Id: I4e00a91ad7d2cfa2ccf669d643a21fda6b3b3fb8
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/348815
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
|
|
If the status of a task to be enqueued indicates reprocessing,
then enqueue the task with a URL whose query param indicates that
proxy fetching should be disabled.
Change-Id: Id7255e861c1dea87d131d83151eb1a46df0ea4ff
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/278712
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
|
|
Remove support for the QueueService config and for queuing tasks for
an AppEngine service.
Change-Id: I05dcdeaef2e378a5261490badd8b11cd68f64429
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/274494
Trust: Jonathan Amsterdam <jba@google.com>
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Julie Qiu <julie@golang.org>
Reviewed-by: Julie Qiu <julie@golang.org>
|
|
The task names we generated for Cloud Tasks included time information,
so that we could control how freqeuently tasks were de-duped. We've
been using a 3-hour window, meaning that if a module's task finished
or was deleted, another task for that module couldn't be added to the
queue (would be de-duped) for 3 hours.
This scheme has the unintended consequence that if a module is still
on the queue after 3 hours, then another task for the same module
could be enqueued (since its task ID would be different). That means
that if the queue ever gets 3 hours behind, it could get filled with
duplicate modules. For example, this morning the queue had about 25,000
tasks, of which about 7,000 were duplicates.
This CL sets a task's name to a function of the module path and
version. That will prevent a task on the queue from being duplicated.
A repeat task can be added about 1 hour after the previous task
finishes or is deleted (see the Task De-duplication section of
https://cloud.google.com/tasks/docs/reference/rpc/google.cloud.tasks.v2#task),
which is fine. It is still possible to add a suffix to the task
name to requeue sooner.
Change-Id: I34ffce5ea67b9e00b88ca4cf38182e34a6ba8657
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/270337
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
|
|
HTTP tasks (but not AppEngine tasks) have a default timeout of 10
minutes. Extend that to the maximum of 30 minutes.
Change-Id: I75efc0523ff83471a0a2a8d453e1284483feefe1
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/258197
Reviewed-by: Julie Qiu <julie@golang.org>
Trust: Jonathan Amsterdam <jba@google.com>
|
|
Arrange for the task queue to authenticate itself to the worker, which
is behind the IAP. This requires adding information to the task that
allows the queue to construct an identity token.
Change-Id: I7e10a2d2d3151852a220f4ad7009df6dda0dd98c
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/254517
Reviewed-by: Julie Qiu <julie@golang.org>
|
|
As staticcheck tells us, the right proto package to use is
google.golang.org/protobuf.proto.
Change-Id: I6e0f987bdd9217c3d26397547a7df250d3e6fd11
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/252838
Reviewed-by: Julie Qiu <julie@golang.org>
|
|
We were calling cmp.Diff on protobuf messages directly, which is not a
good idea. Use proto.Equal instead. Since it doesn't play nicely with
the IgnoreFields option, ignore the Name field in a different way.
Change-Id: Ie0dfa82b2b78c40402d2a05fed9f4dceff2fc2bf
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/252777
Reviewed-by: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
|
|
Instead of setting environment variables and calling config.Init,
which can trigger code paths that can't work on some dev machines,
just create a config in the test.
Change-Id: Ie85dd86540f9bc11481f647a1856d4e4d828f150
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/251497
Reviewed-by: Julie Qiu <julie@golang.org>
Reviewed-by: Miguel Acero <acero@google.com>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
|
|
Allow GCP queues to create a task destined for any URL, not just an
AppEngine service. This lets the worker run somewhere other than
AppEngine, like Kubernetes.
Also, reorganize the GCP type so any bad config is detected at
creation time.
Change-Id: Ic3ff988999517f917d24f07f7fe49df0e4d8bc31
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/251061
Reviewed-by: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
|
|
Add a unified function for creating new queues so that separate
implementations of newQueue in cmd/frontend and cmd/worker can be
removed. Since queue.New contains the necessary logic for deciding
between GCP and InMemory, the two could be made unexported after
updating the corresponding tests.
Fixes golang/go#40097.
Change-Id: Ie509ba39ef293cca3ff95f2ce12833339c0542ea
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/241477
Reviewed-by: Jonathan Amsterdam <jba@google.com>
|
|
If no GO_DISCOVERY_QUEUE_SERVICE is specified, default
to the value of GAE_SERVICE.
Without this, the QueueService config field is empty, so scheduled
tasks are not delivered to the worker.
Change-Id: Ia3d46f122208562c11c76dd0b857a0a5f6e5a822
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/241066
Reviewed-by: Robert Findley <rfindley@google.com>
|
|
At the moment, taskIDChangeInterval is a hardcoded value in
internal/queue. However, we will soon have two task queues running,
which require different change intervals, so this value is now set in
internal/config.
Additionally, the taskIDChangeInterval for the worker is changed to 3
hours.
Change-Id: I498abefce6543005463be7da99a5a778f3a6e973
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/758919
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
|
|
The code for creating a new package is moved to internal/queue. This
allows for adding items to a task queue from the frontend, without
importing the entire internal/etl package.
No logic is changed, except the signature of
queue.InMemoryQueue.process, so that etl.FetchAndUpdateState can be
passed in, and the entire internal/etl package does not need to be
imported by internal/queue, which would lead to a circular import.
Updates b/135954292
Change-Id: I33a6418c73d85e15c4aa5593d633e19eb7e4eb1b
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/671014
Reviewed-by: Jonathan Amsterdam <jba@google.com>
|
