diff options
| author | Jonathan Amsterdam <jba@google.com> | 2021-09-01 07:22:46 -0400 |
|---|---|---|
| committer | Jonathan Amsterdam <jba@google.com> | 2021-09-01 14:16:41 +0000 |
| commit | bb6da5216eba144f9891c1c0dd2228fdbfa45163 (patch) | |
| tree | 230b553713bffd364321db456594aa70026c8dca /internal/fetchdatasource | |
| parent | 6e28a50e59803c0b1f05ca9e18a38d33b9e2868b (diff) | |
| download | go-x-pkgsite-bb6da5216eba144f9891c1c0dd2228fdbfa45163.tar.xz | |
internal/{worker/fetch}: include DB activity in load-shedding
Move the load-shedding logic to the worker and have it span both the
fetch and processing of the module (as previously) as well as
inserting it into the database.
This is a more accurate estimation of load, since running a lot of
concurrent queries definitely slows down processing.
Most of the time this won't make much difference, but under high load,
such as when processing multiple large modules, it will reduce DB
contention and should result in greater throughput.
For golang/go#48010
Change-Id: I7d0922e02d00182e867fd3b29fc284c32ecab5ee
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/346749
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
Diffstat (limited to 'internal/fetchdatasource')
| -rw-r--r-- | internal/fetchdatasource/fetchdatasource.go | 1 |
1 files changed, 0 insertions, 1 deletions
diff --git a/internal/fetchdatasource/fetchdatasource.go b/internal/fetchdatasource/fetchdatasource.go index 84b44c17..e7e38c48 100644 --- a/internal/fetchdatasource/fetchdatasource.go +++ b/internal/fetchdatasource/fetchdatasource.go @@ -128,7 +128,6 @@ func (ds *FetchDataSource) fetch(ctx context.Context, modulePath, version string }() for _, g := range ds.opts.Getters { fr := fetch.FetchModule(ctx, modulePath, version, g, ds.opts.SourceClient) - defer fr.Defer() if fr.Error == nil { m := fr.Module if ds.opts.BypassLicenseCheck { |