| Age | Commit message (Collapse) | Author |
|---|
|
On a mac, brew install postgresql installs postgres without the postgres user: it uses $USER instead. The current tests don't take that into account. This CL changes that by:
- Amending the connecting URI to explicitly specify the db name (the current "" approach makes use of the postgres user, which by coincidence is the same name as the database we want: now instead of we specify the postgres user).
- One user test is amended to use the env var provided user if necessary.
Also adds some logging to make it more clear why failures around the URI are happening.
Change-Id: I03d5b40a62d7034e00f45ba8151292688ab61de4
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/685995
Reviewed-by: Michael Knyszek <mknyszek@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
kokoro-CI: kokoro <noreply+kokoro@google.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
|
|
If a test times out, that implies that it got stuck on something.
By default, the Go testing package dumps goroutines when its own
timeout is passed, which prints a goroutine dump, helping to reveal
what was stuck.
Adding an arbitrary timeout on top of the testing package's own
timeout is, in my experience, almost always counterproductive.
If the arbitrary timeout catches a real hang, it causes the test to
fail instead of dumping goroutines, making it much harder to see what
was stuck. On the other hand, if the timeouts are set aggressively
enough to make the test fail early, they are often too aggressive
for CI testing, causing flakes that then have to be triaged on an
ongoing basis.
On balance, the value of saving a minute or two for developers who
have introduced a hang is not worth the cost of suppressing debugging
information and causing flakes that have to be triaged.
Fixes #61556.
For #59347.
Change-Id: I0263d0d9b18283470f100e5a0155818b87b5312f
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/521837
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Bryan Mills <bcmills@google.com>
Run-TryBot: Bryan Mills <bcmills@google.com>
kokoro-CI: kokoro <noreply+kokoro@google.com>
Reviewed-by: Michael Matloob <matloob@golang.org>
|
|
Add a method for getting information about a DB user. We plan to use
this on the worker to see if we can make better load-shedding
decisions.
For golang/go#48010
Change-Id: I80f2811f657ac47d94446a47a38e00502ae29ae8
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/348933
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
|
|
I noticed that internal/postgres tests timed out in CI. From the
stack traces, it's possible that some deadlock with the poller is
implicated. I'm not sure, but just to rule it out, I'm disabling the
poller during tests.
Change-Id: I6f611ecdba28b176dd01f6e6c673a342086a1a5e
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/293930
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
|
|
To avoid sharing between tests, which can lead to race conditions,
create new sample license slices and structs each time we need them.
This uncovered some problems in the postgres tests.
- The UpsertModule test was changing the top-level module license, but
didn't propagate that change through all the units. We
sample.ReplaceLicense to address that.
- The checkModule was filling in missing license with defaults,
instead of the correct data.
Change-Id: I6074914155eb33d6ed6882f81a9d57b20135e422
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/292929
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
|
|
The parallel tests cause a race condition. Until we can
figure it out, run only one test at a time.
Change-Id: I91bf9a0343bf0b4115b2533882464545679ba6c9
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/292496
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
|
|
Run tests in parallel on separate databases.
With four DBs, the time to run all the package's tests is cut in about
half, from 50s to 25s (the time varies from run to run).
Instead of a global variable holding a single DB, we set up a channel
populated with DBs. Each test receives from the channel to get a DB to
use, and sends it to the channel when done.
Because a test may have to wait to get a DB, we can't set an
individual timeout for some tests because acquiring the DB can take
some time. Individual test timeouts aren't really important, because
`go test` will time out the entire run after 10 minutes anyway.
The only test that can't be run in parallel is TestSearch, because it
reads metrics before and after each sub-test to compute a delta,
thereby implicitly assuming that no other Search calls are running in
parallel.
Change-Id: Icb35dbd2f146e27d82d4eef81343cf9725252155
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/291449
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
|
|
Add GetOldestUnprocessedIndexTime, which computes the index timestamp
of the oldest unprocessed module.
The current time minus this value is a lower bound on the time from
when a module enters the index until it is processed.
Change-Id: I0b74f14e94d8d070496df8a8baa11c31decebfd8
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/278552
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Robert Findley <rfindley@google.com>
|
|
Extract into a separate package the core functionality from
internal/postgres that doesn't depend on our particular schema.
This makes it available for other uses, like devtools commands and etl
autocomplete.
Do the same for testing functionality.
We now have three packages where before we had only one:
- internal/postgres: discovery-specific DB operations and test support
- internal/database: discovery-agnostic DB operations
- internal/testing/dbtest: discovery-agnostic DB test support
Change-Id: I54c59aee328dae71ba6c77170a72e7a83da7c785
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/602327
Reviewed-by: Robert Findley <rfindley@google.com>
|
|
Add context info to the DB.exec and DB.query methods,
and remove it from call sites.
Updates b/139152543.
Change-Id: Id4896f60cb3ab07b358b6db62d561e7101c5091b
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/526008
Reviewed-by: Robert Findley <rfindley@google.com>
|
|
Since we now control the methods of DB, we can remove the unnecessary Context
suffix.
You have to remember to use lower-case "exec" etc. anyway, so this change
doesn't add any additional confusion.
Change-Id: I57bde0dcf359fb37e57457accd170cdddfd2ef3e
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/525153
Reviewed-by: Robert Findley <rfindley@google.com>
|
|
By not embedding it, we don't expose all its methods.
Add selected methods back.
Fixes b/139178399
Change-Id: Ic146dc01c8531e1fd5b56e085f53a7735d44c146
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/525149
Reviewed-by: Robert Findley <rfindley@google.com>
|
|
Updates b/137689600
Change-Id: Iaacee7a0b9ca50c4b665459f3f88bf503ff37bd4
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/509283
Reviewed-by: Julie Qiu <julieqiu@google.com>
|
|
Updates b/135546160
Change-Id: Ic58f94c3b058792e36164203f402e47240f5cd01
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/490956
Reviewed-by: Robert Findley <rfindley@google.com>
|
|
postgres.go is broken up into details.go and insert_version.go:
* insert_version.go now contains InsertVersion and all of the unexported
functions that it uses
* details.go contains all functions needed to render the details view on
the frontend:
* GetPackage
* GetLatestPackage
* GetVersionForPackage
* GetTaggedVersionsForPackageSeries
* GetPseudoVersionsForPackageSeries
* GetImports
* GetImportedBy
* GetLicenses
* GetVersion (used for tests in internal/etl/fetch_test.go)
* postgres.go only contains functions specific to postgres
functionality:
* Open
* Transact
* bulkInsert
* prepareAndExec
* sampleVersion, which was used for testing, is moved to test_helper.go
Tests have also been moved to the corresponding test files.
Change-Id: I1f137c6a3dd99b5f390e3f8c385108fe66e97cf1
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/488917
Reviewed-by: Robert Findley <rfindley@google.com>
|
|
The Go standard library packages are now supported on the discovery
site, under the module "std".
It is also now possible to make fetch requests to the cron for module
versions using a tag or commit hash.
The versions template is also updated based on the current mock.
Updates b/131859847
Updates b/132875076
Updates b/126597197
Change-Id: I01fbbbff5bfec93d0ee4c5048f98ffe2da2f2a6a
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/472904
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
|
|
Postgres has a hard limit of 2^16-1 query parameters, which we were
exceeding for some modules (notably Kubernetes). This CL works around
this limitation by executing multiple queries.
Fixes b/133512292
Change-Id: I17087ea54ddab3515068f1ef131dd2ba87165a5f
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/476518
Reviewed-by: Julie Qiu <julieqiu@google.com>
|
|
As a result of commit ca685af0e, we can now run the fetch action
entirely in-memory (and therefore, on AppEngine rather than K8s). This
opens up the possibility for a significant simplification of the fetch
process, which this CL (largely) implements. At a high level, this
change consists of:
1. Move the fetch handler onto what is currently called the cron
service.
2. Introduce a task-queue abstraction for scheduling tasks that call
back into the fetch endpoint. When running on GCP, this uses the
AppEngine taskqueue package. When running locally, it uses a simple
in-process queue.
3. Replace the /indexupdate/ action with /poll-and-queue/, which both
writes to the module_version_states table, and scheduled a fetch task.
This eliminates the need for the Kubernetes based fetch service, and
allows us to use Google Cloud Tasks for fetch tasks, which has some nice
advantages: built-in exponential back-off and rate limiting, and a nice
dashboard.
It should be noted that the existing /fetchversions/ handler was renamed
to /requeue/ and slightly modified: now it looks for versions that are
eligible for re-processing and reschedules them on the queue, which can
be useful if they have either timed-out of the queue or if the queue had
to be purged for some reason.
This change has been deployed and tested on AppEngine: there were
initially some failures due to memory limits, but those have thus far
disappeared after upgrading to the F4_1G instance class. Note that in
the passive state of no cron tasks we will scale down to zero instances,
and in the active state of many cron requests even the F4_1G instance
will scale up, so it is likely that increasing the instance size has
minimal impact on overall cost, though this has not been rigorously
investigated.
Other failures are rare and appear to be caused by deterministic
failure (e.g. no packages), or due to context timeout.
There were a few other internal changes that had to be made to enable
this CL. Notably:
+ The derrors package was refactored slightly to make it easier to wrap
an error while preserving our internal error semantics, for example
when you want to preserve a NotFound error from a client.
+ The conflict action was surfaced in the bulkInsert API, to allow for
upserting new IndexVersions.
+ All use of the old VersionLogs type were removed, since it was easier
to delete this functionality as part of this CL.
The net result is a rather large changeset, but one that is IMO easier
to review as a whole than to split up.
Fixes b/131614520
Fixes b/131612196
Fixes b/130540724
Updates b/131614050
Fixes b/131615422
Fixes b/132153576
Change-Id: I062995a213c99a660791608e25415b65cd7525a7
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/468418
Reviewed-by: Julie Qiu <julieqiu@google.com>
|
|
This unifies license-related functionality into a new internal/license
package, with the intention of making it easier to understand handling
of licenses. It also makes some type names a little cleaner.
A few minor changes are made along the way (e.g. LicenseInfo ->
Metadata), and tests are added for license matching.
Updates b/131921712
Updates b/131741680
Change-Id: I6a766513e971cea74938fbc60f0220c702e66bcd
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/459926
Reviewed-by: Julie Qiu <julieqiu@google.com>
|
|
The pages on the module view are now listed in order
of the package path to preserve directory structure.
Fixes b/131859852
Change-Id: I1924fd48c310e981e5659a9a3e5c44bb1e6bde1f
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/459561
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
|
|
The following queries have been updated to fetch documentation from the
packages table:
* GetPackage
* GetLatestPackage
* GetVersionForPackage
The documentation tab now displays that data.
Fixes b/131706915
Change-Id: I591f374a355c55b8b937dbf2cfb2de6f105591cd
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/456568
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
|
|
At the moment, there isn't a way to retry versions that have been
inserted into the version log if they are dropped after the
makeNewVersionsTimeout.
A /retry endpoint is added to the proxy index cron, which gets entries
from the version logs that are not present in the versions table and
have not failed, and retries fetching them.
The fetch service will now write errors to the error log.
This job is a temporary solution for populating the database, while the
go/go-discovery-etl design is being implemented.
Change-Id: I389c537068e05341ac79aac734d6df5766a1f278
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/452776
Reviewed-by: Robert Findley <rfindley@google.com>
|
|
This CL implements admittedly complicated logic for handling license
redistributability. It does this in three main steps, which could
conceivably be split into separate CLs if desirable.
1. Add an internal.Package.IsRedistributable method, which reports a
package as redistributable if and only if there is at least one license
permitting redistribution at the module root, and if every directory
containing a detected license file contains at least one license file
permitting redistribution. It is very possible that this logic is will
change in the future.
2. Add behavior in the postgres handler to prune package and version
data that should not be stored for non-redistributable content. As a
precaution it was decided to do this in the data layer rather than at
fetch time.
3. Add handling in the frontend to make it clear when content is not
redistributable.
Because we now have a scenario where we can't show license details, but
still want to surface the license disclaimer, move it to a separate
page.
Also:
+ modify license extraction to store the license path relative to the
module@version subdirectory of the module zip.
+ fix a bug where VersionType was not set on in postgres.GetPackage.
Fixes b/127335320
Updates b/124309095
Change-Id: I36194b78651ccff91b0e06d0d3d4fc639a884565
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/455069
Reviewed-by: Julie Qiu <julieqiu@google.com>
|
|
If a user makes a request to /<import-path>, without the <version> query
param, they will now be shown the latest version of the package.
Fixes b/130215822
Change-Id: Ia24c70ad720fd59f8cdf4dc21cded7beeb032bc5
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/456217
Reviewed-by: Robert Findley <rfindley@google.com>
|
|
This change modifies the postgres test_helper API to make it easier to
use separate databases for each test suite. It also automates the test
database migration, to ensure that tests are run on an up-to-date
database schema.
Updates b/130719094
Change-Id: Iec0c2404b5859f9bb3c403bdec2581b4636f61a6
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/454237
Reviewed-by: Julie Qiu <julieqiu@google.com>
|
|
To be consistent with other version information, use the version_type
stored on the versions table.
Updates b/130367504
Change-Id: I28940a30217f2abd1c688712ab89b5d4dce74d0e
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/453885
Reviewed-by: Julie Qiu <julieqiu@google.com>
|
|
internal.Version has been updated with these fields:
* ReadmeFilePath: displays the file_path of the readme
* ReadmeContents: displays the contents of the readme, previously ReadMe
extractReadmeFromZip now returns both the file_path and contents of the
first README that is detected. The functions extractFile and
containsFile have been removed. Their functionality is now merged
inside extractReadmeFromZip.
readmeHTML now detects if a file is a markdown file, before parsing it
with the blackfriday markdown parser. If it is not, it returns the
readme as plaintext.
Several tests in internal/postgres have been updated to use
postgres.sampleVersions for generating test internal.Versions structs.
Change-Id: Ic0395d7c38090ace431c2ce9468d4ef5070bf2b4
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/450586
Reviewed-by: Robert Findley <rfindley@google.com>
|
|
Break the reference cycle between 'Package' and 'Version', and get close
to the truth in the types returned from the database. To achieve this,
make the following changes:
+ Package type loses its Version reference
+ Version is split into metadata (VersionInfo) and Packages
+ VersionedPackage is introduced to capture the common case where we
want both Package and Version metadata.
Additionally, ensure that Suffix is always set on Packages returned from
the database.
There are still a few fields that can 'lie', notably Package imports and
Version readme, but those are left to a potential later CL.
Updates b/130367504
Change-Id: I392238aca4c335b64766c2ef23d145f462d0ff1b
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/451770
Reviewed-by: Julie Qiu <julieqiu@google.com>
|
|
The query for postgres.Search is updated to use vw_search_results, so
that the search query is easier to read and maintain.
Results with a rank less than 1e-10 are filtered out.
The following fields for postgres.SearchResult are changed:
* Relevance is deprecated
* Rank is added, which is calculated using relevance and num imported by
* NumImportedBy is now used instead of NumImporters, for consistency with
the frontend.
* Total is added, which represents the total number of results from the query
postgres.GetLatestPackageForPaths is deprecated, since it is no longer used.
A bug is fixed in details.tmpl where the active tab in the nav bar was not
rendering for importedby.
Updates b/130090305
Change-Id: I8d72c730bd8667b941ca7a519c23234bd6bd646d
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/451331
Reviewed-by: Robert Findley <rfindley@google.com>
|
|
In many cases, these types were just holders for, respectively,
ModulePath and SeriesPath. Sometimes other fields were populated,
sometimes not. This change removes these types so that it is less
ambiguous what is available when working with our data models.
This also required joining with the modules table to get series_path in
some places. On principle I joined rather than ignore SeriesPath in
tests. We can later revisit simply making 'Series' a method on Version.
In postgres_test, I also introduced a pattern for generating test data
that I would like to consider for a more general test data library:
func sampleMyType(muts ...func(*MyType)) *MyType {...}
Updates b/130367504
Updates b/130719094
Change-Id: I66013fa79699bde9d09a37faef7928577163e0d2
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/451526
Reviewed-by: Julie Qiu <julieqiu@google.com>
|
|
fetchImportersDetails now returns importers for a package.
importedby.tmpl is also updated to display the name and path of a
package's importers.
The tab importers is now importedby, for consistency with the rest of
the codebase.
Fixes b/124309193
Fixes b/126550318
Change-Id: I09f82d4a2b0a60e478e77268638cb7c3ac9cb75e
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/450579
Reviewed-by: Robert Findley <rfindley@google.com>
|
|
fetchImportsDetails now returns imports for a package from the
database. imports.tmpl is also updated to display the name and path of
imports, grouped by whether or not they are in the standard library.
A bug is fixed where the insert for to_name and to_path were
swapped.
Fixes b/124308480
Change-Id: Ic325aee1ee9be37aad7f13948d68ea85461dc92b
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/450576
Reviewed-by: Robert Findley <rfindley@google.com>
|
|
Package imports are now parsed and inserted into
the database during the fetch process.
postgres.GetImports can be used to get imports for a given package from
the database.
Fixes b/126908435
Fixes b/125406053
Change-Id: I663a40ec89eb1fc4e4b76874aba4fcefafeddca0
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/437551
Reviewed-by: Andrew Bonventre <andybons@google.com>
|
|
This method will be used to return full license contents for the
licenses details page.
Fixes b/129372226
Change-Id: I0efda0fad27328eec15a2a8fd1c65ebba1f78a5f
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/450455
Reviewed-by: Julie Qiu <julieqiu@google.com>
|
|
Eliminate usage of grpc.Status (which was just being used to communicate
InvalidArgument), by replacing it with the newly added derrors package.
Tests that were relying on grpc status codes for error assertions had to
be updated as well. I went through a few iterations of how to do this,
and settled on a simple enumeration of errors to be used for testing
only (in test_helper.go).
Also add documentation for postgres.DB and postgres.Open to make golint
happy.
Fixes b/128540225
Change-Id: Ie0b90211ae8d7d45daf980c1cc2701aabd4b9019
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/450158
Reviewed-by: Julie Qiu <julieqiu@google.com>
|
|
postgres.bulkInsert is implemented, which allows for inserting multiple
rows in a single query.
Inserting multiple values with a single insert performed significantly
better than using a PREPARE statment, based on benchmarks in CL 449255.
postgres.versionsDiff is also removed from tests, and cmp.Diff is used instead.
Fixes b/129778736
Updates b/130092669
Change-Id: If38ae56859fa00cfb8ae2cb04b16d275db40f118
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/449253
Reviewed-by: Robert Findley <rfindley@google.com>
|
|
This CL changes our data model so that licenses are associated with
packages rather than module versions, and adds multiple license
detection. As a result, given following file layout
my.mod/LICENSE
my.mod/pkgfoo/COPYING
my.mod/pkgbar/LICENSE.md
both LICENSE and COPYING (but not LICENSE.md) will be associated with
the package my.mod/pkgfoo.
Additionally make some minimal UI changes to support the new data model,
though a proper implementation of the licenses tab is deferred to a
later CL.
Fixes b/130372424
Updates b/129371814
Updates b/129372226
Fixes b/129000846
Fixes b/129372204
Change-Id: Ife58498914fea75e4c58e7c713a8071887f87000
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/447910
Reviewed-by: Julie Qiu <julieqiu@google.com>
|
|
The module page is added, which displays the directory and synopsis
of all packages in the module and the README.
This change includes the following additions:
* frontend.fetchModulePage: returns data needed to populate module.tmpl
* postgres.GetVersionForPackage: returns the version for a given
package_path and version, along with all of the packages for that
version.
Note: At the moment, both the overview page and the module page display
the README. Once a decision is made on where the README will be displayed,
these pages will be updated.
Fixes b/130112117
Fixes b/130111944
Change-Id: Ib240b1744cf66be72a63683804b521beaec62307
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/444468
Reviewed-by: Andrew Bonventre <andybons@google.com>
|
|
Previously, an error occurred when parsing versions with build tags,
such as "v1.2.5+incompatible".
This CL fixes the error and moves parsing major, minor and patch inside
internal/postgres to individual functions.
Fixes #130041347.
Change-Id: I192dad8b1620c2235f19ddc8cf0f04f98c473b39
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/444465
Reviewed-by: Andrew Bonventre <andybons@google.com>
|
|
Use the Timeout middleware to to add a timeout for incoming requests.
As a notable exception, don't do this for module fetch, as we want
module processing to continue even if the client cancels their request
early. Instead, set an explicit timeout for module processing.
Additionally, bind work done on behalf of the request to the request
context. This required switching to the context-aware sql APIs, setting
a context on outbound HTTP requests, and quite a bit of 'ctx' threading.
Fixes b/128689909
Change-Id: Ic732808b6df89a61106f5d14b8a9ecaaefb95c4f
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/443023
Reviewed-by: Julie Qiu <julieqiu@google.com>
|
|
versions tab
fetchVersionsTabPage takes in a package path and a version and then returns
all of the data needed to display the packages that have the same series and
package suffix to make the versions tab content. The data is displayed in a
hierarchical list of major (i.e. "v1") then major and minor (i.e. "1.2") and
lastly the entirety of the version (i.e. "1.2.3" or "1.2.2-alpha.1").
verisons.tmpl is the template that displays all of the data in the
VersionsTabPage struct.
The routing was changed so that MakeFrontendHandlerFunc handles both
/<import path>@<version> which brings up the overview page and
/<import path>@<version>?tab=versions which brings up the versions
tab.
Fixes b/124308701
Change-Id: I92bd86d050ab6e7e669c07a750b325b3e026d052
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/443954
Reviewed-by: Julie Qiu <julieqiu@google.com>
|
|
frontend.fetchSearchPage is implemented, which returns all of the data
needed to generate results for a search page.
postgres.Search is updated so that results with a rank score of 0 are
not returned.
Change-Id: I2e41716ce0cfa9db70b0a84930b4e2568a2c969b
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/443906
Reviewed-by: Robert Findley <rfindley@google.com>
|
|
Truncate all timestamps used in tests to Microsecond precision, so that
precision is not lost when saving and retrieving from Postgres.
Fixes b/129859073
Change-Id: Id677147ac44dc08cd52e0fae9a6d0510b68a666e
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/443892
Reviewed-by: Andrew Bonventre <andybons@google.com>
|
|
This CL edits GetTaggedVersions and GetPseudoVersions so that
they fetch all of the versions of associated packages that
have the same specified series name and package path suffix.
These methods will be used to determine which versions to
display on the versions tab in the frontend.
Fixes b/129553236
Change-Id: I3930868b66f11d955f1ce15307aa21c0599ab999
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/443090
Reviewed-by: Julie Qiu <julieqiu@google.com>
|
|
GetLatestPackageForPaths originally was not pulling expected packages,
due to how the WHERE clause param was being passed in.
It is fixed in this CLs along with the tests.
Change-Id: Ia377175bd6a51fe4fd53ea3d8ccc2c1dfc377ecb
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/443360
Reviewed-by: Channing Kimble-Brown <ckimblebrown@google.com>
|
|
GetTaggedVersions and GetPseudoVersions both return a list of
versions for a specific package. These methods will be used
by the frontend to fetch the versions that will be displayed
in the 'versions' tab.
Fixes b/126550849
Change-Id: I169aad79823d67273cb7ab1f3995b700d75e8ac2
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/441783
Reviewed-by: Julie Qiu <julieqiu@google.com>
|
|
InsertVersion now checks to make sure that the module path is valid
before inserting a version into the database.
parseModulePathAndVersion now checks the module path and returns an
error if the path is invalid.
FetchAndInsertVersion also ensures the module path is valid before
calling InsertVersion.
Fixes b/129269917
Change-Id: I1f9446a671b9af29e48b7cc5554007fda657662a
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/440419
Reviewed-by: Julie Qiu <julieqiu@google.com>
|
|
This change adds GetLatestPackageForPaths which returns the latest
package associated with each path in a list of strings. This method
will eventually be used to get the packages queried by search results.
Updates b/124308701
Fixes b/126714352
Change-Id: I2b6e86aec7b5ba82ed0e45db52784d3cde1b8074
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/439699
Reviewed-by: Julie Qiu <julieqiu@google.com>
|
|
This CL parses the version for the semver columns in
InsertVersion so that we can query the database for packages sorted by
semver.
This also adds the GetLatestPackage which will be used by the frontend
when a user specifies the package path but not the version.
Fixes b/129020053
Change-Id: I8741721f61aa78f0e7dca358345903488df8a6fa
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/438011
Reviewed-by: Julie Qiu <julieqiu@google.com>
|
|
+ Fix a test that fails due to precision differences in time values
in Postgres
+ Provide an example URL when visiting the root path of the fetch
service
+ Update handlers to return a 404 instead of 500 if a user provides
an invalid path
+ Don't require a full URL when parsing module data since the path
is all that's needed
Change-Id: I9f8dc13b0a4995e6cb848f86ad41f6fed2759f82
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/439693
Reviewed-by: Julie Qiu <julieqiu@google.com>
|
