aboutsummaryrefslogtreecommitdiff
path: root/internal/postgres/postgres.go
AgeCommit message (Collapse)Author
2024-02-21internal/postgres: support module@version exclusionsJonathan Amsterdam
Allow a specific module@version to be excluded. Previously, only entire module paths or path prefixes (at every version) could be excluded. This will allow us to exclude specific versions that we can't process, because they are too large for instance. A necessary part of the change is adding a version argument to the IsExcluded method. While we are there, we drop the error return value, since it's always nil. Change-Id: I68fd1c8f66ee654e56bd7dcf230a88959b32bbc8 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/565815 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Michael Matloob <matloob@golang.org> Run-TryBot: Jonathan Amsterdam <jba@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com>
2023-08-22internal/testing/fakedatasource: implement PostgresDB interfaceMichael Matloob
This change adds the rest of the methods from the PostgresDB interface to the fake data source so it can be used to test some functions that expect a postgres db. Most of the methods are left unimplemented. I removed the IsPostgresDB that I originally put on the PostgresDB interface since now we will have other types try to be PostgresDBs. In an upcoming CL, I will extend the DataSource interface so that it's the same as PostgresDB, and just have the frontend handle errors that are errors.ErrUnsupported. The CL then updates tests on some of the methods on the frontend that need a PostgresDB to use the FakeDataSource instead. This CL then removes the code that sets up and runs the internal/frontend tests with a PostgresDB. For golang/go#61399 Change-Id: I3bae5adc5ec03536b2074eee32b4e5558dfa7478 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/520878 Run-TryBot: Michael Matloob <matloob@golang.org> Reviewed-by: Robert Findley <rfindley@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>
2023-07-18internal: add an interface for internal/postgres.DBMichael Matloob
And use that interface to remove the dependency from internal/frontend to internal/postgres. The dependency still exists in the test code. That will be taken care of in another CL. For #61399 Change-Id: Ic1b3c257e6eb34858cf31b7596251aaff71d3482 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/510417 kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Reviewed-by: Michael Matloob <matloob@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Michael Matloob <matloob@golang.org>
2022-11-15all: convert interface{} to anyHana (Hyang-Ah) Kim
Change-Id: I1f3b7cc8899c7707abb01e3d14807c37c3451382 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/449695 TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Run-TryBot: Hyang-Ah Hana Kim <hyangah@gmail.com>
2021-10-21internal/postgres: add defer before wrap function callAntonio Antelo
Change-Id: I0fee50bdf505e55f5cbe9a88b3faef5e87b21fd5 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/357211 Reviewed-by: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org> Trust: Cherry Mui <cherryyz@google.com> Run-TryBot: Cherry Mui <cherryyz@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com>
2021-09-13internal/worker: display DB info on home pageJonathan Amsterdam
Add DB process and lock information to the worker home page. For golang/go#48010 Change-Id: Idab82180a33ce2d00350df0bbfeeb58b2a628ae8 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/349309 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Reviewed-by: Julie Qiu <julie@golang.org>
2021-09-10internal/postgres: get DB user infoJonathan Amsterdam
Add a method for getting information about a DB user. We plan to use this on the worker to see if we can make better load-shedding decisions. For golang/go#48010 Change-Id: I80f2811f657ac47d94446a47a38e00502ae29ae8 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/348933 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com>
2021-07-21internal/{database,postgres}: export CollectStringsJonathan Amsterdam
Export the CollectStrings function and move to the database package so it can be used elsewhere. Change-Id: I1032e4f5042209e42e8f0de9cebfb48653d56f69 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/336209 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-06-09internal/worker: add metric for new unprocessed modulesJonathan Amsterdam
We have a metric for total unprocessed modules, including new ones and those we are reprocessing. Add a metric that tracks only the new ones: those with status 0 or 500 (the latter to count the number of failed new modules). This metric is a better choice for an alert, since during reprocessing we expect a very large total backlog, but we never want a large backlog of new modules. Change-Id: Ibc3cbee1d867f6a454748237352a70cf9eb500c0 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/326290 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-05-27internal/{postgres,worker}: add metric for number unprocessedJonathan Amsterdam
Add a metric that records how many modules need to be processed (those with status codes of 0 or 5xx). That will tell us how much the worker is backed up. Change-Id: I68c47255e3720b5f978037c1251b0e502961d089 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/323109 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-03-04internal/postgres: add collectStringsJonathan Amsterdam
Factor out a common pattern by adding a function that collects strings from a query. Change-Id: Ia9ba535e0deddc79cb0566548ab82d1df0cd52ff Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/298731 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-02-19internal/postgres: disable poller during postgres testsJonathan Amsterdam
I noticed that internal/postgres tests timed out in CI. From the stack traces, it's possible that some deadlock with the poller is implicated. I'm not sure, but just to rule it out, I'm disabling the poller during tests. Change-Id: I6f611ecdba28b176dd01f6e6c673a342086a1a5e Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/293930 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2020-12-17internal/postgres: method to compute index timestamp for latency metricJonathan Amsterdam
Add GetOldestUnprocessedIndexTime, which computes the index timestamp of the oldest unprocessed module. The current time minus this value is a lower bound on the time from when a module enters the index until it is processed. Change-Id: I0b74f14e94d8d070496df8a8baa11c31decebfd8 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/278552 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Robert Findley <rfindley@google.com>
2020-11-04internal/postgres: hardcode unit_id columnJonathan Amsterdam
To facilitate the change in column name from "path_id" to "unit_id", we had dynamically determined the name before each relevant query. Now that the rename has happened, this CL undoes that work, using "unit_id" as a literal string. Change-Id: Iaf0b475504cc7d7b84b3ac61a1df0450ad9a81ed Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/267699 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2020-11-03migrations,internal/postgres: rename path_id columns to unit_idJonathan Amsterdam
Change the foreign-key column name from path_id to unit_id, to match the change in the table from paths to units. Simultaneously change the default column in internal/postgres to unit_id. This doesn't matter for deployed code, because the tables in all environments have at least one row, so the correct column name will be auto-detected. But it matters for tests. Change-Id: I5f090f674eeb6e9c7ec2c7c6cb198e2e5a5fc730 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/267497 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com>
2020-11-03internal/postgres: determine path/unit_id column dynamicallyJonathan Amsterdam
Three tables still have a path_id column that is a foreign key for the "units" (previously "paths") table. It is not possible to rename those columns while keeping the code working: we would have to create a new column, change the code, then delete the old column, all the while keeping the columns in sync. This CL suggests a simpler approach: the code figures out the name of the column on its own. Then we can simply do one migration to rename the all the path_id to unit_id columns, and the code will continue to work. The code becomes uglier, but it is temporary: after we're done, we'll put it back the way it was, with literal column names. Change-Id: Ia4f621e1c0112d378cf63811311a430982c1cb5f Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/267238 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Reviewed-by: Julie Qiu <julie@golang.org>
2020-10-14internal/postgres: initialize excluded stateJonathan Amsterdam
Load the excluded table as soon as we create a DB, to initialize in the in-memory state. Change-Id: I1546566a1cf7c842b101ff8537f0433c8d170e32 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/262417 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2020-10-13internal/postgres: use poller to update excluded listJonathan Amsterdam
Change-Id: I7f2218b3eb035973d3f6bc00a8e5e6a1df2db571 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/261819 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2020-09-09cmd,internal: clean up package documentationJulie Qiu
Several packages were missing package documentation, which is now added. Change-Id: I1a88ebebaee0fddeb5a7bb9e2573375ad887c311 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/253607 Reviewed-by: Jamal Carvalho <jamal@golang.org> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2020-08-25internal,internal/postgres: remove non-redist data on readJonathan Amsterdam
The postgres package removes non-redistributable data during insertion. Also remove it when reading, unless bypassLicenseCheck is true. Handling this in the DataSource layer simplifies frontend code, which can avoid redistributability checks and knowing about the bypass flag except when it constructs its DataSource. As part of this work, move the removal of non-redistributable data to the internal package, so it can eventually be used by the proxy data source. For golang/go#39602 Change-Id: Ia1362ead917b42f844b4c4d25ade74cdcb03d4dc Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/250017 Reviewed-by: Julie Qiu <julie@golang.org>
2020-08-06internal/postgres,etc.: support bypassing license restrictionsJonathan Amsterdam
Add a mode to postgres.DB where data is saved even if the module or package is not redistributable. This "bypassing license restrictions" mode is off by default. Add a flag to the worker binary to turn it on. Used a flag rather than an environment variable so there is no way for ambient state to affect the binary for this important decision. Also: - Add a test to verify that license data is removed/not removed depending on the bypass setting. - Fix a bug where the new model insertion wasn't omitting documentation for non-redistributable modules. - Fix a bug in internal/sample where the top-level readme wasn't getting populated. A later CL will change the frontend to bypass the check. For golang/go#39602 For golang/go#39629 Change-Id: I67a6d24c18f3b93cfbfc9ec2a20159c07a84e077 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/246957 Reviewed-by: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com>
2020-04-23all: rename module to golang.org/x/pkgsiteJulie Qiu
golang.org/x/discovery is renamed to golang.org/x/pkgsite. When the repository is open sourced, it will be hosted at go.googlesource.com/pkgsite. Change-Id: Ifc3b45b771a385b99179e785447f2a87afcacf87 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/724273 Reviewed-by: Jonathan Amsterdam <jba@google.com>
2020-03-27internal/etl: use database.DB instead of sql.DBJonathan Amsterdam
Replace the direct use of sql.DB with the new database.DB. This will give us logging of autocomplete queries. Change-Id: I656e07b42359915a69dfc567c4c6f56d41900aab Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/604404 Reviewed-by: Julie Qiu <julieqiu@google.com> Reviewed-by: Robert Findley <rfindley@google.com>
2020-03-27internal/postgres: replace Open with New(db)Jonathan Amsterdam
The constructor for a postgres.DB takes a *database.DB instead of a driver and connection string. Change-Id: Ief58ce676946fd34cbcafc2b634a772663b6f932 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/604403 Reviewed-by: Robert Findley <rfindley@google.com> Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-03-27internal/database, internal/testing/dbtest: site-agnostic DB functionalityJonathan Amsterdam
Extract into a separate package the core functionality from internal/postgres that doesn't depend on our particular schema. This makes it available for other uses, like devtools commands and etl autocomplete. Do the same for testing functionality. We now have three packages where before we had only one: - internal/postgres: discovery-specific DB operations and test support - internal/database: discovery-agnostic DB operations - internal/testing/dbtest: discovery-agnostic DB test support Change-Id: I54c59aee328dae71ba6c77170a72e7a83da7c785 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/602327 Reviewed-by: Robert Findley <rfindley@google.com>
2020-03-27internal/etl: add an action to populate completion indexesRob Findley
We're going to compute auto-completions using redis sorted sets in the 'HA' redis instance (because we don't want them to be evicted by caching). This data need not be very fresh, and so can be computed by the ETL every so often, perhaps once a day. This CL wires the HA redis instance into the ETL, and adds an action to update these Redis sorted sets. Doing this takes 1-2 minutes and consumes quite a lot of memory, so there are some optimizations used: + the underlying *sql.DB is used for streaming search documents, rather than a method on postgres.DB. There is explanation for this in the code, and additionally I don't think this is so bad as long as it is only done in the ETL. + Completions data is loaded into temporary Redis keys and then renamed to overwrite the keys that will be used by the frontend, in order to minimize disruption while the completion data is being updated. Updates b/143370178 Change-Id: I48caf77eb26d8ea7b60c2e117481bfa98ef089ca Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/602062 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2020-03-27internal/postgres: redact password in error messagesJonathan Amsterdam
Change-Id: I487f1dd3d6e925b1d95a2ddd7726d7a1b4ed9e42 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/599091 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-03-27internal/middleware,etl: add error reporting middlewareRob Findley
A new middleware is added that sends an error report whenever the ETL returns a 500 from a scheduled endpoint. Along the way, refactor main() in cmd/etl to be more consistent about resolving dependencies in helper functions. Fixes b/142731446 Change-Id: Ib44cd40bf48a3c229220c8c571e52ee033047aa5 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/592534 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-03-27internal/postgres: set the postgres statement_timeout parameterJonathan Amsterdam
Set Postgres's statement_timeout parameter. This should prevent runaway statements. We have to set it each time we open a DB connection. Also, the timeout on the fetch context should be related to the statement timeout, so set it to twice the value. Change-Id: I9568ae81b8d2bf86c7f642d5de530acfa7fa90e7 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/583658 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Robert Findley <rfindley@google.com>
2020-03-27discovery: remove unused search codeJulie Qiu
The following code is removed, so that we can drop the documents and mvw_search_documents tables: * postgres.LegacySearch * postgres.RefreshSearchDocuments * postgres.legacyInsertDocuments * postgres.prepareAndExec Updates b/142406961 Updates b/142559211 Change-Id: Iccb5bf930ee7e7e2f7d32283626dd6cfd722ec61 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/575282 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Robert Findley <rfindley@google.com>
2020-03-27internal/postgres: make smaller bulk insert statementsJonathan Amsterdam
Change the maximum number of parameters in a bulk insert statement to 1000. We were allowing up to 64K parameters in a bulk insert statement. Sometimes that was causing the App Engine instance to crash (for example, the Azure SDK for Go, which has over 1600 packages). Also, use execTx to improve log output. Change-Id: Id7c502c9e855cdd9b340ce96a682d486e13ee7f3 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/570666 Reviewed-by: Robert Findley <rfindley@google.com>
2020-03-27internal/postgres: shorten query args in logsJonathan Amsterdam
Instead of putting all the args into a log entry, construct a short string of them. Fixes b/142476090. Change-Id: I67af083c5dafabb6e694f67b53860f28ad2d2015 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/570664 Reviewed-by: Robert Findley <rfindley@google.com>
2020-03-27internal/postgres: add a fast(er) searchRob Findley
A new method is added to postgres.DB, optimistically named 'FastSearch', that implements search hedging. Each FastSearch concurrently queries both all packages as well as popular subsets of packages, so that results can be returned quicker when they consist entirely of popular packages. Additionallly SearchResult.Rank is renamed to 'Score', to differentiate it from ts_rank. The old 'Search' method is kept for now, although 'log' is corrected to 'ln' (this should have little effect on the ordering of results). In a subsequent CL we will switch the default search method to FastSearch, and implement the 'of many' qualifier in the search result summary. Updates b/141182438 Change-Id: I73c6700e09288f3a8bfdb5fa57d0899606e452d2 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/558516 Reviewed-by: Jonathan Amsterdam <jba@google.com>
2020-03-27internal/postgres: log query completion with a structJonathan Amsterdam
Record error and elapsed time. This should make it easier to find slow queries. Change-Id: I363c76039e00850763939d49b367400441cc4c34 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/556730 Reviewed-by: Robert Findley <rfindley@google.com>
2020-03-27internal/postgres: add a search benchmarkRob Findley
A search benchmark is added, to be run against a dev instance of the database. Updates b/141182438 Change-Id: I73e9742cc6697f282a84252ab85b85d7dc0c689f Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/556705 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Julie Qiu <julieqiu@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2020-03-27various: use internal/log packageJonathan Amsterdam
Take advantage of log levels when logging DB queries: use Debug, except for errors. Change-Id: Icd99d1474799b89d2d8149e5f5bb6f417c6448b5 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/556725 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-03-27internal/postgres: take low-order part of instanceIDJonathan Amsterdam
bugfix: take the end of the instance ID, which is random; not the beginning, which seems fixed. Also, given that we only have a handful of instances, 4 characters should suffice. Change-Id: Ibc80f65e335fa2f3ebeb39322f85cb1a1a07a60e Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/556716 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Robert Findley <rfindley@google.com>
2020-03-27internal/postgres: display query errors with the query's unique IDJonathan Amsterdam
Change-Id: I97bbbab39d851fe87b9ab24c6fe78e5c6a7bbdc7 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/556717 Reviewed-by: Julie Qiu <julieqiu@google.com> Reviewed-by: Robert Findley <rfindley@google.com>
2020-03-27internal/postgres: log more queriesJonathan Amsterdam
Change DB.exec to support transactions, and funnel tx.ExecContext calls through it. Also, log bulkInsert queries, being careful to keep the log output to a reasonable size. Change-Id: I356ff3acaec2e20894dddb4d034eba5765009f6a Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/556099 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-03-27internal/postgres: log every DB queryJonathan Amsterdam
Log the start and end of every DB query. Change-Id: If9d053c1ff409ef1ce1fd54ec35e79d24f2d8ab3 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/555198 Reviewed-by: Robert Findley <rfindley@google.com>
2020-03-27internal/postgres: merge latest and versioned handlingRob Findley
getLatestPackage and getLatestVersionInfo are removed, with their functionality inlined into GetPackage and GetVersionInfo, respectively. Additionally, the following cleanup is applied: + sql values are scanned directly into the returned struct + a new helper type is added to support scanning null values into empty strings Fixes b/140558033 Change-Id: I101a475bb4a4a3d70075396d60473e0df0d6c8b0 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/544272 Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-03-27postgres:: add the runQuery functionJonathan Amsterdam
Provide a function to make querying simpler and less error-prone. Make one function use it. Updates b/139401182. Change-Id: Ibc430fa9b36396d3154b5c1f6b9ec86b59893ca8 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/528830 Reviewed-by: Robert Findley <rfindley@google.com>
2020-03-27all: annotate more errorsJonathan Amsterdam
A few stragglers. Fixes b/139152543 Change-Id: Ia87464cfa780e92aaf18e7a5aaa44fe572f055c5 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/528832 Reviewed-by: Robert Findley <rfindley@google.com>
2020-03-27internal/postgres: have db functions return error contextJonathan Amsterdam
Add context info to the DB.exec and DB.query methods, and remove it from call sites. Updates b/139152543. Change-Id: Id4896f60cb3ab07b358b6db62d561e7101c5091b Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/526008 Reviewed-by: Robert Findley <rfindley@google.com>
2020-03-27internal/postgres: rename XXXContext to XXXJonathan Amsterdam
Since we now control the methods of DB, we can remove the unnecessary Context suffix. You have to remember to use lower-case "exec" etc. anyway, so this change doesn't add any additional confusion. Change-Id: I57bde0dcf359fb37e57457accd170cdddfd2ef3e Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/525153 Reviewed-by: Robert Findley <rfindley@google.com>
2020-03-27internal/postgres: make sql.DB a regular fieldJonathan Amsterdam
By not embedding it, we don't expose all its methods. Add selected methods back. Fixes b/139178399 Change-Id: Ic146dc01c8531e1fd5b56e085f53a7735d44c146 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/525149 Reviewed-by: Robert Findley <rfindley@google.com>
2020-03-27discovery: add opencensus instrumentationRob Findley
Opencensus instrumentation is added via the ochttp and ocsql integrations. Additionally, debug pages are served on a separate port to view the opencensus zpage for trace spans and the prometheus page for metrics. If a GCP project is detected, this debug information is exported to Stackdriver. After testing, the dev queue is significantly scaled-up, to accomodate the newly optimized fetch. Updates b/132629518 Change-Id: I7b0aa96e46c30568d320c3820ba55d676f39590b Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/493294 Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-03-27internal/postgres: reorganize packageJulie Qiu
postgres.go is broken up into details.go and insert_version.go: * insert_version.go now contains InsertVersion and all of the unexported functions that it uses * details.go contains all functions needed to render the details view on the frontend: * GetPackage * GetLatestPackage * GetVersionForPackage * GetTaggedVersionsForPackageSeries * GetPseudoVersionsForPackageSeries * GetImports * GetImportedBy * GetLicenses * GetVersion (used for tests in internal/etl/fetch_test.go) * postgres.go only contains functions specific to postgres functionality: * Open * Transact * bulkInsert * prepareAndExec * sampleVersion, which was used for testing, is moved to test_helper.go Tests have also been moved to the corresponding test files. Change-Id: I1f137c6a3dd99b5f390e3f8c385108fe66e97cf1 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/488917 Reviewed-by: Robert Findley <rfindley@google.com>
2020-03-27content,internal: add support for standard library packagesJulie Qiu
The Go standard library packages are now supported on the discovery site, under the module "std". It is also now possible to make fetch requests to the cron for module versions using a tag or commit hash. The versions template is also updated based on the current mock. Updates b/131859847 Updates b/132875076 Updates b/126597197 Change-Id: I01fbbbff5bfec93d0ee4c5048f98ffe2da2f2a6a Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/472904 Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
2020-03-27internal: remove unused internal.VersionSource and ↵Julie Qiu
postgres.LatestProxyIndexUpdate Change-Id: Ie495f97fd4d50b921ff275aef319ca656b342557 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/479758 Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
generated by cgit v1.3 (git 2.53.0) at 2026-04-15 20:02:20 +0000