internal/etl: add an action to populate completion indexes

We're going to compute auto-completions using redis sorted sets in the
'HA' redis instance (because we don't want them to be evicted by
caching). This data need not be very fresh, and so can be computed by
the ETL every so often, perhaps once a day.

This CL wires the HA redis instance into the ETL, and adds an action to
update these Redis sorted sets. Doing this takes 1-2 minutes and
consumes quite a lot of memory, so there are some optimizations used:
 + the underlying *sql.DB is used for streaming search documents, rather
   than a method on postgres.DB. There is explanation for this in the
   code, and additionally I don't think this is so bad as long as it is
   only done in the ETL.
 + Completions data is loaded into temporary Redis keys and then renamed
   to overwrite the keys that will be used by the frontend, in order to
   minimize disruption while the completion data is being updated.

Updates b/143370178

Change-Id: I48caf77eb26d8ea7b60c2e117481bfa98ef089ca
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/602062
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>

1 files changed, 6 insertions, 0 deletions

diff --git a/internal/postgres/postgres.go b/internal/postgres/postgres.go
index d5a6258d..3a958c8c 100644
--- a/internal/postgres/postgres.go
+++ b/internal/postgres/postgres.go
@@ -25,6 +25,12 @@ type DB struct {
 	db *sql.DB
 }
 
+// GetSQLDB returns the underlying SQL database for the postgres instance. This
+// allows the ETL to perform streaming operations on the database.
+func (db *DB) GetSQLDB() *sql.DB {
+	return db.db
+}
+
 func (db *DB) exec(ctx context.Context, query string, args ...interface{}) (res sql.Result, err error) {
 	defer logQuery(query, args)(&err)