internal/bigquery: do not update schema if it has not changed
Even if the schema has not changed, updating it counts as a patch
operation that contributes to the table metadata update quota. We are
seeing dozens of thousands of error logs with exceeded quota. They are
currently not manifesting as actual errors in the pipeline (they are
just logged by bigquery services), but the official documentation
suggests they should. Either way, this CL tries to cut this at its root
by updating table only if the new schema is not the same as the old one.
Change-Id: I94b11c8fd3e12ca2c3649fc5339220f01441d28d
Reviewed-on: https://go-review.googlesource.com/c/pkgsite-metrics/+/525135
Run-TryBot: Zvonimir Pavlinovic <zpavlinovic@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
diff --git a/internal/bigquery/bigquery.go b/internal/bigquery/bigquery.go
index 51a4c1f..59d375a 100644
--- a/internal/bigquery/bigquery.go
+++ b/internal/bigquery/bigquery.go
@@ -158,6 +158,13 @@
return true, c.Table(tableID).Create(ctx, &bq.TableMetadata{Schema: schema})
}
+ if SchemaVersion(schema) == SchemaVersion(meta.Schema) {
+ // The schemas are the same, so we don't need to do anything. In fact, any
+ // update, even an idempotent one, will result in table patching that counts
+ // towards quota limits for table metadata updates.
+ return false, nil
+ }
+
_, err = c.Table(tableID).Update(ctx, bq.TableMetadataToUpdate{Schema: schema}, meta.ETag)
// There is a race condition if multiple threads of control call this function concurrently:
// The table may have changed since Metadata was called above. This error is harmless: it
