Add BigLake configuration validation for catalog creation and updates by davidchaava · Pull Request #5196 · apache/polaris

davidchaava · 2026-07-30T15:30:41Z

Summary

Adds fail-fast validation for BigLake Iceberg REST external catalog configurations during catalog creation and update.

Validation is scoped to GCP-authenticated Iceberg REST catalogs that target the official BigLake endpoint, https://biglake.googleapis.com/iceberg/v1/restcatalog. Other GCP-authenticated Iceberg REST catalogs retain their existing behavior.

Changes

Validates the official BigLake endpoint and supported remote catalog identifiers, including bl://projects/<project-number>/catalogs/<catalog-id>.
Requires a valid header.x-goog-user-project quota-project header.
Preserves compatibility with existing CLI-created catalogs that store the quota-project header in catalog properties.
Updates the CLI to place header.x-goog-user-project in connectionConfigInfo.properties for new GCP Iceberg REST catalogs.
Validates GCS storage locations and the GCS service account when credential vending is enabled.
Applies the same validation rules to catalog creation and update.
Adds unit, management-service, and CLI regression coverage.

Validation

./gradlew :polaris-runtime-service:test --tests org.apache.polaris.service.admin.BigLakeCatalogValidatorTest
pytest tests/test_catalogs_command.py::TestCatalogsCommand::test_external_catalog_gcp -q
Live Polaris to BigLake namespace-listing smoke test using GCP Application Default Credentials.

Related to #5195

davidchaava · 2026-08-05T10:39:26Z

Hi @sungwy ,
Could you please take a look at this PR when you have a chance?

We would especially appreciate your feedback from the GCP/BigLake compatibility perspective.

Thank you.

sungwy · 2026-08-11T18:29:01Z

+      Pattern.compile("^[A-Za-z0-9._-]+$");
+  private static final Pattern GCS_SERVICE_ACCOUNT_PATTERN =
+      Pattern.compile(
+          "^[a-z][a-z0-9-]{4,28}[a-z0-9]@[a-z][a-z0-9-]{4,28}[a-z0-9]\\.iam\\.gserviceaccount\\.com$");


This only accepts the user-managed SA form, so it rejects the Compute-default (…-compute@developer.gserviceaccount.com) and App-Engine-default (@appspot.gserviceaccount.com) SAs which are both valid impersonation targets. GCP's generateAccessToken accepts any SA email, and Polaris' own GcpCredentialsStorageIntegration only checks presence, so this check is stricter than both. Could we accept any valid SA email, or, if the restriction's intentional, reword the error (those defaults are Google SA emails)? WDYT

Good catch! That restriction wasn’t intentional. I’ve relaxed the check so it no longer assumes the user-managed service account format, and added coverage for Compute Engine and App Engine default service accounts. GCP remains the source of truth for whether the account exists and can be impersonated.
Thanks!

…l catalogs

sungwy

Thanks @davidchaava

I took another pass and added a few more review comments. Please let me know when you've taken another pass so I can submit the CI run

sungwy · 2026-08-17T17:33:37Z

        * polaris catalogs list
    """

+    _GCP_QUOTA_PROJECT_PROPERTY = "header.x-goog-user-project"


I think it'd make more sense to move this into constants.py. WDYT?

https://github.com/apache/polaris/blob/main/client/python/apache_polaris/cli/constants.py

Moved GCP_QUOTA_PROJECT_PROPERTY to constants.py

sungwy · 2026-08-17T17:56:53Z

+        || connectionConfig.getAuthenticationParameters().getAuthenticationType()
+            != AuthenticationParameters.AuthenticationTypeEnum.GCP) {
+      return;


I'm a bit confused here. In the federation docs, it says that only Implicit authentication type is supported against BigQuery Metastore. Could we double check this is correct?

https://polaris.apache.org/releases/1.7.0/federation/bigquery-metastore-federation/

Thanks for checking. This validator targets ICEBERG_REST catalogs using GCP authentication against the BigLake REST endpoint. BigQuery Metastore federation uses the separate BIGQUERY connection type and remains IMPLICIT-only. I added a comment clarifying this distinction.

https://github.com/akvelon/polaris/blob/5be34bdf2a14572b3e3c8b91ca6d8564cdac23e3/runtime/service/src/main/java/org/apache/polaris/service/admin/BigLakeCatalogValidator.java#L72-L74

sungwy · 2026-08-17T17:58:16Z

Is this the right place for this validator?

BigQuery Metastore was renamed by Google as Lakehouse Catalog.

I believe in Apache Polaris, code was committed while BigQuery Metastore was the name used by Google.

Shall we put this module into the right path, where BigQuery Metastore related code already exists?
https://github.com/apache/polaris/tree/main/extensions/federation/bigquery

Thanks for raising this. We plan to keep the validator in the admin service because it validates Management API create/update requests for external ICEBERG_REST catalogs targeting the BigLake REST endpoint.

The existing extensions/federation/bigquery module handles the separate optional BIGQUERY connection type for BigQuery Metastore/Lakehouse Catalog federation. Moving this validator there would couple BigLake REST validation to that optional extension and would not cover the actual catalog type used here.

The distinction is reflected in the validator code:
https://github.com/akvelon/polaris/blob/0348c24ace995f90cba72760a95457560f247f5c/runtime/service/src/main/java/org/apache/polaris/service/admin/BigLakeCatalogValidator.java#L61-L87

sungwy · 2026-08-17T18:14:07Z

+      throw new IllegalArgumentException(
+          "Invalid BigLake connectionConfigInfo.properties entry '"
+              + propertyName
+              + "': only '"
+              + QUOTA_PROJECT_HEADER
+              + "' is supported.");


Question: Is there a reason why we block other headers in addition to BLOCKED_HEADER_PROPERTIES?

I think from an API perspective, having an allowlist of headers we support is easier to support and evolve safely with. If that's the intention, I'd want to make sure we are supporting all headers we want in this PR

Yes, this is intentional. For BigLake catalog configuration, Polaris supports only header.x-goog-user-project in connectionConfigInfo.properties. Authorization and credential-vending headers are managed by Polaris and must not be overridden through catalog configuration.

I made the allowlist explicit as SUPPORTED_HEADER_PROPERTIES and added a regression test for unsupported headers:

https://github.com/akvelon/polaris/blob/0348c24ace995f90cba72760a95457560f247f5c/runtime/service/src/main/java/org/apache/polaris/service/admin/BigLakeCatalogValidator.java#L56-L60

sungwy · 2026-08-17T18:17:03Z

+      Pattern.compile("^[a-z][a-z0-9-]{4,28}[a-z0-9]$");
+  private static final Pattern GCP_PROJECT_NUMBER_PATTERN = Pattern.compile("^[1-9][0-9]{5,}$");
+  private static final Pattern BIGLAKE_URI_CATALOG_PATTERN =
+      Pattern.compile("^/[1-9][0-9]{5,}/catalogs/[^/\\s]+$");


I'm not sure if this is correct.

In the docs, it says the warehouse URL is: bl://projects/PROJECT_ID/catalogs/LAKEHOUSE_CATALOG_ID

Where PROJECT_ID is an alphanumeric ID that matches line 44-45 above:

private static final Pattern GCP_PROJECT_ID_PATTERN = Pattern.compile("^[a-z][a-z0-9-]{4,28}[a-z0-9]$");

Thank you. The validator now accepts both GCP project IDs and project numbers in the bl://projects/... warehouse URI. I also added regression coverage for the project ID form in commit 0348c24.

Updated implementation:
https://github.com/akvelon/polaris/blob/0348c24ace995f90cba72760a95457560f247f5c/runtime/service/src/main/java/org/apache/polaris/service/admin/BigLakeCatalogValidator.java#L304-L326

Test:
https://github.com/akvelon/polaris/blob/0348c24ace995f90cba72760a95457560f247f5c/runtime/service/src/test/java/org/apache/polaris/service/admin/BigLakeCatalogValidatorTest.java#L113-L129

davidchaava · 2026-08-18T19:35:10Z

Hi @sungwy,

I’ve gone through your comments and addressed them. Could you please take another look when you have a chance?

Thanks!

github-project-automation Bot added this to Basic Kanban Board Jul 30, 2026

github-project-automation Bot moved this to PRs In Progress in Basic Kanban Board Jul 30, 2026

davidchaava marked this pull request as ready for review July 31, 2026 16:44

sungwy reviewed Aug 13, 2026

View reviewed changes

davidchaava force-pushed the task/add-biglake-configuration-validation branch from 5a981c8 to b9898f5 Compare August 14, 2026 10:18

davidchaava requested a review from sungwy August 14, 2026 12:19

gkalashyan-akv and others added 5 commits August 17, 2026 12:54

Add BigLake configuration validation for catalog creation and updates

0d2b32a

BigLake validation PR improves fail-fast behavior for BigLake externa…

7d61f9a

…l catalogs

Fix BigLake CLI quota project configuration

25bf3cf

Fix BigLake validation CI checks

79835cb

Relax BigLake service account email validation

9127394

gkalashyan-akv force-pushed the task/add-biglake-configuration-validation branch from b9898f5 to 9127394 Compare August 17, 2026 09:03

spotless apply + test fix

bf5d04d

sungwy reviewed Aug 17, 2026

View reviewed changes

Comment thread client/python/apache_polaris/cli/command/catalogs.py Outdated

sungwy reviewed Aug 17, 2026

View reviewed changes

davidchaava added 2 commits August 18, 2026 12:11

Address BigLake validation review feedback

0348c24

Trigger CI

eb2679e

davidchaava force-pushed the task/add-biglake-configuration-validation branch from 8ed4eba to ece243e Compare August 18, 2026 15:31

Fix BigLake merged-config test to hit BigLakeCatalogValidator

5be34bd

davidchaava force-pushed the task/add-biglake-configuration-validation branch from ece243e to 5be34bd Compare August 18, 2026 15:35

davidchaava requested a review from sungwy August 18, 2026 19:35

Uh oh!

Conversation

davidchaava commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Validation

Uh oh!

davidchaava commented Aug 5, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sungwy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidchaava commented Aug 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

davidchaava commented Jul 30, 2026 •

edited

Loading