Skip to content

feat(bigquery): add bigquery_upgrade_manifest_biglake_table sample to… - #14748

Open
iennae wants to merge 2 commits into
mainfrom
feat/bq-upgrade-manifest-biglake-table
Open

iennae wants to merge 2 commits into
mainfrom
feat/bq-upgrade-manifest-biglake-table

Conversation

@iennae

@iennae iennae commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

… upgrade a manifest external table to BigLake with automatic metadata caching

Problem:
The BigQuery documentation (https://cloud.google.com/bigquery/docs/query-open-table-format-using-manifest-files) needs a standalone, tested Python code sample demonstrating how to upgrade a manifest external table to BigLake with automatic metadata caching (region tag bigquery_upgrade_manifest_biglake_table).

Solution:

  • Add upgrade_manifest_biglake_table(project_id: str, dataset_id: str, table_id: str, connection_id: str, manifest_uri: str) -> None in bigquery/biglake/bigquery_upgrade_manifest_biglake_table.py (Upgrade a manifest-backed external table to a BigLake table with AUTOMATIC metadata caching and connection binding.)
  • Add unit tests (bigquery/biglake/bigquery_upgrade_manifest_biglake_table_test.py) covering both the success path and API error handling (GoogleAPICallError)

Description

Fixes #

Checklist

Testing

  • I have tested this change on a live environment and verified it works as intended.

Compliance & Style


Post-Approval Actions

  • Please merge this PR for me once it is approved

@iennae
iennae requested review from a team and chalmerlowe as code owners November 2, 2026 23:27
@snippet-bot

snippet-bot Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Here is the summary of changes.

You are about to add 1 region tag.

This comment is generated by snippet-bot.
If you find problems with this result, please file an issue at:
https://github.com/googleapis/repo-automation-bots/issues.
To update this comment, add snippet-bot:force-run label or use the checkbox below:

  • Refresh this comment

@product-auto-label product-auto-label Bot added api: bigquery Issues related to the BigQuery API. samples Issues that are directly related to samples. labels Nov 2, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new Python sample and associated tests for upgrading a manifest external table to a BigLake table with metadata caching in BigQuery. Feedback on the implementation suggests making the verification query schema-agnostic by using SELECT * instead of hardcoded column names. Additionally, the test configuration in noxfile_config.py should be updated to include Python 3.10 in the test suite, as it is the oldest supported version and should not be ignored.

Comment thread bigquery/biglake/noxfile_config.py Resolved
@iennae
iennae force-pushed the feat/bq-upgrade-manifest-biglake-table branch 3 times, most recently from 7651bd6 to 1139772 Compare November 3, 2026 23:32
max_staleness = INTERVAL 1 HOUR,
metadata_cache_mode = 'AUTOMATIC'
);
SELECT

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@iennae

Thinking out loud

My understanding is that there is a misconception about multi-part SQL statements.

You prolly know this, but in case others come behind us I will provide some context. Some folks believe that multi-part SQL statements don't return results the way we might expect. For example if I included some form of Data Definition Language (DDL) statement like DROP TABLE after the SELECT, we would not get back the results of the SELECT, we would get back an empty list.

This is because:

  • Multi-statement queries are execute as a statement_type = "SCRIPT"
  • In the ScriptOptions object, the key_result_statement property is defined, by default using the enum: KeyResultStatementKind.LAST

For this reason, I wonder if there is merit to separating the DDL portion (CREATE OR REPLACE EXTERNAL TABLE) of this example from the Data Query Language (DQL) portion (SELECT) as part of demonstrating good hygiene.

Something like this so they can be called separately:

# 1. Create or replace the BigLake external table
client.query(ddl_sql).result()

# 2. Query the newly created external table
query_job = client.query(select_sql)
rows = list(query_job.result())

Pros: two calls makes it easier to do troubleshooting if one of the major steps fails. Searching audit logs, job history, etc should be easier especially since customers won't have to drill down into parent jobs to find child jobs.
Cons: There is a small performance hit when a client.query() provisions and registers each top-level job.

LIMIT 10;
"""

try:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

COMMENT

I looked at about seven samples. The ones I saw all appear to have this try/except/except pattern.

I am worried that this degrades the signal-to-noise ratio in our reference samples, documentation, and quickstarts.

I know the Samples Style Guide says to include Error Handling.

The catch blocks take up roughly 20 lines of repetitive boilerplate code that prints to stderr and exits early. I wonder if this could be handled differently.

def test_upgrade_manifest_biglake_table_success(
capsys: pytest.CaptureFixture[str],
) -> None:
with mock.patch("google.cloud.bigquery.Client") as mock_client_cls:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just thinking out loud

In PRs #14740 and #14748, string-target patching is used (e.g. mock.patch("google.cloud.bigquery.Client")). If the sample refactors how Client is imported, string targets can silently break or patch the wrong namespace. (I understand that it is not likely that these samples will be wildly refactored - hence just thinking out loud. I don't think there is a huge ROI in inducing review churn over this.)

To keep target-safety while retaining mock's auto-generated mock instances and call assertions, there may be utility in using mock.patch.object(module.bigquery, "Client") instead of raw string paths.

Alternatively, if we want to match the other PRs, you could use monkeypatch.setattr with a MagicMock.

print(f"Job ID: {query_job.job_id}")
print(f"Manifest URI: {manifest_uri}")
print(f"Statement Type: {query_job.statement_type}")
print("Metadata Cache Mode: AUTOMATIC")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there's no point in printing the metadata cache mode here, because it's a hardcoded value and the response QueryObject doesnt include wether the table created is using metadata cache, it would make more sense to use a get_table() call to inspect the newly created table and inspect the resulted Table object TableObject.external_data_configuration.metadata_cache_mode and print it's value, then it could be asserted during testing.

def test_upgrade_manifest_biglake_table_success(
capsys: pytest.CaptureFixture[str],
) -> None:
with mock.patch("google.cloud.bigquery.Client") as mock_client_cls:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It'd probably be better to patch bigquery_upgrade_manifest_biglage_table.client instance, rather than patch the library.

… upgrade a manifest external table to BigLake with automatic metadata caching

Problem:
The BigQuery documentation (https://cloud.google.com/bigquery/docs/query-open-table-format-using-manifest-files)
needs a standalone, tested Python code sample demonstrating how to
upgrade a manifest external table to BigLake with automatic metadata caching (region tag `bigquery_upgrade_manifest_biglake_table`).

Solution:
- Add `upgrade_manifest_biglake_table(project_id: str, dataset_id: str, table_id: str, connection_id: str, manifest_uri: str) -> None` in `bigquery/biglake/bigquery_upgrade_manifest_biglake_table.py`
  (Upgrade a manifest-backed external table to a BigLake table with AUTOMATIC metadata caching and connection binding.)
- Add unit tests (`bigquery/biglake/bigquery_upgrade_manifest_biglake_table_test.py`) covering both the success
  path and API error handling (`GoogleAPICallError`)
@iennae
iennae force-pushed the feat/bq-upgrade-manifest-biglake-table branch from 1139772 to 42e76b0 Compare November 17, 2026 04:57
…fest_biglake_table

Problem:
Review feedback on this sample pointed out two issues:
- It printed "Metadata Cache Mode: AUTOMATIC" as a hardcoded string, so
  neither the sample nor its test verified that BigQuery applied the
  setting.
- The verification query selected id, label and score, which only exist in
  our test data, so the sample failed against any other manifest table.

Solution:
- After the CREATE statement, read the table back with client.get_table()
  and print the metadataCacheMode from its external data configuration and
  the table's max_staleness. ExternalConfig has no metadata_cache_mode
  property in google-cloud-bigquery 3.47.0, so the value is read from the
  public to_api_repr() output.
- Use SELECT * ... LIMIT 10 so the query works with any schema.
- The test mocks get_table(), asserts it is called with the table ID, and
  checks the reported cache mode and max staleness.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: bigquery Issues related to the BigQuery API. samples Issues that are directly related to samples.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants