Repository navigation
Conversation
A cache miss records the object etag and then reads the bytes. The GET now requires that etag, including when `s3_validate_etag_on_read` or `azure_validate_etag_on_read` is off, so an in-place overwrite is not stored under the old key. A weak etag is not used as a cache key. Co-authored-by: Cursor <cursoragent@cursor.com>
Omitting `expected_cardinality` skipped `PUFFIN_DV_MAX_MATERIALIZED_POSITIONS`. `deserializeRoaringPositionBitmap` now rejects a bitmap above that ceiling before expanding it into positions. Co-authored-by: Cursor <cursoragent@cursor.com>
`PuffinFilesCache` hid non-virtual `clear`, `setMaxSizeInBytes`, and `setMaxCount`, so a `CacheBase` pointer left the footer memo in place. Those methods are now virtual. Co-authored-by: Cursor <cursoragent@cursor.com>
A DAG that only adds columns has no inputs. The cluster protocol treated that as a missing `schema_transform` and dropped it. An empty DAG remains the sentinel for no transform. Co-authored-by: Cursor <cursoragent@cursor.com>
Bucket splitting copies through `clone`. `ObjectInfoInArchive` and `ObjectStorageQueueObjectInfo` fell back to the base copy and dropped the archive identity and `file_metadata`. Co-authored-by: Cursor <cursoragent@cursor.com>
A mismatch used to retry without the pin and could apply a later object under the listed offset, length, and cardinality. `getOrSetDeletionVector` already skips the insert when the loader throws. Co-authored-by: Cursor <cursoragent@cursor.com>
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Codex Review: Something went wrong. Try again later by commenting “@codex review”. ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review |
|
Codex Review: Something went wrong. Try again later by commenting “@codex review”. ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
The debug build treats a missing prototype as an error, so the cluster read-task helper cannot stay at file scope. Co-authored-by: Cursor <cursoragent@cursor.com>
|
@codex review |
|
Codex Review: Something went wrong. Try again later by commenting “@codex review”. ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review |
|
Codex Review: Something went wrong. Try again later by commenting “@codex review”. ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
CI triage for afa617e5 failing checks, 0 caused by this PR. Every one of them fails the same way on other recent PRs into
Details1. This is an access-control problem with the test's named collection or user config. Your diff doesn't touch it, and #2704 fails with exactly the same message. 2. This comes from the S3 write path. The PR only adds a unit test for 3. #2704 fails the same way. 4. Regression oauth / swarms: both runs fail on exactly the same scenarios as the runs for #2700 (certificate backport) and #2680, which don't touch this code. Swarms deserved a closer look because the PR changes About the RecommendationThe PR is fine as far as CI can tell. Merging is safe from a CI point of view, or rerun the failed jobs if you want a cleaner signal. Two of the base-branch failures should get their own tickets: the Evidence: praktika |
|
@codex review |
|
Codex Review: Didn't find any major issues. Already looking forward to the next diff. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
Some of my tests are in progress, in the meantime here is the review Audit review: PR #2715 — Fixes for Iceberg Puffin Delete VectorsPR: #2715 (head Confirmed defects1. Medium — Iceberg tables with more than 100M deleted rows in one data file become unreadableDescription The PR applies the fixed Iceberg always passes the declared The limit is a // PuffinBlockInputFormat.cpp, new in this PR
if (expected_cardinality.has_value() && *expected_cardinality > PUFFIN_DV_MAX_MATERIALIZED_POSITIONS)
throw Exception(ErrorCodes::BAD_ARGUMENTS,
"Deletion vector cardinality {} exceeds materialization limit {}", ...);Steps to reproduce
|
mkmkme
left a comment
There was a problem hiding this comment.
🤖 [Posted by an LLM agent on behalf of @mkmkme]
I found no correctness problems in the changes. The main gap is evidence: nothing tests that a deletion vector changed during a cache miss fails the query and is never cached.
CreateReadBufferCanPinListedEtagWhenValidationIsOff only shows that createReadBuffer sends If-Match when asked. No test covers the wiring from loadDeletionVector, and the Local/Web check in throwIfListedEtagDrifted has no test at all. One gtest on LocalObjectStorage would cover both: record the etag, overwrite the file, call loadDeletionVector with the cache on, then expect an exception and an empty cache and footer memo.
About the 100M-row concern raised earlier: the base branch already rejects it. readDeletionVectorFromPuffin calls checkDeletionVectorBlobReadLimits(length, expected_cardinality), and deserializeDeletionVectorV1Blob checks the same limit, both on antalya-26.8. This PR only adds a duplicate check. A setting to raise the limit could be a separate issue.
Changelog entry: it reads like internals (CacheBase, ObjectInfo::clone, footer memo). Suggestion:
Fixed a race where, with the Puffin files cache enabled, an Iceberg deletion vector overwritten in place between listing and reading could be cached and applied under the old identity even when
s3_validate_etag_on_read/azure_validate_etag_on_readwas off; such reads now fail. Also fixed*Clusterreads of Iceberg tables dropping a schema evolution step that only adds columns.
The docs entry should also say that with the cache on, s3_validate_etag_on_read=0 no longer turns off the check for deletion-vector reads, and that use_puffin_files_cache=0 is the way out. It should also drop HDFS: HDFS etags are never strong, so HDFS never reaches this code.
| if (validate_etag_on_read && object_info.metadata.has_value()) | ||
| /// `pin_to_metadata_etag` is independent of the backend setting: a caller that will cache the | ||
| /// bytes under this etag cannot opt out of the pin. | ||
| if (object_info.metadata.has_value() && (validate_etag_on_read || pin_to_metadata_etag)) |
There was a problem hiding this comment.
🤖 [Posted by an LLM agent on behalf of @mkmkme]
With this, s3_validate_etag_on_read=0 / azure_validate_etag_on_read=0 no longer turn off the check for Puffin cache loads. That's fine, but the error users then get still says "set s3_validate_etag_on_read=0 to disable this check", which won't help them. Can the pinned path point to use_puffin_files_cache=0 instead, or at least can the docs say so?
There was a problem hiding this comment.
Not sure if any docs changes are needed. But I'd double-check if that's intended
| return; | ||
|
|
||
| throw Exception( | ||
| ErrorCodes::BAD_ARGUMENTS, |
There was a problem hiding this comment.
🤖 [Posted by an LLM agent on behalf of @mkmkme]
BAD_ARGUMENTS doesn't fit an object that changed during the read. Callers and retry logic can't tell it apart from a malformed deletion vector. Could this use the same code as the S3/Azure path, or another code that means "changed during read"?
There was a problem hiding this comment.
It's a different. Here is a BAD_ARGUMENT, because changing Etag means that object on S3 was replaced. But Deletion Vector (as well as other Iceberg objects) must be immutable, when need to changer something, new different object must be written as new generation in metadata.
So changing Etag can be result only as a result of anomaly third-party activity on storage. Table is broken in this case.
| const UInt64 position = positionFromKeyAndSubPosition(static_cast<UInt32>(key), sub_position); | ||
| if (position > static_cast<UInt64>(DELETION_VECTOR_MAX_POSITION)) | ||
| throw Exception(ErrorCodes::BAD_ARGUMENTS, "Deletion vector position {} is out of supported range", position); | ||
| if (positions.size() >= PUFFIN_DV_MAX_MATERIALIZED_POSITIONS) |
There was a problem hiding this comment.
🤖 [Posted by an LLM agent on behalf of @mkmkme]
The running-cardinality check above already guarantees this, since the loop runs exactly bitmap.cardinality() times. As written, it adds a branch per position, up to 100M of them. I'd drop it.
| auto transform = std::make_shared<ActionsDAG>(ActionsDAG::deserialize(in, registry, Context::getGlobalContextInstance(), 0)); | ||
|
|
||
| if (!path.empty() && !transform->getInputs().empty()) | ||
| if (!path.empty() && schemaTransformCarriesColumns(*transform)) |
There was a problem hiding this comment.
🤖 [Posted by an LLM agent on behalf of @mkmkme]
An older worker on the same WITH_DATA_LAKE_METADATA protocol still drops an output-only DAG here, because it checks getInputs().empty(). The serializer only rejects protocols below that, so a mixed cluster keeps returning default values instead of initial-default for added columns until every node is upgraded. Should this bump the protocol version? If not, can we document the upgrade caveat? An integration test that reads only an added column through icebergS3Cluster would also be good, since this fix has only unit tests now.
There was a problem hiding this comment.
I added a small note in docs about it.
|
|
||
| /// Bucket splitting copies through `clone`. The base implementation slices to `ObjectInfo` | ||
| /// and drops the archive path, reader, and file info. | ||
| std::shared_ptr<ObjectInfo> clone() const override { return std::make_shared<ObjectInfoInArchive>(*this); } |
There was a problem hiding this comment.
🤖 [Posted by an LLM agent on behalf of @mkmkme]
The only caller of clone is ObjectIteratorSplitByBuckets::next. For an archive entry it calls createReadBuffer(last_object_info->relative_path_with_metadata, ...) first, and ObjectInfoInArchive leaves relative_path empty, so I'd expect that read to fail before clone runs. Queue sources don't use that iterator at all. Keeping the overrides is fine, but can the changelog claim about bucket splitting be backed by a query-level test (s3Cluster over archive.zip :: *.parquet with cluster_table_function_split_granularity='bucket') or dropped?
There was a problem hiding this comment.
I think, it can be left as is.
This was not introduced in current PR. |
- Document in `use_puffin_files_cache` that a cached deletion-vector read is pinned to the etag from the object listing, even when `s3_validate_etag_on_read` or `azure_validate_etag_on_read` is off. A changed object fails the query; disabling those settings does not avoid it, and `use_puffin_files_cache` set to 0 does. - Remove the per-position materialization check in `deserializeRoaringPositionBitmap`. The running-cardinality check before the loop already rejects any bitmap that would exceed `PUFFIN_DV_MAX_MATERIALIZED_POSITIONS`, and the loop emits exactly `cardinality()` positions, so the removed check could not fire. - Document in `docs/en/antalya/protocol.md` that workers built before the output-only `schema_transform` fix drop a transform with no inputs, and that the cluster-protocol version is not bumped for this, so finish the upgrade before running distributed data-lake reads that need such a transform. Related: #2715 Co-authored-by: Cursor <cursoragent@cursor.com>
|
Add some minor changes like fixed descriptions and documentations. |
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Iceberg Puffin deletion-vector cache reads are pinned to the listed etag, including when
s3_validate_etag_on_readorazure_validate_etag_on_readis off. A changed object fails the query instead of being applied under the listed offset, length, and cardinality. Materialization stays withinPUFFIN_DV_MAX_MATERIALIZED_POSITIONSwhen cardinality is omitted.PuffinFilesCacheclear,setMaxSizeInBytes, andsetMaxCountupdate the footer memo throughCacheBase. Cluster reads keep an output-onlyschema_transform. Archive and queue objects keep their type inObjectInfo::clone.Documentation entry for user-facing changes
No new setting or syntax. With the Puffin files cache enabled, a deletion-vector read is tied to the etag from the listing. S3 and Azure send that etag as
If-Matcheven when etag validation is disabled. Local, HDFS, and Web compare the etag after the read. A mismatch raises an error and is not stored in the cache. An empty or weak etag is not used as a cache key. A deletion vector is rejected when it would materialize more than 100000000 positions, whether or not a cardinality was declared.SYSTEM DROP ...cache clear and a runtime cache resize drop memoized Puffin footers together with cached deletion vectors. A distributed read keeps a schema evolution transform that only adds columns. Bucket splitting no longer drops the archive path or the queue file metadata.CI/CD Options
Exclude tests:
Regression jobs to run:
Title: Pin Puffin deletion-vector reads and preserve cluster schema and object identity
Motivation
A Puffin deletion-vector cache miss recorded the object etag and then read the bytes without requiring that etag when
s3_validate_etag_on_readorazure_validate_etag_on_readwas off. An in-place overwrite could be stored under the old key. Local, HDFS, and Web ignoreStoredObject::etag, so the same race existed there even with validation enabled. A first version of the pin retried the read without the etag, which could apply the new object under the listed offset, length, and cardinality.Two related limits were incomplete. Omitting
expected_cardinalityskippedPUFFIN_DV_MAX_MATERIALIZED_POSITIONSand let a compact roaring bitmap expand without a ceiling.PuffinFilesCache::clear,setMaxSizeInBytes, andsetMaxCounthid the non-virtualCacheBasemethods, so a base pointer left the footer memo resident.On the cluster read path, a
schema_transformthat only adds columns has outputs and no inputs. Deserialize treated that the same as the empty DAG used when there is no transform, and the old protocol did not reject it. Bucket splitting copies each object throughObjectInfo::clone.IcebergDataObjectInfoalready copied itself.ObjectInfoInArchiveandObjectStorageQueueObjectInfofell back to the base copy and lost the archive path, reader, and file info, or the queuefile_metadata.Approach
The cache-miss load passes the listed etag into
createReadBuffereven when the validation settings are off. S3 and Azure send it asIf-Match. Local, HDFS, and Web compare the etag after the bytes are read. A mismatch throws.getOrSetDeletionVectordoes not insert when the loader throws, and the query fails instead of reading the current object under the old identity. An empty or weak etag is not a cache key.deserializeRoaringPositionBitmaprejects a bitmap whose cardinality would passPUFFIN_DV_MAX_MATERIALIZED_POSITIONSbefore expanding it, including when no cardinality was declared. A fully empty DAG is still the on-wire stand-in for a missingschema_transform. Any DAG with inputs or outputs is kept, and an old worker protocol rejects it.CacheBase::clear,setMaxCount, andsetMaxSizeInBytesare virtual, andPuffinFilesCacheoverrides them so the footer memo follows the same limits.ObjectInfoInArchiveandObjectStorageQueueObjectInfoimplementclone.Tests
CreateReadBufferCanPinListedEtagWhenValidationIsOffchecks thatIf-Matchis sent only when the pin is requested.PuffinDeletionVectorCardinality.RejectsOmittedCardinalityAboveMaterializationLimitbuilds a run-length bitmap of 100000001 positions and expects a rejection before expansion.PuffinFooterMemo.BasePointerUpdatesFooterMemoclears and shrinks the footer memo through aCacheBasereference.ClusterFunctionReadTaskResponse.RoundTripsOutputOnlySchemaTransformOnSupportedProtocolkeeps an added-column transform, and the old protocol rejects it.ObjectInfoClonechecks that an archive entry and a queue entry stay their own types afterclone.