Repository navigation
Conversation
…e HEAD
Opening an object went through a HEAD (getObjectMetadata) on every read to learn
its size and ETag, and the result was not cached process-wide, so a scan
re-HEADed every candidate file on every query (traced: ~1 S3 HeadObject per
candidate file, e.g. ~1900 HEADs to read ~550 files). The HEAD is also
mandatory-before-cache-lookup, because the filesystem / page / parquet-metadata
caches key on (path, etag) and the etag comes from the HEAD.
Introduce a process-wide identity cache and a single-request metadata fetch:
* `S3::getObjectIdentity` issues one `GetObjectAttributes` request (size + ETag +
multipart part sizes) and falls back to a plain HEAD if the API is
unsupported, denied, or errors - safe against S3-compatible stores. The ETag
is quote-normalized so the GetObjectAttributes and HEAD paths yield the same
identity (they are used as cache keys). Adds the `GetObjectAttributes` wrapper
to `S3::Client` (mirrors `GetObjectTagging`) and the request alias.
* `ObjectStorageIdentityCache` (experimental singleton) maps path ->
{etag, size, part_offsets}. `S3ObjectStorage::getObjectMetadata` consults it on
the identity path (no tags requested): a hit skips the request entirely; a miss
fetches via `getObjectIdentity` and populates it. The tags path and the
credentials-refresh retry are unchanged.
* `ObjectMetadata` gains `part_offsets` (cumulative multipart part offsets),
derived from the per-part sizes; consumers use it to align reads.
* `IcebergDataObjectInfo` no longer pre-populates object metadata from the
manifest, so Iceberg data files also go through this single identity path and
get the real ETag and multipart layout instead of a synthesized identity.
New ProfileEvents: `S3GetObjectAttributes`, `ObjectStorageIdentityCacheHits`,
`ObjectStorageIdentityCacheMisses`.
Note: the identity path returns no user-metadata attributes (GetObjectAttributes
does not provide them); callers that need them still use the tags path. The
cache is an experimental singleton (not Context-managed / not SYSTEM DROP-able)
and should be promoted before productionization.
Signed-off-by: UnamedRus <dtitmoav@gmail.com>
(cherry picked from commit e1b807a)
Replace the process-wide map with a size-bounded CacheBase created from `object_storage_identity_cache_policy`, `_size` and `_size_ratio`, resized on config reload, and cleared by `SYSTEM DROP OBJECT STORAGE IDENTITY CACHE`. S3 reads look it up from the global context and fall back to an uncached request when no context exists. Co-authored-by: Cursor <cursoragent@cursor.com>
… IDENTITY CACHE The new `SYSTEM DROP OBJECT STORAGE IDENTITY CACHE` privilege (added with the Context-managed identity cache) adds a row to `SHOW PRIVILEGES`, so the stateless test reference needs the corresponding line. CI report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2181&sha=5b16210f9e7dbf582ab7e23983337a9244f2b9fe&name_0=PR&name_1=Fast%20test Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: UnamedRus <dtitmoav@gmail.com>
…Attributes behind a read setting Reading object metadata on a lake scan (`StorageObjectStorageSource`) needs the object size and etag (the etag keys the page / filesystem / Parquet-metadata caches). The identity fast path fetched these via `GetObjectAttributes`, which also returns the multipart part layout used for part-aligned reads - but that request is heavier than a plain `HEAD` and is issued per file, so a cold, cache-empty scan paid one `GetObjectAttributes` per object (measured ~2x slower cold on a wide-Iceberg benchmark) for part offsets it usually does not use. Default `S3ObjectStorage::getObjectMetadata` to a plain `HEAD` (size + etag, no part offsets), and add a read setting `object_storage_identity_cache_fetch_part_offsets` (default off) that opts back into `GetObjectAttributes` when part-aligned reads want the layout. A new 3-argument `IObjectStorage::getObjectMetadata` overload carries the flag; its default implementation ignores it, so non-S3 storages and the many callers that do not need part offsets are unchanged. The etag is still returned on both paths, so the object-storage caches keep working. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: UnamedRus <dtitmoav@gmail.com>
The new `ObjectMetadata::part_offsets` field has no default member initializer, so designated initializers of `ObjectMetadata` that omit it (for example in `AzureObjectStorage`) fail with `-Werror,-Wmissing-designated-field-initializers`. Give the field an explicit empty default instead of touching every call site. Co-authored-by: Cursor <cursoragent@cursor.com>
…setting Add the stateless test `05484_object_storage_identity_cache` for the object storage identity cache. It covers a cold read that misses, a warm read that hits without `HeadObject` or `GetObjectAttributes`, the `SYSTEM DROP OBJECT STORAGE IDENTITY CACHE` and `SYSTEM CLEAR OBJECT STORAGE IDENTITY CACHE` commands, and the `object_storage_identity_cache_fetch_part_offsets` setting. The setting was only read in `createReadBuffer`, which runs after the object metadata is already filled, so it never chose `GetObjectAttributes` on the SQL read path. `KeysIterator` now receives the setting and passes it to `getObjectMetadata`, and the empty-metadata branch does the same. Co-authored-by: Cursor <cursoragent@cursor.com>
The identity cache key was the bucket and the object path. Two S3 endpoints can serve buckets with the same name, so a hit from one endpoint answered for the other with its size and ETag. Include the endpoint and the client region in the key. Fields are separated by NUL so that a slash in one field cannot make two keys equal. The new test `05485_object_storage_identity_cache_endpoint_key` reads the same object through `localhost` and `127.0.0.1` and checks that each endpoint misses on its first read and hits on its second. Co-authored-by: Cursor <cursoragent@cursor.com>
The identity cache was keyed by endpoint, region, bucket and path, but nothing removed an entry when this server replaced or deleted the object. A later read then used the old size as the read bound and the old ETag as If-Match, so it returned truncated data or failed the read until eviction or SYSTEM DROP. - A successful single-part or multipart upload stores the new size and ETag, taken from the upload response, in the cache. If the post-upload size check fails, the entry is dropped instead. - A successful copy, a delete, a batch delete (for each key that was removed or already absent), and a conditional delete that reports a token mismatch or not-found drop the entry. - Each key has a generation number. A metadata fetch stores its result only if the generation is unchanged since before its request, so a HEAD that was in flight during a write cannot put the old identity back. - Entries expire after `object_storage_identity_cache_ttl` seconds (default 3600, zero means no expiry). This bounds the staleness when another writer replaces an object. The new test `05486_object_storage_identity_cache_overwrite` inserts a one-row object, overwrites it with three longer rows without clearing the cache, and checks that the next read is a cache hit that returns all three rows. s3_validate_etag_on_read is disabled, so a stale cached size would truncate the read. Co-authored-by: Cursor <cursoragent@cursor.com>
`getObjectIdentity` stripped the quotes from the ETag it read with GetObjectAttributes, and its HEAD fallback did the same. `getObjectInfo` (HEAD) and PutObject keep the quotes. The identity cache stores this value and `ReadBufferFromS3` sends it as `If-Match`, so the two spellings of one object are different bytes. With `object_storage_identity_cache_fetch_part_offsets = 1` the cache held an unquoted ETag, and with `s3_validate_etag_on_read` on the next read of that object could fail although the object did not change. Both paths now store the quoted form. `quoteETag` adds the quotes when they are missing and leaves an already quoted or weak (`W/"..."`) value unchanged. The new test `05487_object_storage_identity_cache_etag_spelling` keeps `s3_validate_etag_on_read` on. It checks that the ETag read through HEAD and through GetObjectAttributes is the same quoted value, and that the following read hits the cache and returns that value. Co-authored-by: Cursor <cursoragent@cursor.com>
`Context::updateObjectStorageIdentityCacheConfiguration` returned at once when the identity cache pointer was unset. Startup leaves the pointer unset when `object_storage_identity_cache_size` is `0`, so a reload that set a non-zero size did nothing and every read stayed uncached until the next process start. A reload now creates the cache when the new size is non-zero, using the reloaded policy, size ratio, and TTL. The size is still capped by the same RAM limit as at startup. A reload that keeps the size at `0` still leaves the cache disabled. The setting description and the `SYSTEM CLEAR|DROP OBJECT STORAGE IDENTITY CACHE` page say that `SYSTEM RELOAD CONFIG` turns the cache on. Co-authored-by: Cursor <cursoragent@cursor.com>
The overwrite fix added a per-key generation map, a TTL with a timestamp on every entry, and a region in the cache key. None of these is needed for the overwrite case: a write, copy, or delete by this server removes the entry, and the next read requests the metadata again. Keeping them made the cache, the S3 write path, and the settings larger than the fix requires. - ObjectStorageIdentityCache is a plain CacheBase again: tryGet and set. The generation map, setIfEpoch, epochOf, replace, invalidate, and the TTL with its timestamp are removed. - The cache key is the endpoint, the bucket, and the path. The region is removed. - A successful upload calls the upload callback without arguments and the callback removes the entry. The size and ETag are no longer copied from the upload response. - A copy, a delete, a batch delete, and a conditional delete remove the entry through invalidateCachedIdentity. - The object_storage_identity_cache_ttl server setting is removed. - Context::setObjectStorageIdentityCache no longer takes a TTL. The reload path that creates the cache when startup left it disabled, and the ETag spelling fix, are unchanged. The test 05486 expects the read after an overwrite to miss and return all three new rows, and the next read to hit. Co-authored-by: Cursor <cursoragent@cursor.com>
…rite A metadata read takes its HEAD before it stores the result. If this server overwrote, copied or deleted the object in that window, the read stored the size and ETag it fetched before the overwrite. The entry was dropped by the overwrite, so nothing removed the stale value again. Each identity-cache key now has a generation. An overwrite, copy or delete bumps it with `invalidate`, which also removes the entry. A metadata read captures the generation before its request and stores the result with `setIfEpoch`, which skips the store when the generation has changed. `clear` drops the generations together with the entries. Add the pause failpoint `object_storage_identity_cache_pause_before_store` between the fetch and the store, and test 05488 uses it to overwrite the object while a read is paused. The next read must miss and return the new rows. Co-authored-by: Cursor <cursoragent@cursor.com>
… cleared `SYSTEM CLEAR OBJECT STORAGE IDENTITY CACHE` removed the per-key generations along with the entries. A key that had been overwritten therefore looked untouched again, and a metadata read that started before the clear and resumed after it could store the size and ETag it had fetched before the overwrite. `clear` now bumps a cache-wide generation and does not reset it. The fetch captures it together with the per-key generation as the new `ObjectStorageIdentityEpoch`, and `setIfEpoch` stores only when both still match. Test 05489 pauses a read after its HEAD, overwrites the object, clears the cache, and resumes the read. The next read must miss and return the new rows. Co-authored-by: Cursor <cursoragent@cursor.com>
Each `invalidate` of a key left a per-key generation record in `epoch_by_key`, and nothing removed it except `clear`. The map grew by one entry for every overwritten, copied or deleted object for the life of the process, outside the cache's size limit. The map is now accounted in bytes and limited to the cache's configured size. When a new record would exceed that limit, the map is dropped and the cache-wide generation is bumped, so every fetch already in flight fails its `setIfEpoch` check and does not store. One record is always kept, so the map never refuses a key even when that single record is larger than the limit. Co-authored-by: Cursor <cursoragent@cursor.com>
The per-key generation map added to keep stale identities out of the identity cache was large and needed its own byte accounting and cap. A single cache-wide epoch covers the same cases with much less code. `invalidate` (overwrite, copy or delete) and `clear` advance the epoch and `invalidate` also removes the entry. A metadata read captures the epoch before its request and stores the result with `setIfEpoch`, which skips the store when the epoch has changed. The epoch is never reset, so a fetch in flight cannot match a later epoch by accident. The trade-off is that any write, copy or delete of any object makes every metadata fetch in flight skip its store. The fetch still returns its result to the caller, and the next read of the key fetches again. Tests 05488 and 05489 keep their scenarios. Only their comments change. Co-authored-by: Cursor <cursoragent@cursor.com>
…ache A read that found the object replaced kept the cached size and ETag. The next read was pinned to that ETag and failed with `S3_OBJECT_CHANGED_DURING_READ` again, and every retry hit the same entry. Nothing in the read path removed it, because a replace made outside this server does not pass through `S3ObjectStorage::writeObject`. `ReadBufferFromS3` now calls a callback when a GET fails `If-Match` or, for an unversioned read, returns a not-found error. `S3ObjectStorage::readObject` passes a callback that invalidates the key, which also advances the cache epoch so a fetch already in flight cannot store the old identity back. `ObjectStorageIdentityCache` also orders stores per key. `beginFetch` hands out a sequence number, and `setIfEpoch` skips a store when a later fetch of the same key has already stored. An older observation therefore cannot overwrite a newer one. Test 05490 replaces the object with a plain PUT, expects the read to fail with `S3_OBJECT_CHANGED_DURING_READ`, and checks that the retry misses, sends a HEAD, and returns the new rows. Co-authored-by: Cursor <cursoragent@cursor.com>
The metadata fetch for a listed `url()` file called the string overload of `getObjectMetadata`. That overload builds a `RelativePathWithMetadata` without `read_source_index`, so `WebObjectStorage::buildURLs` probed every shard. With several shards that expose the same relative path, the size and ETag could come from a different shard than the one that listed the file, and the read bound could be wrong. `IObjectStorage` gains a `getObjectMetadata` overload that takes the whole `RelativePathWithMetadata` and the part-offset flag. Its default forwards to the object overload, so `WebObjectStorage` keeps the index. `S3ObjectStorage` overrides it to pass the flag to its existing fetch. `createReader` and `createReadBuffer` now call this overload. Test 05491 serves the same file from two local HTTP shards with different sizes and checks that the rows from both shards are returned in full. Co-authored-by: Cursor <cursoragent@cursor.com>
A cache hit in `S3ObjectStorage::getObjectMetadata` returned the object with `is_last_modified_known = false`, so the `_time` virtual column became NULL on every read after the first. The identity cache did not store Last-Modified, and leaving the epoch with the flag set would have made the schema and count caches treat the object as older than every cached entry. `ObjectStorageIdentity` now stores the Last-Modified value and whether it is known. The miss path records the HEAD `last_modification_time`. A hit returns the stored time with the flag set. An unknown time still produces an unknown result on hit, so the schema and count caches keep treating the epoch as a missing timestamp. The new field is placed after `fetch_seq`, so the positional initializer for the identity keeps its meaning. Added the test `05492_object_storage_identity_cache_last_modified`, which reads one S3 object twice and requires the same non-NULL `_time` with a cache hit and no HEAD on the second read. Found by the audit of the identity cache PR: #2888 Co-authored-by: Cursor <cursoragent@cursor.com>
…reshness An identity-cache hit now returns the stored Last-Modified, so `_time` is no longer NULL on repeat reads. The same value was also consumed by the schema cache (`ReadBufferIterator`) and the count-from-files cache (`StorageObjectStorageSource`). After an external replace, the stored time is older than the registration time of the cached schema and row count, so both caches would treat the replaced object as unchanged and return the previous result. `ObjectMetadata` gets `is_last_modified_fresh`, which is cleared when the time came from the identity cache, and `lastModifiedConfirmsUnchanged()`. Both caches now require that check instead of only `is_last_modified_known`. A HEAD or listing that just observed the time still allows the caches to use it. A read pinned to the cached ETag still fails with `S3_OBJECT_CHANGED_DURING_READ` after a replace, and the retry loads the new object, so the count is correct after one failed query. Added the test `05493_object_storage_identity_cache_count_after_replace`: it reads `count()` once, replaces the object with a PUT that bypasses the server, expects `S3_OBJECT_CHANGED_DURING_READ`, and then expects the new count. Found by the audit of the identity cache PR: #2888 Co-authored-by: Cursor <cursoragent@cursor.com>
A filesystem cache (and the page cache) is keyed by the object's ETag. When the identity cache supplies that ETag, a fully cached read never issues a GET, so the `If-Match` check in `ReadBufferFromS3` never runs. After an external replace the cache keeps serving the previous bytes under the old ETag, and nothing in the query reports the change. `ObjectMetadata` gets `is_from_identity_cache`, set on an identity-cache hit. `createReadBuffer` checks it when a filesystem or page cache is about to use the ETag as the key, and re-reads the object through the new `getObjectMetadataIgnoringIdentityCache`. That skips the lookup but still stores the result, so the next read is cached again. An unchanged object keeps its ETag and is served from the cache. A replaced object gets a new key and is read from object storage. The added cost is one HEAD per filesystem-cache read whose identity came from the identity cache. A cold read is unchanged. Added the test `05494_object_storage_identity_cache_filesystem_cache_replace`: it reads an object into `cache_for_readbigat`, checks that the next read is a cache hit that still sends a HEAD, replaces the object with a PUT that bypasses the server, and expects the new rows. Found by the audit of the identity cache PR: #2888 Co-authored-by: Cursor <cursoragent@cursor.com>
`GetObjectAttributes` cannot request Last-Modified in this SDK, so a successful part-offset fetch left `last_modification_time` at 0. The identity cache then stored that as "the object has no modification time", and every later `_time` read from the cache was NULL, including reads with part offsets disabled. `S3::ObjectInfo` gets `last_modification_time_observed`, set only by a HEAD response. The identity gets `last_modified_observed`. A cache entry without an observed time is not returned as a complete identity. The read keeps the cached part offsets and issues a HEAD for the time. The stored entry then carries the HEAD's time. The part offsets are kept only while the ETag is unchanged. A HEAD that really returns a zero modification time is still cached as unknown. Added the failpoint `object_storage_identity_cache_skip_last_modified`, which simulates an attributes response without Last-Modified, and the test `05495_object_storage_identity_cache_part_offsets_time`. It reads one object with part offsets enabled and expects `_time` to be NULL. It then disables the failpoint and expects a non-NULL `_time` with a HEAD. Found by the audit of the identity cache PR: #2888 Co-authored-by: Cursor <cursoragent@cursor.com>
…bjectAttributes A filesystem or page cache hit re-reads the identity before using its ETag as the cache key. When the cached identity came with part offsets, that re-read used GetObjectAttributes. That call does not return Last-Modified, so the revalidating read reported `_time` as NULL and the store replaced the cached time with "not observed". The revalidation now uses a HEAD, which returns the ETag and Last-Modified. `ObjectStorageIdentityCache::setIfEpoch` keeps the part offsets already stored when the fetch saw the same ETag, because a HEAD does not return them. A different ETag still drops them. Adds test 05496 covering the primed, revalidated, and later reads. Co-authored-by: Cursor <cursoragent@cursor.com>
…ity-cache ETag A footer-only Parquet `count()` reads the row count from the parquet metadata cache and never issues a GET. `ParquetMetadataCache` keys by ETag, and the ETag came from the identity cache. After an external replace, the count returned the old row count, and `s3_validate_etag_on_read` never ran because no GET was sent. When the read is a Parquet count and `use_parquet_metadata_cache` is enabled, `createReadBuffer` now revalidates an identity-cache ETag with a HEAD, using the same path as the filesystem and page caches. Parquet reads that GET the data still check the ETag there and are unchanged. Adds test 05497 that counts a Parquet object, replaces it externally, and expects the next count to return the new row count after a HEAD. Co-authored-by: Cursor <cursoragent@cursor.com>
The identity cache had a second source of identity: `GetObjectAttributes`, which carried multipart part offsets, and a `last_modified_observed` flag because that request does not return Last-Modified. That path was not in any release, so it is removed rather than kept as dead complexity. Every identity-cache entry now comes from a HEAD, which always returns size, ETag and Last-Modified. `getObjectMetadata` no longer takes `fetch_part_offsets`, `setIfEpoch` only checks the epoch and `fetch_seq`, and the `object_storage_identity_cache_fetch_part_offsets` setting, the `object_storage_identity_cache_skip_last_modified` failpoint, and the `S3GetObjectAttributes` and `DiskS3GetObjectAttributes` events are removed. The S3 client's `getObjectIdentity` wrapper is removed. The fixes from earlier commits are kept: a cache hit reports Last-Modified with `is_last_modified_fresh` set to false, and filesystem, page and Parquet footer-only count revalidation issue a HEAD before using a cached ETag. Tests 05495 and 05496 covered only the part-offset path and are removed. Tests 05484, 05485, 05486 and 05490 no longer count `GetObjectAttributes`. Co-authored-by: Cursor <cursoragent@cursor.com>
An identity-cache hit returns the previous size and ETag. Two decisions used that metadata without a GET, so an external replace could be dropped unread: - `s3_skip_empty_files` (on by default) skipped a file whose cached size was 0, in `StorageObjectStorageSource::createReader` and `ReadBufferIterator`. - A query condition cache hit that dropped every row group, or every row group left after the bucket filter, skipped a file whose cached ETag was the key. A new helper `revalidateIdentityCachedObjectMetadata` replaces an identity-cache hit with a HEAD before those decisions. Data-lake keys do not contain the ETag, so they are not revalidated for the condition cache. Reads that do not skip still use the cache and do not pay the HEAD. Adds test 05498 that replaces an empty object and a Parquet object that the condition cache would have skipped, then expects the new rows after a HEAD. Co-authored-by: Cursor <cursoragent@cursor.com>
…acement The empty-file check in createReader runs once, before the query condition cache loop. A condition-cache hit that drops the whole file re-HEADs the object, which can reveal that the replacement is empty. That path then fell through and opened the object, and formats such as Parquet throw on an empty file. Apply the empty-file skip again after the re-HEAD, honoring `s3_skip_empty_files`. Extend the 05498 test with an empty Parquet replacement behind a condition-cache hit. Co-authored-by: Cursor <cursoragent@cursor.com>
…alya-26.8/object-storage-identity-cache Co-authored-by: Cursor <cursoragent@cursor.com>
…aded-virtual The private three-argument `S3ObjectStorage::getObjectMetadata` shared its name with the `IObjectStorage::getObjectMetadata` overloads, so clang's `-Woverloaded-virtual` (an error under `-Werror`) reported it as hiding the two-argument `RelativePathWithMetadata` overload. Rename it to `getObjectMetadataImpl`; the public overrides keep their signatures. Co-authored-by: Cursor <cursoragent@cursor.com>
… cache test The stateless runner can randomize `read_from_filesystem_cache_if_exists_otherwise_bypass_cache` to 1. In that mode a cache miss reads from S3 and does not store the segment. Test 05494 clears `cache_for_readbigat`, so the cold read leaves nothing behind, and the cached read misses and issues a `S3GetObject`. The test expects zero GETs there. Set the setting to 0 in the test's read settings, as other filesystem cache tests do, so a miss populates the cache and the warm read is a cache hit. Co-authored-by: Cursor <cursoragent@cursor.com>
…s outside ClickHouse The identity cache keeps the ETag of an object. A test that replaces an object in MinIO without going through ClickHouse leaves a stale ETag behind. The next read sends it as If-Match, the server returns 412, and the query fails with `S3_OBJECT_CHANGED_DURING_READ` (surfaced as `INCORRECT_DATA` for Iceberg manifests). - `test_database_iceberg`: clear the identity cache after the manifest rewrites in `test_select_manifest_without_sequence_number` and `test_select_manifest_without_snapshot_id`. - `test_iceberg_inverted_manifest_bounds`: clear the identity cache after each `fput_object` rewrite of the manifests. - `test_storage_gcp_auth`: a warm read hits the identity cache and skips HEAD, so one GET adds one token request. Expect 3 token requests instead of 4 after the second read. - `04033_s3_brace_glob_no_extra_head`: tag `no-parallel` and clear the identity cache first. The test counts HEAD requests, and a hit from an earlier test would make the count 0. - `05491_url_object_metadata_shard`: the test server answers Range requests with 206 and Content-Range. A 200 response triggers `Unable to read with range header`, and the stateless runner fails a test on any client stderr. Integration shard failures (4/8 and 2/8 of amd_asan_ubsan, db disk, old analyzer): https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2888&sha=0ffd9d6cc9a390df868d26bc5496fa564e54a908&name_0=PR&name_1=Integration%20tests%20%28amd_asan_ubsan%2C%20db%20disk%2C%20old%20analyzer%2C%204%2F8%29 https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2888&sha=0ffd9d6cc9a390df868d26bc5496fa564e54a908&name_0=PR&name_1=Integration%20tests%20%28amd_asan_ubsan%2C%20db%20disk%2C%20old%20analyzer%2C%202%2F8%29 Stateless tests: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2888&sha=0ffd9d6cc9a390df868d26bc5496fa564e54a908&name_0=PR&name_1=Stateless%20tests%20%28amd_asan_ubsan%2C%20cas%20s3%20storage%2C%20parallel%2C%201%2F2%29 PR: #2888 Co-authored-by: Cursor <cursoragent@cursor.com>
CI triage for 96dba9eSummary: 7 failing checks (5 test checks, plus 2 Grype checks counted once)
1. 🔴
|
The identity cache reused an object's size, ETag and Last-Modified across S3 principals. A read signed with a different secret therefore skipped the HEAD, and the failing GET returned the signature error instead of `HTTP response code: 403`, which `test_sts_smoke_env_credentials_disabled_no_opt_in` checks. Store a SipHash of the access key, secret and session token in each entry, and hit only when it matches. A different principal misses and sends its own HEAD. An entry learned by another principal is left in place. Related: #2888 Co-authored-by: Cursor <cursoragent@cursor.com>
|
@codex review |
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9f77d4e86b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| /// Computed only for a lookup. `getCredentials` resolves an assumed role, and a different | ||
| /// secret must not reuse this entry. The STS test mock returns one access key for every session. | ||
| const UInt64 credential_hash = s3CredentialHash(*client.get()); | ||
| if (auto cached = identity_cache->tryGet(identity_key, credential_hash)) |
There was a problem hiding this comment.
Revalidate cache hits when ETag checking is disabled
When s3_validate_etag_on_read=0 (including profiles using pre-26.7 compatibility), an external overwrite never invalidates this entry: the subsequent GET has no If-Match or response-ETag comparison, while createReadBuffer uses the cached size as the read bound. Replacing an object with a larger version therefore causes every later query to read only the old prefix indefinitely, rather than merely opting out of concurrent-overwrite validation. Bypass or revalidate identity-cache hits before reading when ETag validation is disabled.
Useful? React with 👍 / 👎.
…alidation With `s3_validate_etag_on_read = 0` (including `compatibility` before 26.7) the GET carries no `If-Match` and its response ETag is not compared. A cached size and ETag then stay in use after another client replaces the object: - A small file is prefetched up to `getFileSize()`, the cached size, so a larger replacement is returned as the previous prefix, and the next query does the same while the entry stays cached. - Parquet, ORC, and Arrow use the cached size as the file length. `S3ObjectStorage::getObjectMetadataImpl` now skips the identity-cache lookup when the query context has `s3_validate_etag_on_read = 0`. It still HEADs and stores the result, so a later read with validation on can hit. `StorageObjectStorageSource::createReadBuffer` also re-HEADs a cache hit when the reading context does not validate the ETag, in case the hit was taken on a thread without that setting. Test `05499_object_storage_identity_cache_unvalidated_replace` replaces a cached object externally and checks that a read with validation off returns the new rows with a HEAD, and that the next read with validation on is a cache hit. PR: #2888 Co-authored-by: Cursor <cursoragent@cursor.com>
…eletes `BackupWriterS3` writes, copies, and deletes objects through `WriteBufferFromS3`, `copyS3File`, `copyS3FileRange`, `copyDataToS3File`, and `deleteFileFromS3` directly. Those paths do not go through `S3ObjectStorage`, so they never dropped the entry in `ObjectStorageIdentityCache`. A key that a backup overwrote kept its previous ETag. The next read with `s3_validate_etag_on_read = 1` sent that ETag in `If-Match`, got a 412, and failed with `S3_OBJECT_CHANGED_DURING_READ` until `rejectCachedIdentity` dropped the entry. `BackupWriterS3` now invalidates the key after each mutation: - `writeFile` and `writeFileIfNotExists` through the `WriteBufferFromS3` completion callback, which also runs when the post-upload size check fails - `copyFileFromDisk`, `copyFile`, and `copyDataToFile` after the copy returns - `removeFile` and `removeFiles` after the delete returns Test `05500_object_storage_identity_cache_backup_replace` caches a backup object, writes a second backup to the same URL, and checks that the next read succeeds and issues a `HEAD`. PR: #2888 Co-authored-by: Cursor <cursoragent@cursor.com>
…ter the object was written `WriteBufferFromS3::finalizeImpl` ran `on_upload_complete` only after `task_tracker->waitAll()` and the post-upload size check. An exception from an upload task skipped it. `PutObject` or `CompleteMultipartUpload` can commit the object and still report an error, for example when the response is lost, so the cached identity kept the previous ETag. The next read with `s3_validate_etag_on_read = 1` then failed with `S3_OBJECT_CHANGED_DURING_READ` once. The upload worker now sets `object_creation_request_sent` immediately before each object-creating request. `finalizeImpl` runs `waitAll` and the size check inside one `try`, and its `catch` calls `on_upload_complete` when that flag is set, then rethrows. A part upload that fails before any object-creating request leaves the flag unset, so the entry for the still-current object is kept. `ExceptionOnCompleteMPURunsUploadCallback` checks that a failed `CompleteMultipartUpload` runs the callback once and still throws. PR: #2888 Co-authored-by: Cursor <cursoragent@cursor.com>
… the object was created `copyS3File`, `copyS3FileRange`, and `copyDataToS3File` invalidate the destination only when they return. `BackupWriterS3` and `S3ObjectStorage::copyObject` / `copyObjectToAnotherObjectStorage` relied on that. If `CompleteMultipartUpload`, `PutObject`, or `CopyObject` committed the destination and the response was lost, the call threw before the invalidation, so the cached identity kept the previous ETag. The next read with `s3_validate_etag_on_read = 1` sent it in `If-Match` and failed with `S3_OBJECT_CHANGED_DURING_READ` once. The copy helpers now take an `OnObjectCreationFailure` callback. `UploadHelper` sets `object_creation_request_sent` immediately before each object-creating request, and `performCopy` calls the callback from its `catch` only when that flag is set. A failure before any such request, for example a part upload that fails first, leaves the entry for the still-current object in place. `BackupWriterS3` passes the same invalidation it runs after a successful copy, and the disk copy paths in `S3ObjectStorage` do the same. The success path is unchanged. Tests `CopyDataToS3FileRunsCallbackWhenCompleteFails` and `CopyDataToS3FileSkipsCallbackWhenPartUploadFails` check that a failed `CompleteMultipartUpload` runs the callback once and a failed `UploadPart` does not. PR: #2888 Co-authored-by: Cursor <cursoragent@cursor.com>
…t move `ObjectStorageQueuePostProcessor::moveS3Objects` copies each object into another bucket with `copyS3File` directly, not through `S3ObjectStorage::copyObject`. Nothing dropped the destination's entry in `ObjectStorageIdentityCache`, so a key read earlier through `s3()` or an S3 table kept its previous ETag. After the move overwrote it, the next read with `s3_validate_etag_on_read = 1` sent that ETag in `If-Match`, got a 412, and failed with `S3_OBJECT_CHANGED_DURING_READ` once. The move now passes an `on_object_creation_failure` callback to `copyS3File` and calls the same invalidation after the copy returns. The callback runs only when `CopyObject`, `PutObject`, or `CompleteMultipartUpload` was sent and the copy then threw. A failure before that leaves the entry for the still-current object in place. A move within the same bucket is unchanged, because it already goes through `S3ObjectStorage::copyObject`. Azure moves do not use this cache. PR: #2888 Co-authored-by: Cursor <cursoragent@cursor.com>
The backup writer, the queue move, and the S3 disk each looked up the global context and the identity cache and invalidated one key. That lookup is now `invalidateObjectStorageIdentityCache`, declared next to `makeObjectStorageIdentityCacheKey` in `ObjectStorageIdentityCache.h` and defined in `ObjectStorageIdentityCache.cpp`. It does nothing when the global context or the cache is not set. `BackupIO_S3.cpp` drops its local copy. `ObjectStorageQueuePostProcessor::moveS3Objects` calls the shared function from its invalidation lambda. `S3ObjectStorage::invalidateCachedIdentity` forwards to it with the disk's endpoint and bucket, and the read and write buffer callbacks call it with copies of the endpoint, bucket, and path, because those buffers can outlive the storage. No behavior change. PR: #2888 Co-authored-by: Cursor <cursoragent@cursor.com>
`S3ObjectStorage::getObjectMetadataImpl` reads `Setting::s3_validate_etag_on_read` to decide whether to consult the identity cache, but the file never declared that setting in its `namespace Setting` block. The build failed with "no member named 's3_validate_etag_on_read' in namespace 'DB::Setting'". Add the same `extern` declaration the other object-storage readers already use. The setting itself is defined in `Settings.cpp`. PR: #2888 Co-authored-by: Cursor <cursoragent@cursor.com>
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Object storage identity cache
Documentation entry for user-facing changes
Part of #2181 by @UnamedRus
The change is an in-memory cache of S3 object identity so a repeated open does not send a
HEADfor size, ETag, and Last-Modified.What is stored
ObjectStorageIdentityCacheis a process-wideCacheBase(SLRU by default) owned by the globalContext. Each entry is one object on one store:The value does not include tags, user metadata, or part offsets. A miss is a plain
HEAD(S3::getObjectInfo). A request that asks for tags does not read or update the cache.tryGetObjectMetadatadoes not use the cache either.A hit sets
ObjectMetadata::is_from_identity_cache. When the stored Last-Modified is known, the hit also setsis_last_modified_freshto false._timemay still show that time, but the schema cache and the count cache do not treat it as proof that the object is unchanged (lastModifiedConfirmsUnchanged).When the cache is used
S3ObjectStorage::getObjectMetadataconsults the cache only for an identity-only read (with_tagsfalse).getObjectMetadataIgnoringIdentityCachealways issues theHEADand still stores the result.Before the
HEAD, the fetch takesbeginFetch(an epoch and a per-fetch sequence).setIfEpochstores the result only when that epoch is still current and no later fetch of the same key has stored. One global epoch covers the whole cache: any invalidation rejects every in-flight store, and the next read of those keys sendsHEADagain. A fetch that started earlier cannot overwrite a fetch of the same key that started later.Iceberg data files are not filled from the manifest. The comment in
IcebergDataObjectInfoleaves identity empty so the first metadata read goes through this cache and the ETag used by the filesystem, page, and Parquet metadata caches is the one returned by S3.url()keepsread_source_indexon the object when metadata is fetched. The same relative path can come from different expanded URL shards (http://{host1,host2}/...). The schema and count cache key includes that index, andWebObjectStorage::readObjectuses it to pick the shard instead of treating every shard as a failover of the first one. Test05491_url_object_metadata_shard.Invalidation
S3ObjectStorage::invalidateCachedIdentitydrops the entry and advances the epoch. It runs when this server removes, conditionally deletes, or finishes writing the object, and after a successful copy, on the destination.WriteBufferFromS3takes anon_upload_completecallback and runs it after the upload has replaced the object, including when the post-upload size check fails.A
GETthat sees the object change also drops the entry, throughReadBufferFromS3::rejectCachedIdentity:S3_OBJECT_CHANGED_DURING_READ)GETreturns 404SYSTEM CLEAR OBJECT STORAGE IDENTITY CACHEandSYSTEM DROP OBJECT STORAGE IDENTITY CACHEdrop every entry and advance the epoch. The privilege isSYSTEM DROP OBJECT STORAGE IDENTITY CACHE, grouped underSYSTEM DROP CACHE. A fetch paused across the clear cannot store afterwards (failpointobject_storage_identity_cache_pause_before_store).An overwrite made by another client is not visible to this invalidation. The next read that needs the bytes still compares ETags (
s3_validate_etag_on_read, default on) and the paths below re-HEADbefore they would skip the object or serve it from another cache.Re-HEAD before a decision that would not GET
A hit is enough for a read that still sends
GET: If-Match rejects a replaced object. It is not enough when the server would skip theGET.revalidateIdentityCachedObjectMetadatareturns immediately unlessis_from_identity_cacheis set. Otherwise it sendsHEADthroughgetObjectMetadataIgnoringIdentityCache, using the archive path when the object is an archive, and replaces the metadata. After that,is_from_identity_cacheis false, so a later step does notHEADagain.s3_skip_empty_filesis on (the default), the size is known, and the cached size is 0,createReaderre-HEADs before skipping. The same check is inReadBufferIteratorfor schema inference.HEADs once and looks up again. The condition-cache key is path plus ETag. If the refreshed size is known and 0 ands3_skip_empty_filesis on, the file is skipped. A partial hit still reads and relies on If-Match. Data-lake condition-cache keys do not include the ETag, and those scans are not re-HEADed for this cache.createReadBufferre-HEADs when the metadata came from the identity cache, the size is known, and the ETag can be a cache key. The filesystem cache key is a hash of path plus ETag. An unchanged object keeps the key. A replaced object gets a new key and is read again. Other Parquet reads still check the ETag onGET.Configuration and observability
Server settings, also applied by
SYSTEM RELOAD CONFIG:object_storage_identity_cache_size, default 64 MiB.0disables the cache. A reload with a non-zero size creates the cache when startup left it disabled.object_storage_identity_cache_policy, defaultSLRUobject_storage_identity_cache_size_ratio, default0.5Profile events:
ObjectStorageIdentityCacheHits,ObjectStorageIdentityCacheMisses.Current metrics:
ObjectStorageIdentityCacheBytes,ObjectStorageIdentityCacheCells.If there is no global context yet, the metadata request is issued uncached.
docs/reference/statements/system.mdxdocumentsSYSTEM CLEAR|DROP OBJECT STORAGE IDENTITY CACHE.The private three-argument metadata method is
getObjectMetadataImpl. The namegetObjectMetadatawould hide the two-argumentIObjectStorage::getObjectMetadataand fail the build with-Werror,-Woverloaded-virtual.Tests
Stateless tests under
tests/queries/0_stateless/, allno-fasttest(mock S3 atlocalhost:11111),no-parallel(the cache is process-wide):05484_object_storage_identity_cacheHEAD.05485_object_storage_identity_cache_endpoint_key05486_object_storage_identity_cache_overwrite05488_object_storage_identity_cache_stale_storeHEADdoes not overwrite a newer one.05489_object_storage_identity_cache_clear_during_fetchSYSTEM CLEAR OBJECT STORAGE IDENTITY CACHEduring a fetch rejects the store.05490_object_storage_identity_cache_external_replace05491_url_object_metadata_shardurl()metadata fetch keeps the shard index.05492_object_storage_identity_cache_last_modified05493_object_storage_identity_cache_count_after_replace05494_object_storage_identity_cache_filesystem_cache_replaceHEAD. A replaced object is read again. The read settings pinread_from_filesystem_cache_if_exists_otherwise_bypass_cacheto 0, because the stateless runner can turn that setting on and then a miss is not stored.05497_object_storage_identity_cache_parquet_count_after_replaceHEADs before using a cached ETag.05498_object_storage_identity_cache_skip_replacedHEADbefore the skip. Replacing the object with an empty Parquet file is skipped instead of opened.01271_show_privilegesand04836_system_cache_on_cluster_access_typeslist the new system-query privilege.Scope that stayed out
The cache does not store tags or user metadata. A metadata read that needs them always goes to S3.
tryGetObjectMetadatais unchanged and does not fill the cache. A read that ignores a missing file uses that path when metadata is absent. A later re-HEADof an entry that is already cached usesgetObjectMetadataand throws if the object has been deleted.Virtual columns
_size,_etag, and_timeon a hit show the stored identity. They update when a path above re-HEADs or when theGETrejects the ETag.CI/CD Options
Exclude tests:
Regression jobs to run: