Skip to content

Object storage identity cache - #2888

Draft
ianton-ru wants to merge 39 commits into
antalya-26.8from
feature/antalya-26.8/object-storage-identity-cache
Draft

ianton-ru wants to merge 39 commits into
antalya-26.8from
feature/antalya-26.8/object-storage-identity-cache

Conversation

@ianton-ru

@ianton-ru ianton-ru commented Nov 17, 2026 •

Copy link
Copy Markdown

Changelog category (leave one):

  • New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Object storage identity cache

Documentation entry for user-facing changes

Part of #2181 by @UnamedRus

The change is an in-memory cache of S3 object identity so a repeated open does not send a HEAD for size, ETag, and Last-Modified.

What is stored

ObjectStorageIdentityCache is a process-wide CacheBase (SLRU by default) owned by the global Context. Each entry is one object on one store:

  • key: endpoint, bucket, and path, joined with NUL bytes, so two endpoints that reuse a bucket name do not share an entry
  • value: ETag, size, whether the size is known, Last-Modified in seconds, and whether that time is known

The value does not include tags, user metadata, or part offsets. A miss is a plain HEAD (S3::getObjectInfo). A request that asks for tags does not read or update the cache. tryGetObjectMetadata does not use the cache either.

A hit sets ObjectMetadata::is_from_identity_cache. When the stored Last-Modified is known, the hit also sets is_last_modified_fresh to false. _time may still show that time, but the schema cache and the count cache do not treat it as proof that the object is unchanged (lastModifiedConfirmsUnchanged).

When the cache is used

S3ObjectStorage::getObjectMetadata consults the cache only for an identity-only read (with_tags false). getObjectMetadataIgnoringIdentityCache always issues the HEAD and still stores the result.

Before the HEAD, the fetch takes beginFetch (an epoch and a per-fetch sequence). setIfEpoch stores the result only when that epoch is still current and no later fetch of the same key has stored. One global epoch covers the whole cache: any invalidation rejects every in-flight store, and the next read of those keys sends HEAD again. A fetch that started earlier cannot overwrite a fetch of the same key that started later.

Iceberg data files are not filled from the manifest. The comment in IcebergDataObjectInfo leaves identity empty so the first metadata read goes through this cache and the ETag used by the filesystem, page, and Parquet metadata caches is the one returned by S3.

url() keeps read_source_index on the object when metadata is fetched. The same relative path can come from different expanded URL shards (http://{host1,host2}/...). The schema and count cache key includes that index, and WebObjectStorage::readObject uses it to pick the shard instead of treating every shard as a failover of the first one. Test 05491_url_object_metadata_shard.

Invalidation

S3ObjectStorage::invalidateCachedIdentity drops the entry and advances the epoch. It runs when this server removes, conditionally deletes, or finishes writing the object, and after a successful copy, on the destination. WriteBufferFromS3 takes an on_upload_complete callback and runs it after the upload has replaced the object, including when the post-upload size check fails.

A GET that sees the object change also drops the entry, through ReadBufferFromS3::rejectCachedIdentity:

  • If-Match returns 412
  • the response ETag does not match the expected ETag (S3_OBJECT_CHANGED_DURING_READ)
  • an unversioned GET returns 404

SYSTEM CLEAR OBJECT STORAGE IDENTITY CACHE and SYSTEM DROP OBJECT STORAGE IDENTITY CACHE drop every entry and advance the epoch. The privilege is SYSTEM DROP OBJECT STORAGE IDENTITY CACHE, grouped under SYSTEM DROP CACHE. A fetch paused across the clear cannot store afterwards (failpoint object_storage_identity_cache_pause_before_store).

An overwrite made by another client is not visible to this invalidation. The next read that needs the bytes still compares ETags (s3_validate_etag_on_read, default on) and the paths below re-HEAD before they would skip the object or serve it from another cache.

Re-HEAD before a decision that would not GET

A hit is enough for a read that still sends GET: If-Match rejects a replaced object. It is not enough when the server would skip the GET.

revalidateIdentityCachedObjectMetadata returns immediately unless is_from_identity_cache is set. Otherwise it sends HEAD through getObjectMetadataIgnoringIdentityCache, using the archive path when the object is an archive, and replaces the metadata. After that, is_from_identity_cache is false, so a later step does not HEAD again.

  • Empty file. If s3_skip_empty_files is on (the default), the size is known, and the cached size is 0, createReader re-HEADs before skipping. The same check is in ReadBufferIterator for schema inference.
  • Query condition cache. For a non-data-lake object, a lookup that would drop the whole object re-HEADs once and looks up again. The condition-cache key is path plus ETag. If the refreshed size is known and 0 and s3_skip_empty_files is on, the file is skipped. A partial hit still reads and relies on If-Match. Data-lake condition-cache keys do not include the ETag, and those scans are not re-HEADed for this cache.
  • Filesystem cache, page cache, and a Parquet footer-only count. createReadBuffer re-HEADs when the metadata came from the identity cache, the size is known, and the ETag can be a cache key. The filesystem cache key is a hash of path plus ETag. An unchanged object keeps the key. A replaced object gets a new key and is read again. Other Parquet reads still check the ETag on GET.

Configuration and observability

Server settings, also applied by SYSTEM RELOAD CONFIG:

  • object_storage_identity_cache_size, default 64 MiB. 0 disables the cache. A reload with a non-zero size creates the cache when startup left it disabled.
  • object_storage_identity_cache_policy, default SLRU
  • object_storage_identity_cache_size_ratio, default 0.5

Profile events: ObjectStorageIdentityCacheHits, ObjectStorageIdentityCacheMisses.
Current metrics: ObjectStorageIdentityCacheBytes, ObjectStorageIdentityCacheCells.

If there is no global context yet, the metadata request is issued uncached.

docs/reference/statements/system.mdx documents SYSTEM CLEAR|DROP OBJECT STORAGE IDENTITY CACHE.

The private three-argument metadata method is getObjectMetadataImpl. The name getObjectMetadata would hide the two-argument IObjectStorage::getObjectMetadata and fail the build with -Werror,-Woverloaded-virtual.

Tests

Stateless tests under tests/queries/0_stateless/, all no-fasttest (mock S3 at localhost:11111), no-parallel (the cache is process-wide):

Test What it checks
05484_object_storage_identity_cache A second read of an unchanged object is an identity-cache hit and does not send another HEAD.
05485_object_storage_identity_cache_endpoint_key The same bucket and path on two endpoints are different keys.
05486_object_storage_identity_cache_overwrite A write by this server drops the entry.
05488_object_storage_identity_cache_stale_store An older in-flight HEAD does not overwrite a newer one.
05489_object_storage_identity_cache_clear_during_fetch SYSTEM CLEAR OBJECT STORAGE IDENTITY CACHE during a fetch rejects the store.
05490_object_storage_identity_cache_external_replace An external replace is detected on read and the cached identity is dropped.
05491_url_object_metadata_shard url() metadata fetch keeps the shard index.
05492_object_storage_identity_cache_last_modified A hit returns Last-Modified, and that time is not a freshness proof for the schema and count caches.
05493_object_storage_identity_cache_count_after_replace A count after an external replace does not keep the previous row count.
05494_object_storage_identity_cache_filesystem_cache_replace An unchanged object is served from the filesystem cache after a re-HEAD. A replaced object is read again. The read settings pin read_from_filesystem_cache_if_exists_otherwise_bypass_cache to 0, because the stateless runner can turn that setting on and then a miss is not stored.
05497_object_storage_identity_cache_parquet_count_after_replace A Parquet footer-only count re-HEADs before using a cached ETag.
05498_object_storage_identity_cache_skip_replaced A cached empty size, and a condition-cache hit that would drop the file, re-HEAD before the skip. Replacing the object with an empty Parquet file is skipped instead of opened.

01271_show_privileges and 04836_system_cache_on_cluster_access_types list the new system-query privilege.

Scope that stayed out

The cache does not store tags or user metadata. A metadata read that needs them always goes to S3.

tryGetObjectMetadata is unchanged and does not fill the cache. A read that ignores a missing file uses that path when metadata is absent. A later re-HEAD of an entry that is already cached uses getObjectMetadata and throws if the object has been deleted.

Virtual columns _size, _etag, and _time on a hit show the stored identity. They update when a path above re-HEADs or when the GET rejects the ETag.

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Unit tests
  • Performance tests
  • Aarch64 tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • CAS (content-addressed storage; Antalya only)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • OAuth (5m)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

UnamedRus and others added 6 commits November 16, 2026 13:02
…e HEAD

Opening an object went through a HEAD (getObjectMetadata) on every read to learn
its size and ETag, and the result was not cached process-wide, so a scan
re-HEADed every candidate file on every query (traced: ~1 S3 HeadObject per
candidate file, e.g. ~1900 HEADs to read ~550 files). The HEAD is also
mandatory-before-cache-lookup, because the filesystem / page / parquet-metadata
caches key on (path, etag) and the etag comes from the HEAD.

Introduce a process-wide identity cache and a single-request metadata fetch:

* `S3::getObjectIdentity` issues one `GetObjectAttributes` request (size + ETag +
  multipart part sizes) and falls back to a plain HEAD if the API is
  unsupported, denied, or errors - safe against S3-compatible stores. The ETag
  is quote-normalized so the GetObjectAttributes and HEAD paths yield the same
  identity (they are used as cache keys). Adds the `GetObjectAttributes` wrapper
  to `S3::Client` (mirrors `GetObjectTagging`) and the request alias.

* `ObjectStorageIdentityCache` (experimental singleton) maps path ->
  {etag, size, part_offsets}. `S3ObjectStorage::getObjectMetadata` consults it on
  the identity path (no tags requested): a hit skips the request entirely; a miss
  fetches via `getObjectIdentity` and populates it. The tags path and the
  credentials-refresh retry are unchanged.

* `ObjectMetadata` gains `part_offsets` (cumulative multipart part offsets),
  derived from the per-part sizes; consumers use it to align reads.

* `IcebergDataObjectInfo` no longer pre-populates object metadata from the
  manifest, so Iceberg data files also go through this single identity path and
  get the real ETag and multipart layout instead of a synthesized identity.

New ProfileEvents: `S3GetObjectAttributes`, `ObjectStorageIdentityCacheHits`,
`ObjectStorageIdentityCacheMisses`.

Note: the identity path returns no user-metadata attributes (GetObjectAttributes
does not provide them); callers that need them still use the tags path. The
cache is an experimental singleton (not Context-managed / not SYSTEM DROP-able)
and should be promoted before productionization.

Signed-off-by: UnamedRus <dtitmoav@gmail.com>
(cherry picked from commit e1b807a)
Replace the process-wide map with a size-bounded CacheBase created from
`object_storage_identity_cache_policy`, `_size` and `_size_ratio`, resized on
config reload, and cleared by `SYSTEM DROP OBJECT STORAGE IDENTITY CACHE`.
S3 reads look it up from the global context and fall back to an uncached
request when no context exists.

Co-authored-by: Cursor <cursoragent@cursor.com>
… IDENTITY CACHE

The new `SYSTEM DROP OBJECT STORAGE IDENTITY CACHE` privilege (added with the
Context-managed identity cache) adds a row to `SHOW PRIVILEGES`, so the stateless
test reference needs the corresponding line.

CI report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2181&sha=5b16210f9e7dbf582ab7e23983337a9244f2b9fe&name_0=PR&name_1=Fast%20test

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: UnamedRus <dtitmoav@gmail.com>
…Attributes behind a read setting

Reading object metadata on a lake scan (`StorageObjectStorageSource`) needs the object size and etag
(the etag keys the page / filesystem / Parquet-metadata caches). The identity fast path fetched these
via `GetObjectAttributes`, which also returns the multipart part layout used for part-aligned reads -
but that request is heavier than a plain `HEAD` and is issued per file, so a cold, cache-empty scan
paid one `GetObjectAttributes` per object (measured ~2x slower cold on a wide-Iceberg benchmark) for
part offsets it usually does not use.

Default `S3ObjectStorage::getObjectMetadata` to a plain `HEAD` (size + etag, no part offsets), and add
a read setting `object_storage_identity_cache_fetch_part_offsets` (default off) that opts back into
`GetObjectAttributes` when part-aligned reads want the layout. A new 3-argument
`IObjectStorage::getObjectMetadata` overload carries the flag; its default implementation ignores it,
so non-S3 storages and the many callers that do not need part offsets are unchanged. The etag is still
returned on both paths, so the object-storage caches keep working.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: UnamedRus <dtitmoav@gmail.com>
The new `ObjectMetadata::part_offsets` field has no default member
initializer, so designated initializers of `ObjectMetadata` that omit it
(for example in `AzureObjectStorage`) fail with
`-Werror,-Wmissing-designated-field-initializers`. Give the field an
explicit empty default instead of touching every call site.

Co-authored-by: Cursor <cursoragent@cursor.com>
…setting

Add the stateless test `05484_object_storage_identity_cache` for the
object storage identity cache. It covers a cold read that misses, a warm
read that hits without `HeadObject` or `GetObjectAttributes`, the
`SYSTEM DROP OBJECT STORAGE IDENTITY CACHE` and
`SYSTEM CLEAR OBJECT STORAGE IDENTITY CACHE` commands, and the
`object_storage_identity_cache_fetch_part_offsets` setting.

The setting was only read in `createReadBuffer`, which runs after the
object metadata is already filled, so it never chose `GetObjectAttributes`
on the SQL read path. `KeysIterator` now receives the setting and passes
it to `getObjectMetadata`, and the empty-metadata branch does the same.

Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Workflow [PR], commit [9990639]

ianton-ru and others added 23 commits November 17, 2026 14:32
The identity cache key was the bucket and the object path. Two S3
endpoints can serve buckets with the same name, so a hit from one
endpoint answered for the other with its size and ETag. Include the
endpoint and the client region in the key. Fields are separated by NUL
so that a slash in one field cannot make two keys equal.

The new test `05485_object_storage_identity_cache_endpoint_key` reads
the same object through `localhost` and `127.0.0.1` and checks that
each endpoint misses on its first read and hits on its second.

Co-authored-by: Cursor <cursoragent@cursor.com>
The identity cache was keyed by endpoint, region, bucket and path, but
nothing removed an entry when this server replaced or deleted the
object. A later read then used the old size as the read bound and the
old ETag as If-Match, so it returned truncated data or failed the read
until eviction or SYSTEM DROP.

- A successful single-part or multipart upload stores the new size and
  ETag, taken from the upload response, in the cache. If the
  post-upload size check fails, the entry is dropped instead.
- A successful copy, a delete, a batch delete (for each key that was
  removed or already absent), and a conditional delete that reports a
  token mismatch or not-found drop the entry.
- Each key has a generation number. A metadata fetch stores its result
  only if the generation is unchanged since before its request, so a
  HEAD that was in flight during a write cannot put the old identity
  back.
- Entries expire after `object_storage_identity_cache_ttl` seconds
  (default 3600, zero means no expiry). This bounds the staleness when
  another writer replaces an object.

The new test `05486_object_storage_identity_cache_overwrite` inserts a
one-row object, overwrites it with three longer rows without clearing
the cache, and checks that the next read is a cache hit that returns
all three rows. s3_validate_etag_on_read is disabled, so a stale cached
size would truncate the read.

Co-authored-by: Cursor <cursoragent@cursor.com>
`getObjectIdentity` stripped the quotes from the ETag it read with
GetObjectAttributes, and its HEAD fallback did the same. `getObjectInfo`
(HEAD) and PutObject keep the quotes. The identity cache stores this
value and `ReadBufferFromS3` sends it as `If-Match`, so the two
spellings of one object are different bytes. With
`object_storage_identity_cache_fetch_part_offsets = 1` the cache held
an unquoted ETag, and with `s3_validate_etag_on_read` on the next read
of that object could fail although the object did not change.

Both paths now store the quoted form. `quoteETag` adds the quotes when
they are missing and leaves an already quoted or weak (`W/"..."`) value
unchanged.

The new test `05487_object_storage_identity_cache_etag_spelling` keeps
`s3_validate_etag_on_read` on. It checks that the ETag read through HEAD
and through GetObjectAttributes is the same quoted value, and that the
following read hits the cache and returns that value.

Co-authored-by: Cursor <cursoragent@cursor.com>
`Context::updateObjectStorageIdentityCacheConfiguration` returned at
once when the identity cache pointer was unset. Startup leaves the
pointer unset when `object_storage_identity_cache_size` is `0`, so a
reload that set a non-zero size did nothing and every read stayed
uncached until the next process start.

A reload now creates the cache when the new size is non-zero, using the
reloaded policy, size ratio, and TTL. The size is still capped by the
same RAM limit as at startup. A reload that keeps the size at `0`
still leaves the cache disabled.

The setting description and the `SYSTEM CLEAR|DROP OBJECT STORAGE
IDENTITY CACHE` page say that `SYSTEM RELOAD CONFIG` turns the cache on.

Co-authored-by: Cursor <cursoragent@cursor.com>
The overwrite fix added a per-key generation map, a TTL with a
timestamp on every entry, and a region in the cache key. None of these
is needed for the overwrite case: a write, copy, or delete by this
server removes the entry, and the next read requests the metadata
again. Keeping them made the cache, the S3 write path, and the
settings larger than the fix requires.

- ObjectStorageIdentityCache is a plain CacheBase again: tryGet and
  set. The generation map, setIfEpoch, epochOf, replace, invalidate,
  and the TTL with its timestamp are removed.
- The cache key is the endpoint, the bucket, and the path. The region
  is removed.
- A successful upload calls the upload callback without arguments and
  the callback removes the entry. The size and ETag are no longer
  copied from the upload response.
- A copy, a delete, a batch delete, and a conditional delete remove the
  entry through invalidateCachedIdentity.
- The object_storage_identity_cache_ttl server setting is removed.
- Context::setObjectStorageIdentityCache no longer takes a TTL.

The reload path that creates the cache when startup left it disabled,
and the ETag spelling fix, are unchanged.

The test 05486 expects the read after an overwrite to miss and return
all three new rows, and the next read to hit.

Co-authored-by: Cursor <cursoragent@cursor.com>
…rite

A metadata read takes its HEAD before it stores the result. If this server
overwrote, copied or deleted the object in that window, the read stored the
size and ETag it fetched before the overwrite. The entry was dropped by the
overwrite, so nothing removed the stale value again.

Each identity-cache key now has a generation. An overwrite, copy or delete
bumps it with `invalidate`, which also removes the entry. A metadata read
captures the generation before its request and stores the result with
`setIfEpoch`, which skips the store when the generation has changed.
`clear` drops the generations together with the entries.

Add the pause failpoint `object_storage_identity_cache_pause_before_store`
between the fetch and the store, and test 05488 uses it to overwrite the
object while a read is paused. The next read must miss and return the new
rows.

Co-authored-by: Cursor <cursoragent@cursor.com>
… cleared

`SYSTEM CLEAR OBJECT STORAGE IDENTITY CACHE` removed the per-key generations
along with the entries. A key that had been overwritten therefore looked
untouched again, and a metadata read that started before the clear and
resumed after it could store the size and ETag it had fetched before the
overwrite.

`clear` now bumps a cache-wide generation and does not reset it. The fetch
captures it together with the per-key generation as the new
`ObjectStorageIdentityEpoch`, and `setIfEpoch` stores only when both still
match.

Test 05489 pauses a read after its HEAD, overwrites the object, clears the
cache, and resumes the read. The next read must miss and return the new rows.

Co-authored-by: Cursor <cursoragent@cursor.com>
Each `invalidate` of a key left a per-key generation record in
`epoch_by_key`, and nothing removed it except `clear`. The map grew by one
entry for every overwritten, copied or deleted object for the life of the
process, outside the cache's size limit.

The map is now accounted in bytes and limited to the cache's configured
size. When a new record would exceed that limit, the map is dropped and
the cache-wide generation is bumped, so every fetch already in flight
fails its `setIfEpoch` check and does not store. One record is always
kept, so the map never refuses a key even when that single record is
larger than the limit.

Co-authored-by: Cursor <cursoragent@cursor.com>
The per-key generation map added to keep stale identities out of the
identity cache was large and needed its own byte accounting and cap.
A single cache-wide epoch covers the same cases with much less code.

`invalidate` (overwrite, copy or delete) and `clear` advance the epoch
and `invalidate` also removes the entry. A metadata read captures the
epoch before its request and stores the result with `setIfEpoch`, which
skips the store when the epoch has changed. The epoch is never reset, so
a fetch in flight cannot match a later epoch by accident.

The trade-off is that any write, copy or delete of any object makes every
metadata fetch in flight skip its store. The fetch still returns its result
to the caller, and the next read of the key fetches again.

Tests 05488 and 05489 keep their scenarios. Only their comments change.

Co-authored-by: Cursor <cursoragent@cursor.com>
…ache

A read that found the object replaced kept the cached size and ETag. The
next read was pinned to that ETag and failed with
`S3_OBJECT_CHANGED_DURING_READ` again, and every retry hit the same entry.
Nothing in the read path removed it, because a replace made outside this
server does not pass through `S3ObjectStorage::writeObject`.

`ReadBufferFromS3` now calls a callback when a GET fails `If-Match` or, for
an unversioned read, returns a not-found error. `S3ObjectStorage::readObject`
passes a callback that invalidates the key, which also advances the cache
epoch so a fetch already in flight cannot store the old identity back.

`ObjectStorageIdentityCache` also orders stores per key. `beginFetch` hands
out a sequence number, and `setIfEpoch` skips a store when a later fetch of
the same key has already stored. An older observation therefore cannot
overwrite a newer one.

Test 05490 replaces the object with a plain PUT, expects the read to fail
with `S3_OBJECT_CHANGED_DURING_READ`, and checks that the retry misses,
sends a HEAD, and returns the new rows.

Co-authored-by: Cursor <cursoragent@cursor.com>
The metadata fetch for a listed `url()` file called the string overload of
`getObjectMetadata`. That overload builds a `RelativePathWithMetadata`
without `read_source_index`, so `WebObjectStorage::buildURLs` probed every
shard. With several shards that expose the same relative path, the size
and ETag could come from a different shard than the one that listed the
file, and the read bound could be wrong.

`IObjectStorage` gains a `getObjectMetadata` overload that takes the whole
`RelativePathWithMetadata` and the part-offset flag. Its default forwards to
the object overload, so `WebObjectStorage` keeps the index. `S3ObjectStorage`
overrides it to pass the flag to its existing fetch. `createReader` and
`createReadBuffer` now call this overload.

Test 05491 serves the same file from two local HTTP shards with different
sizes and checks that the rows from both shards are returned in full.

Co-authored-by: Cursor <cursoragent@cursor.com>
A cache hit in `S3ObjectStorage::getObjectMetadata` returned the object
with `is_last_modified_known = false`, so the `_time` virtual column
became NULL on every read after the first. The identity cache did not
store Last-Modified, and leaving the epoch with the flag set would have
made the schema and count caches treat the object as older than every
cached entry.

`ObjectStorageIdentity` now stores the Last-Modified value and whether it
is known. The miss path records the HEAD `last_modification_time`. A hit
returns the stored time with the flag set. An unknown time still produces
an unknown result on hit, so the schema and count caches keep treating
the epoch as a missing timestamp.

The new field is placed after `fetch_seq`, so the positional initializer
for the identity keeps its meaning.

Added the test `05492_object_storage_identity_cache_last_modified`, which
reads one S3 object twice and requires the same non-NULL `_time` with a
cache hit and no HEAD on the second read.

Found by the audit of the identity cache PR:
#2888

Co-authored-by: Cursor <cursoragent@cursor.com>
…reshness

An identity-cache hit now returns the stored Last-Modified, so `_time`
is no longer NULL on repeat reads. The same value was also consumed by
the schema cache (`ReadBufferIterator`) and the count-from-files cache
(`StorageObjectStorageSource`). After an external replace, the stored
time is older than the registration time of the cached schema and row
count, so both caches would treat the replaced object as unchanged and
return the previous result.

`ObjectMetadata` gets `is_last_modified_fresh`, which is cleared when the
time came from the identity cache, and `lastModifiedConfirmsUnchanged()`.
Both caches now require that check instead of only
`is_last_modified_known`. A HEAD or listing that just observed the time
still allows the caches to use it.

A read pinned to the cached ETag still fails with
`S3_OBJECT_CHANGED_DURING_READ` after a replace, and the retry loads the
new object, so the count is correct after one failed query.

Added the test `05493_object_storage_identity_cache_count_after_replace`:
it reads `count()` once, replaces the object with a PUT that bypasses the
server, expects `S3_OBJECT_CHANGED_DURING_READ`, and then expects the
new count.

Found by the audit of the identity cache PR:
#2888

Co-authored-by: Cursor <cursoragent@cursor.com>
A filesystem cache (and the page cache) is keyed by the object's ETag.
When the identity cache supplies that ETag, a fully cached read never
issues a GET, so the `If-Match` check in `ReadBufferFromS3` never runs.
After an external replace the cache keeps serving the previous bytes
under the old ETag, and nothing in the query reports the change.

`ObjectMetadata` gets `is_from_identity_cache`, set on an identity-cache
hit. `createReadBuffer` checks it when a filesystem or page cache is
about to use the ETag as the key, and re-reads the object through the
new `getObjectMetadataIgnoringIdentityCache`. That skips the lookup but
still stores the result, so the next read is cached again. An unchanged
object keeps its ETag and is served from the cache. A replaced object
gets a new key and is read from object storage.

The added cost is one HEAD per filesystem-cache read whose identity came
from the identity cache. A cold read is unchanged.

Added the test `05494_object_storage_identity_cache_filesystem_cache_replace`:
it reads an object into `cache_for_readbigat`, checks that the next read
is a cache hit that still sends a HEAD, replaces the object with a PUT
that bypasses the server, and expects the new rows.

Found by the audit of the identity cache PR:
#2888

Co-authored-by: Cursor <cursoragent@cursor.com>
`GetObjectAttributes` cannot request Last-Modified in this SDK, so a
successful part-offset fetch left `last_modification_time` at 0. The
identity cache then stored that as "the object has no modification
time", and every later `_time` read from the cache was NULL, including
reads with part offsets disabled.

`S3::ObjectInfo` gets `last_modification_time_observed`, set only by a
HEAD response. The identity gets `last_modified_observed`. A cache entry
without an observed time is not returned as a complete identity. The
read keeps the cached part offsets and issues a HEAD for the time. The
stored entry then carries the HEAD's time. The part offsets are kept only
while the ETag is unchanged. A HEAD that really returns a zero
modification time is still cached as unknown.

Added the failpoint `object_storage_identity_cache_skip_last_modified`,
which simulates an attributes response without Last-Modified, and the
test `05495_object_storage_identity_cache_part_offsets_time`. It reads one
object with part offsets enabled and expects `_time` to be NULL. It then
disables the failpoint and expects a non-NULL `_time` with a HEAD.

Found by the audit of the identity cache PR:
#2888

Co-authored-by: Cursor <cursoragent@cursor.com>
…bjectAttributes

A filesystem or page cache hit re-reads the identity before using its ETag as the cache key.
When the cached identity came with part offsets, that re-read used GetObjectAttributes. That
call does not return Last-Modified, so the revalidating read reported `_time` as NULL and the
store replaced the cached time with "not observed". The revalidation now uses a HEAD, which
returns the ETag and Last-Modified.

`ObjectStorageIdentityCache::setIfEpoch` keeps the part offsets already stored when the fetch
saw the same ETag, because a HEAD does not return them. A different ETag still drops them.

Adds test 05496 covering the primed, revalidated, and later reads.

Co-authored-by: Cursor <cursoragent@cursor.com>
…ity-cache ETag

A footer-only Parquet `count()` reads the row count from the parquet metadata cache and never
issues a GET. `ParquetMetadataCache` keys by ETag, and the ETag came from the identity cache. After
an external replace, the count returned the old row count, and `s3_validate_etag_on_read` never
ran because no GET was sent.

When the read is a Parquet count and `use_parquet_metadata_cache` is enabled, `createReadBuffer`
now revalidates an identity-cache ETag with a HEAD, using the same path as the filesystem and page
caches. Parquet reads that GET the data still check the ETag there and are unchanged.

Adds test 05497 that counts a Parquet object, replaces it externally, and expects the next count
to return the new row count after a HEAD.

Co-authored-by: Cursor <cursoragent@cursor.com>
The identity cache had a second source of identity: `GetObjectAttributes`, which carried multipart
part offsets, and a `last_modified_observed` flag because that request does not return Last-Modified.
That path was not in any release, so it is removed rather than kept as dead complexity.

Every identity-cache entry now comes from a HEAD, which always returns size, ETag and Last-Modified.
`getObjectMetadata` no longer takes `fetch_part_offsets`, `setIfEpoch` only checks the epoch and
`fetch_seq`, and the `object_storage_identity_cache_fetch_part_offsets` setting, the
`object_storage_identity_cache_skip_last_modified` failpoint, and the `S3GetObjectAttributes` and
`DiskS3GetObjectAttributes` events are removed. The S3 client's `getObjectIdentity` wrapper is removed.

The fixes from earlier commits are kept: a cache hit reports Last-Modified with
`is_last_modified_fresh` set to false, and filesystem, page and Parquet footer-only count
revalidation issue a HEAD before using a cached ETag. Tests 05495 and 05496 covered only the
part-offset path and are removed. Tests 05484, 05485, 05486 and 05490 no longer count
`GetObjectAttributes`.

Co-authored-by: Cursor <cursoragent@cursor.com>
An identity-cache hit returns the previous size and ETag. Two decisions used that metadata
without a GET, so an external replace could be dropped unread:

- `s3_skip_empty_files` (on by default) skipped a file whose cached size was 0, in
  `StorageObjectStorageSource::createReader` and `ReadBufferIterator`.
- A query condition cache hit that dropped every row group, or every row group left after the
  bucket filter, skipped a file whose cached ETag was the key.

A new helper `revalidateIdentityCachedObjectMetadata` replaces an identity-cache hit with a
HEAD before those decisions. Data-lake keys do not contain the ETag, so they are not revalidated
for the condition cache. Reads that do not skip still use the cache and do not pay the HEAD.

Adds test 05498 that replaces an empty object and a Parquet object that the condition cache
would have skipped, then expects the new rows after a HEAD.

Co-authored-by: Cursor <cursoragent@cursor.com>
…acement

The empty-file check in createReader runs once, before the query condition
cache loop. A condition-cache hit that drops the whole file re-HEADs the
object, which can reveal that the replacement is empty. That path then fell
through and opened the object, and formats such as Parquet throw on an empty
file. Apply the empty-file skip again after the re-HEAD, honoring
`s3_skip_empty_files`.

Extend the 05498 test with an empty Parquet replacement behind a
condition-cache hit.

Co-authored-by: Cursor <cursoragent@cursor.com>
…alya-26.8/object-storage-identity-cache

Co-authored-by: Cursor <cursoragent@cursor.com>
…aded-virtual

The private three-argument `S3ObjectStorage::getObjectMetadata` shared its
name with the `IObjectStorage::getObjectMetadata` overloads, so clang's
`-Woverloaded-virtual` (an error under `-Werror`) reported it as hiding the
two-argument `RelativePathWithMetadata` overload. Rename it to
`getObjectMetadataImpl`; the public overrides keep their signatures.

Co-authored-by: Cursor <cursoragent@cursor.com>
… cache test

The stateless runner can randomize `read_from_filesystem_cache_if_exists_otherwise_bypass_cache` to 1.
In that mode a cache miss reads from S3 and does not store the segment. Test 05494 clears
`cache_for_readbigat`, so the cold read leaves nothing behind, and the cached read misses and issues
a `S3GetObject`. The test expects zero GETs there.

Set the setting to 0 in the test's read settings, as other filesystem cache tests do, so a miss
populates the cache and the warm read is a cache hit.

Co-authored-by: Cursor <cursoragent@cursor.com>
…s outside ClickHouse

The identity cache keeps the ETag of an object. A test that replaces an object in MinIO
without going through ClickHouse leaves a stale ETag behind. The next read sends it as If-Match,
the server returns 412, and the query fails with `S3_OBJECT_CHANGED_DURING_READ` (surfaced as
`INCORRECT_DATA` for Iceberg manifests).

- `test_database_iceberg`: clear the identity cache after the manifest rewrites in
  `test_select_manifest_without_sequence_number` and `test_select_manifest_without_snapshot_id`.
- `test_iceberg_inverted_manifest_bounds`: clear the identity cache after each `fput_object` rewrite
  of the manifests.
- `test_storage_gcp_auth`: a warm read hits the identity cache and skips HEAD, so one GET adds one
  token request. Expect 3 token requests instead of 4 after the second read.
- `04033_s3_brace_glob_no_extra_head`: tag `no-parallel` and clear the identity cache first. The
  test counts HEAD requests, and a hit from an earlier test would make the count 0.
- `05491_url_object_metadata_shard`: the test server answers Range requests with 206 and
  Content-Range. A 200 response triggers `Unable to read with range header`, and the stateless runner
  fails a test on any client stderr.

Integration shard failures (4/8 and 2/8 of amd_asan_ubsan, db disk, old analyzer):
https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2888&sha=0ffd9d6cc9a390df868d26bc5496fa564e54a908&name_0=PR&name_1=Integration%20tests%20%28amd_asan_ubsan%2C%20db%20disk%2C%20old%20analyzer%2C%204%2F8%29
https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2888&sha=0ffd9d6cc9a390df868d26bc5496fa564e54a908&name_0=PR&name_1=Integration%20tests%20%28amd_asan_ubsan%2C%20db%20disk%2C%20old%20analyzer%2C%202%2F8%29
Stateless tests: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2888&sha=0ffd9d6cc9a390df868d26bc5496fa564e54a908&name_0=PR&name_1=Stateless%20tests%20%28amd_asan_ubsan%2C%20cas%20s3%20storage%2C%20parallel%2C%201%2F2%29

PR: #2888

Co-authored-by: Cursor <cursoragent@cursor.com>
@ianton-ru

Copy link
Copy Markdown
Author

@blau-ai

@blau-ai

blau-ai commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

CI triage for 96dba9e

Summary: 7 failing checks (5 test checks, plus 2 Grype checks counted once)

Check Failing test Verdict
Integration tests (amd_asan_ubsan, targeted) and (db disk, old analyzer, 1/8) test_storage_s3/test_sts.py::test_sts_smoke_env_credentials_disabled_no_opt_in 🔴 PR-caused
Integration tests (amd_asan_ubsan, db disk, old analyzer, 3/8) test_s3_cluster/test.py::test_graceful_shutdown 🟠 Probably PR-caused: needs a closer look
Stateless tests (amd_debug, distributed plan, s3 storage, parallel) 03572_export_merge_tree_part_limits_and_table_functions 🟡 Probably flaky (not fully cleared)
Regression release swarms /swarms/feature/node failure/cpu overload* ⚪ Pre-existing: also fails on #2901
Regression release iceberg_2 /iceberg/export partition/* ⚪ Bug in the regression suite (Python TypeError)
Grype Scan (keeper, server-alpine) 4 high/critical CVEs in images ⚪ Not caused by this PR (no Docker or dependency changes)

1. 🔴 test_sts_smoke_env_credentials_disabled_no_opt_in: PR-caused

It fails the same way in two different jobs (targeted, and db disk 1/8). Both shards are green on #2901 and #2890 against the same base.

test_sts.py:180   assert "HTTP response code: 403" in ei.value.stderr
E   ... Code: 499. DB::Exception: The request signature we calculated does not match the signature you provided ...
    4. src/IO/ReadBufferFromS3.cpp:726: DB::ReadBufferFromS3::sendRequest(...)
    5. src/IO/ReadBufferFromS3.cpp:581: DB::ReadBufferFromS3::initialize(...)

Why: this is the control step of the test. The same object, test_sts_prod_shape.csv, was just read with valid credentials (role session miniorole). Then it is read again with a wrong role_session_name, which must be rejected. Before this PR, the wrong-credentials query failed on the HEAD request. A HEAD error has no response body, so the error text was HTTP response code: 403. With this PR, the object's size/ETag comes from ObjectStorageIdentityCache, so no HEAD is sent. The first request that fails is the GET (see the ReadBufferFromS3::sendRequest frame), and it returns the XML error SignatureDoesNotMatch instead.

The root cause is that the cache key is makeObjectStorageIdentityCacheKey(endpoint, bucket, path) and does not include the credentials. An identity learned by one principal is reused for a request signed by a different (here, unauthorised) principal.

Suggested fix (author's call):

  • Preferred: add a credential fingerprint to the key in ObjectStorageIdentityCache.h:53. For example, use the access key id, or role_arn + session name / external_id for STS, or a hash of the client's auth config. Separate the parts with \0 like the other fields. Then a cached entry is only reused by the same principal, the HEAD permission check comes back, and this test passes unchanged. Without this, a user with no permission on an object can still get its size/ETag/Last-Modified from the cache. That matters for paths that never GET the object (_size and similar virtual columns, count/metadata-only paths, schema inference with cached schemas), and it is a behaviour change from upstream.
  • Minimum: if sharing across credentials is intended, relax the assertion to accept either error ("HTTP response code: 403" or "SignatureDoesNotMatch" / "signature we calculated does not match"). Say in the PR description that the cache skips the HEAD authorization check.

2. 🟠 test_s3_cluster::test_graceful_shutdown: probably PR-caused

test_s3_cluster/test.py:2171   assert errors == 0
E   assert 2 == 0          (first run)
E   assert 1 == 0          (retry, also failed)
  • It failed twice in this job, both in the first run and the retry. The same shard (db disk, old analyzer, 3/8) is green on Antalya 26.8 Backport of #131001 - Reuse the GCM cipher context between rows in encrypt/decrypt #2901 and Antalya-26.8 - Drop CAS log configs on installed-binary integration nodes #2890 (Failures: 0/732, 0/769).
  • The job log contains no DB::Exception. So the errors are most likely queries that returned a different result than expected (the test counts both mismatches and exceptions), not queries that raised.
  • The input files data/graceful/* are static. The test calls SYSTEM STOP SWARM MODE and then kills a node while s3Cluster queries are running, so objects get re-dispatched to other replicas. This PR changes how sources open objects (StorageObjectStorageSource.cpp, ReadBufferIterator.cpp, Utils.h). A cached identity used on the re-dispatch path is a plausible cause, but I have not confirmed the mechanism.
  • Next step: run test_graceful_shutdown locally on the PR binary twice, once as is and once with <object_storage_identity_cache_size>0</object_storage_identity_cache_size> on the cluster nodes. If it only fails with the cache enabled, it is confirmed. Then compare how files are distributed across replicas in system.query_log / the server logs, and check whether a file is read twice or skipped after the swarm stops.

3. 🟡 03572_export_merge_tree_part_limits_and_table_functions: probably flaky

---- Count files in big_destination_max_bytes, should be 5 (4 parquet, 1 commit)
-5
+4
Randomized settings diagnosis: Runs: 8, Failed: 1, Passed: 7 → "Failure is flaky"

The row count right after is correct (4194304), so the data was exported in full and only one file is missing from the s3(..., '**') listing. This looks like a race between "exports completed" and the commit file becoming visible, and it passed 7 of 8 reruns. It failed only once, and this PR touches the S3 read/glob path, so I can't fully clear it. If it fails again on the next run, look at it together with #2. Next step: safe to re-run.

4. ⚪ Regression release swarms: pre-existing

Fails in /swarms/feature/node failure/cpu overload and cpu overload all swarm nodes with Code: 209 ... Timeout exceeded while reading from socket (... receive timeout 300000 ms). The same two scenarios also fail on #2901 (8a76e39), which does not touch object storage. Not caused by this PR.

5. ⚪ Regression release iceberg_2: bug in the regression suite

File ".../iceberg/tests/export_partition/feature.py", line 70, in _load_modules
    Feature(
TypeError: name must be specified

Every /iceberg/export partition/* scenario errors in about 1 ms, before any query runs. This is a Python bug in the regression repo's export_partition/feature.py, independent of the ClickHouse binary. Not caused by this PR.

6. ⚪ Grype scans (keeper, server-alpine)

Grype Scan Completed with 4 high/critical vulnerabilities in the image packages. The PR does not change any Docker or dependency files. Not caused by this PR.

Still pending when I checked: Build (wasm_parser), SQLLogic test, SQLStorm test.


Action items for the PR: fix #1 (include the credentials in the identity cache key, or at least update the test), then confirm or rule out #2 with the cache disabled. Everything else can be re-run or ignored.

@ianton-ru ianton-ru added antalya port-antalya PRs to be ported to all new Antalya releases antalya-26.8 labels Nov 18, 2026
The identity cache reused an object's size, ETag and Last-Modified across S3 principals. A read signed with a different secret therefore skipped the HEAD, and the failing GET returned the signature error instead of `HTTP response code: 403`, which `test_sts_smoke_env_credentials_disabled_no_opt_in` checks.

Store a SipHash of the access key, secret and session token in each entry, and hit only when it matches. A different principal misses and sends its own HEAD. An entry learned by another principal is left in place.

Related: #2888
Co-authored-by: Cursor <cursoragent@cursor.com>
@ianton-ru

Copy link
Copy Markdown
Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-19T10:57:28.742794Z 9f77d4e Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9f77d4e86b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

/// Computed only for a lookup. `getCredentials` resolves an assumed role, and a different
/// secret must not reuse this entry. The STS test mock returns one access key for every session.
const UInt64 credential_hash = s3CredentialHash(*client.get());
if (auto cached = identity_cache->tryGet(identity_key, credential_hash))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Revalidate cache hits when ETag checking is disabled

When s3_validate_etag_on_read=0 (including profiles using pre-26.7 compatibility), an external overwrite never invalidates this entry: the subsequent GET has no If-Match or response-ETag comparison, while createReadBuffer uses the cached size as the read bound. Replacing an object with a larger version therefore causes every later query to read only the old prefix indefinitely, rather than merely opting out of concurrent-overwrite validation. Bypass or revalidate identity-cache hits before reading when ETag validation is disabled.

Useful? React with 👍 / 👎.

ianton-ru and others added 7 commits November 19, 2026 12:59
…alidation

With `s3_validate_etag_on_read = 0` (including `compatibility` before 26.7) the GET carries no
`If-Match` and its response ETag is not compared. A cached size and ETag then stay in use after
another client replaces the object:

- A small file is prefetched up to `getFileSize()`, the cached size, so a larger replacement is
  returned as the previous prefix, and the next query does the same while the entry stays cached.
- Parquet, ORC, and Arrow use the cached size as the file length.

`S3ObjectStorage::getObjectMetadataImpl` now skips the identity-cache lookup when the query context
has `s3_validate_etag_on_read = 0`. It still HEADs and stores the result, so a later read with
validation on can hit.

`StorageObjectStorageSource::createReadBuffer` also re-HEADs a cache hit when the reading context
does not validate the ETag, in case the hit was taken on a thread without that setting.

Test `05499_object_storage_identity_cache_unvalidated_replace` replaces a cached object externally and
checks that a read with validation off returns the new rows with a HEAD, and that the next read with
validation on is a cache hit.

PR: #2888

Co-authored-by: Cursor <cursoragent@cursor.com>
…eletes

`BackupWriterS3` writes, copies, and deletes objects through `WriteBufferFromS3`, `copyS3File`,
`copyS3FileRange`, `copyDataToS3File`, and `deleteFileFromS3` directly. Those paths do not go through
`S3ObjectStorage`, so they never dropped the entry in `ObjectStorageIdentityCache`.

A key that a backup overwrote kept its previous ETag. The next read with `s3_validate_etag_on_read = 1`
sent that ETag in `If-Match`, got a 412, and failed with `S3_OBJECT_CHANGED_DURING_READ` until
`rejectCachedIdentity` dropped the entry.

`BackupWriterS3` now invalidates the key after each mutation:

- `writeFile` and `writeFileIfNotExists` through the `WriteBufferFromS3` completion callback, which also
  runs when the post-upload size check fails
- `copyFileFromDisk`, `copyFile`, and `copyDataToFile` after the copy returns
- `removeFile` and `removeFiles` after the delete returns

Test `05500_object_storage_identity_cache_backup_replace` caches a backup object, writes a second backup
to the same URL, and checks that the next read succeeds and issues a `HEAD`.

PR: #2888

Co-authored-by: Cursor <cursoragent@cursor.com>
…ter the object was written

`WriteBufferFromS3::finalizeImpl` ran `on_upload_complete` only after `task_tracker->waitAll()` and the
post-upload size check. An exception from an upload task skipped it. `PutObject` or
`CompleteMultipartUpload` can commit the object and still report an error, for example when the
response is lost, so the cached identity kept the previous ETag. The next read with
`s3_validate_etag_on_read = 1` then failed with `S3_OBJECT_CHANGED_DURING_READ` once.

The upload worker now sets `object_creation_request_sent` immediately before each object-creating
request. `finalizeImpl` runs `waitAll` and the size check inside one `try`, and its `catch` calls
`on_upload_complete` when that flag is set, then rethrows. A part upload that fails before any
object-creating request leaves the flag unset, so the entry for the still-current object is kept.

`ExceptionOnCompleteMPURunsUploadCallback` checks that a failed `CompleteMultipartUpload` runs the
callback once and still throws.

PR: #2888

Co-authored-by: Cursor <cursoragent@cursor.com>
… the object was created

`copyS3File`, `copyS3FileRange`, and `copyDataToS3File` invalidate the destination only when they
return. `BackupWriterS3` and `S3ObjectStorage::copyObject` / `copyObjectToAnotherObjectStorage` relied
on that. If `CompleteMultipartUpload`, `PutObject`, or `CopyObject` committed the destination and the
response was lost, the call threw before the invalidation, so the cached identity kept the previous
ETag. The next read with `s3_validate_etag_on_read = 1` sent it in `If-Match` and failed with
`S3_OBJECT_CHANGED_DURING_READ` once.

The copy helpers now take an `OnObjectCreationFailure` callback. `UploadHelper` sets
`object_creation_request_sent` immediately before each object-creating request, and `performCopy`
calls the callback from its `catch` only when that flag is set. A failure before any such request,
for example a part upload that fails first, leaves the entry for the still-current object in place.

`BackupWriterS3` passes the same invalidation it runs after a successful copy, and the disk copy
paths in `S3ObjectStorage` do the same. The success path is unchanged.

Tests `CopyDataToS3FileRunsCallbackWhenCompleteFails` and `CopyDataToS3FileSkipsCallbackWhenPartUploadFails`
check that a failed `CompleteMultipartUpload` runs the callback once and a failed `UploadPart` does not.

PR: #2888

Co-authored-by: Cursor <cursoragent@cursor.com>
…t move

`ObjectStorageQueuePostProcessor::moveS3Objects` copies each object into another bucket with `copyS3File`
directly, not through `S3ObjectStorage::copyObject`. Nothing dropped the destination's entry in
`ObjectStorageIdentityCache`, so a key read earlier through `s3()` or an S3 table kept its previous ETag.
After the move overwrote it, the next read with `s3_validate_etag_on_read = 1` sent that ETag in `If-Match`,
got a 412, and failed with `S3_OBJECT_CHANGED_DURING_READ` once.

The move now passes an `on_object_creation_failure` callback to `copyS3File` and calls the same
invalidation after the copy returns. The callback runs only when `CopyObject`, `PutObject`, or
`CompleteMultipartUpload` was sent and the copy then threw. A failure before that leaves the entry for the
still-current object in place. A move within the same bucket is unchanged, because it already goes through
`S3ObjectStorage::copyObject`. Azure moves do not use this cache.

PR: #2888

Co-authored-by: Cursor <cursoragent@cursor.com>
The backup writer, the queue move, and the S3 disk each looked up the global context and the identity
cache and invalidated one key. That lookup is now `invalidateObjectStorageIdentityCache`, declared next to
`makeObjectStorageIdentityCacheKey` in `ObjectStorageIdentityCache.h` and defined in
`ObjectStorageIdentityCache.cpp`. It does nothing when the global context or the cache is not set.

`BackupIO_S3.cpp` drops its local copy. `ObjectStorageQueuePostProcessor::moveS3Objects` calls the
shared function from its invalidation lambda. `S3ObjectStorage::invalidateCachedIdentity` forwards to it
with the disk's endpoint and bucket, and the read and write buffer callbacks call it with copies of the
endpoint, bucket, and path, because those buffers can outlive the storage.

No behavior change.

PR: #2888

Co-authored-by: Cursor <cursoragent@cursor.com>
`S3ObjectStorage::getObjectMetadataImpl` reads `Setting::s3_validate_etag_on_read` to decide whether to
consult the identity cache, but the file never declared that setting in its `namespace Setting` block.
The build failed with "no member named 's3_validate_etag_on_read' in namespace 'DB::Setting'".

Add the same `extern` declaration the other object-storage readers already use. The setting itself is
defined in `Settings.cpp`.

PR: #2888

Co-authored-by: Cursor <cursoragent@cursor.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

antalya antalya-26.8 port-antalya PRs to be ported to all new Antalya releases

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants