[SteNicholas]

Purpose

Linked issue: close #1379

DataFusion SQL rejected DELETE on every primary-key table, although the primary-key writer already accepts DELETE records. This adds SQL DELETE for primary-key tables using merge-engine=deduplicate, following Spark's DeleteFromPaimonTableCommand#performPrimaryKeyDelete.

Brief change log

• Route DELETE on primary-key tables to execute_primary_key_delete, which scans the logical rows matching the WHERE clause, marks them with a DELETE _VALUE_KIND column and writes them through TableWrite, so the tombstones keep the primary-key, partition and sequence-field values. Commit them through the existing commit path and return the matched-row count; a delete without matches publishes no snapshot.
• Plan the scan against one captured snapshot: the FROM source must be the target table itself, not a view or a time-traveled copy, and self-references in the predicate, including subqueries, read the same snapshot. Concurrent upserts follow the normal primary-key merge rules, including sequence.field ordering.
• Write all tombstones before committing, so commit retries reuse the prepared files instead of rescanning. Close the writer when the scan or write fails to discard unprepared output; when the commit fails, keep the prepared files and report the table and commit user, since the snapshot may have been published.
• Reject other merge engines, ignore-delete including its fallback keys, rowkind.field, query-auth and DELETE clauses beyond a single target with WHERE before scanning or writing.

Tests

• Unit tests in crates/integrations/datafusion/src/delete/tests.rs: snapshot pinning of the target and predicate self-references, replaced targets, commit retries without rescanning, uncertain commits, tombstone write failure cleanup and rejection of unsupported options before scanning.
• Integration tests in crates/integrations/datafusion/tests/pk_delete_tests.rs: predicates, NULLs, no-match and never-written tables, subqueries, partitions, composite keys, dynamic buckets, quoted names, sequence fields and sort orders, concurrent upserts, scan failure cleanup, existing target guards and rejected merge engines, options and syntax.
• Replace test_delete_rejects_primary_key_table and test_pk_aggregation_rejects_delete.

API and Format

No API or storage format change.

Documentation

Update docs/src/sql.md with the supported primary-key tables, snapshot, concurrency and retry semantics, and the unsupported options.

🤖 Generated with Claude Code

[SteNicholas]

Ping @JingsongLi.