[HMDank]

This pull request introduces an end-to-end event stream analytics pipeline, enabling Redpanda event data to be ingested, stored as Parquet files in cloud storage, and queried using Arrow Flight SQL via GizmoSQL. The changes add new services, configuration, and documentation to support this workflow.

Event Stream Analytics Pipeline

• Added the redpanda-connect service, which ingests events from Redpanda, transforms the data, and writes Parquet files to S3-compatible cloud storage (R2). Includes configuration for S3 credentials and data schema.
• Added the gizmosql service, which allows querying the event stream's Parquet lakehouse using Arrow Flight SQL. Includes secure credential handling and a custom entrypoint script for environment setup and templating.

[Copilot]

🟡 Changes recommended

Active critical security, connectivity, event-processing, and storage-configuration issues remain, along with a YAML ordering nit.

8 open issues

• Disable query logging to prevent credential exposure · New
• Use the configured PostgreSQL database name · New
• Prevent credentials from appearing in logged SQL · New
• Pass S3_ENDPOINT_URL into the connector environment · New
• Subscribe to topics produced by the configured connector · New
• Parse the Debezium event shape before filtering · New
• Use auto as the R2 signing region · New
• Move secrets before volumes to preserve key ordering · New

What changed in this PR

Adds an end-to-end event-stream analytics pipeline for Vibetype, connecting Redpanda events to Parquet files in S3-compatible storage and exposing the lakehouse through GizmoSQL and Grafana.

Changes:

• Adds Redpanda Connect ingestion, event transformation, Parquet encoding, and cloud-storage configuration.
• Adds GizmoSQL initialization, DuckLake maintenance, PostgreSQL integration, and credential handling.
• Integrates Grafana, production deployment settings, shared secrets, environment configuration, and documentation.

File	Description
src/​production/​redpanda-connect/​compose.yaml	Configures production update behavior for Redpanda Connect.
src/​production/​gizmosql/​compose.yaml	Configures production update behavior for GizmoSQL services.
src/​development/​sqitch/​compose.yaml	Provides GizmoSQL PostgreSQL role secrets to migrations.
src/​development/​redpanda-connect/​configurations/​config.yaml	Defines event ingestion, filtering, transformation, Parquet encoding, and S3-compatible storage output.
src/​development/​redpanda-connect/​compose.yaml	Adds the Redpanda Connect service with configuration and storage credential wiring.
src/​development/​postgres/​compose.yaml	Adds PostgreSQL secrets for GizmoSQL database roles.
src/​development/​grafana/​configurations/​provisioning/​datasources/​gizmosql.yaml	Provisions the GizmoSQL Grafana datasource.
src/​development/​grafana/​compose.yaml	Installs the GizmoSQL Grafana plugin and provides its credentials.
src/​development/​gizmosql/​configurations/​maintenance.sql	Registers landing files and performs lakehouse erasure and metadata cleanup.
src/​development/​gizmosql/​configurations/​maintenance.sh	Runs periodic lakehouse maintenance and checkpoint operations.
src/​development/​gizmosql/​configurations/​init.sql.template	Initializes DuckLake storage, catalog tables, schemas, views, and retention settings.
src/​development/​gizmosql/​configurations/​entrypoint.sh	Loads secrets and renders the GizmoSQL initialization script.
src/​development/​gizmosql/​compose.yaml	Defines GizmoSQL and maintenance services with secret, volume, and health-check wiring.
src/​development/​compose.yaml	Declares shared cloud-storage secrets for the development stack.
src/​development/​.env.template	Adds the configurable S3-compatible storage endpoint.
artifacts/​docs/​README.md	Documents the new event-streaming, storage, and GizmoSQL services.

_{Review effort: Lite}

[dargmuesli]

Reviewed against head 2fab18e2, including the companion branches (the debezium outbox routing, the sqitch ducklake and outbox migrations, and the vibetype outbox work). dargstack validate passes and the committed docs match a fresh dargstack document run.

Three defects will keep the pipeline dark or silently wrong; details are in the line comments. A few whole-PR notes:

• Merge order: the consumed topics only exist once #294 and the sqitch outbox table migrations (not yet on sqitch main) land, plus the vibetype outbox branch. Until then the connector consumes empty topics. Worth a line in the PR body.
• The production deltas for gizmosql, gizmosql-maintenance and redpanda-connect omit the json-file log limits that 24 of the other 28 production services set; with PRINT_QUERIES=1 the gizmosql service will log a lot. Consider matching the house pattern.
• The bucket name test is used for both landing files and lakehouse data in development and production. Presumably intentional for now, but worth a sentence in the PR body.
• escape_sed_replacement in the gizmosql entrypoint escapes & and \ but not the | delimiter; the comment claims arbitrary values stay safe, which isn't quite true. Harmless for today's value types.
• Strategic note: GizmoSQL is now in maintenance mode; the project README points users to Hardcharger. Not a blocker, but this PR establishes the lakehouse foundation on a maintained-no-further project, so that trade-off should be a conscious decision.

Things I verified that are OK, so the older threads can close:

• The hardcoded DATABASE 'ducklake' is correct: sqitch main has database_ducklake.sql creating the ducklake database owned by the gizmosql role.
• Query logging is safe: GizmoSQL redacts CREATE SECRET statements in its logs, and the ATTACH is credential-free by design.
• The Grafana $__file{} expander is valid, trims the trailing newline dargstack writes into secret files, and the jsonData field names match the plugin's own provisioning example.
• Cross-session access works: the server is a single DuckDB instance; ATTACH is instance-scoped and plain CREATE SECRET lands in the instance-scoped temporary store, so the maintenance and Grafana sessions see the lakehouse catalog and the R2/postgres secrets.
• gizmosql_client ships in the full image and reads GIZMOSQL_PASSWORD; the redpanda-connect image is busybox (wget available), exposes 4195, and its default CMD picks up the mounted /connect.yaml.
• glob('r2://…') works via httpfs, the DuckLake option names check out, and the account.deleted envelope does carry data.account_id, so the erasure query is correct.