Announcing Supermetal Kafka Target

Supermetal now outputs to Kafka with two message formats: a Debezium-compatible envelope for existing pipelines, and a compact native format for new builds.

Under the hood, both are powered by source connectors written from scratch using Apache Arrow in Rust. No JVM. No Kafka Connect. No GC pauses. No snapshot locks, no connector state in Kafka topics.

Initial syncs that take hours or days with Debezium complete in a fraction of the time.

Capabilities:

JSON and Avro serialization with Confluent Schema Registry support
Support for Debezium configuration options including tombstones, headers, decimal/binary/time precision handling, and transaction metadata
Optional transactional writes aligned to source database transaction boundaries

Debezium Compatible Format

{
    "payload": {
        "before": null,
        "after": {
            "id": 1,
            "first_name": "Anne",
            "last_name": "Kretchmar",
            "email": "[email protected]"
        },
        "source": {
            "version": "3.4.1.Final",
            "connector": "postgresql",
            "name": "PostgreSQL_server",
            "ts_ms": 1559033904863,
            "ts_us": 1559033904863123,
            "ts_ns": 1559033904863123000,
            "snapshot": true,
            "db": "postgres",
            "sequence": "[\"24023119\",\"24023128\"]",
            "schema": "public",
            "table": "customers",
            "txId": 555,
            "lsn": 24023128,
            "xmin": null
        },
        "op": "c",
        "ts_ms": 1559033904863,
        "ts_us": 1559033904863841,
        "ts_ns": 1559033904863841257
    }
}

Supermetal Format

{
    "id": 1,
    "first_name": "Anne",
    "last_name": "Kretchmar",
    "email": "[email protected]",
    "_sm_version": 1559033904863841257,
    "_sm_deleted": false
}

Correctness

Supermetal prioritizes correctness above all. Our testing includes end-to-end type fuzzing from source to target database and sqllogictest style test suites, more details published soon.

We run source databases through Supermetal into a target database, then assert the replicated data matches exactly. For Kafka, we run Debezium alongside Supermetal and compare the output. The results are often byte-for-byte identical and always semantically equivalent, accounting for minor serialization differences.

Performance

Supermetal separates compute from IO. Per message serialization to JSON or Avro is CPU intensive. By moving serialization batches to a work-stealing pool, we use all available CPU cores for compute while IO proceeds independently.

Benchmark: Postgres to Kafka

Snapshot Throughput

10 tables × 10M rows each · Peak 1.9M rows/sec

Duration1:50

Rows108M

CDC Latency Under Load

p100 latency <1s up to 15K rows/sec

p100 Latency<1s

Target Load

WAL Throughput

p100 Latency

Sub-second latency up to 15K rows/sec. Above 20K/sec, single-threaded Postgres logical decoding saturates. The Breakdown view shows Read Latency (the time delta between source commit and arrival at Supermetal) accounts for 98% of the end to end latency.

Get started in minutes

curl -fsSL https://trial.supermetal.io/install.sh | sh

iwr -useb https://trial.supermetal.io/install.ps1 | iex

Direct download

Questions? Check out our docs or reach out to us.