DatriseAI-first ETL
ENESPT

Upsert generator

Upsert into ClickHouse: INSERT into ReplacingMergeTree + FINAL

Name your table, key and columns and get the statement ClickHouse actually accepts, with a guard so an older row never overwrites a newer one. Below it: how the mechanic works, what breaks, and how Datrise loads ClickHouse incrementally.

Generate the statement

The URL updates as you type; share it to hand someone the exact form.

Statement · INSERT into ReplacingMergeTree + FINAL

-- Declare the target once; the ORDER BY key is the dedupe key and
-- the version column decides which duplicate survives a merge:
--   ENGINE = ReplacingMergeTree(updated_at) ORDER BY (id)

INSERT INTO deals (id, name, stage, amount, owner_id, updated_at)
SELECT id, name, stage, amount, owner_id, updated_at
FROM deals_staging;

-- Reads must ask for the collapsed view; merges are asynchronous.
SELECT id, name, stage, amount, owner_id, updated_at
FROM deals FINAL;

How the upsert works in ClickHouse

ClickHouse does not upsert on write. The idiom is a ReplacingMergeTree table: you INSERT every version of a row, and background merges later collapse duplicates that share the ORDER BY key, keeping the one with the highest version column (updated_at in the generated statement). Until a merge runs, both versions exist, so reads use FINAL or argMax to see the current one.

That trade is what makes ClickHouse fast at ingest: inserts are append-only and cheap, and deduplication is deferred. Insert in large batches, thousands of rows at a time or with async_insert enabled, because each INSERT creates a data part and a table with too many parts throttles writes. Partition by month and order by (key, updated_at) so range scans and merges stay local.

Before you run it

  • There is no UPDATE-on-insert: ReplacingMergeTree keeps every version until a background merge collapses duplicates by ORDER BY key, keeping the row with the highest version column.
  • Query with FINAL (or GROUP BY key with argMax) to see one row per key before merges catch up. OPTIMIZE TABLE … FINAL forces a merge but rewrites the whole part; do not run it per batch.
  • Insert in large batches (thousands of rows, or async_insert=1). Each small INSERT creates a part, and too many parts throttles the table.

Questions people ask

Why do I still see two rows for one key?

Merges are asynchronous. Query with FINAL, or GROUP BY key and take argMax(column, updated_at), until the background merge collapses them.

Should I run OPTIMIZE TABLE … FINAL after each load?

No. It forces a full merge of every part and is expensive on large tables. Let merges run in the background and read with FINAL.

How do deletes work?

ReplacingMergeTree accepts an is_deleted column (ClickHouse 23.2+): insert the row with is_deleted = 1 and a newer version, and merges drop it. Lightweight DELETE also works but marks rows rather than removing them immediately.

The same generator for other destinations

Skip writing the merge at all

Datrise lands CRM and SaaS entities into ClickHouse with this exact mechanic, a watermark on updated-at, and typed columns, so the statement above is what runs on your behalf. Join the waitlist to get early access.

Browse the integration catalog