Gerador de upsert

Upsert no ClickHouse: INSERT into ReplacingMergeTree + FINAL

Nomeie tabela, chave e colunas e receba a instrução que o ClickHouse realmente aceita, com uma guarda para que uma linha antiga nunca sobrescreva uma nova. Abaixo: como a mecânica funciona, o que quebra e como a Datrise carrega o ClickHouse de forma incremental.

As notas técnicas desta página estão em inglês.

Gere a instrução

Instrução · INSERT into ReplacingMergeTree + FINAL

-- Declare the target once; the ORDER BY key is the dedupe key and
-- the version column decides which duplicate survives a merge:
--   ENGINE = ReplacingMergeTree(updated_at) ORDER BY (id)

INSERT INTO deals (id, name, stage, amount, owner_id, updated_at)
SELECT id, name, stage, amount, owner_id, updated_at
FROM deals_staging;

-- Reads must ask for the collapsed view; merges are asynchronous.
SELECT id, name, stage, amount, owner_id, updated_at
FROM deals FINAL;

Como o upsert funciona no ClickHouse

ClickHouse does not upsert on write. The idiom is a ReplacingMergeTree table: you INSERT every version of a row, and background merges later collapse duplicates that share the ORDER BY key, keeping the one with the highest version column (updated_at in the generated statement). Until a merge runs, both versions exist, so reads use FINAL or argMax to see the current one.

That trade is what makes ClickHouse fast at ingest: inserts are append-only and cheap, and deduplication is deferred. Insert in large batches, thousands of rows at a time or with async_insert enabled, because each INSERT creates a data part and a table with too many parts throttles writes. Partition by month and order by (key, updated_at) so range scans and merges stay local.

Antes de executar

There is no UPDATE-on-insert: ReplacingMergeTree keeps every version until a background merge collapses duplicates by ORDER BY key, keeping the row with the highest version column.
Query with FINAL (or GROUP BY key with argMax) to see one row per key before merges catch up. OPTIMIZE TABLE … FINAL forces a merge but rewrites the whole part; do not run it per batch.
Insert in large batches (thousands of rows, or async_insert=1). Each small INSERT creates a part, and too many parts throttles the table.

Perguntas frequentes

Why do I still see two rows for one key?

Merges are asynchronous. Query with FINAL, or GROUP BY key and take argMax(column, updated_at), until the background merge collapses them.

Should I run OPTIMIZE TABLE … FINAL after each load?

No. It forces a full merge of every part and is expensive on large tables. Let merges run in the background and read with FINAL.

How do deletes work?

ReplacingMergeTree accepts an is_deleted column (ClickHouse 23.2+): insert the row with is_deleted = 1 and a newer version, and merges drop it. Lightweight DELETE also works but marks rows rather than removing them immediately.

O mesmo gerador para outros destinos

Pule a escrita do merge

A Datrise entrega entidades de CRM e SaaS no ClickHouse com exatamente esta mecânica, uma marca d'água sobre updated-at e colunas tipadas, então a instrução acima é a que roda por você. Entre na lista de espera para acesso antecipado.

Explorar o catálogo de integrações