Duckle

Quick links

Get started

• What is Duckle?
• What's new in v0.5.6
• Quickstart (60 s)
• Download / Install
• Build from source
• Run your first pipeline

Use the product

• Meet Duckie (AI)
• How to use Duckle
• Recipes / examples
• In-app Git (GitHub/GitLab)
• Workspace + Git flow
• Schedules
• Server deployment
• MCP server: connect Claude, Cursor or any agent
• Connection management
• Context variables

Reference

• Capabilities matrix
• Sources
• Transforms
• Sinks
• Data quality
• Custom code
• Control flow
• Advanced settings
• Engines
• Configuration

Resources

• Architecture
• Clean data for AI
• Performance tips
• FAQ
• Troubleshooting
• CI / CD
• Status
• Roadmap
• Contributing
• Sponsor Duckle
• License
• Releases
• Roadmap doc
• Contributing doc

What is Duckle?

A visual data pipeline studio that runs on your laptop. Drag sources, transforms, validators, and sinks onto a canvas. Wire them together. Press Run. Duckle compiles the graph to SQL and executes it through a real columnar engine, with live previews, generated SQL on every node, and zero hidden state.

In short: a free, open-source, local-first alternative to hosted ETL platforms like Fivetran and Airbyte, that can also run dbt on DuckDB inside the same tool.

Three things make Duckle different from the heavyweights and the toy ETL tools:

1. An AI assistant that ships in the box. Describe the pipeline you want in English; Duckie writes the JSON and drops it onto the canvas. The model runs locally - no API key, no telemetry, no cloud round-trip.
2. 356 components ready at install time. Files, lakehouses, SQL databases, warehouses, NoSQL, vector DBs, streaming brokers, SaaS REST/GraphQL APIs, even FTP and IMAP - working today, not coming-soon.
3. A self-contained binary you can audit. ~65 MB download. Engines install on first launch. Workspaces are plain files in a folder you choose. Diff them, branch them, ship them.

What's new in v0.5.6

Real-time WebSocket connectors, a geospatial projection toolkit, streaming XML at multi-GB scale, and a scheduler that finally speaks your local time.

• WebSocket source and sink (#192). src.websocket connects to a ws:// or wss:// endpoint, optionally sends a subscribe frame, and collects up to maxMessages frames into rows; snk.websocket pushes each row back out as a text frame. For live market data, sensor streams, and real-time dashboards.
• Geospatial projection toolkit (#188, #189, #190). Define Projection stamps a CRS onto geometry that has none, Reproject Geometry transforms between CRS with the target CRS preserved on the output, and Create Geometry builds a GEOMETRY column from X/Y, WKT, or WKB.
• Dynamic time offsets in templates (#191). ${date+1d}, ${now-2h}, ${timestamp+30m}, and chained forms like ${date+1d-2h}, all resolved against a single run instant so every reference in a run stays consistent.
• Streaming, multi-GB XML (#186). src.xml streams local, sftp://, and https:// inputs with automatic gzip / zip handling, and honours a declared schema instead of buffering the document.
• Salesforce Bulk sink snk.salesforce.bulk (#164). Migration-scale write-back over Bulk API 2.0: DuckDB streams the upstream to CSV on disk and each ≤90 MB part runs the async job lifecycle (create → upload → poll → fetch result sets), so a multi-million-row load never lands in memory.
• Local-time cron and a minute-accurate countdown (#194). Cron expressions now evaluate in the machine's local time zone, so "Every day at 03:00" means 03:00 where you are. The Next countdown shows hours and minutes ("in 1 h 7 min"), and a 5-field cron is accepted instead of silently never firing.
• OAuth client credentials for any REST source (#195). Token minting is no longer Salesforce-only: give any REST connector a token URL and it mints a fresh access token per run. Suits a Xero Custom Connection, which needs HTTP Basic client auth.
• JSON transform xf.jq (#173) for per-row jq over a JSON column, and CRS-aware measurements (#177) that pick planar or spheroid math from the column's own CRS.

Fixes: saved S3 and other non-Salesforce connection references resolve at run time (#185), Duckie honours a configured custom AI endpoint (#183), the Linux window stays resizable after un-maximize (#182), an externally set DUCKLE_DUCKDB_BIN wins over the bundled binary (#179), and a zero-row result no longer collapses to a lone json column (#170). macOS engine signing was also hardened so llama-server launches from a downloaded build (#89) - reports from macOS users welcome, as it could not be exercised on the Windows dev box.

Full notes: see the v0.5.6 release.

Meet Duckie - the local AI pipeline assistant

Describe what you need. Duckie writes the pipeline.

The sidebar on the right is Duckie AI Assistant - powered by Qwen 2.5 Coder 1.5B running through llama.cpp, downloaded once (~1.1 GB) and then run entirely on your CPU. Ask in plain English; Duckie streams back a valid Duckle pipeline definition. One click drops it onto the canvas, ready to inspect, tweak, and run.

Why Duckle is different

Benchmark

The most common job in data engineering: load a 20M-row CSV into DuckDB. One identical 2.49 GB file (20M rows of TPC-H lineitem, 16 typed columns), every tool measured at its best configuration, wall-clock time to land the data as a table.

How it was measured

• Machine: Intel Core i7-13650HX (14C / 20T), 24 GB RAM, NVMe SSD, Windows 11, DuckDB 1.5.4. Duckle, dlt and ingestr ran here.
• Best config per tool: dlt used the Arrow plus parquet loader path; Talend and Informatica used their bulk output connectors at max config (their default row-by-row sinks were 5-7x slower). Nothing was left on a slow default to pad the gap.
• Talend and Informatica ran on a separate 8 GB VPS, so per-tool peak RAM was not captured for those two.
• Airbyte (source-file to destination-duckdb) is scaled from measured 2M and 5M runs at a steady ~18k rows/s, and it also needs an always-on ~8 GB platform just to start.
• Wall-clock time, peak working-set of the whole process tree.

Why Duckle is this fast: its 15.69s sits right on top of raw DuckDB's own load floor (~16s to fully parse and write all 20M typed rows into an on-disk table). Duckle wraps the engine with pipelines, connectors, and a UI, then gets out of its way. That is the entire design goal. A read-only scan or aggregate over the same CSV is far faster still; this benchmark measures the heavier "materialize it as a table" job that every ETL tool here performs.

Status

Duckle is in public beta. The visual designer, the DuckDB execution engine, the scheduler, the cloud connectors, and the Duckie AI assistant all work today and are covered by 170+ integration tests across Linux, macOS, and Windows. The catalog is still growing and APIs may evolve before 1.0, but the day-to-day surface is stable enough for real work.

Scope, stated plainly: Duckle is a single-machine, embedded studio. If you outgrow one box, point Duckle's output at the system that scales (a warehouse, an object store, a lakehouse). It will not pretend to be a cluster.

The component palette ships 363 nodes so the roadmap is visible in the product itself:

• 356 available runs on the DuckDB engine today
• 3 preview is configurable in the designer (drag, wire, set properties); execution is being wired engine-by-engine
• 15 planned is reserved in the palette but not yet executable - see docs/roadmap.md

Screenshots

Real pipelines, built and run in Duckle - not mockups.

Capabilities

Duckle is not a CSV tool with extras. It reads a broad set of formats and sources, ships a deep transform library, and writes to files, databases, object storage, vector DBs, message buses, and email.

Sources (101 available)

For CSV / TSV sources, the Schema panel accepts an optional per-column Format (a strptime token string such as %d/%m/%Y) on Date and Timestamp columns. Several date columns can each parse a different layout in one read - the column is read as text and re-parsed with its own format, working around DuckDB's single global date format. A value that does not match its format becomes null rather than failing the run. Set a Date or Timestamp column's Format to excel to convert Excel day-serials correctly. CSV sources also surface ignoreErrors (skip unparseable rows) and nullPadding (pad short rows with nulls) toggles in the GUI.

For JSON sources, a Format selector picks how the file is read (auto / array / JSON Lines / object), and a skip malformed records toggle drops records that fail to parse instead of failing the run.

Transforms (130 available)

All 6 AI transforms ship today. Three need a model API (LLM, Classify, Embeddings) and ride the apiKey-in-props pattern; three are pure-local (Chunk, PII Redact, Dedupe).

Data quality (27 available)

Validators split their input: passing rows continue on the main port, failures route to a reject port you can sink, count, or inspect.

Custom code (6 available)

Sinks (63 available)

Database sinks support an optional dead-letter (validate-before-insert) step: rows that do not match the declared column types are split off to a dead-letter file (parquet / csv / json) and only the clean rows are inserted.

Control flow (18 available)

Advanced settings (per-node)

Every node has an Advanced tab with fields the engine honours at run time:

Orchestration and workspace

Clean data before it reaches your AI

Models inherit the quality of their inputs. RAG indexes, embedding stores, and training sets quietly accumulate duplicates, nulls, malformed rows, mixed encodings, and inconsistent schemas. Duckle is built to scrub that data before it lands in a vector store:

• Deduplicate with exact Distinct, Uniqueness, and Fuzzy Deduplicate (Jaro-Winkler / Levenshtein); use Record Match to find near-duplicate pairs with a similarity score
• Semantic dedupe with xf.ai.dedupe over a precomputed embedding column
• Profile + describe every column up front (Column Profile, Describe, Histogram) so issues surface before they reach a model
• Validate and filter malformed, empty, or out-of-range records and route failures to a reject port
• Normalize types, encodings, casing, and null handling across messy sources (Standardize, Cast, regex / string transforms)
• Redact PII (emails, phones, SSNs, credit cards) via xf.ai.pii before embedding
• Chunk + embed long text via xf.ai.chunk -> xf.ai.embed for RAG indexing
• Classify rows with an LLM (xf.ai.classify constrains the model to one of N user-supplied categories)
• Retrieve with both halves of hybrid search, locally, no model API required: Vector Similarity Search (cosine / L2 / inner product) and Full-Text Search (BM25)
• Land it in your store - pgvector ships, and Pinecone, Qdrant, Weaviate, Milvus all have working sinks that POST batches through each vendor's HTTP API

Engines

Duckle ships a thin shell and installs its engines on first launch.

First-launch extension pre-fetch

When the installer downloads the DuckDB CLI it also pre-fetches the extensions Duckle uses, with per-extension progress, so the first time you touch a Postgres source or an Iceberg table there is no surprise network hop mid-pipeline:

httpfs (S3 / GCS / HTTP), `azu