Sources
Sources
Postgres tables, Postgres changes, Snowflake tables and queries, ODBC databases, BigQuery tables and queries, files in S3 or a folder, Google Sheets, and Python sources: the kinds of source and the settings they share.
A source is where data comes from. SQLLoad has eight kinds:
kind |
Reads | Tables | Declared in |
|---|---|---|---|
postgres |
Tables of a Postgres database, with binary COPY |
One per matching source table | sqlload.toml |
postgres_cdc |
A Postgres database’s inserts, updates and deletes, from a logical replication slot, after a first snapshot | One per matching source table | sqlload.toml |
snowflake |
From 0.26.0. Tables and views of a Snowflake database, and query results, unloaded as Parquet | One per matching table, and one per query | sqlload.toml |
odbc |
From 0.27.0. Tables, views and query results of any database with an ODBC driver, read as Arrow | One per matching table, and one per query | sqlload.toml |
bigquery |
From 0.30.0. Tables, views and query results of BigQuery, read as Arrow through the Storage Read API | One per matching table, and one per query | sqlload.toml |
files |
Parquet, CSV, JSON or NDJSON files in S3 or a local folder | One per source | sqlload.toml |
google_sheets |
From 0.27.0. Tabs or ranges of a Google Sheets spreadsheet, typed | One per tab or range | sqlload.toml |
| Python | Whatever a Python function yields: dicts, Arrow data, JSON bytes, HTTP jobs | One per decorated function | Python, under python_paths |
Every source loads into a target: its destination, or the project’s default_target. Settings
that shape tables (write_strategy, unique_key, cursor_column and the rest) can be set on the
source as defaults for all its tables, and per table. See
sqlload.toml for the full list.
Each kind has its own page:
- Postgres: choosing tables and columns, masking, and how tables are read.
- Postgres changes: streaming changes from a replication slot into DuckDB, Snowflake, Databricks or Parquet files, and what to watch on the server. To set one up, on a schedule or continuously, see Capture Postgres changes.
- Snowflake, from 0.26.0: choosing tables, query tables, read windows around the time of the run, and where unloaded files go.
- ODBC, from 0.27.0: connections, drivers, choosing tables, query tables and value sizes.
- BigQuery, from 0.30.0: choosing tables, query tables, types, cost and permissions.
- Files: formats, partitions, which files load, and files the destination reads itself.
- Google Sheets: tabs and ranges, headers, and how cells are typed.
- Python: any API, SaaS or custom source as a table, including REST APIs that run entirely in Rust. Every function is in the Python API.
