BigQuery and Databricks are new job destinations in 1.7, alongside S3, SFTP and email. A destination doesn't ask for a hostname or a key of its own, it points at a connection you've already set up under Databases and reuses it.
Before you start
- A working BigQuery or Databricks connection (see the connect tutorials). Pipelines tier.
Steps
- Click the + button next to Destinations.
- Under TYPE, choose BigQuery or Databricks.
- Pick your connection in Connection.
- Enter the Target Table (e.g.
dataset.table_name). - Choose Write Mode: Append (INSERT) or Replace (TRUNCATE + INSERT).
- Name it and click Save Destination.
- When scheduling a job, pick this destination under Database.
Replace runs TRUNCATE then INSERT, which is a destructive operation on the target table. If you're not sure whether Append or Replace is right, start with Append; you can switch a destination's write mode later without losing the connection.
A destination is a saved shape, not a live pipe: it remembers the connection, the target table and the write mode, so setting up the same delivery twice for two different jobs is a matter of picking the same destination from a list rather than filling in the fields again. That's the same idea Reusable Destinations covers for every destination type, not just warehouse tables.
Worth deciding up front whether the target table is one this job owns outright or one other processes also write to. Replace is fine for a table only this job populates, a nightly snapshot for example, but it will erase anything another job or a person wrote there between runs. Append is the safer default whenever more than one thing might be writing to the same table.
The account or service principal behind the connection needs write access on the target, which is a separate grant from whatever lets it read. BigQuery Data Editor and Databricks MODIFY are the roles to check first if a job that used to only query now fails once you point a destination at the same connection.
If you need something more than a plain append or a full replace, a merge that inserts new rows and updates matching ones on a key column, that's what Data Sync is for rather than a job destination. A job destination is deliberately simple: two write modes, no field mapping, no match key. Reach for Data Sync once the delivery needs to be smarter than that.
If something goes wrong
| If you see | Fix |
|---|---|
| No BigQuery connections available. Add a BigQuery connection under Databases first. | Add the connection first (tutorial 1). |
| Job fails on write | Give the account write access (BigQuery Data Editor, or MODIFY in Databricks). |
| Rows disappeared | Replace empties the table first. Use Append to keep history. |