Catch bad Databricks data before anyone else.

Before you start

A working Databricks connection with a running SQL warehouse, and QueryFlow Pipelines.

Step by step

1. Run a data quality check as a query in the SQL Editor against your Databricks connection.
2. Click Watch this on the result.
3. Choose Row count or A value depending on what the check returns.
4. Set a Condition that represents “bad,” like Greater than 0.
5. Set Check every, pick Destinations, and Save.

A worked example: catching duplicate keys

On a Databricks table that should have exactly one row per customer:

SELECT COUNT(*) AS duplicate_customers
FROM (
  SELECT customer_id
  FROM main.crm.customers
  GROUP BY customer_id
  HAVING COUNT(*) > 1
);

Watch A value on duplicate_customers, Condition Greater than 0, Check every 1d. A dedup bug in an upstream sync shows up the next check instead of surfacing weeks later as a mismatched customer count in a board deck.

A second worked example: an out-of-range value

For a metrics table where a percentage column should never exceed 100:

SELECT COUNT(*) AS bad_rows
FROM main.analytics.daily_metrics
WHERE conversion_rate > 100 OR conversion_rate < 0;

Same pattern, A value, Greater than 0, whatever cadence matches how often the table refreshes.

Check it worked

Run preview confirms the check currently returns a clean baseline before you save. Check now afterward confirms a real check fires and the destination receives it.

Troubleshooting

A third worked example: nulls after a schema change

A recent migration added a required region column, but some rows written by an older job version still come through null:

SELECT COUNT(*) AS missing_region
FROM main.sales.orders
WHERE region IS NULL AND order_date >= current_date() - INTERVAL 1 DAYS;

Watching this catches straggler writes from an un-updated job long after the migration itself is done, which a one-time backfill check wouldn't.

BigQuery version

The same pattern applies on BigQuery. See catch bad BigQuery data first for the equivalent setup.