Data Quality Overview in Databricks

Brief about Data Quality

Six Dimensions of Data Quality

Many tools and technical documents classify data quality as an aspect of the Six-dimensions Model, as shown below.

While the six dimensions define how data quality is evaluated, practical implementations require technical mechanisms to detect and monitor these issues within the data platform. And the three most common method include:

Rule-based / Constraint Validation: Data is validated against predefined rules or constraints, such as schema checks, NOT NULL conditions, uniqueness, or business logic rules.

Statistical Monitoring: Key dataset metrics (e.g., row count, null ratio, value distribution) are monitored to detect abnormal changes in data behavior. Alerts or anomaly detection can identify unusual patterns that may indicate pipeline failures or upstream issues. In many cases, these metrics act as proxies for data correctness, where significant deviations from historical baselines signal potential data quality problems even when explicit validation rules are not violated.

Data Reconciliation: Data from different systems or pipeline stages is compared to ensure consistency. Examples include row count checks, aggregate comparisons, or source-to-target validation.

In practice, reconciliation is often the most complex form of data quality validation because it requires understanding both the data pipeline and the underlying business logic. Transformations, filtering, and aggregations may change the structure or semantics of the data between stages, so reconciliation checks must account for these changes rather than simply comparing datasets directly.

Implementation Approaches to Data Quality

In practice, data quality systems are often implemented using two complementary approaches: reactive monitoring and proactive validation.

Reactive monitoring (Data observability)

This approach focuses on analyzing data after it has already been written to storage systems such as lakehouse tables. In this approach, monitoring tools periodically scan datasets, compute metrics, and detect anomalies based on historical patterns. This method is useful for identifying unexpected changes in data behavior without requiring modifications to existing pipelines.

However, reactive monitoring detects issues only after the data has already been produced, and it may require additional compute resources to repeatedly scan large datasets.

Proactive validation (Pipeline enforcement)

This approach shifts data quality checks earlier in the data lifecycle by embedding validation logic directly into data pipelines. In this model, pipelines include explicit data quality checks such as schema validation, null checks, or constraint assertions. If a validation rule fails, the pipeline can stop execution and prevent incorrect data from being published.

While this approach can prevent errors before they propagate downstream, it requires pipeline developers to adopt data quality libraries or frameworks and implement validation logic within their code.

In practice, modern data platforms often combine both approaches: pipeline-level validation prevents obvious data errors, while monitoring systems observe datasets over time to detect unexpected anomalies.

Validation Perspectives

Data quality validation can generally be viewed from two complementary perspectives: rule-based validation and data observation.

Rule-based validation (Rules Engine)

This perspective focuses on validating data against predefined rules or constraints. These rules usually represent business logic or structural requirements, such as NOT NULL conditions, uniqueness, referential integrity, or domain constraints. Rule-based validation is effective for enforcing explicit correctness requirements and preventing invalid records from entering the dataset.

Metrics-based monitoring (Data observation)

This perspective focuses on observing dataset-level metrics and identifying abnormal patterns over time. Instead of validating individual records, the system monitors indicators such as row count, null ratio, value distribution, or freshness to detect potential data quality issues.

Relationship Between Rules and Metrics

In practice, these two approaches are closely related. Many simple validation rules can be expressed as metrics (for example, a NOT NULL rule can be monitored through the null ratio of a column). However, metrics-based monitoring cannot fully replace rule-based validation because some business constraints require explicit logical checks.

At the same time, metrics-based approaches are often more effective for detecting unexpected issues that cannot be predefined as rules. By analyzing historical patterns and baseline behavior, statistical monitoring can identify anomalies such as sudden volume drops/increases, distribution shifts, or unusual data patterns that traditional rule engines may fail to detect.

Implementing Data Quality in Databricks

Built-in Data Quality Capabilities in Databricks

Databricks provides several built-in capabilities that can be used to implement data quality checks within the Lakehouse platform.

Table Constraints

Databricks supports standard SQL constraint to automatically identify a data set that contains errors and prevent it from being inserted into a table. Two types of constraints are supported with Delta tables:

• NOT NULL: prevents any NULL values from being inserted into the column.
• CHECK: requires that the specified Boolean expression must be true for each input row.

While table constraints provide a simple mechanism for enforcing basic data validation, they also introduce several limitations in practice.

First, business rules may evolve frequently. Embedding such rules directly into table definitions can make them harder to maintain and adapt when business logic changes over time.

Second, strict constraints can prevent data from being written to a table if validation conditions are not satisfied. While this behavior helps ensure correctness, it may also reduce flexibility in a Lakehouse environment where ingesting imperfect data and handling validation within downstream processing is sometimes preferred.

For more detail: https://learn.microsoft.com/en-us/azure/databricks/tables/constraints

Delta Live Tables (DLT) / Spark Declarative Pipelines (SDP)

Delta Live Tables (DLT) (whose underlying concepts have been contributed to Apache Spark as Spark Declarative Pipelines – SDP) provides a number of features for building and managing data pipelines in Databricks. Among these features, SDP includes decorators such as @dp.expect that allow developers to define data quality expectations on DataFrames.

During pipeline execution, these expectations are incorporated into the Spark execution plan. In practice, this is implemented by rewriting the Spark logical plan to include validation expressions, enabling records to be checked as part of the pipeline execution.

However, these validations are defined directly within the pipeline logic. As a result, their implementation and maintenance are largely controlled by the pipeline developers, and platform-level systems have limited ability to enforce or manage these checks independently.

This mechanism aligns with the proactive (pipeline-level) data quality approach described earlier, where validation rules are embedded directly into data pipelines to detect and block incorrect data before it is published.

For more detail: https://learn.microsoft.com/en-us/azure/databricks/ldp/expectations

Anomaly Detection

Databricks provides built-in anomaly detection capabilities in Unity Catalog for monitoring data quality across tables within a schema. This feature automatically analyzes historical table activity to identify unusual changes in data behavior.

The monitoring primarily focuses on signals such as freshness (whether a table is updated according to its expected cadence) and completeness (unexpected changes in row volume). In some cases, additional indicators such as null ratios per column may also be evaluated.

The monitoring process is executed through background jobs running on serverless compute. These jobs analyze historical commit information and data volume patterns to build a baseline of expected behavior for each table. Databricks applies an intelligent scanning strategy, where the monitoring frequency is automatically determined based on factors such as table update cadence and downstream usage patterns.

This capability is fully managed by Databricks. Monitoring is enabled at the schema level in Unity Catalog, meaning all tables within the schema are included by default, with the option to exclude specific tables if necessary. The monitoring jobs are executed using serverless compute managed by Databricks. As a result, users have limited visibility or control over the execution schedule and compute resources used, which can make the associated cost model less transparent.

For more detail: https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/data-quality-monitoring/anomaly-detection/

Data Profiling

Databricks provides data profiling capabilities within Data Quality Monitoring to compute dataset statistics and track data characteristics over time. Profiling allows users to define profiles (a set of metrics to compute) attached to specific tables. Databricks periodically refreshes these profiles by running compute jobs to calculate the defined metrics.

Different profiling modes are supported depending on the type of dataset being monitored. For example, snapshot profiling performs a full scan of the table to compute metrics for the entire dataset, while time-series profiling computes metrics over recent time windows and therefore requires the dataset to contain a timestamp column. Another mode, inference profiling, is designed for monitoring machine learning inference tables.

The computed metrics are stored in two Delta tables: a profile metrics table containing summary statistics and a drift metrics table containing information about changes in data distribution. Databricks automatically generates dashboards based on these tables to visualize trends in data quality over time. These tables can also be queried directly or used to configure alerts for proactive monitoring. Overall, this feature aligns with the data observability perspective described earlier.

However, there are several considerations. Profiles are defined per table, which means monitoring many tables may require multiple profiling jobs. Each profile execution uses serverless compute to calculate metrics, which may lead to increasing costs when applied at scale.

In addition, Databricks generates a relatively large set of metrics by default. In many practical scenarios, only a subset of these metrics may be required for monitoring purposes. As a result, computing unnecessary metrics could introduce additional compute overhead.

For more detail: https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/data-quality-monitoring/data-profiling/

External Data Quality Tools Integrated with Databricks

In addition to built-in capabilities, several external data quality frameworks can be integrated with Databricks to provide more advanced validation and monitoring features.

Deequ

Deequ is an open-source Apache Spark library developed by AWS that lets users define data quality rules and run quality checks on large datasets using Spark jobs. As a library rather than a standalone system, it supports various implementation patterns, such as embedding validation in runtime data pipelines or running separate Spark applications to monitor dataset quality periodically.

Deequ allows developers to define validation rules via a programmatic API or a declarative rule language (DQDL). These rules translate into analyzers that compute dataset statistics and data quality metrics. Deequ uses these metrics to perform rule validation, anomaly detection, and automatic constraint suggestion.

Deequ includes performance optimizations for large datasets. It can execute multiple analyzers in a single dataset scan (scan sharing), reuse previously computed metrics through its metrics repository, and support incremental computation for partitioned or growing datasets.

Deequ is only a library, not a complete data quality platform, so organizations must add components like rule management, scheduling, and alerting to build a full framework. It primarily supports batch validation on existing Spark DataFrames and lacks native support for proactive validation during ingestion or real-time streaming validation. In addition, the project does not always track the latest Apache Spark releases, which may lead to compatibility issues with some newer Databricks Runtime versions.

Great Expectations

Great Expectations (GE) is a framework that takes a different approach than Deequ. Instead of metrics-first, GE follows an expectation-first model, where developers define high-level data expectations such as column completeness, value ranges, or schema structure through declarative configurations.

These expectations are intentionally closer to natural language and business semantics than low-level rules. Rather than expressing validation as technical conditions or metric thresholds, expectations encode what “good data” should look like in a more human-readable and domain-aligned way. As a result, each expectation carries richer context — including intent, description, and validation outcome — allowing the system to produce more informative outputs such as detailed validation reports, documentation pages, and data quality summaries.

GE provides a library to integrate quality checks into pipeline runtime reactively or to run separate jobs. It also includes built-in UI capabilities for visualizing validation results and generating documentation, which helps teams communicate dataset reliability more effectively.

For developers, GE is relatively user-friendly due to its declarative and semantic design. However, this abstraction comes with trade-offs. On Databricks, GE is not cost-efficient because each expectation often results in a separate dataset scan. In addition, its expectation-centric model is less suited for anomaly detection or use cases that rely on historical behavior and statistical baselines.

Soda

Soda is a data quality platform that allows users to define dataset checks using SodaCL, a declarative language for specifying rules such as row counts, null ratios, or value constraints. When executed on Databricks or other SQL-based systems, these checks are translated into SQL queries that compute metrics and validate the defined conditions.

In the open-source version, scan results and profiling metrics are typically stored locally (for example as JSON files), while centralized monitoring, dashboards, and alerting are mainly provided through Soda Cloud. Scan execution is usually orchestrated through external workflow tools such as Airflow or Lakeflow.

While SodaCL is primarily designed as a validation DSL rather than a general-purpose metric computation layer, Soda still provides lower-level APIs that can be used to compute metrics directly. However, when relying mainly on SodaCL, metric computation often becomes implicit or tied to validation logic.

For example, placeholder checks (such as sum(column) > -999999999) may be introduced to force metric computation during scans. In addition, for more complex or custom metrics, separate Spark jobs are sometimes used to compute these metrics explicitly, with SodaCL then consuming the results for validation.

Custom Spark-based Data Quality Checks

In some scenarios, built-in capabilities or external frameworks may not fully cover complex data quality requirements. In these cases, teams may implement custom Spark-based validation logic directly within their data platform.

These custom checks can follow two common approaches. The first is proactive validation, where data quality checks are embedded directly into data pipelines. For example, additional validation tasks may be added to ETL workflows to verify constraints such as row counts, null ratios, schema consistency, or business rules before publishing data to downstream tables. In some cases, teams may also use data quality libraries to implement these checks in a more structured way.

The second approach is reactive monitoring, where dedicated quality jobs periodically scan datasets to compute metrics, profile data characteristics, and detect anomalies over time. These jobs typically calculate indicators such as row counts, completeness, distribution statistics, or freshness metrics and generate alerts if abnormal changes are detected.

In practice, custom data quality implementations often combine both approaches: pipeline-level validation prevents obvious errors from propagating downstream, while periodic monitoring jobs provide broader observability across datasets.

For example, we implement a dedicated data quality task within each pipeline that computes dataset metrics and publishes them to a centralized metrics table. These metrics are then aggregated and monitored across pipelines to provide a simple unified view of data quality signals across the platform.

Comparison of Data Quality Approaches

Summary

Overall, besides providing an overview of common data quality concepts, this document highlights two key perspectives for implementing data quality systems: rule-based validation and metrics-based monitoring, as well as two complementary implementation approaches: proactive validation and reactive monitoring.

The tools discussed in this document represent different implementations across these two dimensions rather than fundamentally different approaches. Some tools focus primarily on rule validation (e.g., table constraints, Great Expectations), while others emphasize metrics computation and anomaly detection (e.g., Databricks profiling, Soda, or Deequ).

In practice, building a complete data quality platform - including centralized rule management, metrics storage, monitoring, and alerting - often requires combining multiple tools or implementing additional platform components, since most existing solutions only address part of the overall data quality lifecycle.

References

https://www.databricks.com/discover/pages/data-quality-management#principles-of-data-quality