Data Engineering / Field notes
Fresh data. But is it ready for a decision?
A green refresh indicator tells a reader that something ran. It does not, by itself, show that every required observation arrived or that the resulting metric is safe to use.
Freshness answers a timing question
Source freshness checks compare the age of loaded data with configured thresholds. They help identify delayed inputs. The expected schedule still needs to reflect the source: an hourly feed and a monthly release should not share the same timing rule.
dbt: source freshness ↗Completeness answers a coverage question
Imagine a simplified daily feed that should contain 24 hourly observations. The newest observation arrives on time, but two earlier hours are missing. A recent timestamp does not resolve those gaps. For real time-series data, derive the expected intervals from the requested window and timezone rules; do not assume every local day has 24 hours.
Consistency answers a modelling question
dbt data tests can check assertions such as unique keys, required values, accepted categories and relationships. Those checks describe specific expectations; passing them does not prove every business rule is correct. Add checks that match the decision the dataset supports.
dbt: data tests ↗Decide what failure means
For a cost comparison, missing prices may make the result unusable. For an operational overview, a clearly marked partial result may still be useful. Define that policy before a failure occurs. A useful publication design validates a candidate dataset before replacing the currently served version and makes the last successful update visible.
Show the reader what they need
A compact status area can answer three questions: when was the data last validated, which period is covered, and are any required checks failing? Detailed logs can serve the maintainer. The reader needs enough context to decide whether to use the result, wait for an update or investigate an exception.