Checks
Checks are data integrity tests that are defined in build.yml
. They are run at package build time to ensure that all consumers of the data package only receive data that comply to the given checks.
Checks can be used to prevent model drift and data deployment errors that result from using data that do not fit an expected profile.
Known issues
Syntax
Checks are defined in a top-level dictionary called
checks:
qc.data
is an automatic variable that contains the node's data in pandas data frameThe full pandas expression syntax is supported
Standard Python can be inlined with YAML's
|
operator (see below)
Functions (qc.*
)
qc.*
)Signature | Description |
| Check that |
| Checks that all column values are in the list (and vice versa), or calls a lambda on the column |
| Print line numbers of rows that match |
| Check that column values fall within [ |
| Check that all column values match |
| Check that all column values contain substing |
| Not yet supported. Check that all column datetimes conform to |
COL_REGEX
is a string literal or regular expression that matches one or more columns; the corresponding check is applied to each matching column
Example
Source data: sales.xls from Tableau Community
Last updated