A build.yml
file specifies the structure, type, and names for package contents.
Below is the general syntax of the build.yml
file:
contents:GROUP_NAME:# transform: optional. applies recursively to child nodes. If 'transform'# is omitted, Quilt will select a transform by file extension, falling# back on the "id" transform, which stores the raw data.transform: {id | csv | tsv | ssv | xls | xlsx}# kwargs: optional. applies recursively to child nodes.kwargs:# Each KEYWORD: VALUE pair is passed directly to the transform function.# For example, with csv this is pandas.read_csv.KEYWORD: VALUE # keyword arguments for transform function/methodDATA_NAME:# file: required. Relative path from base of package dir.file: FILE_PATH# transform: optional. If given, overrides GROUP_NAME's transform.transform: {id, csv, tsv, ssv, xls, xlsx} # optional# if transform is omitted,kwargs: # overrides GROUP_NAME's kwargs for this node.KEYWORD: VALUE # optional. keyword arguments toGLOB_GROUP: # Container node for nodes created from matching filesQUOTED_GLOB_PATH: # standard glob path, such as "*.csv"transform: csv # set transform for all matched files (optional)kwargs: # set kwargs for all matched files (otional)KEYWORD_ARG: VALUE
Example build.yml
:
contents:data_example: # create a node named 'data_example'transform: csv # Read files using csv readerkwargs: # optionalheader: # optional: no header rowsep: "," # optional: set field separatorchild:file: data/foo.txt # parsed as CSV, no headeranother_child:file: data/bar.txt # parsed as CSV, no headerchild_from_elsewhere:# parsed as TSV (from kwargs set here) no header (from parent's kwargs)file: data2/bar.txtkwargs:sep: '\t' # Use tab as separator.glob_example: # create a node named 'glob example'# assuming these files exist: 'somedir/foo.xls', 'somedir/subdir/bar.csv', 'somedir/baz.tsv''somedir/**/*': # create nodes 'foo', 'bar', and 'baz'# matched files parsed as CSV, no headerglob_example_2: # create a node named 'glob_example_2'# assuming these files exist: 'chars/bella.txt', 'chars/edward.txt', 'chars/old/esme.txt''chars/*.txt':transform: tsv # create nodes 'bella' and 'edward'# matched files parsed as TSV, no header
file
- required for leaf nodes; specifies where source file lives on disk
transform
- specifies how the file will be parsed
kwargs
- these options are passed through to the parser (usually pandas.read_csv
so that users can skip lines, type columns, specify delimiters, and much more)
checks
- experimental data unit tests
environments
- experimental environments for checks
package
- experimental source specifier includes an existing package or sub-package in the build tree (see Package Composition)
*?[!]
- any character in this group will initiate glob-style pattern matching
transform
and kwargs
can be provided at the group level, in which case they apply to all descendants until and unless overridden.
By default, quilt build
converts some file types (e.g., csv, tsv) to Pandas DataFrames using pandas.read_csv
. Sometimes, usually due to columns of mixed types, pandas will throw an exception during quilt build
. In such cases it's helpful to include column types in build.yml
by adding a dtype
parameter:
contents:iris:file: iris.datatransform: csvkwargs:header:dtype:sepal_length: floatsepal_width: floatpetal_length: floatpetal_width: floatclass: str
dtype
takes a dict where keys are column names and values are valid Pandas column types:
int
bool
float
complex
str
unicode
buffer
See also dtypes.
If a string containing wildcards is used as a node name, it will be matched against the build directory. The filename of any matching path, minus the extension, will be used as the nodename. As when specifying a single data node, kwargs
and transform
may be used to specify how the file should be read.
The following standard wildcard strings are accepted:
**
- Match current dir and all subdirs, recursively
*
- Match any one or more characters
?
- Match any single character
[X]
(where X is any number of characters) - Match any one character contained in X
[!X]
(where X is any number of characters) - Exclude any one character contained in X
All wildcards except **
act in the current directory alone, so *
does not match subdir/foo/file.ext
, but subdir/foo/*
and **/*.ext
do.
If provided, transform
and any specific kwargs
are used with each matched file. Otherwise, the parent (or default) transform
and kwargs
will be used.
Finally, if matching results in identical node names, the nodes are renamed in a consistent manner (paths are sorted lexicographically), and any duplicate names are numbered. So for files "foo.txt" and "subdir/foo.txt", the result is "foo" (from foo.txt) and "foo_2" (from "subdir/foo.txt"). The naming behavior is consistent across platforms.