Work with packages

Read a package
Packages may contain data of any size or type. A given package instance--specified by a hash, tag, or version--is immutable for reproducibility.
Install a data package from  user uciml
1
$ quilt install uciml/iris
Copied!
Note: most Quilt commands are available both on the command line and in Python.
You can install a package as follows:
1
import quilt
2
quilt.install("uciml/iris")
Copied!
Import the package
1
$ python
2
>>> from quilt.data.uciml import iris
3
>>> iris
4
<PackageNode 'Users/YOU/quilt_packages/uciml/iris'>
5
raw/
6
tables/
7
README
8
>>> iris.tables.bezdek_iris() # this is a pandas DataFrame
9
    sepal_length  sepal_width  petal_length  petal_width  label
10
0  5.1           3.5          1.4           0.2          Iris-setosa
11
1  4.9           3.0          1.4           0.2          Iris-setosa
12
2  4.7           3.2          1.3           0.2          Iris-setosa
13
...
Copied!
Read more about the uciml/iris package on its landing page, or browse packages on Quilt.
Edit a package
Start by installing and importing the package you wish to modify:
1
import quilt
2
quilt.install("uciml/wine")
3
from quilt.data.uciml import wine
Copied!
Alternatively, you can build an empty package and import it for editing:
1
import quilt
2
quilt.build("USER/FOO")
3
from quilt.data.USER import FOO
Copied!
Update: As of version 2.9.9, easiest method to edit a package is to use subpackage build and push.
Edit dataframe nodes
Use the Pandas API to edit existing dataframes:
1
df = wine.tables.wine()
2
hue = df['Hue']
3
df['HueNormalized'] = (hue - hue.min())/(hue.max() - hue.min())
Copied!
Add package nodes
Use the _set helper method on the top-level package node to create new groups and data nodes:
1
import pandas as pd
2
df = pd.DataFrame(dict(x=[1, 2, 3]))
3
# insert a dataframe at wine.mygroup.data()
4
wine._set(["mygroup", "data"], df) 
5
# insert a file at wine.mygroup.anothergroup.blob()
6
wine._set(["mygroup", "anothergroup", "blob"], "localpath/file.txt") #
Copied!
Delete package nodes
Use del to delete attributes:
1
del wine.raw.wine
Copied!
Edit metadata
Use the _meta attribute to attach any JSON-serializable dictionary of metadata to a group or a data node:
1
wine.mygroup._meta['foo'] = 'bar'
2
wine.mygroup._meta['created'] = time.time()
Copied!
Data nodes contain a built-in key _meta['_system'] with information such as the original file path. You may access it, but any modifications to it may be lost.
Persist changes
At this point, your changes only exist in memory. To persist your changes, read on to learn about build and push.
Build a package
Building a package creates a local bundle of serialized data. $ quilt ls displays your local packages and their location on disk.
There are three ways to build data packages with Quilt:
1.Implicitly with quilt build USR/PKG DIRECTORY. Implicit builds are good for taking quick snapshots of unstructured data like images or text files. Quilt serializes columnar formats formats (xls, csv, tsv, etc.) to data frames; all other files will be copied "as is".
2.Explicitly with quilt build USR/PKG FILE.YML. Explicit builds allow fine-grained control over package names, types, and contents.
3.One the fly, in Python
Each of the above methods for building packages is supported in Python and on the command line.
Implicit builds
To implicitly build a package of unserialized data:
1
quilt build USR/PKG DIRECTORY
Copied!
Everything in DIR and it's subdirectories will be packaged into USR/PKG.
To publish your package:
1
quilt push USR/PKG --public
Copied!
Users on Individual and Business plans can omit the --public flag to create private packages.
Explicit builds
Explicit builds cue from a YAML file, conventionally called build.yml.
1
quilt build USR/PKG BUILD.YML
Copied!
build.yml specifies the structure and contents of a package.
quilt generate creates a build.yml file
An easy way to create a build.yml file is as follows:
1
quilt generate DIR
Copied!
The above command creates build.yml and README.md files that you can modify to your liking. A README.md file is highly recommended as it populates your package landing page with documentation. See the API section for more on how README markdown is converted to HTML.
See build.yml syntax for more.
Directory and file naming in quilt generate
Directories and files that start with a numeric character or underscore will be prefixed with the letter n. If a name collision results, the build will fail with an error.
If two files have the same path and root name, but different file extensions (foo.txt, foo.csv), the extensions will be appended as follows: foo_txt, foo_csv. If, after appending, there remains a name collision, the build will fail with an error.
Build on the fly
1
# start with an empty package
2
quilt.build("akarve/foo")
3
# put some data in it
4
import pandas as pd
5
from quilt.data.akarve import foo
6
df = pd.DataFrame(data=[1,2,3])
7
foo._set(['bar'], df)
8
foo.bar()
9
# Output:
10
# 0
11
# 0    1
12
# 1    2
13
# 2    3
Copied!
Valid package names
Package handles take the form USER_NAME/PACKAGE_NAME. The package name and all of its children must be valid Python identifiers:
Start with a letter
Contain only alphanumerics and underscore
The above criteria ensure that packages can be accessed with Python's dot operator.
Push a package
Pushing a package stores a built package in a server-side registry. Push a package to back up changes or share your package with others.
1
$ quilt login # requires free account
2
$ quilt push USR/PKG --public
Copied!
Or, in Python:
1
# log in to the registry (requires a free account)
2
quilt.login()
3
# push it to the registry
4
quilt.push("USR/PKG", is_public=True)
Copied!
Users on Individual and Business plans can omit is_public=True to create private packages.