Introduction

master status

Manage data like code

Quilt provides versioned, reusable building blocks for analysis in the form of data packages. A data package may contain data of any type or any size.

Quilt does for data what package managers do for code: provide a centralized store of record.

Demo

Benefits

• Reproducibility - Imagine source code without versions. Ouch. Why live with un-versioned data? Versioned data makes analysis reproducible by creating unambiguous references to potentially complex data dependencies.

• Collaboration and transparency - Data likes to be shared. Quilt offers a unified catalog for finding and sharing data.

• Auditing - the registry tracks all reads and writes so that admins know when data are accessed or changed.

• Less data prep - the registry abstracts away network, storage, and file format so that users can focus on what they wish to do with the data.

• De-duplication - Data are identified by their SHA-256 hash. Duplicate data are written to disk once, for each user. As a result, large, repeated data fragments consume less disk and network bandwidth.

• Faster analysis - Serialized data loads 5 to 20 times faster than files. Moreover, specialized storage formats like Apache Parquet minimize I/O bottlenecks so that tools like Presto DB and Hive run faster.

Key concepts

Data package

A Quilt data package is a tree of data wrapped in a Python module. You can think of a package as a miniature, virtualized filesystem accessible to a variety of languages and platforms.

Each Quilt package has a unique handle of the form USER_NAME/PACKAGE_NAME.

Packages are stored in a server-side registry. The registry controls permissions and stores package meta-data, such as the revision history. Each package has a web landing page for documentation, like this one for uciml/iris.

The data in a package are tracked in a hash tree. The tophash for the tree is the hash of all hashes of all data in the package. The combination of a package handle and tophash form a package instance. Package instances are immutable.

Leaf nodes in the package tree are called fragments or objects. Installed fragments are de-duplicated and kept in a local object store.

Package lifecycle

Core commands

build creates a package

build hashes and serializes data. All data and metadata are tracked in a hash-tree that specifies the structure of the package.

By default:

• Unstructured and semi-structured data are copied "as is" (e.g. JSON, TXT)

• Tabular file formats (like CSV, TSV, XLS, etc.) are parsed with

  pandas and serialized to Parquet with

  pyarrow.

You may override the above defaults, for example if you wish data to remain in its original format, with the transform: id setting in build.yml.

push stores a package in a server-side registry

Packages are registered against a Flask/MySQL endpoint that controls permissions and keeps track of where data lives in blob storage (S3 for the Free tier).

install downloads a package

After a permissions check the client receives a signed URL to download the package from blob storage.

Installed packages are stored in a local quilt_modules folder. Type $ quilt ls to see where quilt_modules is located.

import exposes your package to code

Quilt data packages are wrapped in a Python module so that users can import data like code: from quilt.data.USER_NAME import PACKAGE_NAME.

Data import is lazy to minimize I/O. Data are only loaded from disk if and when the user references the data (usually by adding parenthesis to a package path, pkg.foo.bar()).

Service

Quilt is offered as a managed service at quiltdata.com. Alternatively, users can run their own registries (refer to the registry documentation).

Architecture

Quilt consists of three components. See the contributing docs for further details.