version-5.0.x
Walkthrough
Catalog
Advanced
Git-like operations for datasets and Jupyter notebooks
quilt3 provides a simple command-line for versioning large datasets and storing them in Amazon S3. There are only two commands you need to know:pushcreates a new package revision in an S3 bucket that you designateinstalldownloads data from a remote package to disk
Why not use Git?
In short, neither Git nor Git LFS have the capacity or performance to function as a repository for data. S3, on the other hand, is widely used, fast, supports versioning, and currently stores some trillions of data objects.
Similar concerns apply when baking datasets into Docker containers: images bloat and slow container operations down.
Pre-requisites
You will need either an AWS account, credentials, and an S3 bucket, OR a Quilt enterprise stack with at least one bucket. In order to read from and write to S3 with
quilt3, you must first do one of the following:- OR, if and only if your company runs a Quilt enterprise stack, run the following:1pip install quilt32quilt3 config https://yourquilt.yourcompany.com3quilt3 loginCopied!
Install a package
A Quilt package contains any collection of data (usually as files), metadata, and documentation that you specify.
Let's get a data package from S3 and write it
quilt-hurdat/data .1
mkdir reef-check
2
cd reef-check
3
quilt3 install \
4
"akarve/reef-check" \
5
--registry s3://quilt-example \
6
--dest .
Copied!
Now you've got data in the current working directory.
1
ls
2
CA-06-california-counties.json quilt_summarize.json urchins-interactive.json
3
README.md reef-check.ipynb urchins2006-2019.parquet
Copied!
Creating your first package
Now let's imagine that we've modified this data locally. We save our Jupyter notebook and push the results back to Quilt:
1
# Be sure to substitute YOUR_NAME and YOUR_BUCKET with the desired strings
2
quilt3 push \
3
YOUR_NAME/reef-check \
4
--dir . \
5
--registry s3://YOUR_BUCKET \
6
--message "Initial commit of reef data"
Copied!
Quilt will then print out something like the following:
1
Package YOUR_NAME/[email protected] pushed to s3://YOUR_BUCKET
2
Successfully pushed the new package to https://yourquilt.yourocmpany.com/b/YOUR_NAME/packages/akarve/reef-check
Copied!
List the packages in a bucket
1
quilt3 list-packages s3://YOUR_BUCKET
Copied!
In the Quilt catalog, you will now see a new package revision, complete with a README, data grid preview, and an interactive visualization in Altair.
Learn more
Those are the basics of reading and writing Quilt packages with the CLI. See the CLI reference for more.
Last modified 5d ago