quilt
Search…
CLI, environment

catalog

1
usage: quilt3 catalog [-h] [--detailed_help] [--host HOST] [--port PORT]
2
[--no-browser]
3
[navigation_target]
4
5
Run Quilt catalog locally
6
7
positional arguments:
8
navigation_target Which page in the local catalog to open. Leave blank to
9
go to the catalog landing page, pass in an s3 url (e.g.
10
's3://bucket/myfile.txt') to go to file viewer, or pass
11
in a package name in the form 'BUCKET:USER/PKG' to go to
12
the package viewer.
13
14
optional arguments:
15
-h, --help show this help message and exit
16
--detailed_help Display detailed information about this command and then
17
exit
18
--host HOST Bind socket to this host
19
--port PORT Bind to a socket with this port
20
--no-browser Don't open catalog in a browser after startup
Copied!
Run the Quilt catalog on your machine. Running quilt3 catalog launches a Python webserver on your local machine that serves a catalog web app and provides required backend services using temporary AWS credentials. Temporary credentials are derived from your default AWS credentials (or active AWS_PROFILE) using boto3.sts.get_session_token. For more details about configuring and using AWS credentials in boto3, see the AWS documentation: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html

Previewing files in S3

The Quilt catalog allows users to preview files in S3 by downloading and processing/converting them inside the Python webserver running on local machine. Neither your AWS credentials nor data requested goes through any third-party cloud services aside of S3.

config

1
usage: quilt3 config [-h] [--set KEY=VALUE [KEY=VALUE ...]] [catalog_url]
2
3
Configure Quilt
4
5
positional arguments:
6
catalog_url URL of catalog to config with, or empty string to
7
reset the config
8
9
optional arguments:
10
-h, --help show this help message and exit
11
--set KEY=VALUE [KEY=VALUE ...]
12
Set a number of key-value pairs for config_values(do
13
not put spaces before or after the = sign). If a value
14
contains spaces, you should define it with double
15
quotes: foo="this is a sentence". Note that values are
16
always treated as strings.
Copied!

config-default-remote-registry

1
usage: quilt3 config-default-remote-registry [-h] default_remote_registry
2
3
Configure default remote registry for Quilt
4
5
positional arguments:
6
default_remote_registry
7
The default remote registry to use, e.g. s3://quilt-ml
8
9
optional arguments:
10
-h, --help show this help message and exit
Copied!

disable-telemetry

1
usage: quilt3 disable-telemetry [-h]
2
3
Disable anonymous usage metrics
4
5
optional arguments:
6
-h, --help show this help message and exit
Copied!

install

1
usage: quilt3 install [-h] [--registry REGISTRY] [--top-hash TOP_HASH]
2
[--dest DEST] [--dest-registry DEST_REGISTRY]
3
[--path PATH]
4
name
5
6
Install a package
7
8
positional arguments:
9
name Name of package, in the USER/PKG format
10
11
optional arguments:
12
-h, --help show this help message and exit
13
--registry REGISTRY Registry where package is located, usually s3://MY-
14
BUCKET. Defaults to the default remote registry.
15
--top-hash TOP_HASH Hash of package to install. Defaults to latest.
16
--dest DEST Local path to download files to.
17
--dest-registry DEST_REGISTRY
18
Registry to install package to. Defaults to local
19
registry.
20
--path PATH If specified, downloads only PATH or its children.
Copied!

list-packages

1
usage: quilt3 list-packages [-h] registry
2
3
List all packages in a registry
4
5
positional arguments:
6
registry Registry for packages, e.g. s3://quilt-example
7
8
optional arguments:
9
-h, --help show this help message and exit
Copied!

login

1
usage: quilt3 login [-h]
2
3
Log in to configured Quilt server
4
5
optional arguments:
6
-h, --help show this help message and exit
Copied!

logout

1
usage: quilt3 logout [-h]
2
3
Log out of current Quilt server
4
5
optional arguments:
6
-h, --help show this help message and exit
Copied!

push

1
usage: quilt3 push --dir DIR [-h] [--registry REGISTRY] [--dest DEST]
2
[--message MESSAGE] [--meta META] [--workflow WORKFLOW]
3
[--force]
4
name
5
6
Pushes the new package to the remote registry
7
8
positional arguments:
9
name Name of package, in the USER/PKG format
10
11
required arguments:
12
--dir DIR Directory to add to the new package
13
14
optional arguments:
15
-h, --help show this help message and exit
16
--registry REGISTRY Registry where to create the new package. Defaults to
17
the default remote registry.
18
--dest DEST Where to copy the objects in the package
19
--message MESSAGE The commit message for the new package
20
--meta META Sets package-level metadata. Format: A json string with
21
keys in double quotes '{"key": "value"}'
22
--workflow WORKFLOW Workflow ID or empty string to skip workflow
23
validation. If not specified, the default workflow will
24
be used.
25
--force Skip the parent top hash check and create a new
26
revision even if your local state is behind the remote
27
registry.
Copied!

verify

1
usage: quilt3 verify [-h] --registry REGISTRY --top-hash TOP_HASH --dir DIR
2
[--extra-files-ok]
3
name
4
5
Verify that package contents matches a given directory
6
7
positional arguments:
8
name Name of package, in the USER/PKG format
9
10
optional arguments:
11
-h, --help show this help message and exit
12
--registry REGISTRY Registry where package is located, usually s3://MY-
13
BUCKET
14
--top-hash TOP_HASH Hash of package to verify
15
--dir DIR Directory to verify
16
--extra-files-ok Whether extra files in the directory should cause a
17
failure
Copied!

Environment variables

QUILT_DISABLE_CACHE

Turn off cache. Defaults to False.
1
$ export QUILT_DISABLE_CACHE=true
Copied!

QUILT_DISABLE_USAGE_METRICS

Disable anonymous usage collection. Defaults to False
1
$ export QUILT_DISABLE_USAGE_METRICS=true
Copied!

QUILT_MANIFEST_MAX_RECORD_SIZE

Maximum size of a record in package manifest. Setting this variable is strongly discouraged. Defaults to 1_000_000.

QUILT_MINIMIZE_STDOUT

Turn off TQDM progress bars for log files. Defaults to False
1
$ export QUILT_MINIMIZE_STDOUT=true
Copied!

QUILT_TRANSFER_MAX_CONCURRENCY

Number of threads for file transfers. Defaults to 10.
This variable could be tried for improving file transfer rate. The optimal value depends on network bandwidth, CPU performance, file sizes, etc.
1
$ export QUILT_TRANSFER_MAX_CONCURRENCY=20
Copied!

XDG_*

Quilt uses appdirs for Python to determine where to write data. You can therefore override the following path constants with environment variables using the XDG standard (see appdirs docs).
For instance, AWS Lambda requires the user to use tmp/* as the scratch directory. You can override quilt3.util.CACHE_PATH, so that quilt3 install will succeed in Lambda, by setting the XDG_CACHE_HOME environment variable.

Constants (see util.py for more)

  • APP_AUTHOR
  • APP_NAME
  • BASE_DIR - Base directory of the application
  • BASE_PATH - Base pathlib path for the application directory
  • CACHE_PATH - Pathlib path for the user cache directory
  • CONFIG_PATH - Base pathlib path for the application configuration file
  • OPEN_DATA_URL - Application data url
  • PACKAGE_NAME_FORMAT - Regex for legal package names
  • TEMPFILE_DIR_PATH - Base pathlib path for the application tempfiles