Frequently Asked Questions
p = quilt3.Package()
p.push("USR/PKG", message="MSG", registry="s3://BUCKET")
Quilt packages are one level of abstraction above S3 object versions. Object versions track mutations to a single file, whereas a quilt package references a collection files and assigns this collection a unique version.
It is strongly recommended that you enable object versioning on the S3 buckets that you push Quilt packages to. Object versioning ensures that mutations to every object are tracked, and provides some protection against deletion.
Yes, to find bugs and prioritize features.
You can disable anonymous usage collection with an environment variable:
quilt3.disable_telemetry()to persistently disable anonymous usage statistics.
- 1.Go to CloudFormation > Stacks > YourQuiltStack > Outputs
- 2.Copy the row labeled TemplateBuildMetadata
- 3."git_revision" is your template version
Yes. Follow these steps:
- 1.Run your compute in the same region as your S3 bucket (as opposed to a local machine or foreign region)—I/O is much faster.
- 2.Use a larger instance with more vCPUs.
You can script the Quilt CLI directly from your shell environment and chain it with your R scripts to create a unified workflow:
quilt3 install my-package # download Quilt data package
[Run R commands or scripts] # modify the data in Quilt data package using R
quilt3 push --dir path/to/remote-registry my-package # upload Quilt data package to the remote registry
You may have a test data package that you wish to delete at some point to ensure your data repository is clean and organized. Please do this very carefully! In favor of immutability, Quilt makes deletion a bit tricky. First, note that
quilt3.Package.deleteonly deletes the package manifest, not the underlying objects. If you wish to delete the entire package and its objects, delete the objects first.
Warning: the objects you delete will be lost forever. Ditto for the package revision.
To delete, first browse the package then walk it, deleting its entry objects as follows:
import quilt3 as q3
s3 = boto3.client("s3")
reg = "s3://quilt-bio-staging"
pname = "akarve/delete-object"
p = q3.Package.browse(pname, registry=reg)
for (k, e) in p.walk():
pk = e.physical_key
s3.delete_object(Bucket=pk.bucket, Key=pk.path, VersionId=pk.version_id)
You can then follow the above with
q3.delete_package(pname, registry=reg, top_hash=p.top_hash).
Be sure to run
quilt3 logoutif you've previously logged in.
Select among multiple profiles in your shell as follows:
This allows for extremely granular querying of your data package name, metadata, and contents and includes logical operators, comparison functions, conditional expressions, mathematical functions, bitwise functions, date and time functions and operators, regular expression functions, and aggregate functions. Please review the references linked below to learn more.
regexp_extract_all(string, pattern)Return the substring(s) matched by the regular expression
SELECT regexp_extract_all('1a 2b 14m', '\d+');
Yes. Quilt is built on top of Amazon S3, and has the same character limitations. Although any UTF-8 character is supported in an object key name (filename), using certain characters can result in problems with some applications and protocols. The following guideline will help you maximize compliance. For a comprehensive list of safe characters, characters that might require special handling, and characters to avoid, please review the official Amazon S3 documentation linked below.
- Alphanumeric characters:
- Special characters:
- Exclamation point (
- Hyphen (
- Underscore (
- Period (
- Asterisk (
- Single quote (
- Open parenthesis (
- Close parenthesis (
Optional additional features (such as automated data packaging) require additional IPs.