nf-quilt3is evolving rapidly and therefore is not yet recommended for production work. You may encounter bugs and missing functionality.
Nextflow is a bioinformatics workflow manager that enables the development of portable and reproducible workflows. The software is used by scientists and engineers to write, deploy and share data-intensive, highly scalable, workflows on any infrastructure.
nf-quilt3is a plugin developed by Quilt Data that enables you read and write directly to Quilt data packages using
quiltURLs wherever you currently use
To add the
nf-quilt3plugin to your workflow, you need Nextflow 22.09 (or later) and Python 3.9 (or later).
This plugin uses the
quilt3CLI to call the Quilt API. You must install the
quilt3module and ensure the CLI is in your path:
pip3 install quilt3
which quilt3 #e.g., /usr/local/bin/quilt3
Next, create a Quilt URL for the S3 bucket where you want to store (and eventually read) your results. You must also specify a package name containing exactly one '/', such as
instrument/experimentFinally, run your Nextflow pipeline with your config file, setting that URL as your output directory, .e.g.:
nextflow run nf-core/sarek -profile test,docker -plugins nf-quilt3 --outdir quilt+s3://raw-bucket#package=nf-quilt/sarek&path=.
Note that you won't need the '-plugins' option if you modify
Add the following snippet to your
nextflow.configto enable the plugin (or just that one 'id' if you already have other plugins):
In the future, you will be able to use that package as input to future jobs, e.g.:
nextflow run my/analysis --indir quilt+s3://raw-bucket#package=experiment/instrument --outdir quilt+s3://prod-bucket#package=experiment/analysis
Run the following command in the project root directory (ie. where the file
- 1.Clone the Nextflow repository into a sibling directory, .e.g:
git clone --depth 1 https://github.com/nextflow-io/nextflow ../nextflow
- 1.Compile the plugin alongside the Nextflow code:
- 1.Run Nextflow with the plugin, using
./launch.shas a drop-in replacement for the
nextflowcommand, and adding the option
-plugins nf-quilt3to load the plugin:
./launch.sh run nextflow-io/hello -plugins nf-quilt3
The project should be hosted in a GitHub repository whose name should match the name of the plugin, that is the name of the directory in the
Follow these steps to package, upload and publish the plugin:
- 1.Create a file named
gradle.propertiesin the project root containing the following attributes (this file should not be committed to Git):
github_organization: the GitHub organisation where the plugin repository is hosted.
github_username: The GitHub username granting access to the plugin repository.
github_access_token: The GitHub access token required to upload and commit changes to the plugin repository.
github_commit_email: The email address associated with your GitHub account.
- 1.Use the following command to package and create a release for your plugin on GitHub: