Nextflow

nf-quilt is prerelease technology and evolving rapidly. For the latest updates and issues, please visit the GitHub repository: quiltdata/nf-quilt

What is Nextflow?

Nextflow is a bioinformatics workflow manager that enables the development of portable and reproducible workflows. The software is used by scientists and engineers to write, deploy and share data-intensive, highly scalable, workflows on any infrastructure.

Overview

Nextflow plugin for interacting with quilt packages as a FileSystem.

nf-quilt is a plugin developed by Quilt Data that enables you read and write directly to Quilt data packages using quilt URLs wherever you currently use s3, az or gs URLs.

Getting Started

To add the nf-quilt plugin to your workflow, you need Nextflow 22.09 (or later) and Python 3.9 (or later).

Quilt Configuration

This plugin uses the quilt3 CLI to call the Quilt API. You must install the quilt3 module and ensure the CLI is in your path:

pip3 install quilt3
which quilt3 #e.g., /usr/local/bin/quilt3

Reading and Writing Quilt URLs

Next, create a Quilt URL for the S3 bucket where you want to store (and eventually read) your results. You must also specify a package name containing exactly one '/', such as instrument/experiment Finally, run your Nextflow pipeline with your config file, setting that URL as your output directory, .e.g.:

nextflow run nf-core/sarek -profile test,docker -plugins nf-quilt --outdir quilt+s3://raw-bucket#package=nf-quilt/sarek&path=.

Pipeline Configuration

Note that you won't need the '-plugins' option if you modify nextflow.config

Add the following snippet to your nextflow.config to enable the plugin (or just that one 'id' if you already have other plugins):

plugins {
    id 'nf-quilt'
}

In the future, you will be able to use that package as input to future jobs, e.g.:

nextflow run my/analysis --indir quilt+s3://raw-bucket#package=experiment/instrument --outdir quilt+s3://prod-bucket#package=experiment/analysis

Development

Based on nf-hello

Unit testing

Run the following command in the project root directory (ie. where the file settings.gradle is located):

make check

Testing and debugging

1. Clone the Nextflow repository into a sibling directory, .e.g:

git clone --depth 1 https://github.com/nextflow-io/nextflow ../nextflow

2. Compile the plugin alongside the Nextflow code:

make compile

3. Run Nextflow with the plugin, using ./launch.sh as a drop-in replacement for the nextflow command, and adding the option -plugins nf-quilt to load the plugin:

./launch.sh run nextflow-io/hello -plugins nf-quilt

Package, upload and publish

The project should be hosted in a GitHub repository whose name should match the name of the plugin, that is the name of the directory in the plugins folder (e.g. nf-quilt).

Follow these steps to package, upload and publish the plugin:

1. Create a file named gradle.properties in the project root containing the following attributes (this file should not be committed to Git):

• github_organization: the GitHub organisation where the plugin repository is hosted.

• github_username: The GitHub username granting access to the plugin repository.

• github_access_token: The GitHub access token required to upload and commit changes to the plugin repository.

• github_commit_email: The email address associated with your GitHub account.

2. Use the following command to package and create a release for your plugin on GitHub:

./gradlew :plugins:nf-quilt:upload

3. Create a pull request against nextflow-io/plugins to make the plugin accessible to Nextflow.

References

• Nextflow

• nf-quilt