Using Nextflow with Flux
New in version 22.11.0-edge.
Overview
The Flux Framework is a modern resource manager that can span the space between cloud and HPC. If your center does not provide Flux, you can build Flux yourself and launch it as a job using your resource manager of choice (e.g. SLURM or a cloud provider).
In the docker/flux
directory we provide a Dockerfile for interacting with Flux along with a VSCode Developer Container environment that you can put at the root of the project to be provided with a Flux agent and the dependencies needed to build Nextflow. There are two ways to use this:
Build a container from scratch and bind your code to it (e.g. for development or testing)
Use VSCode and DevContainers to create a more seamless environment
Both strategies are described below. For this tutorial, you will generally want to prepare a pipeline to use the flux
executor, create an environment with Flux, start a Flux instance, and interact with it.
Prepare your pipeline
To run your pipeline with Flux, you’ll want to specify it in your config. Here is an example nextflow.config
:
manifest {
mainScript = 'demo.nf'
homePage = 'https://github.com/nextflow-io/nextflow/tree/master/docker/flux'
description = 'Demo using Nextflow with Flux'
}
process {
executor = 'flux'
}
For additional Flux settings, see the Flux Executor section.
Here is an example pipeline that we will use:
workflow {
breakfast = channel.of '🥞️', '🥑️', '🥧️', '🍵️', '🍞️'
haveMeal(breakfast)
}
process haveMeal {
debug true
input:
val food
script:
"""
printf '$food for breakfast!'
"""
}
Prepare your environment
You can either build the Docker image from the root of the Nextflow repository:
$ docker build -f docker/flux/.devcontainer/Dockerfile --platform linux/amd64 -o type=docker -t nextflow-flux .
And then shell into the container for a development environment. You’ll need to bind the present working directory to /code
to see your local changes in the container:
$ docker run -it -v $PWD:/code nextflow-flux
You can also move the .devcontainer
directory to the root of your repository, and open it in VSCode:
$ cp -R docker/flux/.devcontainer .devcontainer
Then open in VSCode, and select Re-open in container:
$ code .
Then you should be able to open a terminal (Terminal -> New Terminal) to interact with the command line. Try running make
again! Whichever of these two approaches you take, you should be in a container environment with the flux
command available.
Start a Flux instance
Once in your container, you can start an interactive Flux instance (from which you can submit jobs on the command line to test with Nextflow) as follows:
$ flux start --test-size=4
Getting familiar with Flux
Here is an example of submitting a job and getting the log for it.
First submit the job:
$ flux submit echo "HELLO MOTO"
ƒEzWqspb
Then get the log for it:
$ flux job attach ƒEzWqspb
HELLO MOTO
Try submitting a longer job:
$ flux submit sleep 60
And then seeing it in the jobs listing.
$ flux jobs
JOBID USER NAME ST NTASKS NNODES TIME INFO
ƒ4tkMUAAT root sleep R 1 1 2.546s ab6634a491bb
Submitting with Nextflow
Prepare your nextflow.config
and demo.nf
in the same directory.
$ ls .
demo.nf nextflow.config
Finally, run the pipeline with Flux:
$ nextflow -c nextflow.config run demo.nf
N E X T F L O W ~ version 22.10.0
Launching `demo.nf` [clever_blackwell] DSL2 - revision: f8cda838cb
executor > flux (5)
[4c/f162db] process > haveMeal (3) [100%] 5 of 5 ✔
🥞️ for breakfast!
🍞️ for breakfast!
🍵️ for breakfast!
🥑️ for breakfast!
🥧️ for breakfast!