Configuration

Configuration file

When a pipeline script is launched, Nextflow looks for configuration files in multiple locations. Since each configuration file may contain conflicting settings, they are applied in the following order (from lowest to highest priority):

Parameters defined in pipeline scripts (e.g. main.nf)
The config file $HOME/.nextflow/config
The config file nextflow.config in the project directory
The config file nextflow.config in the launch directory
Config file specified using the -c <config-file> option
Parameters specified in a params file (-params-file option)
Parameters specified on the command line (--something value)

When more than one of these options for specifying configurations are used, they are merged, so that the settings in the first override the same settings appearing in the second, and so on.

Tip

You can use the -C <config-file> option to use a single configuration file and ignore all other files.

Syntax

A Nextflow configuration file is a simple text file containing a set of properties defined using the syntax:

name = value

Please note, string values need to be wrapped in quotation characters while numbers and boolean values (true, false) do not. Also note that values are typed. This means that, for example, 1 is different from '1' — the former is interpreted as the number one, while the latter is interpreted as a string value.

Variables

Configuration properties can be used as variables in the configuration file by using the usual $propertyName or ${expression} syntax.

For example:

propertyOne = 'world'
anotherProp = "Hello $propertyOne"
customPath = "$PATH:/my/app/folder"

Please note, the usual rules for String interpolation are applied, thus a string containing a variable reference must be wrapped in double-quote chars instead of single-quote chars.

The same mechanism allows you to access environment variables defined in the hosting system. Any variable name not defined in the Nextflow configuration file(s) is interpreted to be a reference to an environment variable with that name. So, in the above example, the property customPath is defined as the current system PATH to which the string /my/app/folder is appended.

Comments

Configuration files use the same conventions for comments used by the Groovy or Java programming languages. Thus, use // to comment a single line, or /* .. */ to comment a block on multiple lines.

Includes

A configuration file can include one or more configuration files using the keyword includeConfig. For example:

process.executor = 'sge'
process.queue = 'long'
process.memory = '10G'

includeConfig 'path/foo.config'

When a relative path is used, it is resolved against the actual location of the including file.

Constants

The following constants are globally available in a Nextflow configuration file:

baseDir: Deprecated since version 20.04.0.; Alias for projectDir.
launchDir: The directory where the workflow was launched.
projectDir: The directory where the main script is located.

Config scopes

Configuration settings can be organized in different scopes by dot prefixing the property names with a scope identifier, or grouping the properties in the same scope using the curly brackets notation. For example:

alpha.x = 1
alpha.y = 'string value..'

beta {
     p = 2
     q = 'another string ..'
}

See Configuration options for the full list of config settings.

Parameters

Pipeline parameters can be defined in the config file using the params scope:

params.custom_param = 123
params.another_param = 'string value .. '

params {
    alpha_1 = true
    beta_2 = 'another string ..'
}

Process configuration

The process scope allows you to specify process directives separately from the pipeline code.

For example:

process {
    executor = 'sge'
    queue = 'long'
    clusterOptions = '-pe smp 10 -l virtual_free=64G,h_rt=30:00:00'
}

By using this configuration, all processes in your pipeline will be executed through the SGE cluster, with the specified settings.

Process selectors

The withLabel selectors allow the configuration of all processes annotated with a label directive as shown below:

process {
    withLabel: big_mem {
        cpus = 16
        memory = 64.GB
        queue = 'long'
    }
}

The above configuration example assigns 16 cpus, 64 Gb of memory and the long queue to all processes annotated with the big_mem label.

In the same manner, the withName selector allows the configuration of a specific process in your pipeline by its name. For example:

process {
    withName: hello {
        cpus = 4
        memory = 8.GB
        queue = 'short'
    }
}

The withName selector applies both to processes defined with the same name and processes included under the same alias. For example, withName: hello will apply to any process originally defined as hello, as well as any process included under the alias hello.

Furthermore, selectors for the alias of an included process take priority over selectors for the original name of the process. For example, given a process defined as foo and included as bar, the selectors withName: foo and withName: bar will both be applied to the process, with the second selector taking priority over the first.

Tip

Label and process names do not need to be enclosed with quotes, provided the name does not include special characters (-, !, etc) and is not a keyword or a built-in type identifier. When in doubt, you can enclose the label name or process name with single or double quotes.

Selector expressions

Both label and process name selectors allow the use of a regular expression in order to apply the same configuration to all processes matching the specified pattern condition. For example:

process {
    withLabel: 'foo|bar' {
        cpus = 2
        memory = 4.GB
    }
}

The above configuration snippet sets 2 cpus and 4 GB of memory to the processes annotated with a label foo and bar.

A process selector can be negated prefixing it with the special character !. For example:

process {
    withLabel: 'foo' { cpus = 2 }
    withLabel: '!foo' { cpus = 4 }
    withName: '!align.*' { queue = 'long' }
}

The above configuration snippet sets 2 cpus for the processes annotated with the foo label and 4 cpus to all processes not annotated with that label. Finally it sets the use of long queue to all process whose name does not start with align.

Selector priority

Process configuration settings are applied to a process in the following order (from lowest to highest priority):

Process configuration settings (without a selector)
Process directives in the process definition
withLabel selectors matching any of the process labels
withName selectors matching the process name
withName selectors matching the process included alias
withName selectors matching the process fully qualified name

For example:

process {
    cpus = 4
    withLabel: foo { cpus = 8 }
    withName: bar { cpus = 16 }
    withName: 'baz:bar' { cpus = 32 }
}

With the above configuration:

All processes will use 4 cpus (unless otherwise specified in their process definition).
Processes annotated with the foo label will use 8 cpus.
Any process named bar (or imported as bar) will use 16 cpus.
Any process named bar (or imported as bar) invoked by a workflow named baz with use 32 cpus.

Config profiles

Configuration files can contain the definition of one or more profiles. A profile is a set of configuration attributes that can be selected during pipeline execution by using the -profile command line option.

Configuration profiles are defined by using the special scope profiles, which group the attributes that belong to the same profile using a common prefix. For example:

profiles {

    standard {
        process.executor = 'local'
    }

    cluster {
        process.executor = 'sge'
        process.queue = 'long'
        process.memory = '10GB'
    }

    cloud {
        process.executor = 'cirrus'
        process.container = 'cbcrg/imagex'
        docker.enabled = true
    }

}

This configuration defines three different profiles: standard, cluster, and cloud, that each set different process configuration strategies depending on the target runtime platform. The standard profile is used by default when no profile is specified.

Tip

Multiple configuration profiles can be specified by separating the profile names with a comma, for example:

nextflow run <your script> -profile standard,cloud

Danger

When using the profiles feature in your config file, do NOT set attributes in the same scope both inside and outside a profiles context. For example:

process.cpus = 1

profiles {
  foo {
    process.memory = '2 GB'
  }

  bar {
    process.memory = '4 GB'
  }
}

In the above example, the process.cpus attribute is not correctly applied because the process scope is also used in the foo and bar profiles.

Workflow handlers

Workflow event handlers can be defined in the config file, which is useful for handling pipeline events without having to modify the pipeline code:

workflow.onComplete = {
    // any workflow property can be used here
    println "Pipeline complete"
    println "Command line: $workflow.commandLine"
}

workflow.onError = {
    println "Error: something when wrong"
}

See Workflow handlers for more information.