nf-core/configs: LRZ CM4 Configuration

About

All nf-core pipelines have been successfully configured for use on the CoolMuc4 cluster that is provided by the Leibniz Rechenzentrum (LRZ) of the Bavarian Academy of Sciences, located in Garching, Germany..

NB: You will need an account to use the LRZ Linux cluster.

Usage

To use, run the pipeline with -profile lrz_cm4. This will download and launch the lrz_cm4.config.

We recommend using nextflow >= 25.04.2 with apptainer (1.3.4) for containerization. These are available as modules (please confirm the module name using module avail):

## Load Nextflow and apptainer environment modules
module load nextflow/25.04.2 apptainer/1.3.4

Details

NB: Please note that running nextflow on a login node is not permitted.

NB: Please note that it is not possible to run nextflow with the SLURM executor in a job, compute nodes cannot submit jobs.

Instead of having nextflow run on a login node and submit jobs to the SLURM scheduler, the nextflow head job, coordinating the workflow, has to run inside a SLURM-job and job scheduling is done ‘inside’ the SLURM job using the flux or local executors. This is outlined here and implemented in -profile lrz_cm4. By default, this uses the flux executor, if you would prefer to use the local executor, please use -profile lrz_cm4,local. Independent of the executor used, task memory limits will be set through apptainer.

Serial / cm4_tiny / terramem

Run nextflow inside a SLURM job using either local or flux for job scheduling within the SLURM allocation. In case the cm4_tiny partition of the cm4 cluster, the serial partition of serial cluster, or terramem partition of the inter cluster is to be used (i.e. if the job requires less 1 full node) please prepare a script similar to the one below:

NB: This config assumes that memory is not requested explicitly, and computes the memory resourceLimit as 4.5GB * number of CPUs

#! /bin/bash
#SBATCH -D .
#SBATCH -J nextflow_run
#SBATCH --get-user-env
#SBATCH -M cm4                  # for serial: serial here; for terramem: inter
#SBATCH -p cm4_tiny             # for serial: serial here; for terramem: terramem_inter
#SBATCH --cpu-per-task=100      # Please see https://doku.lrz.de/job-processing-on-the-linux-cluster-10745970.html for partition limits
#SBATCH --ntasks=1
#SBATCH --export=NONE
#SBATCH --time=24:00:00
 
module load flux
 
flux start
nextflow run nf-core/rnaseq \
    -profile test,lrz_cm4

In case the scheduling should not be done via flux, but local, please use:

#! /bin/bash
#SBATCH -D .
#SBATCH -J nextflow_run
#SBATCH --get-user-env
#SBATCH -M cm4                  # for serial: serial here; for terramem: inter
#SBATCH -p cm4_tiny             # for serial: serial here; for terramem: terramem_inter
#SBATCH --cpu-per-task=100      # Please see https://doku.lrz.de/job-processing-on-the-linux-cluster-10745970.html for partition limits
#SBATCH --ntasks=1
#SBATCH --export=NONE
#SBATCH --time=24:00:00
 
nextflow run nf-core/rnaseq \
    -profile test,lrz_cm4,local

cm4_std

NB: If more than one node is used, make sure to use flux for execution.

On the cm4_std partition of the cm4 cluster, full (exclusive) nodes are scheduled. Use

#! /bin/bash
#SBATCH -D .
#SBATCH -J nextflow_run
#SBATCH --get-user-env
#SBATCH -M cm4              # if cores <= 112 go to cm4_tiny
#SBATCH -p cm4_std          # if cores <= 112 go to cm4_tiny
#SBATCH --qos=cm4_std       # if tiny, QOS is not required
#SBATCH --nodes=2           # 2 nodes (maximum: 4)
#SBATCH --ntasks-per-node=1
#SBATCH --export=NONE
#SBATCH --time=24:00:00
 
module load flux
 
flux start
nextflow run nf-core/rnaseq \
    -profile test,lrz_cm4

this script is to be submitted via sbatch. The correct resource limits are applied based on the number of requested nodes (which are fetched from the environment).

Config file

See config file on GitHub

conf/lrz_cm4
/* ----------------------------------------------------
 * Nextflow config file for the LRZ cm4 cluster
 * ----------------------------------------------------
 */
 
manifest {
    name = 'LRZ CM4 Configuration'
    author = 'Amit Fenn, Niklas Schandry, Frederik Dröst'
    homePage = 'plantmicrobe.de'
    description = 'Configuration for LRZ CM4 cluster'
}
 
params {
    // Configuration metadata
    config_profile_name = 'LRZ CM4'
    onfig_profile_description = 'LRZ CM4 configuration'
    config_profile_contact = 'Amit Fenn (@amitfenn); Niklas Schandry(@nschan)'
    config_profile_url = 'https://doku.lrz.de/job-processing-on-the-linux-cluster-10745970.html/'
    config_version = '1.0.0'
    // Default output directory (relative to launch directory)
    outdir = 'results'
}
 
apptainer {
    enabled = true
    autoMounts = true
    runOptions = { "--memory ${task.memory} " }
}
 
process {
    beforeScript = 'module load apptainer/1.3.4'
    executor = 'flux'
    resourceLimits = [
        // The below checks if the queue is cm4_std, which means that for serial or cm4_tiny this is taken from SLURM_*_PER_NODE
        cpus:   System.getenv("SLURM_JOB_PARTITION") == 'cm4_std' ? 224 * System.getenv('SLURM_NNODES').toInteger()     : (System.getenv("SLURM_CPUS_ON_NODE") ? System.getenv("SLURM_CPUS_ON_NODE").toInteger() : 128),
        memory: System.getenv("SLURM_JOB_PARTITION") == 'cm4_std' ? 488.GB * System.getenv('SLURM_NNODES').toInteger()  : (System.getenv("SLURM_CPUS_ON_NODE") ? (System.getenv("SLURM_CPUS_ON_NODE").toInteger() * 4500.MB) : 96.GB),
    ]
}
 
profiles {
    local {
        process {
            executor = 'local'
        }
    }
}
 
trace {
    enabled = true
    overwrite = true
}
 
report {
    enabled = true
    overwrite = true
}
 
timeline {
    enabled = true
    overwrite = true
}
 
dag {
    enabled = true
    overwrite = true
}
On this page