Neumann - Submit all Sub-directories automatically

It is often necessary to run many dozen of (StarCCM) simulations which are very similar to each other,
for example, the task to record the characteristics of systems of a pump, or a sensitivity analyses on input parameter.
Many of our CFD is run on HPC cluster, such as Neumann.
For each simulation job, a script has to be provided to tell the workload manager (SLURM) what to do.

It can become quite tedious (and maddening) to prepare a sbatch file for each simulation.
For this reason, scripting this process is the less error-prone way.
This page describes a fully-automated solution, while a semi-automatic solution is also given on this wiki.

If you want to run StarCCM .sim files just sequentially, you can get away with a macro to be found in the Steve-Portal:
How can I sequentially run a set of simulations that I have prepared?

Although, StarCCM is used as an example here, you can apply the same structure to any other kind of job, such as these.

Prepare and Submit all sub-directories at once

Assume, you want to perform StarCCM simulations which all need the same resources, and have the same file name.
An corresponding file tree is given below. In this example, a sbatch script differs only in the location where the simulation has to be run.

The script explaind here does the following steps:

  1. it copys a template into each sub-directory,
  2. and replaces a placeholder with the absolute path of the sub-directory.
  3. Then, it copies each sbatch script to a specified location in home,
  4. and submits them, one after another.

Copying the sbatch script to home is necessary because there is a tripwire in Neumann's configuration. All sbatch scripts have to be submitted from the home-directory, and not from scratch. These are located on different hard-drives. The SLURM programm is running in home and can get confused by scripts started from scratch.

The assumed file tree is shown here. The sbatch template and the prepAndSubmit.sh script need to be in the parent directory.

File Tree:
./
  sim1/
    my.sim
  sim2/
    my.sim
  prepAndSubmit.sh
  starccm-job.sh
prepAndSubmit.sh
#!/bin/bash
## SE 05/2018
##
## Select a secure location in your home directory for the sbatch file
homeprojdir="/home/$USER/projectdir"
## Sbatch Template name
sbjob="starccm-job.sh"
 
# make dir if homeproject is not present
if [ ! -d "$homeprojdir" ]; then  mkdir -p $homeprojdir; fi
## Catching current dir
calldir=$(pwd)
echo "Batch submitting $( ls */| wc -l) sub diretories"
## Iterating through all sub-directories
for subdir in */; do
        ## Storing absolute path of current sub directory
        subAbsDir=$calldir/$subdir
        ## Copying template into current sub directory
        cp $sbjob $subdir/$sbjob
        ## Replacing placeholder in sub directory copy
        sed -i "s#ABSOLUTPATHSEDPLACEHOLDER#$subAbsDir#g" $subdir/$sbjob
        ## Copying sbatch file into home tree and submit from there
        cp $subdir/$sbjob $homeprojdir
        cd $homeprojdir
        echo "Submitting now $calldir/$subdir"
        sbatch $sbjob
        cd $calldir
done

The Sbatch Template has to have the place holder string ABSOLUTPATHSEDPLACEHOLDER to be set as work directory.

starccm-job.sh
#!/bin/bash
# Version 15.01.2018
#################### Job Settings #############################################################
# Specific Commands for the work load manager SLURM are lines beginning with "#SBATCH"
#SBATCH -J multirun           # Setting the display name for the submission
#SBATCH -N 1                  # Number of nodes to reserve, -N 2-5  for variable number of requested node count
#SBATCH --ntasks-per-node 16  # typically 16, range: 1..16 (max 16 cores per node)
#SBATCH -t 001:00:00          # set walltime in hours, format:    hhh:mm:ss, days-hh, days-hhh:mm:ss
#SBATCH -p short              # Desired Partition, alternatively comment this line out and submit the script with 'sbatch -p big jobscript.sh'
 
#################### Simulation Settings ######################################################
 
## Work directory. Directory of your simulation file. Should be in the scratch directory, not in home.
WORKDIR="ABSOLUTPATHSEDPLACEHOLDER"
 
## Simulation File (location from work directory)
SIMULATIONFILE="my.sim"
 
## Macro file (location from work directory)
MACROFILE="macro.java"
 
[...]

go to https://wikis.ovgu.de/hpc/doku.php?id=guide:jobscript:starccm to get a complete and up-to-date script

Seperate steps

Separating the above steps allow manual corrections to the sbatch scripts before submitting, or rerunning the simulations.

Just prepare scripts

justPrepare.sh
#!/bin/bash
sbjob="starccm-job.sh"
 
calldir=$(pwd)
for sub in */; do
        cp $sbjob $sub/$sbjob
        subDir=$calldir/$sub
        sed -i "s#ABSOLUTEPATHSEDPLACEHOLDER#$subDir#g" $sub/$sbjob
done

Just submit sub-directories

justSubmit.sh
#!/bin/bash
homeProjDir="~/project/"
sbjob="starccm-job.sh"
 
callDir=$(pwd)
for sub in */; do
        cp $sub/$sbjob $homeProjDir/$sbjob
        cd $homeProjDir
        echo "Submitting now $callDir/$sub"
        sbatch $sbjob
        cd $callDir
done
guide/neumann/submitallsubdirectories.txt · Last modified: 2018/05/14 15:53 by seengel
Back to top
CC Attribution-Share Alike 3.0 Unported
chimeric.de = chi`s home Valid CSS Driven by DokuWiki do yourself a favour and use a real browser - get firefox!! Recent changes RSS feed Valid XHTML 1.0