# Neumann - Submit all Sub-directories automatically

It is often necessary to run many dozen of (StarCCM) simulations which are very similar to each other,
for example, the task to record the characteristics of systems of a pump, or a sensitivity analyses on input parameter.
Many of our CFD is run on HPC cluster, such as Neumann.
For each simulation job, a script has to be provided to tell the workload manager (SLURM) what to do.

It can become quite tedious (and maddening) to prepare a sbatch file for each simulation.
For this reason, scripting this process is the less error-prone way.
This page describes a fully-automated solution, while a semi-automatic solution is also given on this wiki.

If you want to run StarCCM .sim files just sequentially, you can get away with a macro to be found in the Steve-Portal:
How can I sequentially run a set of simulations that I have prepared?

Although, StarCCM is used as an example here, you can apply the same structure to any other kind of job, such as these.

## Prepare and Submit all sub-directories at once

Assume, you want to perform StarCCM simulations which all need the same resources, and have the same file name.
An corresponding file tree is given below. In this example, a sbatch script differs only in the location where the simulation has to be run.

The script explaind here does the following steps:

1. it copys a template into each sub-directory,
2. and replaces a placeholder with the absolute path of the sub-directory.
3. Then, it copies each sbatch script to a specified location in home,
4. and submits them, one after another.

Copying the sbatch script to home is necessary because there is a tripwire in Neumann's configuration. All sbatch scripts have to be submitted from the home-directory, and not from scratch. These are located on different hard-drives. The SLURM programm is running in home and can get confused by scripts started from scratch.

The assumed file tree is shown here. The sbatch template and the prepAndSubmit.sh script need to be in the parent directory.

File Tree:
./
sim1/
my.sim
sim2/
my.sim
prepAndSubmit.sh
starccm-job.sh
prepAndSubmit.sh
#!/bin/bash
## SE 05/2018
##
## Select a secure location in your home directory for the sbatch file
homeprojdir="/home/$USER/projectdir" ## Sbatch Template name sbjob="starccm-job.sh" # make dir if homeproject is not present if [ ! -d "$homeprojdir" ]; then  mkdir -p $homeprojdir; fi ## Catching current dir calldir=$(pwd)
echo "Batch submitting $( ls */| wc -l) sub diretories" ## Iterating through all sub-directories for subdir in */; do ## Storing absolute path of current sub directory subAbsDir=$calldir/$subdir ## Copying template into current sub directory cp$sbjob $subdir/$sbjob
## Replacing placeholder in sub directory copy
sed -i "s#ABSOLUTPATHSEDPLACEHOLDER#$subAbsDir#g"$subdir/$sbjob ## Copying sbatch file into home tree and submit from there cp$subdir/$sbjob$homeprojdir
cd $homeprojdir echo "Submitting now$calldir/$subdir" sbatch$sbjob
cd $calldir done The Sbatch Template has to have the place holder string ABSOLUTPATHSEDPLACEHOLDER to be set as work directory. starccm-job.sh #!/bin/bash # Version 15.01.2018 #################### Job Settings ############################################################# # Specific Commands for the work load manager SLURM are lines beginning with "#SBATCH" #SBATCH -J multirun # Setting the display name for the submission #SBATCH -N 1 # Number of nodes to reserve, -N 2-5 for variable number of requested node count #SBATCH --ntasks-per-node 16 # typically 16, range: 1..16 (max 16 cores per node) #SBATCH -t 001:00:00 # set walltime in hours, format: hhh:mm:ss, days-hh, days-hhh:mm:ss #SBATCH -p short # Desired Partition, alternatively comment this line out and submit the script with 'sbatch -p big jobscript.sh' #################### Simulation Settings ###################################################### ## Work directory. Directory of your simulation file. Should be in the scratch directory, not in home. WORKDIR="ABSOLUTPATHSEDPLACEHOLDER" ## Simulation File (location from work directory) SIMULATIONFILE="my.sim" ## Macro file (location from work directory) MACROFILE="macro.java" [...] go to https://wikis.ovgu.de/hpc/doku.php?id=guide:jobscript:starccm to get a complete and up-to-date script ## Seperate steps Separating the above steps allow manual corrections to the sbatch scripts before submitting, or rerunning the simulations. ### Just prepare scripts justPrepare.sh #!/bin/bash sbjob="starccm-job.sh" calldir=$(pwd)
for sub in */; do
cp $sbjob$sub/$sbjob subDir=$calldir/$sub sed -i "s#ABSOLUTEPATHSEDPLACEHOLDER#$subDir#g" $sub/$sbjob
done

### Just submit sub-directories

justSubmit.sh
#!/bin/bash
homeProjDir="~/project/"
sbjob="starccm-job.sh"

callDir=$(pwd) for sub in */; do cp$sub/$sbjob$homeProjDir/$sbjob cd$homeProjDir
echo "Submitting now $callDir/$sub"
sbatch $sbjob cd$callDir
done