multiple-job-dirs.shtml

---
layout: default
title: Submitting Multiple Jobs in Individual Directories
---

<br>
This guide demonstrates how to submit multiple jobs, using a specific 
directory structure.  It is relevant to: 
<ul>
<li>Researchers who have used CHTC's "ChtcRun" tools in the past</li>
<li>Anyone who wants to submit multiple jobs, where each job has its 
own directory for input/output files on the submit server.  </li>
</ul>

<h2>1. Software and Input Preparation</h2>

<p>The first time you submit jobs, you will need to prepare a portable version 
of your software and a script (what we call the job's "executable") that runs 
your code.  We have guides for preparing: </p>

<ul>
<li><a href="{{'/matlab-jobs' | relative_url }}">Matlab</a></li>
<li><a href="{{'/python-jobs' | relative_url }}">Python</a></li>
<li><a href="{{'/r-jobs' | relative_url }}">R</a></li>
</ul>

<p>Choose the right guide for you and follow the directions for compiling 
your code (Matlab) or building an installation (Python, R).  Also follow 
the instructions for writing a shell script that runs your program. These 
are typically steps 1 and 2 of the above guides.</p>

<h2>2. Directory Structure</h2>

Once you've prepared your code and script, create the same directory structure 
that you would normally use with ChtcRun.  For a single batch of jobs, the 
directories will look like this: 

<pre class="other">
project_name/
    run_code.sh
    shared/
        scripts, code_package
        shared_input
    job1/
	input/
	    job_input
    job2/
        input/
	    job_input
    job3/
	input/
	    job_input
</pre>

<p>You'll want to put all your code and files required for 
every job in <code>shared/</code> and individual input files in the individual 
job directories in an <code>input</code> folder.  In the 
submit file below, it matters that the individual job directories 
start with the word "job".  </p>

<h2>3. Submit File</h2>

<blockquote><b>Note: if you are submitting more than 10,000 jobs at once, 
you'll need to use a different submit file.  Please email the CHTC 
Research Computing Facilitators at <a href="mailto:chtc@cs.wisc.edu">chtc@cs.wisc.edu</a>
if this is the case! </b></blockquote>

<p>Your submit file, which should go in your main project directory, should look 
like this: </p>

<pre class="sub">
# Specify the HTCondor Universe (vanilla is the default and is used
#  for almost all jobs), the desired name of the HTCondor log file,
#  and the desired name of the standard error and standard output file.  
universe = vanilla
log = <i>process</i>.log
error = <i>process</i>.err
output = <i>process</i>.out
#
# Specify your executable (single binary or a script that runs several
#  commands) and arguments
executable = <i>run_code</i>.sh
# arguments = arguments to your script go here
#
# Specify that HTCondor should transfer files to and from the
#  computer where each job runs. 
should_transfer_files = YES
when_to_transfer_output = ON_EXIT
# Set the submission directory for each job with the $(directory)
# variable (set below in the queue statement).  Then transfer all 
# files in the shared directory, and from the input folder in the 
# submission directory
initialdir = <i>$(directory)</i>
transfer_input_files = <i>../shared/</i>,<i>input/</i>
#
# Tell HTCondor what amount of compute resources
#  each job will need on the computer where it runs.
request_cpus = <i>1</i>
request_memory = <i>1GB</i>
request_disk = <i>1GB</i>
#
# Create a job for each "job" directory.
queue directory matching job*
</pre>

<p>You <b>must</b> change the name of the executable to your own script, 
and in certain cases, add arguments. </p>

<p>Note that the final line matches the pattern of your directory names 
created in the second step.  You can use a different name for the directories
(like <code>data</code> or <code>seed</code>), but you should use whatever 
word they share in the final queue statement in place of "job".  </p>

<p>Jobs can then be submitted as described in our <a href="{{'/helloworld' | relative_url }}">
Introduction to HTC Guide</a>, using <code>condor_submit</code>.