submit_variations.shtml

---
layout: default
title: HTCondor Submit File Variations
---

<br>
<b>The examples and information in this guide work best for the below cases*:</b>
<ol>
<li>Submission to an HTCondor System with file transfer (rather than a shared filesystem).</li>
<li>Submission to an HTCondor System that is unix-based (Linux or Mac operating system, as Windows may have important differences).</li>
</ol>
<small>(*see the HTCondor software manual and online examples from other organizations for cases outside of those we cover on the CHTC website)</small>
<br>
<br>
<b>To best understand the below information, users should already have an understanding of:</b>
<ol>
<li>Using the command line to: navigate within directories, create/copy/move/delete files and directories, and run their intended programs (aka "executables").</li>
<!-- coming soon <li>The CHTC’s Roadmap for High Throughput Computing</li> -->
<li><a href="{{'/helloworld' | relative_url }}">The CHTC's Intro to Running HTCondor Jobs</a></li>
<li>The importance of indicating the below submit file lines, which are left
 out of the below examples for simplicity:
<pre>
universe = vanilla    # or other
request_memory = <MB>
request_disk = <KB>
request_cpus = <positive integer>
should_transfer_files = YES
when_to_transfer_output = ON_EXIT    # or other
</ol>  
<h2>Click on the links below to go directly to the appropriate example</h2>
<ol>
<li><b><a href="#int-files">Organizing many jobs with integer-numbered filenames</a></b></li>
<li><b><a href="#int-dir">Organizing job files within integer-numbered directories</a></b></li>
<ul>
<li><a href="#int-dir-process">Numbered directories using $(Process)</a></li>
<li><a href="#int-dir-initialdir">Numbered directories using $(Process) as the InitialDir</a></li>
</ul>
<li><b><a href="#own-variables">Using your own non-HTCondor variables in the submit file</a></b></li>
<li><b><a href="#notint">Jobs with un-numbered data</a></b></li>
<ul>
<li><a href="#notint-files">Using a different, non-numbered input file for each job</a></li>
<li><a href="#notint-dir">Using a different, non-numbered directory for each job</a></li>
</ul>
</ol>

<!-- Submitting Many Jobs with a Single Submit File -->

<br>
<a name="int-files"></a>
<h1> 1. Organizing many jobs with integer-numbered filenames: </h1>

<p>This example is most similar to our
 <a href="{{'/helloworld' | relative_url }}">Intro to Running HTCondor
 Jobs</a> and uses HTCondor's $(Process) variable to indicate
 filename differences for each job. For "queue <i>N</i>" in the submit file,
 the $(Process) variable will start with
 "0" and end with <i>N</i>-1, such that you can use $(Process) to indicate
 filenames and filepaths. </p>

<b>Submit file:</b>
<pre>
executable = my.exec
arguments = $(Process).data comparison.data $(Process).out
output = $(Process).out
error = $(Process).err
log = $(Process).log
transfer_input_files = $(Process).data,comparison.data
queue 3
</pre>

<!-- this "directory" class for these tables is at the end of web.css-->
<table class="directory">
<tr>  
<th>Submit directory before submitting</th>
<th>After completion</th>
</tr>
<tr>
<td class="directory"><pre>
(submit_dir)/	submit.file
		my.exec
		comparison.data
		0.data
		1.data
		2.data
</pre></td>
<td class="directory"><pre>
(submit_dir)/ 	submit.file
		my.exec
		comparison.data
		0.data  0.out  0.err  0.log
		1.data  1.out  1.err  1.log
		2.data  2.out  2.err  2.log
</pre> </td>
</tr>
</table>
			        
<b>Job working directory after input transfer, before execution</b>
<pre>
(working_dir)/	my.exec
		0.data
		comparison.data
		(other HTCondor tracking files)
</pre>

<br>
<a name="int-dir"></a>
<h1>2. Organizing job files within integer-numbered sub-directories</h1>

<a name="int-dir-process"></a>
<h2>A. Numbered directories using $(Process)</h2>
<p>The below example uses $(Process) to organize input, error, output,
 and log files into numbered directories. Note that other files produced
 within the job will still return to the submission directory.</p>
<b>Submit file:</b>
<pre>
executable = my.exec
arguments = input.data comparison.data $(Process).out
output = $(Process)/job.out
error = $(Process)/job.err
log = $(Process)/job.log
transfer_input_files = $(Process)/input.data,comparison.data
queue 3
</pre>

<table class="directory">
<tr>  
<th>Submit directory before submitting</th>
<th>After completion</th>
</tr>
<tr>
<td class="directory"><pre>
(submit_dir)/	submit.file
		my.exec
		comparison.data
		0/	input.data
		1/	input.data
		2/	input.data
</pre></td>
<td class="directory"><pre>
(submit_dir)/	submit.file
		my.exec
		comparison.data
		0.out	1.out	2.out
		0/	input.data  job.out
			job.err     job.log
		1/	input.data  job.out
			job.err     job.log
		2/	input.data  job.out
			job.err     job.log
</pre> </td>
</tr>
</table>

<b>Job working directory after input transfer, before execution</b>
<pre>
(working_dir)/  my.exec
		input.data
		comparison.data
		(other HTCondor tracking files)
</pre>

<a name="int-dir-initialdir"></a>
<h2>B. Numbered directories using $(Process) as the InitialDir</h2>
<p>Most users prefer using "IntitialDir" to organize all job data, including
 files created within the job, as opposed to the previous method. Below,
 HTCondor's "InitialDir"
 variable is used to set the initial directory of <i>each</i> job, establishing
 the location for log/output/error, as well as input files needed by the job and
 any files produced by the job.<br>
<b>NOTE that HTCondor still expects the executable to exist relative to
 submission directory.</b></p>

<b>Submit file:</b>
<pre>
InitialDir = $(Process)
executable = my.exec
arguments = input.data comparison.data my.out
output = job.out
error = job.err
log = job.log
transfer_input_files = input.data,../comparison.data
queue 3
</pre>

<table class="directory">
<tr>  
<th>Submit directory before submitting</th>
<th>After completion</th>
</tr>
<tr>
<td class="directory"><pre>
(submit_dir)/	submit.file
		my.exec
		comparison.data
		0/	input.data
		1/	input.data
		2/	input.data
</pre></td>
<td class="directory"><pre>
(submit_dir)/	submit.file
		my.exec
		comparison.data
		0/	input.data  job.out  my.out
			job.err     job.log
		1/	input.data  job.out  my.out
			job.err     job.log
		2/	input.data  job.out  my.out
			job.err     job.log
</pre> </td>
</tr>
</table>

<b>Job working directory after input transfer, before execution</b>
<pre>
(working_dir)/ 	my.exec
		input.data
		comparison.data
		(other HTCondor tracking files)
</pre>

<br>
<a name="own-variables"></a>
<h1>3. Using your own non-HTCondor variables in the submit file </h1>

<p>You can define your own variable(s), named as you like, for use
 throughout the submit file. Just make sure to not use a built-in
 variable that is already used by HTCondor ("output" for example).</p>

<b>Submit file:</b>
<pre>
# My variables:
myvar = compare-states
sharedfile = comparison.data
#
executable = $(myvar).exec
arguments = $(Process).data $(sharedfile) $(myvar).out
output = $(Process).out
error = $(Process).err
log = $(Process).log
transfer_input_files = $(Process).data,$(sharedfile)
queue 3
</pre>

<table class="directory">
<tr>  
<th>Submit directory before submitting</th>
<th>After completion</th>
</tr>
<tr>
<td class="directory"><pre>
(submit_dir)/	submit.file
		compare-states.exec
		comparison.data
		0.data
		1.data
		2.data
</pre></td>
<td class="directory"><pre>
(submit_dir)/	submit.file
		compare-states.exec
		comparison.data
		0.data  0.out  0.err  0.log
		1.data  1.out  1.err  1.log
		2.data  2.out  2.err  2.log
</pre> </td>
</tr>
</table>

<b>Job working directory after input transfer, before execution</b>
<pre>
(working_dir)/	my.exec
		0.data
		comparison.data
		(other HTCondor tracking files)
</pre>

<br>
<a name="notint"></a>
<h1>4. Jobs with non-numbered data</h1>


<p>Rather than creating multiple submit files for individual jobs
 or having to number your files, it is possible to write a single
 submit file that uses different non-numbered files by adding
  multiple "queue" statements.

<p>Essentially, the submit file establishes a set of
 overall submit file lines, but in combination with a set of
 submit file lines that are redefined with each instance of
 "queue <i>N</i>".
</p>

<a name="notint-files"></a>
<h2>Using a different, non-numbered input file for each job </h2>

<p>A. Using the ideas from the previous example with non-HTCondor
 submit file variables, you can redefine
 your own non-HTCondor variable for each instance of "queue":</p>

<b>Submit file:</b>
<pre>
executable = my.exec
arguments = $(state).data comparison.data my_$(state).out
output = $(state).out
error = $(state).err
log = $(state).log
transfer_input_files = $(state).data,comparison.data

state = iowa
queue 1

state = missouri
queue 1

state = wisconsin
queue 1
</pre>


<table class="directory">
<tr>  
<th>Submit directory before submitting</th>
<th>After completion</th>
</tr>
<tr>
<td class="directory"><pre>
(submit_dir)/	submit.file
		my.exec
		comparison.data
		iowa.data
		missouri.data
		wisconsin.data
</pre></td>
<td class="directory"><pre>
(submit_dir)/ 	submit.file
		my.exec
		comparison.data
		iowa.err	iowa.data  
		iowa.log	iowa.out
                missouri.err    missouri.data
                missouri.log    missouri.out
		wisconsin.err	wisconsin.data  
		wisconsin.log	wisconsin.out
		my_iowa.out
		my_missouri.out
		my_wisconsin.out
</pre> </td>
</tr>
</table>

<b>Job working directory after input transfer, before execution</b>
<pre>
(working_dir)/  my.exec
                iowa.data
                comparison.data
                (other HTCondor tracking files)
</pre>

<a name="notint-dir"></a>
<h2>B. Using a different non-numbered directory for each job</h2>

<p>In this example, we once again use multiple "queue" statements,
 but redefine the "InitialDir" each time.
</p>

<b>Submit file:</b>
<pre>
executable = my.exec
arguments = state.data comparison.data my_$(InitialDir).out
output = job.out
error = job.err
log = job.log
transfer_input_files = input.data,../comparison.data

InitialDir = iowa
queue 1

InitialDir = missouri
queue 1

InitialDir = wisconsin
queue 1
</pre>

<table class="directory">
<tr>  
<th>Submit directory before submitting</th>
<th>After completion</th>
</tr>
<tr>
<td class="directory"><pre>
(submit_dir)/	submit.file
		my.exec
		comparison.data
		iowa/	   state.data
		missouri/  state.data
		wisconsin/ state.data
</pre></td>
<td class="directory"><pre>
(submit_dir)/	submit.file
		my.exec
		comparison.data
		iowa/  	   state.data	job.out
			   job.err	job.log
			   my_iowa.out
		missouri/  state.data	job.out
			   job.err	job.log
			   my_missouri.out
		wisconsin/ state.data	job.out
			   job.err	job.log
			   my_wisconsin.out
</pre> </td>
</tr>
</table>

<b>Job working directory after input transfer, before execution</b>
<pre>
(working_dir)/  my.exec
                missouri.data
                comparison.data
                (other HTCondor tracking files)
</pre>

<!-- V. Separately organizing Output, Error, and Log files -->