docs update

Author: nsimakov

CHANGELOG.md

@@ -1,6 +1,17 @@
-Application Kernels Remote Runner
-=========================================
+# Application Kernels Remote Runner (AKRR ) Change Log
 
-## 2019-10-21 v2.1.0
+## 2020-08-10 v2.1.0
 
-- Many improvements and bug fixes over v1.0 
+- new CLI interface. Single entry to all routines: ```akrr [-v] <command> [arguments]```.
+- Converted to python3.
+- RPM installation is available for CentOS 7.8.
+- New AppKernels: HPCG, MDTest and ENZO.
+- New resource types support: OpenStack and stand alone machine without queue system (shell). 
+- Batch appkernels task submission, for example run all appkernels 10 times each as soon as possible.
+  Usable for checking performance before/after update.
+- Docker container based appkernel execution (HPCC on single node on docker enabled resources).
+- Many other improvements and bug fixes.
+
+## 2015-08-15 v1.0
+
+- Initial AKRR v1.0 release.

README.md

@@ -14,7 +14,7 @@ a given run frequency. Accordingly, through XDMoD, system managers have the
 ability to proactively monitor system performance as opposed to having to rely 
 on users to report failures or underperforming hardware and software.
 
-* [Overview](docs/AKRR_Overview.md)
+* [Overview](docs/index.md)
 * [Download](docs/AKRR_Download.md)
 * [Installation](docs/AKRR_Install.md)
 * [Update](docs/AKRR_Update.md)

docs/AKRR_Add_Resource.md

@@ -1,12 +1,15 @@
-This section describes how to add traditional HPC resource to AKRR. On 
-addition of OpenStack check 
-[Adding OpenStack Resource and Application Kernels Deployment](AKRR_Add_OpenStack_Resource_and_AppKernels.md.md)
+This section describes how to add traditional HPC resource to AKRR. AKRR also support
+execution on systems without queueing system and [OpenStack](AKRR_Add_OpenStack_Resource_and_AppKernels.md.md) (limited support at this point).  
+
 
 # Adding a New HPC Resource
 
-Addition of new HPC resource to AKRR consists of two steps: configuration of
-new resource and deployment of AKRR's HPC resource-side scripts and application 
-inputs. The last step also performs installation validation.
+HPC resource is added in two steps:
+
+1) Configuration of new resource
+2) Deployment of AKRR's HPC resource-side scripts and application inputs.
+
+The last step also performs installation validation.
 
 From the AKRR point of view, an HPC resource is a distinct and **homogeneous 
 set** of computational nodes. The resource name should reflect such a set. For 
@@ -72,10 +75,10 @@ XDMoD name._
 
 > **Tips and Tricks**
 >
-> If resource headnode do not reply on pinging use __--no-ping__ do disable that check.
+> If resource headnode do not reply on pinging use __--no-ping__ argument do disable that check.
 >
 > If your system is fairly non-standard (for example non-default port for ssh, 
-> usage of globus-ssh for access and similar) you can use __--minimalistic__ option.
+> usage of globus-ssh for access and similar) you can use __--minimalistic__ argument.
 > This option sets a minimalistic interactive session and the generated 
 > configuration file must be manually edited.
 
@@ -182,12 +185,11 @@ and check to make sure that only the key(s) you wanted were added.
     and move to resource validation and deployment step.
     i.e. execute:
         akrr resource deploy -r ub-hpc
-        
 ```
 
 > **Tips and Tricks**
 >
-> reducing number of ssh connection
+> **Reducing number of ssh connection**:
 > AKRR would generate a large number of ssh connections. If you don't want to 
 > stress you headnode in this manner you can set ssh to reuse the connections. Add 
 > following to ~/.ssh/config :
@@ -287,20 +289,8 @@ batchJobHeaderTemplate template variable
 will become "#SBATCH --nodes=2" in batch job script if application kernel should 
 run on two nodes.
 
-In order to enter shell curly brackets they should be enter as double curly 
-brackets. All double curly brackets will be replaced with single curly bracket 
-during batch job script generation.
-
-> Example:
->
-> "awk "{{for (i=0;i<$_TASKS_PER_NODE;++i)print}}"
-> 
-> in  template variable will become:
->
-> "awk "{for (i=0;i<$_TASKS_PER_NODE;++i)print}"
->
-> in  batch job script.
->
+In order to enter curly brackets itself they should be enter as double curly 
+brackets (i.e. ${{ENV_VAR}} in template will be ${ENV_VAR} in resulting script).
 
 The commented parameters will assume default values. Below is the description of 
 the parameters and their default values:
@@ -324,16 +314,18 @@ the parameters and their default values:
 | **Batch job script settings**                                                                                                                                                                                                                                      |
 | batch_scheduler                       | N        | Scheduler type: slurm or pbs. sge might work as well but was not tested                                                                                                                         | Must be set   |
 | batch_job_header_template             | N        | Header for batch job script. Describe the resources requests and set AKRR_NODELIST environment variable containing list of all nodes.See below for more detailed information.                   | Must be set   |
+| max_number_of_active_tasks            | Y        | Maximal number of active tasks, default is -1, that is no limits                                                                                                                                | -1            |
 
 
-## How to set _batch_job_header_template_
-
 _batch_job_header_template _is a template used in the generation of batch job 
 scripts. It specifies the resources (e.g. number of nodes) and other parameters 
 used by scheduler.
 
 The following are instructions on how to convert batch job script header to _batch_job_header_template_. 
+
+<!---
 For more details see [Batch Job Script Generation](AKRR_Batch_Job_Script_Generation.md). 
+--->
 
 Below is a batch script which execute NAMD application on resorch which use Slurm:
 
@@ -401,9 +393,11 @@ below:
 
 Now, we can generate test application kernel batch job script and visually 
 inspect it for mistake presence. Run:
+
 ```bash
 akrr task new --dry-run --gen-batch-job-only -r <resource_name> -a test -n 2
 ```
+
 This command will generate batch job script and output it to standard output. 
 Below is example of the output
 

docs/AKRR_Batch_Job_Script_Generation.md

@@ -0,0 +1,22 @@
+# How AKRR Generate Batch Scripts
+
+AKRR is designed with need to execute multiple application kernels on multiple HPC resources. 
+The complexity of this task in part comes from
+the fact that various HPC resources can be severely different in hardware and software. 
+For example, HPC resources can use different queueing
+systems, different vendors and versions of MPI, BLAS and others libraries. 
+Furthermore, some applications can be compiled in number of way
+which will affect how they should be executed (e.g. TCP/IP vs MPI). 
+This makes impossible to create a single job script which will work on all
+platforms. 
+A large set of separate job scripts for every application kernel on every HPC resource would be unbearable to maintain. 
+AKRR
+addresses variability of batch job scripts by the use of templates. 
+
+Job script is created from template for every new task.
+The root template for batch job script is located at 
+$AKRR_SRC/akrr/default_conf/default.resource.conf and listed bellow:
+
+```python
+
+```

docs/AKRR_Deployment_of_Application_Kernel_on_Resource.md

@@ -34,9 +34,13 @@ rarely installed system-wide and thus they need to be installed first.
 
 ## Generate Initiate Configuration File
 
-The initial configuration file is generated with 
-_akrr app add_ command. It will generate an initial 
-configuration file and place it to $AKRR_HOME/cfg/resource/<app kernelname>.conf.
+The initial configuration file is generated.
+
+```bash
+akrr app add -r <resource_name> -a <appkernel_name>
+``` 
+It will generate an initial 
+configuration file and place it to _$AKRR_HOME/etc/resource/<resource_name>/<appkernel_name>.conf_.
 
 # Edit Configuration File
 
@@ -47,39 +51,54 @@ application kernel on this particular machine/resource.  
 # Generate Batch Job Script and Execute it Manually (Optional)
 
 The purpose of this step is to ensure that the configuration lead to a correct 
-(and workable) batch job script. First the batch job script is generated with 
-**'akrr_ctl.sh batch_job'**. Then this script is executed in an interactive 
-session (this improves the turn-around in case of errors). If the script fails 
-to execute, the issues can be fixed first in that script itself and then merged 
-with the configuration file.
+(and workable) batch job script. First the batch job script is generated as:
+```bash
+# only print batch job script
+akrr task new --dry-run --gen-batch-job-only -r <resource_name> -a <appkernel_name> -n <number_of_nodes>
+# generate batch job script and copy it to resource (without running it)
+akrr task new --gen-batch-job-only -r <resource_name> -a <appkernel_name> -n <number_of_nodes>
+```
+
+Then this script is submitted manually ot executed in an interactive session 
+(this improves the turn-around in case of errors). If the script fails 
+to execute, the issues can be fixed first in that script itself followed by respective updates in configuration file.
 
 This step is somewhat optional because it is very similar to the next step. 
 However the opportunity to work in an interactive session will often improve the 
 turn-around time because there is no need to stay in queue for each iteration.
 
 # Perform Validation Run
 
-For this step **appkernel_validation.py** utility is used to validate 
-application kernel installation on particular resource. It execute the 
-application kernel and analyses its results. If it fails the problems need to be 
+This step validates application kernel installation on the resource. 
+
+```bash
+akrr app validate -r <resource_name> -a <appkernel_name> -n <number_of_nodes>
+```
+
+It execute the application kernel and analyses its results. If it fails the problems need to be 
 fixed and another round of validation should be performed
 
 # Schedule regular execution of application kernel
 
 Finally, if validation was successful the application kernel can be submited for 
 regular execution on that resource.
 
+```bash
+akrr task new -r <resource_name> -a <appkernel_name> -n <list of nodes counts> -p <periodicity> \
+    -s <first submit date-time>
+```
+
 # Details on the Individual Application Kernels Deployment
 
 * [NAMD Deployment](AKRR_NAMD_Deployment.md)
 * [HPCC Deployment](AKRR_HPCC_Deployment.md)
+* [HPCG Deployment](AKRR_HPCG_Deployment.md)
 * [IMB Deployment](AKRR_IMB_Deployment.md)
 * [IOR Deployment](AKRR_IOR_Deployment.md)
-* [HPCG Deployment](AKRR_HPCG_Deployment.md)
+* [MDTest Deployment](AKRR_MDTest_Deployment.md)
 * [NWChem Deployment](AKRR_NWChem_Deployment.md)
 * [GAMESS Deployment](AKRR_GAMESS_Deployment.md)
 * [Enzo Deployment](AKRR_Enzo_Deployment.md)
-
 * [Creating New Application Kernel](AKRR_Creating_New_Application_Kernel.md)
 
 

docs/AKRR_Download.md

@@ -1,6 +1,5 @@
 # AKRR Download
 
-RPM package and source code of upcomming new version of AKRR (2.0) will be 
-available upon release at <https://github.com/ubccr/akrr/releases>.
+RPM package and source code of resent releases is available at <https://github.com/ubccr/akrr/releases>.
 
-Next: [AKRR installation](AKRR_Install.md)
+Next: [AKRR installation](AKRR_Install.md)

docs/AKRR_ENZO_Deployment.md

@@ -308,6 +308,9 @@ akrr task new -r $RESOURCE -a $APPKER -n 1,2,4,8
 
 #Start daily execution from today on nodes 1,2,4,8 and distribute execution time between 1:00 and 5:00
 akrr task new -r $RESOURCE -a $APPKER -n 1,2,4,8 -t0 "01:00" -t1 "05:00" -p 1
+
+# Run on all nodes count 20 times (default number of runs to establish baseline)
+akrr task new -r $RESOURCE -a $APPKER -n 1,2,4,8 --n-runs 20
 ```
 
 see [Scheduling and Rescheduling Application Kernels](AKRR_Tasks_Scheduling.md) and 

docs/AKRR_GAMESS_Deployment.md

@@ -351,6 +351,9 @@ akrr task new -r $RESOURCE -a $APPKER -n 1,2,4,8
 
 #Start daily execution from today on nodes 1,2,4,8 and distribute execution time between 1:00 and 5:00
 akrr task new -r $RESOURCE -a $APPKER -n 1,2,4,8 -t0 "01:00" -t1 "05:00" -p 1
+
+# Run on all nodes count 20 times (default number of runs to establish baseline)
+akrr task new -r $RESOURCE -a $APPKER -n 1,2,4,8 --n-runs 20
 ```
 
 see [Scheduling and Rescheduling Application Kernels](AKRR_Tasks_Scheduling.md) and 

docs/AKRR_HPCC_Deployment.md

@@ -356,6 +356,9 @@ akrr task new -r $RESOURCE -a $APPKER -n 1,2,4,8
 
 #Start daily execution from today on nodes 1,2,4,8 and distribute execution time between 1:00 and 5:00
 akrr task new -r $RESOURCE -a $APPKER -n 1,2,4,8 -t0 "01:00" -t1 "05:00" -p 1
+
+# Run on all nodes count 20 times (default number of runs to establish baseline)
+akrr task new -r $RESOURCE -a $APPKER -n 1,2,4,8 --n-runs 20
 ```
 
 see [Scheduling and Rescheduling Application Kernels](AKRR_Tasks_Scheduling.md) and 

docs/AKRR_HPCG_Deployment.md

@@ -264,6 +264,9 @@ akrr task new -r $RESOURCE -a $APPKER -n 1,2,4,8
 
 #Start daily execution from today on nodes 1,2,4,8 and distribute execution time between 1:00 and 5:00
 akrr task new -r $RESOURCE -a $APPKER -n 1,2,4,8 -t0 "01:00" -t1 "05:00" -p 1
+
+# Run on all nodes count 20 times (default number of runs to establish baseline)
+akrr task new -r $RESOURCE -a $APPKER -n 1,2,4,8 --n-runs 20
 ```
 
 see [Scheduling and Rescheduling Application Kernels](AKRR_Tasks_Scheduling.md) and 

docs/AKRR_IMB_Deployment.md

@@ -295,6 +295,9 @@ akrr task new -r $RESOURCE -a $APPKER -n 2,4,8
 
 #Start daily execution from today on nodes 2,4,8 and distribute execution time between 1:00 and 5:00
 akrr task new -r $RESOURCE -a $APPKER -n 2,4,8 -t0 "01:00" -t1 "05:00" -p 1
+
+# Run on all nodes count 20 times (default number of runs to establish baseline)
+akrr task new -r $RESOURCE -a $APPKER -n 2,4,8 --n-runs 20
 ```
 
 see [Scheduling and Rescheduling Application Kernels](AKRR_Tasks_Scheduling.md) and 

docs/AKRR_IOR_Deployment.md

@@ -41,7 +41,7 @@ installed parallel HDF5 library than you might want to skip it.
 Below are brief notes on parallel hdf5 installation,  
 [http://www.hdfgroup.org/HDF5/](http://www.hdfgroup.org/HDF5/) for HDF5 installation details.
 
-ior-3.2.0 does not work yet with hdf5-1.10.5, so use hdf5-1.8.*.
+> **Note:** ior-3.2.0 does not work with hdf5-1.10.\*, so use hdf5-1.8.\* for that version or use development version of ior (3.3-dev).
 
 **On target resource:**
 ```bash
@@ -799,6 +799,9 @@ akrr task new -r $RESOURCE -a $APPKER -n 1,2,4,8
 
 #Start daily execution from today on nodes 1,2,4,8 and distribute execution time between 1:00 and 5:00
 akrr task new -r $RESOURCE -a $APPKER -n 1,2,4,8 -t0 "01:00" -t1 "05:00" -p 1
+
+# Run on all nodes count 20 times (default number of runs to establish baseline)
+akrr task new -r $RESOURCE -a $APPKER -n 1,2,4,8 --n-runs 20
 ```
 
 see [Scheduling and Rescheduling Application Kernels](AKRR_Tasks_Scheduling.md) and 

docs/AKRR_MDTest_Deployment.md

@@ -283,6 +283,9 @@ akrr task new -r $RESOURCE -a $APPKER -n 1,2,4,8
 
 #Start daily execution from today on nodes 1,2,4,8 and distribute execution time between 1:00 and 5:00
 akrr task new -r $RESOURCE -a $APPKER -n 1,2,4,8 -t0 "01:00" -t1 "05:00" -p 1
+
+# Run on all nodes count 20 times (default number of runs to establish baseline)
+akrr task new -r $RESOURCE -a $APPKER -n 1,2,4,8 --n-runs 20
 ```
 
 see [Scheduling and Rescheduling Application Kernels](AKRR_Tasks_Scheduling.md) and 

docs/AKRR_NAMD_Deployment.md

@@ -504,11 +504,14 @@ DONE, you can move to next step!
 Now this application kernel can be submitted for regular execution:
 
 ```bash
-#Perform a test run on all nodes count
+# Perform a test run on all nodes count
 akrr task new -r $RESOURCE -a $APPKER -n 1,2,4,8
 
 #Start daily execution from today on nodes 1,2,4,8 and distribute execution time between 1:00 and 5:00
 akrr task new -r $RESOURCE -a $APPKER -n 1,2,4,8 -t0 "01:00" -t1 "05:00" -p 1
+
+# Run on all nodes count 20 times (default number of runs to establish baseline)
+akrr task new -r $RESOURCE -a $APPKER -n all --n-runs 20
 ```
 
 see [Scheduling and Rescheduling Application Kernels](AKRR_Tasks_Scheduling.md) and 

docs/AKRR_NWChem_Deployment.md

@@ -307,6 +307,9 @@ akrr task new -r $RESOURCE -a $APPKER -n 1,2,4,8
 
 #Start daily execution from today on nodes 1,2,4,8 and distribute execution time between 1:00 and 5:00
 akrr task new -r $RESOURCE -a $APPKER -n 1,2,4,8 -t0 "01:00" -t1 "05:00" -p 1
+
+# Run on all nodes count 20 times (default number of runs to establish baseline)
+akrr task new -r $RESOURCE -a $APPKER -n 1,2,4,8 --n-runs 20
 ```
 
 see [Scheduling and Rescheduling Application Kernels](AKRR_Tasks_Scheduling.md) and