05_pkg.qmd

---
title: "Managing Julia packages on HPC"
engine: knitr
---

## Overview 

Installing a package in Julia includes two steps (1) downloading (2) pre-compiling.
In a laptop, both are done in one shot using 

```julia
]
activate experiment_repo/julia_env
add NameOfPackage
```

In an HPC setup, step (2) requires computational effort hence is best done in compute node. However 
in some HPC setups such as Sockeye, compute nodes do not have internet access, so the 
process needs to be split. We cover here some utilities that facilitate this.


## Julia environment

A Julia environment is a specification of all package 
dependencies and their versions. 

By convention, we will store it in a 
directory called `experiment_repo/julia_env`.


## Adding a package

There are two syntaxes in Julia to install packages: the 
interactive Julia package manager or using programmatic Pkg 
syntax. We cover both below. 

In both syntaxes, we first need to tell Julia which environment 
to use. This is called "activating" an environment.


### Interactive Julia package manager

The most common method is to use the interactive Julia package 
manager. To start it, type `]` followed by `enter`. 

Activating is done with the `activate` keyword. 

You can then add a package using the `add` command. The argument of 
`add` can be either a registered Julia package, for example 
we show here how to add the 
[Pigeons package](https://github.com/Julia-Tempering/Pigeons.jl) 
for MCMC sampling:

```julia
ENV["JULIA_PKG_PRECOMPILE_AUTO"]=0 # Hold off precompile since we are in login node
]
activate experiment_repo/julia_env
add Pigeons
```

Alternatively, the package can be a git repository, optionally with a specific 
commit/tag:

```julia
add Example@0.5
add Example#master
add Example#c37b675
add https://github.com/JuliaLang/Example.jl#master
add git@github.com:JuliaLang/Example.jl.git
add "git@github.com:JuliaLang/Example.jl.git"#master
add https://github.com/Company/MonoRepo:juliapkgs/Package.jl
```

To exit the interactive Julia package manager, type `control-C`.


### Programmatic interface

Alternatively, a programmatic interface is also available 
for scripting:

```{julia output = F}
ENV["JULIA_PKG_PRECOMPILE_AUTO"]=0 # Hold off precompile since we are in login node
using Pkg 
Pkg.activate("experiment_repo/julia_env")
Pkg.add("Pigeons")
```


## Precompilation on HPC

Once we have downloaded the packages in the login node, 
we now turn to the task of performing pre-compilation. 

```{bash}
cd experiment_repo 
./nextflow run nf-nest/pkg.nf -profile cluster
```

Optionally, you can append an argument to 
specify the number of threads 
to request and use during pre-compilation: e.g. add `--nPrecompileThreads 20` 
to request 20 threads instead of the default of 10. 

## Testing your Julia environment interactively

In the code above, we have added the package 
[Pigeons](https://github.com/Julia-Tempering/Pigeons.jl) as an example. 

To interactively test an installed package, simply activate the 
environment from a Julia session:

```{julia output = F}
using Pkg
Pkg.activate("experiment_repo/julia_env")

using Pigeons
pigeons(target = toy_mvn_target(1000))
```


## More information

See [Pkg.jl documentation](https://pkgdocs.julialang.org/v1/).