Ray tutorial

[keshavb96]

This document presents a detailed tutorial on how Ray can be used together with JAX to achieve fault tolerant training.

[gspschmid]

Tried this out and needed to make a few changes to get the container ready (see inline comments).

Once that was set up I ran into the following error on a viking node:

$ docker build -t ray_resiliency_example -f Dockerfile .
...
$ docker run --gpus=all --name resilient_jax --network=host --security-opt seccomp=unconfined --cap-add SYS_PTRACE -it --shm-size=50g --ulimit memlock=-1 ray_resiliency_example
root@viking-prod-283:/ray_resiliency_example# ./launch_ray_job.sh
...
redis.exceptions.ConnectionError: Error 111 connecting to 10.78.2.240:6380. Connection refused.

Full log here: https://gist.github.com/gspschmid/ff1d8e7873a5010d880cc8350bf314f1

Nvm, launch_ray_job.sh had the line to launch redis commented out, it seems to work after uncommenting that!

[gspschmid]

One comment re triggering multiple non-rank-0 failures