InstaScale is a controller that works with Multi-cluster-app-dispatcher (MCAD) to get aggregated resources available in the kubernetes cluster without creating pending pods. It uses machinesets to launch instances on cloud provider to be added to the Kubernetes cluster.

Key features:

• Acquires aggregated heterogenous instances needed for workload execution.
• Does not clog Kubernetes control plane.
• Works with your Kubernetes scheduling system to schedule pods on aggregated resources.
• Terminates instances on workload completion.

InstaScale and MCAD interaction

• User submits Multi GPU job(s)
• Job(s) lands in MCAD queue
• When resources are not available it triggers scaling i.e. calls InstaScale
• InstaScale looks at resource requests specified by the user and matches those with the desired Machineset(s) to get nodes.
• After InstaScal-ing, when aggregate resources are available to run the job MCAD dispatches the job.
• When job completes, resources obtained for the job are released.

Development

Pre-requisites

• Installed go version 1.18
• Running OpenShift cluster

Building

• To build locally : make build
• To run locally : make run

Image creation

• To build and release a docker image for controller : make IMG=quay.io/project-codeflare/instascale:<TAG> docker-build docker-push
• Note that the other contents of the Makefile (as well as the config and bin dirs) exist for future operator development, and are not currently utilized

Deployment

• Deploy MCAD using steps here.
• Deploy InstaScale using commands below:

git clone https://github.com/project-codeflare/instascale.git
cd deployment/
oc apply -f instascale-configmap.yaml
oc apply -f instascale-sa.yaml
oc apply -f instascale-clusterrole.yaml
oc apply -f instascale-clusterrolebinding.yaml
oc apply -f deployment.yaml

Running an InstaScale deployment locally with Visual Studio Code

• Deploy MCAD using steps here.
• Deploy InstaScale by following the steps below:

git clone https://github.com/project-codeflare/instascale.git
cd deployment/
oc apply -f instascale-configmap.yaml
oc apply -f instascale-sa.yaml
oc apply -f instascale-clusterrole.yaml
oc apply -f instascale-clusterrolebinding.yaml

• In Visual Studio Code update .vscode/launch.json so that "KUBECONFIG" points to your Kubernetes config file.
  
• If you changed the namespace in instascale-configmap.yaml update the args[] in launch.json to include --configs-namespace="<YOUR_NAMESPACE>".
  
• You can now run the local deployment with the debugger.

Testing

Run tests with command:

go test -v ./controllers/

Release process

Prerequisite:

• Build and release MCAD
• Make sure that MCAD version is published on Go package site

1. Run instascale-release.yml action.
2. Verify that instascale-release.yml action passed successfully.