docker-compose: complement the docker-compose install description for multi-node cluster scenario

[cherusk]

Is your feature request related to a problem? Please describe.

The k3s docker-compose stack does not behave well on nixos hosts for a multi node cluster on a singlehost.

It has many loose components that can fail independently, plugin loads, user limits etc.

NIgh they offer a docker-compose reference stack that also describes how worker should be configured.

Describe the solution you would like

You have already a document that describes a docker-compose fashioned install for a cluster:

https://docs.k0sproject.io/head/k0s-in-docker/#use-docker-compose-alternative

But this k0s cluster would be comprised of only one controller node.

I'd like to add worker nodes.

It might be possible to get the docker-compose.yaml into a workable shape by inferring from the cli based docker worker cluster inclusion.

But could't you make the that more convenient for consumers of k0s by offering a reference, functional docker-compose file for a cluster.

Describe alternatives you've considered

k3s docker-compose -> https://github.com/k3s-io/k3s/blob/master/docker-compose.yml

Additional context

No response

[adlion]

I am using this on my docker compose for bringing up master and worker nodes

services:
  kube0:
    container_name: kube0
    image: docker.io/k0sproject/k0s:v1.31.5-k0s.0
    command: |
      /bin/sh -c "
      rm -rf /shared/join-token.env && 
      k0s controller --config=/etc/k0s/config.yaml &
      sleep 5 &&
      mkdir -p /shared &&
      k0s token create --role=worker --expiry=1h > /shared/join-token.env &&
      wait
      "
    hostname: kube0
    privileged: true
    cgroup: host
    deploy:
      resources:
        limits:
          cpus: '1.0'
          memory: 1g
        reservations:
          cpus: '0.25'
          memory: 100m
    volumes:
      - "kube0_shared:/shared"
      - "kube0:/var/lib/k0s"
      - "./k0s.config.yaml:/etc/k0s/config.yaml:ro"
    ports:
      - "6443:6443"
    network_mode: "bridge"

  kube1:
    image: docker.io/k0sproject/k0s:v1.31.5-k0s.0
    container_name: kube1
    hostname: kube1
    cgroup: host
    deploy:
      resources:
        limits:
          cpus: '1.0'
          memory: 1g
        reservations:
          cpus: '0.25'
          memory: 100m
    privileged: true
    ports:
      - "32080:32080"
      - "32443:32443"
      - "32081:32081"
    depends_on:
      - kube0
    volumes:
      - "kube0_shared:/shared"
      - "kube1:/var/lib/k0s"
      - "./worker_join.sh:/shared/worker_join.sh"
    network_mode: "bridge"
    command: /bin/sh /shared/worker_join.sh
  
  kube2:
    image: docker.io/k0sproject/k0s:v1.31.5-k0s.0
    container_name: kube2
    hostname: kube2
    cgroup: host
    deploy:
      resources:
        limits:
          cpus: '1.0'
          memory: 1g
        reservations:
          cpus: '0.25'
          memory: 100m
    privileged: true
    depends_on:
      - kube0
    volumes:
      - "kube0_shared:/shared"
      - "kube2:/var/lib/k0s"
      - "./worker_join.sh:/shared/worker_join.sh"
    network_mode: "bridge"
    command: /bin/sh /shared/worker_join.sh


volumes:
  kube0_shared:
  kube0:
  kube1:
  kube2:

And the script

#!/bin/sh
# Sleep 10 seconds to leave master to delete the previous shared token for workers
sleep 10
# Wait until the token file is available and valid
while true; do
  if [ -f /shared/join-token.env ]; then
    echo "Token file detected. Reading contents..."
    TOKEN=$(cat /shared/join-token.env | tr -d '\r\n')
    if [ -n "$TOKEN" ]; then
      echo "Valid token found!"
      echo "DEBUG: Token is: $TOKEN"
      break
    else
      echo "Token file exists but is empty. Retrying..."
    fi
  else
    echo "Waiting for token file to be created..."
  fi
  sleep 2
done

# Start the join with the retrieved token
echo "Join token available"
# exec k0s controller --no-taints --token-file "/shared/join-token.env" --enable-worker
exec k0s worker --token-file "/shared/join-token.env"

So far it has worked without any issue

[cherusk]

@adlion Thanks, I'll try your configuration reference.

[cherusk]

For me there was no way to get a docker compose k0s stack workable.

Especially the dynamic volume allocation posed an issue, even via openebs.

I have resorted to a minimal kind stack on NIXos: https://github.com/godon-dev/godon-charts/blob/main/kind_config.yaml

That is working as expected with minimal configuration needed.