Support for running Multi-Cluster Argo Workflows

[dmildh-absci]

Hello Karmada Gurus! I am in the process of seeing if I can get Karmada working to support a Multi-Cluster setup similar to how one of the Kubestellar folks documented this here. So far I have the kind cluster setup from your repo that stood up the control cluster with the Karmada api components and the 3 member servers. I have also installed Argo workflows on the member clusters and the Karama-host and current am able to submit the below sample workflow and see it get created on the member servers but what I am trying to do is similar to the Kubesteller article in which you would preferably have a single workflow UI that can show you the status of all of the member cluster's workflows and not quite sure if this is supported. I think this could be a really useful use case for those that are trying to set this up along with some supporting docs and some example code.

I have diagrammed out the design below to help clarify what I think it would looks like but not sure yet about the Argo Workflows server component and where that sits.

Here is the example workflow and propagation policy I am using.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  labels:
    app: sample0
  name: sample0
  namespace: argo
spec:
  parallelism: 1
  serviceAccountName: default
  entrypoint: volumes-existing-example
  tolerations:
  - key: 'key'
    operation: 'Equal'
    value: 'value'
    effect: 'NoSchedule'

  templates:
  - name: volumes-existing-example
    steps:
    - - name: hello
        template: whalesay

  - name: whalesay
    container:
      image: docker/whalesay:latest
      command: [cowsay]
      args: ["hello world"]
      resources:
        limits:
          memory: 1Gi
        requests:
          memory: 1Mi

---
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: sample0
  namespace: argo
  labels:
    app: sample0
spec:
  propagateDeps: true
  resourceSelectors:
    - apiVersion: argoproj.io/v1alpha1
      kind: Workflow
      name: sample0
  placement:
    clusterAffinity:
      clusterNames:
        - member1
        - member3
    replicaScheduling:
      replicaSchedulingType: Duplicated

If anyone can provide any guidance on this or can help please let me know.

[likku123]

I am early looking to implement the same approach as well . Installing Argoworkflow in Karmada control plane and submitting the jobs to members clusters which does not have the Argoworkflow installed.

@RainbowMango Any pointers on this ask ?

[chaosi-zju]

I am early looking to implement the same approach as well . Installing Argoworkflow in Karmada control plane and submitting the jobs to members clusters which does not have the Argoworkflow installed.

I used to achieved this purpose

but we can only achieve simple work through examples like DAG

as for complex example like Artifacts, we can't, since we can't resolve such scene that the output of one step is exactly the input of another step.

[likku123]

@chaosi-zju

Is it possible to share a sample workflow yaml ? I just want to make sure what changes in have to make to run the workflow in multi cluster.

Thank you !

[RainbowMango]

I have also installed Argo workflows on the member clusters and the Karama-host and current am able to submit the below sample workflow and see it get created on the member servers but what I am trying to do is similar to the Kubesteller article in which you would preferably have a single workflow UI that can show you the status of all of the member cluster's workflows and not quite sure if this is supported.

Hi @dmildh-absci Interesting use case, and thanks @likku123 for bringing up.

Just went through the docs and I don't see any gap for doing the same thing with Karmada.
Regarding the status reflection, we achieve that by declaring a ResourceInterpreter .

In addition for the Workflow, Karmada maintains a built-in ResourceInterpreters here , seems all we need to add two more operations InterpretStatus and AggregateStatus.

Let me know if you are still interested in this. Glad to talk more.

[likku123]

Thank you ! @RainbowMango For the docs and pointers. Regarding schedulers in Karmada is it preferable to use volcano because of its active development around the custom schedulers? Is it required to install volcano-global for multi cluster or Karmada with volcano is sufficient ?