Multi-Cluster Job dispatching on Kueue

Run Deployment in Multi-Cluster

Mon, 17 Feb 2025 00:00:00 +0000

Before you begin

Check the MultiKueue installation guide on how to properly setup MultiKueue clusters.
Deployments receive live status updates through the status from the remote Pods created on the worker cluster.

Pod integration requirements

Since Kueue v0.15, you don’t need to explicitly enable "pod" integration to use the "deployment" integration.

Run Plain Pod in Multi-Cluster

Mon, 17 Feb 2025 00:00:00 +0000

Before you begin

Check the MultiKueue installation guide on how to properly setup MultiKueue clusters.
Follow steps in Run Plain Pods to learn how to enable and configure the pod integration.

Pods created on the manager cluster are automatically gated and receive live status updates from their remote counterparts

Run StatefulSet in Multi-Cluster

Wed, 15 Jan 2025 00:00:00 +0000

Before you begin

Check the MultiKueue installation guide on how to properly setup MultiKueue clusters.

Pod integration requirements

Since Kueue v0.15, you don’t need to explicitly enable "pod" integration to use the "statefulset" integration.

For Kueue v0.14 and earlier, "pod" integration must be explicitly enabled.

Run Kubernetes Job in Multi-Cluster

Tue, 05 Nov 2024 00:00:00 +0000

Before you begin

Check the MultiKueue installation guide on how to properly setup MultiKueue clusters.

The current status of a Job being executed by MultiKueue on a worker cluster is live-updated on the management cluster.

This gives the users and automation tools the ability to track the progress of Job status (.status) without lookup to the worker cluster, making MultiKueue transparent from that perspective.

Run TrainJobs in Multi-Cluster

Thu, 06 Nov 2025 00:00:00 +0000

Before you begin

Check the MultiKueue installation guide on how to properly setup MultiKueue clusters.

For the ease of setup and use we recommend using at least Kueue v0.11.0 and for Kubeflow Trainer at least v2.0.0.

See Trainer Installation for installation and configuration details of Kubeflow Trainer v2.

Run AppWrappers in Multi-Cluster

Mon, 23 Jun 2025 00:00:00 +0000

Before you begin

Check the MultiKueue installation guide on how to properly setup MultiKueue clusters.

For the ease of setup and use we recommend using at least Kueue v0.11.0 and at least AppWrapper Operator v1.1.1.

See AppWrapper Quick-Start Guide for installation and configuration details of the AppWrapper Operator.

Run Jobsets in Multi-Cluster

Wed, 25 Sep 2024 00:00:00 +0000

Before you begin

Check the MultiKueue installation guide on how to properly setup MultiKueue clusters.

For the ease of setup and use we recommend using at least Kueue v0.8.1 and JobSet v0.6.0 (see JobSet Installation for more details).

Run Jobsets example

Once the setup is complete you can test it by running the jobset-sample.yaml Job.

Run KubeRay Jobs in Multi-Cluster

Wed, 19 Mar 2025 00:00:00 +0000

Before you begin

Check the MultiKueue installation guide on how to properly setup MultiKueue clusters.

For the ease of setup and use we recommend using at least Kueue v0.11.0 and for KubeRay Operator at least v1.3.1.

See KubeRay Operator Installation for installation and configuration details of KubeRay Operator.

Run Kubeflow Jobs (v1) in Multi-Cluster

Wed, 25 Sep 2024 00:00:00 +0000

Note

This page covers legacy Kubeflow Trainer v1 jobs (PyTorchJob, TFJob, XGBoostJob, etc.).

For Kubeflow Trainer v2 TrainJob, see Run TrainJobs in Multi-Cluster.

Before you begin

Check the MultiKueue installation guide on how to properly setup MultiKueue clusters.

For the ease of setup and use we recommend using at least Kueue v0.11.0 and for Kubeflow Trainer at least v1.9.0.

Run External Framework Jobs in Multi-Cluster

Fri, 19 Sep 2025 00:00:00 +0000

Feature state alpha since Kueue v0.14

Before you begin

Check the MultiKueue installation guide on how to properly setup MultiKueue clusters.
Enable the MultiKueueAdaptersForCustomJobs feature gate. This feature is in Alpha and is disabled by default.

To enable the feature, add the --feature-gates=MultiKueueAdaptersForCustomJobs=true flag to the Kueue controller manager’s arguments. For example, you can apply it to your deployment with the following command:

Run MPIJobs in Multi-Cluster

Fri, 25 Oct 2024 00:00:00 +0000

Before you begin

Check the MultiKueue installation guide on how to properly setup MultiKueue clusters.

For the proper setup and use it is required using at least Kueue v0.9.0 and for MPI Operator at least v0.6.0.

Installation on the Clusters

Note

While both MPI Operator and Trainer must be running on the same cluster, there are special steps that has to be applied to Trainer deployment. See Working alongside MPI Operator for more details.

See MPI Operator Installation for installation and configuration details of MPI Operator.

Run LeaderWorkerSet in Multi-Cluster

Fri, 16 Jan 2026 00:00:00 +0000

Before you begin

Check the MultiKueue installation guide on how to properly setup MultiKueue clusters.

How it works

LeaderWorkerSet creates one Workload per replica, which requires special handling in MultiKueue.