Troubleshooting on Kueue

Troubleshooting Jobs

Thu, 21 Mar 2024 00:00:00 +0000

This doc is about troubleshooting pending Kubernetes Jobs, however, most of the ideas can be extrapolated to other supported job types.

See the Kueue overview to visualize the components that collaborate to run a Job.

In this document, assume that your Job is called my-job and it’s in the my-namespace namespace.

Identifying the Workload for your Job

For each Job, Kueue creates a Workload object to hold the details about the admission of the Job, whether it was admitted or not.

Troubleshooting Queues

Thu, 21 Mar 2024 00:00:00 +0000

Why no workloads are admitted in the LocalQueue?

The status of the LocalQueue includes details of any configuration problems on the LocalQueue, as part of the Active condition.

Run the following command to see the status of the LocalQueue:

kubectl get localqueue -n my-namespace my-local-queue -o yaml

The status of the LocalQueue will be similar to the following:

Troubleshooting Provisioning Request in Kueue

Mon, 20 May 2024 00:00:00 +0000

This document helps you troubleshoot ProvisioningRequests, an API defined by ClusterAutoscaler.

Kueue creates ProvisioningRequests via the Provisioning Admission Check Controller, and treats them like an Admission Check. In order for Kueue to admit a Workload, the ProvisioningRequest created for it needs to succeed.

Before you begin

Before you begin troubleshooting, make sure your cluster meets the following requirements:

Troubleshooting Pods

Thu, 21 Mar 2024 00:00:00 +0000

This doc is about troubleshooting plain Pods when directly managed by Kueue, in other words, Pods that are not managed by kubernetes Jobs or supported CRDs.

Note

This doc focuses on the behavior of Kueue when managing Pods that is different from other job integrations. You can read Troubleshooting Jobs for more general troubleshooting steps.

Is my Pod managed directly by Kueue?

Kueue adds the label kueue.x-k8s.io/managed with value true to Pods that it manages. If the label is not present on a Pod, it means that Kueue is not going to admit or account for the resource usage of this Pod directly.

Troubleshooting delete ClusterQueue

Wed, 28 Aug 2024 00:00:00 +0000

Deleting a ClusterQueue might require additional steps when it has active Workloads.

This guide provides a clear approach to safely deleting a ClusterQueue, explaining the process and offering alternative methods to handle specific scenarios.

For this example, assume your ClusterQueue is named my-cq.

Understanding `kubectl delete clusterqueue`

When you run the following command:

Troubleshooting on Kueue

Troubleshooting Jobs

Identifying the Workload for your Job

Troubleshooting Queues

Why no workloads are admitted in the LocalQueue?

Troubleshooting Provisioning Request in Kueue

Before you begin

Troubleshooting Pods

Note

Is my Pod managed directly by Kueue?

Troubleshooting delete ClusterQueue

Understanding kubectl delete clusterqueue

Understanding `kubectl delete clusterqueue`