Run StatefulSet

This page shows how to leverage Kueue’s scheduling and resource management capabilities when running StatefulSets.

We demonstrate how to support scheduling StatefulSets in Kueue based on the Plain Pod Group integration, where StatefulSet is represented as a single independent workload.

This guide is for serving users that have a basic understanding of Kueue. For more information, see Kueue’s overview.

Before you begin

1. Learn how to install Kueue with a custom manager configuration.

2. Ensure that you have the v1/statefulset integration enabled, for example:

   apiVersion: config.kueue.x-k8s.io/v1beta1
   kind: Configuration
   integrations:
     frameworks:
      - "pod" # required by statefulset
      - "statefulset"
   

   Also, follow steps in Run Plain Pods to learn how to enable the v1/pod integration and how to configure it using the podOptions field.

3. Check Administer cluster quotas for details on the initial Kueue setup.

Running a StatefulSet admitted by Kueue

When running a StatefulSet on Kueue, take into consideration the following aspects:

a. Queue selection

The target local queue should be specified in the metadata.labels section of the StatefulSet configuration.

metadata:
   labels:
      kueue.x-k8s.io/queue-name: user-queue

b. Configure the resource needs

The resource needs of the workload can be configured in the spec.template.spec.containers.

spec:
  template:
     spec:
      containers:
       - resources:
           requests:
             cpu: 3

c. Scaling

Currently, scaling operations on StatefulSets are not supported. This means you cannot perform scale up or scale down operations directly through Kueue.

Example

Here is a sample StatefulSet:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: sleep-statefulset
  labels:
    app: sleep
    kueue.x-k8s.io/queue-name: user-queue
spec:
  replicas: 3
  selector:
    matchLabels:
      app: sleep
  template:
    metadata:
      labels:
        app: sleep
    spec:
      containers:
        - name: sleep
          image: "gcr.io/k8s-staging-perf-tests/sleep:v0.1.0@sha256:8d91ddf9f145b66475efda1a1b52269be542292891b5de2a7fad944052bab6ea"
          args:
            - "5m"
          resources:
            requests:
              cpu: "3"
  serviceName: "sleep"

You can create the StatefulSet using the following command:

kubectl create -f sample-statefulset.yaml