Workload-Aware Scheduling in Kubernetes 1.37

In Kubernetes 1.36, we gave the scheduler a group primitive. Workload became a static policy template, PodGroup became the decoupled runtime object, and the gang plugin learned to hold pods at PreEnqueue and Permit until a quorum could be met. The Job controller could create those objects for you behind the WorkloadWithJob feature gate.

That answered “what the API is”. It left a second question open: “how does every controller adopt it without each one reinventing the same plumbing?”.

If the answer is “read the KEPs and hand-assemble Workload and PodGroup objects yourself”, we recreate the exact fragmentation Workload-Aware Scheduling (WAS) was meant to end, one layer up. The Job controller would carry one dialect, JobSet another, LeaderWorkerSet a third, and every out-of-tree controller its own.

Kubernetes 1.37 closes that gap with three things: a set of reusable building block APIs, a shared workloadbuilder Go library, and an updated, explicit Job integration built on both. This post walks through all three, and then through the path an out-of-tree controller takes to plug into WAS using them.

Building blocks APIs

The building blocks APIs are strongly-typed Go structs in `scheduling.k8s.io/v1alpha3` that expresses one scheduling concept. Controller authors embed these structs into their own API types as fields; the `workloadbuilder` library compiles them. Users never author raw `Workload` objects — they fill in fields on a resource they already understand.

The type names follow two conventions. Leaf-level blocks are prefixed `WorkloadPodGroup…` (for example `WorkloadPodGroupSchedulingPolicy`) and the multi-level variants are prefixed `WorkloadCompositePodGroup…`.

The key design decision was “a block imposes no fixed top-level shape”. Each controller picks the field names and structure idiomatic to its own API. Reusing the standard names is recommended, so a user who configured gang scheduling on one controller recognizes it on another, but at the same time it is a convention, not an enforced schema. The `Job` API, for instance, groups all four blocks under `spec.scheduling` via a single type:

Another controller might nest the same blocks per component of a multi-part workload, or support only a subset. There are four leaf blocks.

Scheduling policy

The same `basic` and `gang` policies as a `PodGroup`’s `spec.schedulingPolicy`. The one difference is that the block’s `gang.minCount` is optional — leave it unset and the controller supplies a domain-appropriate default. The Job controller defaults it to the Job’s `parallelism`.

Scheduling constraints

Topology co-location: a node-label key naming the domain (a rack, a zone) that every pod in the group must share, with at most one topology constraint per group. Constraints are immutable in the compiled `Workload`, so controllers should freeze the field after creation.

Disruption mode

Whether the group’s pods may be disrupted individually (`single`) or only as a unit (`all`). The library rejects combinations that are not meaningful — notably `all` disruption on a `basic` policy, because the preemption unit must not be larger than the scheduling unit. A group scheduled pod-by-pod has no group-level unit to disrupt.

Resource claims

Which DRA claims are shared by every pod in the group rather than allocated per pod. Each entry names the claim within the group and points at an existing `ResourceClaim` or a `ResourceClaimTemplate`; a group may declare at most four. Pods consume the group’s devices by declaring a matching claim in their own spec, using the same name and referring to the same object.

Composite building blocks

Multi-level controllers that orchestrate other controllers — JobSet creating Jobs, disaggregated inference on LeaderWorkerSet that coordinates a group of groups. For that layer the API provides an analogous set of primitives prefixed `WorkloadCompositePodGroup…`. They mirror the leaf blocks, with one telling difference: the composite gang policy uses `minGroupCount`. The minimum number of child groups that must be schedulable together — in place of the leaf’s `minCount`. Keeping leaf and composite types distinct lets each level of the hierarchy evolve independently.

The `workloadbuilder` library

Embedding building blocks tells the scheduler what you want. `workloadbuilder` is how you turn that intent into the `Workload` and `PodGroup` objects the scheduler actually reads, so no controller reimplements defaulting, validation, and template compilation.

It is built for both in-tree and out-of-tree controllers. The Job controller uses it; an out-of-tree controller such as JobSet or Kubeflow TrainJob vendors it like any other Go dependency. It ships from the staging/component-helpers.

In 1.37, it consumes the `v1alpha3` building blocks and compiles them into `v1beta1` `Workload` and `PodGroup` objects; `CompositePodGroup` objects remain `v1alpha3`.

The `WorkloadItem` tree

A controller describes its workload as a tree of `WorkloadItem` nodes, one per logical component. A node with no children becomes a single `PodGroupTemplate`; a node with children becomes a `CompositePodGroupTemplate` over them** — which is exactly how a multi-level controller represents a group of groups.

Each node carries three things:

default config: the controller’s own defaults for anything the user leaves unset (this is where an unconfigured Job stays on `basic`)
input: the user’s intent from the controller’s API, recorded together with the field path it lives at, so validation errors point at the exact field the user set
callbacks (optional): adjustments to the merged configuration, such as filling an unset gang `minCount` from the Job’s parallelism.

That field-path detail is the quiet win: errors surface against the user’s own CRD field, not against some synthetic `Workload` they never wrote.

The builder flow

Hand the tree to a `Builder` and work through four calls:

NewBuilder constructs the builder from the tree, plus the name, namespace, and owner reference for the object to produce. The owner becomes the `Workload`’s controller reference, used for discovery and garbage collection.
Validate resolves the tree and reports problems as a list of field errors your controller returns from its own validation.
BuildWorkload compiles the tree into a `Workload`. The result is cached, so several PodGroups can be created from one compiled result.
NewPodGroup creates a runtime `PodGroup` from one of the compiled templates, named.

Validation

Validate checks in two layers:

Structural validation of the blocks themselves which include required fields, value ranges, one-of unions, immutability that is generated from the API types via declarative validation.
Controller-policy checks that declarative validation can’t express — the allow-lists below, and cross-field rules such as rejecting `all` disruption alongside `basic`.

Validation also differs between create and update: on update the library enforces the fields frozen after creation, which means you pass the previously stored configuration alongside the new one.

Whether you want the first layer depends on where you run:

Out-of-tree controllers leave declarative validation on (the default), nothing else applies those structural rules to a custom resource, so one `Validate` call covers both layers.
In-tree controllers set DisableDeclarativeValidation, because the API server already runs declarative validation on the embedded blocks while validating the parent object. Skipping the first layer avoids checking the same fields twice.

Allow-lists

Because the block types are shared across controllers, future releases may add options that don’t make sense for every controller. `workloadbuilder` uses an allow-list model: a controller declares the policies and disruption modes it supports, and `Validate` rejects anything outside that set, reporting the error at the offending block’s field path.

Options are therefore denied by default. When a new policy is introduced upstream, an existing controller keeps rejecting it until its maintainers extend the allow-list — which for an out-of-tree controller also means updating its vendored copy of the library. So we can be sure that new scheduling behavior never leaks silently into a controller that hasn’t opted in.

Generating PodGroups from an existing Workload

When the Workload already exists (compiled by a parent controller, or created by hand) a child controller that only manages the runtime `PodGroup` uses `NewBuilderFromExistingWorkload`. It creates `PodGroup`s from the supplied `Workload` under its own owner reference, and does not validate or recompile anything, so the existing `Workload` is never recompiled.

This is the seam that makes the parent/child split below work.

Job controller integration - now explicit

The 1.36 Job integration was implicit: with `WorkloadWithJob` enabled, an eligible Job silently grew a `Workload` and `PodGroup` from a set of conditions. 1.37 makes the intent explicit and the implementation shared — the Job controller compiles its objects through `workloadbuilder`, exactly like an external controller would.

An explicit `.spec.scheduling` block

Users state scheduling intent directly on the Job. Omit the block and the Job defaults to `Basic` — the ordinary pod-by-pod outcome is unchanged.

The four `.spec.scheduling` fields map one-to-one onto the leaf building blocks. All of them are immutable after creation except `schedulingPolicy.gang.minCount`, which stays mutable to scale a gang elastically.

What the controller does with it

When the controller processes that Job, it:

Creates a `Workload` in the same namespace, with a `podGroupTemplate` compiled from `.spec.scheduling` (here, a gang policy with `minCount: 8`).
Creates a PodGroup from that template, the runtime scheduling unit, carrying an inline copy of the policy.
Creates pods with `spec.schedulingGroup.podGroupName` set, linking each pod to its group.

Discovery runs through spec references — `Workload.spec.controllerRef` and `PodGroup.spec.workloadRef` — not by name. Objects the controller creates carry an `ownerReferences` entry back to the Job, so they are garbage collected when the Job is deleted. A `Basic` Job (no `.spec.scheduling`) still gets a `Workload` and `PodGroup`, so the observable objects are consistent regardless of policy.

Elastic gangs

Because `gang.minCount` is mutable, you can rescale a gang in place: set `.spec.scheduling.schedulingPolicy.gang.minCount` directly, or change `.spec.parallelism` when `minCount` is unset. The controller recompiles the `Workload` and re-syncs the `PodGroup` size without recreating the Job; changes apply to pods evaluated in future scheduling cycles. A `minCount` greater than `parallelism` is rejected — that gang can never be satisfied.

Alpha limitations

Each Job maps to exactly one `PodGroup`; all its pods share a single scheduling policy.
Only `schedulingPolicy.gang.minCount` is mutable; every other `.spec.scheduling` field is frozen after creation.
Suspended Jobs retain their `Workload` and `PodGroup` — they are not deleted on suspend or recreated on resume.

Integrating an out-of-tree controller

This is the section the release is really for. Here is the path for a controller you own, a custom trainer, a serving operator, or an existing project like JobSet, LeaderWorkerSet, or Kueue adopting the native primitive.

Embed building blocks in your API: Add the `v1alpha3` blocks you support as fields on your CRD, named to fit your users’ mental model — the `JobSchedulingConfiguration` grouping is a good template to copy.
Build a WorkloadItem tree in your reconciler: One node for a flat workload; nested nodes for a multi-level one. Record each block with its field path so errors land on the user’s field.
Set your validation posture: Out-of-tree, leave declarative validation on. Declare your allow-lists (`AllowedPolicies`, `AllowedDisruptionModes`) so unsupported options are rejected up front.
Compile and create: `NewBuilder` → `Validate` → `BuildWorkload` → `NewPodGroup`. Create or patch the objects, then stamp your pods with `spec.schedulingGroup.podGroupName`.
Let the scheduler do the rest: Gang, topology, disruption, and preemption behavior come from the blocks you attached you implemented none of it.

What you are explicitly not doing: writing a gang controller, inventing a `PodGroup` dialect, or coupling your users to a specific backend scheduler.

The parent/child split, and how the Job controller already does it

The most useful pattern for multi-level controllers is visible in how the Job controller behaves under a parent such as JobSet. When a Job carries an `ownerReference` to a parent that compiles the `Workload`, the Job controller defers `Workload` ownership** to that parent. What it does about the runtime `PodGroup` depends on what the parent delegates:

If the parent sets the `scheduling.k8s.io/group-template-name` annotation on the Job, the Job controller creates and owns the `PodGroup`, mapped to the parent’s named `PodGroupTemplate`.
Otherwise the parent owns both objects; the Job controller creates neither and simply discovers and uses the existing ones when creating pods.
And if the Job’s pod template already sets `spec.template.spec.schedulingGroup`, the Job controller stays out of it entirely — you manage the `Workload`/`PodGroup` lifecycle yourself.

That is the whole out-of-tree contract in miniature: a parent controller uses `BuildWorkload` once to compile the tree, names its `PodGroupTemplate`s, and lets child controllers attach runtime `PodGroup`s via `NewBuilderFromExistingWorkload` — no child ever recompiles the parent’s `Workload`.

CompositePodGroup for tree-shaped workloads

Flat `PodGroup`s are enough for a single gang. Modern AI workloads are not flat: JobSet is a set of child jobs; disaggregated inference on LeaderWorkerSet is a leader plus workers with distinct roles.

CompositePodGroup lets a controller express that hierarchy directly, so the grouping the scheduler sees mirrors the grouping the workload actually has — a parent `WorkloadItem` compiles to a `CompositePodGroupTemplate` over its children, and the composite gang policy’s `minGroupCount` says how many child groups must land together.

Coming to the ecosystem

The in-tree primitives only matter if the controllers people actually run adopt them. The integrations with the broader ecosystem are already in flight. The WAS working group is now working with several controller communities, with positive feedback and no identified blockers for Beta. Three are worth calling out (alongside KubeRay and TrainJob, which are in the same cohort).

JobSet: The direction is for JobSet to compile a single parent Workload with named PodGroupTemplates, then let each child Job attach its runtime PodGroup through the delegation path shown earlier — the scheduling.k8s.io/group-template-name annotation and NewBuilderFromExistingWorkload.With CompositePodGroup expressing the group-of-groups. The child-Job seam already exists in the 1.37 Job controller, which is why JobSet is the most direct next consumer.
LeaderWorkerSet (LWS): LWS is a primary driver for the composite building blocks and CompositePodGroup, where a single minGroupCount expresses “the leader and its workers are admitted as one unit,” rather than LWS re-implementing that coordination itself.
Kueue: (the goal here is composition, not replacement) WAS provides the native gang primitive, and Kueue stays the queueing and quota layer on top. The work is clean interop, so a Kueue-admitted workload can express its gang through the in-tree Workload/PodGroup contract instead of a Kueue-specific one — letting admission, quota, and kube-scheduler’s gang stack rather than each re-implementing all-or-nothing placement.

If you maintain one of these or another workload controller, this is the moment to bring its shape to the working group, while the v1alpha3 surface is still malleable.

Try it yourself

Every scenario in this post is a runnable manifest in the companion repo blog-demo → was-1.37. Bring up a local `kind` cluster with the right feature gates, then work through the scenarios with the narrated `verify.sh`.

The most instructive first exercise is the explicit `.spec.scheduling` Job (`02-gang-job.yaml`). Apply it, then watch the controller emit the `Workload` and `PodGroup` and wire the pods to the group.