Distribute workload with placement selected managed clusters
The Placement
API is used to dynamically select a set of ManagedCluster
in
one or multiple ManagedClusterSets
so that the workloads can be deployed to
these clusters.
If you define a valid Placement
, the placement controller generates a
corresponding PlacementDecision
with the selected clusters listed in the
status. As an end-user, you can parse the selected clusters and then operate on
the target clusters. You can also integrate a high-level workload orchestrator
with the PlacementDecision
to leverage its scheduling capabilities.
For example, with OCM addon policy
installed, a Policy
that includes a Placement
mapping can distribute the
Policy
to the managed clusters.
For details see this example.
Some popular open source projects also integrate with the Placement
API. For
example Argo CD, it can leverage the
generated PlacementDecision
to drive the assignment of Argo CD Applications to a
desired set of clusters, details see this example.
And KubeVela, as an implementation of
the open application model, also will take advantage of the Placement
API for
workload scheduling.
And in this article, we want to show you how to use clusteradm
to deploy
ManifestWork
to Placement
selected clusters.
Prerequisites
Before starting with the following steps, suggest you understand the content below.
Placement: The
Placement
API is used to dynamically select a set ofManagedCluster
in one or multipleManagedClusterSets
so that higher-level users can either replicate Kubernetes resources to the member clusters or run their advanced workload i.e. multi-cluster scheduling.ManifestWork: A custom resource in the hub cluster that groups a list of Kubernetes resources together and is meant for dispatching them into the managed cluster if the
ManifestWork
is created in a validcluster namespace
.
Deploy manifestwork to placement selected managed clusters
In deploy Kubernetes resources to the managed clusters,
it shows you how to use clusteradm
to create a ManifestWork
and deploy it
onto a specific managed clusters. As Placement
can dynamically select a set of
ManagedCluster
, the next steps will show you how clusteradm
leverages
placement scheduling ability and dynamically deploy ManifestWork
to a set of
managed clusters.
Following setup dev environment by kind to prepare an environment.
curl -sSL https://raw.githubusercontent.com/open-cluster-management-io/OCM/main/solutions/setup-dev-environment/local-up.sh | bash
Confirm there are 2
ManagedCluster
and a defaultManagedClusterSet
created.$ clusteradm get clusters NAME ACCEPTED AVAILABLE CLUSTERSET CPU MEMORY KUBERNETES VERSION cluster1 true True default 24 49265496Ki v1.23.4 cluster2 true True default 24 49265496Ki v1.23.4 $ clusteradm get clustersets NAME BOUND NAMESPACES STATUS default 2 ManagedClusters selected
Bind the default
ManagedClusterSet
to defaultNamespace
.clusteradm clusterset bind default --namespace default
$ clusteradm get clustersets NAME BOUND NAMESPACES STATUS default default 2 ManagedClusters selected
Note: click here to see more details about how to operate
ManagedClusterSet
usingclusteradm
.Create a
Placement
placement1 to select the two clusters in defaultManagedClusterSet
.cat << EOF | kubectl apply -f - apiVersion: cluster.open-cluster-management.io/v1beta1 kind: Placement metadata: name: placement1 namespace: default spec: numberOfClusters: 2 clusterSets: - default EOF
Use
clusteradm
command to createManifestWork
my-first-work withPlacement
placement1.clusteradm create work my-first-work -f work.yaml --placement default/placement1
The
work.yaml
contains kubernetes resource definitions, for sample:$ cat work.yaml apiVersion: v1 kind: ServiceAccount metadata: namespace: default name: my-sa --- apiVersion: apps/v1 kind: Deployment metadata: namespace: default name: nginx-deployment labels: app: nginx spec: replicas: 3 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: serviceAccountName: my-sa containers: - name: nginx image: nginx:1.14.2 ports: - containerPort: 80
Check the
ManifestWork
, it should be distributed to both cluster1 and cluster2.$ kubectl get manifestwork -A NAMESPACE NAME AGE cluster1 my-first-work 28s cluster2 my-first-work 28s
Update the
Placement
placement1 to select only one managed cluster.kubectl patch placement placement1 --patch '{"spec": {"clusterSets": ["default"],"numberOfClusters": 1}}' --type=merge
As the placement decision changes, running below command to reschedule
ManifestWork
my-first-work to the newly selected cluster.clusteradm create work my-first-work -f work.yaml --placement default/placement1 --overwrite
Check the
ManifestWork
again, now it’s only deployed to cluster1.$ kubectl get manifestwork -A NAMESPACE NAME AGE cluster1 my-first-work 18m
What happens behind the scene
The main idea is that clusteradm
parse the selected clusters generated by
Placement
, and fill in that as ManifestWork
namespace. Then create the
ManifestWork
and it would be distributed to a set of clusters.
Let’s see more details.
Placement
placement1 generates aPlacementDecision
placement1-decision-1.$ kubectl get placementdecision -n default -l cluster.open-cluster-management.io/placement=placement1 -oyaml apiVersion: v1 items: - apiVersion: cluster.open-cluster-management.io/v1beta1 kind: PlacementDecision metadata: creationTimestamp: "2022-07-06T15:03:12Z" generation: 1 labels: cluster.open-cluster-management.io/placement: placement1 name: placement1-decision-1 namespace: default ownerReferences: - apiVersion: cluster.open-cluster-management.io/v1beta1 blockOwnerDeletion: true controller: true kind: Placement name: placement1 uid: aa339f57-0eb7-4260-8d4d-f30c1379fd35 resourceVersion: "47679" uid: 9f948619-1647-429d-894d-81e11dd8bcf1 status: decisions: - clusterName: cluster1 reason: "" - clusterName: cluster2 reason: "" kind: List metadata: resourceVersion: "" selfLink: ""
clusteradm
get thePlacementDecision
generated byPlacement
placement1 with labelcluster.open-cluster-management.io/placement: placement1
, reference code. Then parse the clusterName cluster1 and cluster2, fill in that asManifestWork
namespace, reference code. Then installsManifestWork
to namespace cluster1 and cluster2, which will finally be distributed to the two clusters.$ kubectl get manifestwork -A NAMESPACE NAME AGE cluster1 my-first-work 28s cluster2 my-first-work 28s