Distribute workload with placement selected managed clusters

The Placement API is used to dynamically select a set of ManagedCluster in one or multiple ManagedClusterSets so that the workloads can be deployed to these clusters.

If you define a valid Placement, the placement controller generates a corresponding PlacementDecision with the selected clusters listed in the status. As an end-user, you can parse the selected clusters and then operate on the target clusters. You can also integrate a high-level workload orchestrator with the PlacementDecision to leverage its scheduling capabilities.

For example, with OCM addon policy installed, a Policy that includes a Placement mapping can distribute the Policy to the managed clusters, details see this example.

Some popular open source projects also integrate with the Placement API. For example Argo CD, it can leverage the generated PlacementDecision to drive the assignment of Argo CD Applications to a desired set of clusters, details see this example. And KubeVela, as an implementation of the open application model, also will take advantage of the Placement API for workload scheduling.

And in this article, we want to show you how to use clusteradm to deploy ManifestWork to Placement selected clusters.

Prerequisites

Before starting with the following steps, suggest you understand the content below.

Placement: The Placement API is used to dynamically select a set of ManagedCluster in one or multiple ManagedClusterSets so that higher-level users can either replicate Kubernetes resources to the member clusters or run their advanced workload i.e. multi-cluster scheduling.
ManifestWork: A custom resource in the hub cluster that groups a list of Kubernetes resources together and is meant for dispatching them into the managed cluster if the ManifestWork is created in a valid cluster namespace.

Deploy manifestwork to placement selected managed clusters

In deploy Kubernetes resources to the managed clusters, it shows you how to use clusteradm to create a ManifestWork and deploy it onto a specific managed clusters. As Placement can dynamically select a set of ManagedCluster, the next steps will show you how clusteradm leverages placement scheduling ability and dynamically deploy ManifestWork to a set of managed clusters.

Following setup dev environment by kind to prepare an environment.

curl -sSL https://raw.githubusercontent.com/open-cluster-management-io/OCM/main/solutions/setup-dev-environment/local-up.sh | bash

Confirm there are 2 ManagedCluster and a default ManagedClusterSet created.

$ clusteradm get clusters
NAME       ACCEPTED   AVAILABLE   CLUSTERSET   CPU   MEMORY       KUBERNETES VERSION
cluster1   true       True        default      24    49265496Ki   v1.23.4
cluster2   true       True        default      24    49265496Ki   v1.23.4

$ clusteradm get clustersets
NAME      BOUND NAMESPACES   STATUS
default                      2 ManagedClusters selected

Bind the default ManagedClusterSet to default Namespace.

clusteradm clusterset bind default --namespace default

$ clusteradm get clustersets
NAME      BOUND NAMESPACES   STATUS
default   default            2 ManagedClusters selected

Note: click here to see more details about how to operate ManagedClusterSet using clusteradm.

Create a Placement placement1 to select the two clusters in default ManagedClusterSet.

cat << EOF | kubectl apply -f -
apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Placement
metadata:
  name: placement1
  namespace: default
spec:
  numberOfClusters: 2
  clusterSets:
    - default
EOF

Use clusteradm command to create ManifestWork my-first-work with Placement placement1.

clusteradm create work my-first-work -f work.yaml --placement default/placement1

The work.yaml contains kubernetes resource definitions, for sample:

$ cat work.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: default
  name: my-sa
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: default
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      serviceAccountName: my-sa
      containers:
        - name: nginx
          image: nginx:1.14.2
          ports:
            - containerPort: 80

Check the ManifestWork, it should be distributed to both cluster1 and cluster2.

$ kubectl get manifestwork -A
NAMESPACE   NAME            AGE
cluster1    my-first-work   28s
cluster2    my-first-work   28s

Update the Placement placement1 to select only one managed cluster.

kubectl patch placement placement1 --patch '{"spec": {"clusterSets": ["default"],"numberOfClusters": 1}}' --type=merge

As the placement decision changes, running below command to reschedule ManifestWork my-first-work to the newly selected cluster.
```
clusteradm create work my-first-work -f work.yaml --placement default/placement1 --overwrite
```

Check the ManifestWork again, now it’s only deployed to cluster1.

$ kubectl get manifestwork -A
NAMESPACE   NAME            AGE
cluster1    my-first-work   18m

What happens behind the scene

The main idea is that clusteradm parse the selected clusters generated by Placement, and fill in that as ManifestWork namespace. Then create the ManifestWork and it would be distributed to a set of clusters. Let’s see more details.

Placement placement1 generates a PlacementDecision placement1-decision-1.

$ kubectl get placementdecision -n default -l cluster.open-cluster-management.io/placement=placement1 -oyaml
apiVersion: v1
items:
- apiVersion: cluster.open-cluster-management.io/v1beta1
  kind: PlacementDecision
  metadata:
    creationTimestamp: "2022-07-06T15:03:12Z"
    generation: 1
    labels:
      cluster.open-cluster-management.io/placement: placement1
    name: placement1-decision-1
    namespace: default
    ownerReferences:
    - apiVersion: cluster.open-cluster-management.io/v1beta1
      blockOwnerDeletion: true
      controller: true
      kind: Placement
      name: placement1
      uid: aa339f57-0eb7-4260-8d4d-f30c1379fd35
    resourceVersion: "47679"
    uid: 9f948619-1647-429d-894d-81e11dd8bcf1
  status:
    decisions:
    - clusterName: cluster1
      reason: ""
    - clusterName: cluster2
      reason: ""
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

clusteradm get the PlacementDecision generated by Placment placement1 with label cluster.open-cluster-management.io/placement: placement1, reference code. Then parse the clusterName cluster1 and cluster2, fill in that as ManifestWork namespace, reference code. Then installs ManifestWork to namespace cluster1 and cluster2, which will finally be distributed to the two clusters.
```
$ kubectl get manifestwork -A
NAMESPACE   NAME            AGE
cluster1    my-first-work   28s
cluster2    my-first-work   28s
```