Distribute workload with placement selected managed clusters

The Placement API is used to dynamically select a set of ManagedCluster in one or multiple ManagedClusterSets so that the workloads can be deployed to these clusters.

If you define a valid Placement, the placement controller generates a corresponding PlacementDecision with the selected clusters listed in the status. As an end-user, you can parse the selected clusters and then operate on the target clusters. You can also integrate a high-level workload orchestrator with the PlacementDecision to leverage its scheduling capabilities.

For example, with OCM addon policy installed, a Policy that includes a Placement mapping can distribute the Policy to the managed clusters. For details see this example.

Some popular open source projects also integrate with the Placement API. For example Argo CD, it can leverage the generated PlacementDecision to drive the assignment of Argo CD Applications to a desired set of clusters, details see this example. And KubeVela, as an implementation of the open application model, also will take advantage of the Placement API for workload scheduling.

And in this article, we want to show you how to use clusteradm to deploy ManifestWork to Placement selected clusters.

Prerequisites

Before starting with the following steps, suggest you understand the content below.

  • Placement: The Placement API is used to dynamically select a set of ManagedCluster in one or multiple ManagedClusterSets so that higher-level users can either replicate Kubernetes resources to the member clusters or run their advanced workload i.e. multi-cluster scheduling.

  • ManifestWork: A custom resource in the hub cluster that groups a list of Kubernetes resources together and is meant for dispatching them into the managed cluster if the ManifestWork is created in a valid cluster namespace.

Deploy manifestwork to placement selected managed clusters

In deploy Kubernetes resources to the managed clusters, it shows you how to use clusteradm to create a ManifestWork and deploy it onto a specific managed clusters. As Placement can dynamically select a set of ManagedCluster, the next steps will show you how clusteradm leverages placement scheduling ability and dynamically deploy ManifestWork to a set of managed clusters.

  1. Following setup dev environment by kind to prepare an environment.

    curl -sSL https://raw.githubusercontent.com/open-cluster-management-io/OCM/main/solutions/setup-dev-environment/local-up.sh | bash
    
  2. Confirm there are 2 ManagedCluster and a default ManagedClusterSet created.

    $ clusteradm get clusters
    NAME       ACCEPTED   AVAILABLE   CLUSTERSET   CPU   MEMORY       KUBERNETES VERSION
    cluster1   true       True        default      24    49265496Ki   v1.23.4
    cluster2   true       True        default      24    49265496Ki   v1.23.4
    
    $ clusteradm get clustersets
    NAME      BOUND NAMESPACES   STATUS
    default                      2 ManagedClusters selected
    
  3. Bind the default ManagedClusterSet to default Namespace.

    clusteradm clusterset bind default --namespace default
    
    $ clusteradm get clustersets
    NAME      BOUND NAMESPACES   STATUS
    default   default            2 ManagedClusters selected
    

    Note: click here to see more details about how to operate ManagedClusterSet using clusteradm.

  4. Create a Placement placement1 to select the two clusters in default ManagedClusterSet.

    cat << EOF | kubectl apply -f -
    apiVersion: cluster.open-cluster-management.io/v1beta1
    kind: Placement
    metadata:
      name: placement1
      namespace: default
    spec:
      numberOfClusters: 2
      clusterSets:
        - default
    EOF
    
  5. Use clusteradm command to create ManifestWork my-first-work with Placement placement1.

    clusteradm create work my-first-work -f work.yaml --placement default/placement1
    

    The work.yaml contains kubernetes resource definitions, for sample:

    $ cat work.yaml
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      namespace: default
      name: my-sa
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      namespace: default
      name: nginx-deployment
      labels:
        app: nginx
    spec:
      replicas: 3
      selector:
        matchLabels:
          app: nginx
      template:
        metadata:
          labels:
            app: nginx
        spec:
          serviceAccountName: my-sa
          containers:
            - name: nginx
              image: nginx:1.14.2
              ports:
                - containerPort: 80
    
  6. Check the ManifestWork, it should be distributed to both cluster1 and cluster2.

    $ kubectl get manifestwork -A
    NAMESPACE   NAME            AGE
    cluster1    my-first-work   28s
    cluster2    my-first-work   28s
    
  7. Update the Placement placement1 to select only one managed cluster.

    kubectl patch placement placement1 --patch '{"spec": {"clusterSets": ["default"],"numberOfClusters": 1}}' --type=merge
    
  8. As the placement decision changes, running below command to reschedule ManifestWork my-first-work to the newly selected cluster.

    clusteradm create work my-first-work -f work.yaml --placement default/placement1 --overwrite
    
  9. Check the ManifestWork again, now it’s only deployed to cluster1.

    $ kubectl get manifestwork -A
    NAMESPACE   NAME            AGE
    cluster1    my-first-work   18m
    

What happens behind the scene

The main idea is that clusteradm parse the selected clusters generated by Placement, and fill in that as ManifestWork namespace. Then create the ManifestWork and it would be distributed to a set of clusters. Let’s see more details.

  1. Placement placement1 generates a PlacementDecision placement1-decision-1.

    $ kubectl get placementdecision -n default -l cluster.open-cluster-management.io/placement=placement1 -oyaml
    apiVersion: v1
    items:
    - apiVersion: cluster.open-cluster-management.io/v1beta1
      kind: PlacementDecision
      metadata:
        creationTimestamp: "2022-07-06T15:03:12Z"
        generation: 1
        labels:
          cluster.open-cluster-management.io/placement: placement1
        name: placement1-decision-1
        namespace: default
        ownerReferences:
        - apiVersion: cluster.open-cluster-management.io/v1beta1
          blockOwnerDeletion: true
          controller: true
          kind: Placement
          name: placement1
          uid: aa339f57-0eb7-4260-8d4d-f30c1379fd35
        resourceVersion: "47679"
        uid: 9f948619-1647-429d-894d-81e11dd8bcf1
      status:
        decisions:
        - clusterName: cluster1
          reason: ""
        - clusterName: cluster2
          reason: ""
    kind: List
    metadata:
      resourceVersion: ""
      selfLink: ""
    
  2. clusteradm get the PlacementDecision generated by Placement placement1 with label cluster.open-cluster-management.io/placement: placement1, reference code. Then parse the clusterName cluster1 and cluster2, fill in that as ManifestWork namespace, reference code. Then installs ManifestWork to namespace cluster1 and cluster2, which will finally be distributed to the two clusters.

    $ kubectl get manifestwork -A
    NAMESPACE   NAME            AGE
    cluster1    my-first-work   28s
    cluster2    my-first-work   28s
    
Last modified September 20, 2024: Refactor using docsy. (#429) (3f36d75)