Open Cluster Management (OCM) is a powerful, modular, extensible platform for Kubernetes multi-cluster orchestration. Learning from the past failing lesson of building Kubernetes federation systems in the Kubernetes community, in OCM we will be jumping out of the legacy centric, imperative architecture of Kubefed v2 and embracing the “hub-agent” architecture which is identical to the original pattern of “hub-kubelet” from Kubernetes. Hence, intuitively in OCM our multi-cluster control plane is modeled as a “Hub” and on the other hand each of the clusters being managed by the “Hub” will be a “Klusterlet” which is obviously inspired from the original name of “kubelet”. Here’s a more detailed clarification of the two models we will be frequently using throughout the world of OCM:

Hub Cluster: Indicating the cluster that runs the multi-cluster control plane of OCM. Generally the hub cluster is supposed to be a light-weight Kubernetes cluster hosting merely a few fundamental controllers and services.
Klusterlet: Indicating the clusters that being managed by the hub cluster. Klusterlet might also be called “managed cluster” or “spoke cluster”. The klusterlet is supposed to actively pulling the latest prescriptions from the hub cluster and consistently reconciles the physical Kubernetes cluster to the expected state.

“Hub-spoke” architecture

Benefiting from the merit of “hub-spoke” architecture, in abstraction we are de-coupling most of the multi-cluster operations generally into (1) computation/decision and (2) execution, and the actual execution against the target cluster will be completely off-loaded into the managed cluster. The hub cluster won’t directly request against the real clusters, instead it just persists its prescriptions declaratively for each cluster, and the klusterlet will be actively pulling the prescriptions from the hub and doing the execution. Hence, the burden of the hub cluster will be greatly relieved because the hub cluster doesn’t need to either deal with flooding events from the managed clusters or be buried in sending requests against the clusters. Imagine in a world where there’s no kubelet in Kubernetes and its control plane is directly operating the container daemons, it will be extremely hard for a centric controller to manage a cluster of 5k+ nodes. Likewise, that’s how OCM trying to breach the bottleneck of scalability, by dividing and offloading the execution into separated agents. So it’s always feasible for a hub cluster to accept and manage thousand-ish clusters.

Each klusterlet will be working independently and autonomously, so they have a weak dependency to the availability of the hub cluster. If the hub goes down (e.g. during maintenance or network partition) the klusterlet or other OCM agents working in the managed cluster are supposed to keep actively managing the hosting cluster until it re-connects. Additionally if the hub cluster and the managed clusters are owned by different admins, it will be easier for the admin of the managed cluster to police the prescriptions from the hub control plane because the klusterlet is running as a “white-box” as a pod instance in the managed cluster. Upon any accident, the klusterlet admin can quickly cut off the connection with the hub cluster without shutting the whole multi-cluster control plane down.

The “hub-agent” architecture also minimized the requirements in the network for registering a new cluster to the hub. Any cluster that can reach the endpoint of the hub cluster will be able to be managed, even a random KinD sandbox cluster on your laptop. That is because the prescriptions are effectively pulled from the hub instead of pushing. In addition to that, OCM also provides a addon named “cluster-proxy” which automatically manages a reverse proxy tunnel for proactive access to the managed clusters by leveraging on the Kubernetes’ subproject konnectivity.

Modularity and extensibility

Not only OCM will bring you a fluent user-experience of managing a number of clusters on ease, but also it will be equally friendly to further customization or second-time development. Every functionality working in OCM is expected to be freely-pluggable by modularizing the atomic capability into separated building blocks, except for the mandatory core module named registration which is responsible for controlling the lifecycle of a managed controller and exporting the elementary ManagedCluster model.

Another good example surfacing our modularity will be the placement, a standalone module focusing at dynamically selecting the proper list of the managed clusters from the user’s prescription. You can build any advanced multi-cluster orchestration on the top of placement, e.g. multi-cluster workload re-balancing, multi-cluster helm charts replication, etc. On the other hand if you’re not satisfied by the current capacities from our placement module, you can quickly opt-out and replace it with your customized ones, and reach out to our community so that we can converge in the future if possible.

Concepts

Cluster registering: “double opt-in handshaking”

Practically the hub cluster and the managed cluster can be owned/maintained by different admins, so in OCM we clearly separated the roles and make the cluster registration require approval from the both sides defending from unwelcome requests. In terms of terminating the registration, the hub admin can kick out a registered cluster by denying the rotation of hub cluster’s certificate, on the other hand from the perspective of a managed cluster’s admin, he can either brutally deleting the agent instances or revoking the granted RBAC permissions for the agents. Note that the hub controller will be automatically preparing environment for the newly registered cluster and cleaning up neatly upon kicking a managed cluster.

Cluster registration security model

The worker cluster admin can list and read any managed cluster’s CSR, but those CSR cannot be used to impersonate due to the fact that CSR only contains the certificate. The client authentication requires both the key and certificate. The key is stored in each managed cluster, and it will not be transmitted across the network.

The worker cluster admin cannot approve his or her own cluster registration by default. Two separate RBAC rules are needed to approve a cluster registration. The permission to approve the CSR and the permission to accept the managed cluster. Only the cluster admin on hub has both permissions and can accept the cluster registration request. The second accept permission is gated by a webhook.

Cluster namespace

Kubernetes has a native soft multi-tenancy isolation in the granularity of its namespace resources, so in OCM, for each of the managed cluster we will be provisioning a dedicated namespace for the managed cluster and grants sufficient RBAC permissions so that the klusterlet can persist some data in the hub cluster. This dedicated namespace is the “cluster namespace” which is majorly for saving the prescriptions from the hub. e.g. we can create ManifestWork in a cluster namespace in order to deploy some resources towards the corresponding cluster. Meanwhile, the cluster namespace can also be used to save the uploaded stats from the klusterlet e.g. the healthiness of an addon, etc.

Addons

Addon is a general concept for the optional, pluggable customization built over the extensibility from OCM. It can be a controller in the hub cluster, or just a customized agent in the managed cluster, or even the both collaborating in peers. The addons are expected to implement the ClusterManagementAddon or ManagedClusterAddOn API of which a detailed elaboration can be found here.

Building blocks

The following is a list of commonly-used modules/subprojects that you might be interested in the journey of OCM:

Registration

The core module of OCM manages the lifecycle of the managed clusters. The registration controller in the hub cluster can be intuitively compared to a broker that represents and manages the hub cluster in terms of cluster registration, while the registration agent working in the managed cluster is another broker that represents the managed cluster. After a successful registration, the registration controller and agent will also be consistently probing each other’s healthiness. i.e. the cluster heartbeats.

Work

The module for dispatching resources from the hub cluster to the managed clusters, which can be easily done by writing a ManifestWork resource into a cluster namespace. See more details about the API here.

Placement

Building custom advanced topology across the clusters by either grouping clusters via the labels or the cluster-claims. The placement module is completely decoupled from the execution, the output from placement will be merely a list of names of the matched clusters in the PlacementDecision API, so the consumer controller of the decision output can reactively discovery the topology or availability change from the managed clusters by simply list-watching the decision API.

Application lifecycle

The application lifecycle defines the processes that are used to manage application resources on your managed clusters. A multi-cluster application uses a Kubernetes specification, but with additional automation of the deployment and lifecycle management of resources to individual clusters. A multi-cluster application allows you to deploy resources on multiple clusters, while maintaining easy-to-reconcile service routes, as well as full control of Kubernetes resource updates for all aspects of the application.

Governance and risk

Governance and risk is the term used to define the processes that are used to manage security and compliance from the hub cluster. Ensure the security of your cluster with the extensible policy framework. After you configure a hub cluster and a managed cluster, you can create, modify and delete policies on the hub and apply policies to the managed clusters.

Registration operator

Automating the installation and upgrading of a few built-in modules in OCM. You can either deploy the operator standalone or delegate the registration operator to the operator lifecycle framework.

1.2 - ClusterClaim

What is ClusterClaim?

ClusterClaim is a cluster-scoped API available to users on a managed cluster. The ClusterClaim objects are collected from the managed cluster and saved into the status of the corresponding ManagedCluster object on the hub.

Usage

ClusterCaim is used to specify additional properties of the managed cluster like the clusterID, version, vendor and cloud provider. We defined some reserved ClusterClaims like id.k8s.io which is a unique identifier for the managed cluster.

In addition to the reserved ClusterClaims, users can also customize 20 ClusterClaims by default. The maximum count of customized ClusterClaims can be configured via the flag max-custom-cluster-claims of registration agent on the managed cluster.

The ClusterClaim with the label open-cluster-management.io/spoke-only will not be synced to the status of ManagedCluster.

Example

Here is a ClusterClaim example specifying a id.k8s.io:

apiVersion: cluster.open-cluster-management.io/v1alpha1
kind: ClusterClaim
metadata:
  name: id.k8s.io
spec:
  value: myCluster

After applying the ClusterClaim above to any managed cluster, the value of the ClusterClaim is reflected in the ManagedCluster on the hub cluster:

apiVersion: cluster.open-cluster-management.io/v1
kind: ManagedCluster
metadata: ...
spec: ...
status:
  clusterClaims:
    - name: id.k8s.io
      value: myCluster

1.3 - ManagedCluster

What is ManagedCluster?

ManagedCluster is a cluster scoped API in the hub cluster representing the registered or pending-for-acceptance Kubernetes clusters in OCM. The klusterlet agent working in the managed cluster is expected to actively maintain/refresh the status of the corresponding ManagedCluster resource on the hub cluster. On the other hand, removing the ManagedCluster from the hub cluster indicates the cluster is denied/exiled from the hub cluster. The following is the introduction of how the cluster registration lifecycle works under the hood:

Cluster registration and acceptance

Bootstrapping registration

Firstly, the cluster registration process should be initiated by the registration agent which requires a bootstrap kubeconfig e.g.:

apiVersion: v1
kind: Secret
metadata:
  name: bootstrap-hub-kubeconfig
  namespace: open-cluster-management-agent
type: Opaque
data:
  kubeconfig: <base64-encoded kubeconfig>

A minimal RBAC permission required for the subject in the bootstrap kubeconfig will be:

CertificateSigningRequest’s “get”, “list”, “watch”, “create”, “update”.
ManagedCluster’s “get”, “list”, “create”, “update”

Note that ideally the bootstrap kubeconfig is supposed to live shortly (hour-ish) after signed by the hub cluster so that it won’t be abused by unwelcome clients.

Last but not least, you can always live an easier life by leveraging OCM’s command-line tool clusteradm to manage the whole registration process.

Approving registration

When we’re registering a new cluster into OCM, the registration agent will be starting by creating an unaccepted ManagedCluster into the hub cluster along with a temporary CertificateSigningRequest (CSR) resource. The cluster will be accepted by the hub control plane, if the following requirements is meet:

The CSR is approved and signed by any certificate provider setting filling .status.certificate with legit X.509 certificates.
The ManagedCluster resource is approved by setting .spec.hubAcceptsClient to true in the spec.

Note that the cluster approval process above can be done by one-line:

$ clusteradm accept --clusters <cluster name>

Upon the approval, the registration agent will observe the signed certificate and persist them as a local secret named “hub-kubeconfig-secret” (by default in the “open-cluster-management-agent” namespace) which will be mounted to the other fundamental components of klusterlet such as the work agent. In a word, if you can find your “hub-kubeconfig-secret” successfully present in your managed cluster, the cluster registration is all set!

Overall the registration process in OCM is called double opt-in mechanism, which means that a successful cluster registration requires both sides of approval and commitment from the hub cluster and the managed cluster. This will be especially useful when the hub cluster and managed clusters are operated by different admins or teams. In OCM, we assume the clusters are mutually untrusted in the beginning then set up the connection between them gracefully with permission and validity under control.

Note that the functionality mentioned above are all managed by OCM’s registration sub-project, which is the “root dependency” in the OCM world. It includes an agent in the managed cluster to register to the hub and a controller in the hub cluster to coordinate with the agent.

Cluster heartbeats and status

By default, the registration will be reporting and refreshing its healthiness state to the hub cluster on a one-minute basis, and that interval can be easily overridden by setting .spec.leaseDurationSeconds on the ManagedCluster.

In addition to that, a few commonly-used information will also be reflected in the status of the ManagedCluster, e.g.:

  status:
    version:
      kubernetes: v1.20.11
    allocatable:
      cpu: 11700m
      ephemeral-storage: "342068531454"
      hugepages-1Gi: "0"
      hugepages-2Mi: "0"
      memory: 17474228Ki
      pods: "192"
    capacity:
      cpu: "12"
      ephemeral-storage: 371168112Ki
      hugepages-1Gi: "0"
      hugepages-2Mi: "0"
      memory: 23777972Ki
      pods: "192"
    conditions: ...

Cluster taints and tolerations

To support filtering unhealthy/not-reporting clusters and keep workloads from being placed in unhealthy or unreachable clusters, we introduce the similar concept of taint/toleration in Kubernetes. It also allows user to add a customized taint to deselect a cluster from placement. This is useful when the user wants to set a cluster to maintenance mode and evict workload from this cluster.

In OCM, Taints and Tolerations work together to allow users to control the selection of managed clusters more flexibly.

Taints of ManagedClusters

Taints are properties of ManagedClusters, they allow a Placement to repel a set of ManagedClusters. A Taint includes the following fields:

Key (required). The taint key applied to a cluster. e.g. bar or foo.example.com/bar.
Value (optional). The taint value corresponding to the taint key.
Effect (required). The Effect of the taint on Placements that do not tolerate the taint. Valid effects are
- NoSelect. It means Placements are not allowed to select a cluster unless they tolerate this taint. The cluster will be removed from the placement decision if it has already been selected by the Placement.
- PreferNoSelect. It means the scheduler tries not to select the cluster, rather than prohibiting Placements from selecting the cluster entirely. (This is not implemented yet, currently clusters with effect PreferNoSelect will always be selected.)
- NoSelectIfNew. It means Placements are not allowed to select the cluster unless: 1) they tolerate the taint; 2) they have already had the cluster in their cluster decisions;
TimeAdded (required). The time at which the taint was added. It is set automatically and the user should not to set/update its value.

Builtin taints to reflect the status of ManagedClusters

There are two builtin taints, which will be automatically added to ManagedClusters, according to their conditions.

cluster.open-cluster-management.io/unavailable. The taint is added to a ManagedCluster when it is not available. To be specific, the cluster has a condition ‘ManagedClusterConditionAvailable’ with status of ‘False’. The taint has the effect NoSelect and an empty value. Example,

apiVersion: cluster.open-cluster-management.io/v1
kind: ManagedCluster
metadata:
 name: cluster1
spec:
 hubAcceptsClient: true
 taints:
   - effect: NoSelect
     key: cluster.open-cluster-management.io/unavailable
     timeAdded: '2022-02-21T08:11:54Z'

cluster.open-cluster-management.io/unreachable. The taint is added to a ManagedCluster when it is not reachable. To be specific,
- 1. The cluster has no condition ‘ManagedClusterConditionAvailable’;
- 1. Or the status of condition ‘ManagedClusterConditionAvailable’ is ‘Unknown’; The taint has the effect NoSelect and an empty value. Example,
```
apiVersion: cluster.open-cluster-management.io/v1
kind: ManagedCluster
metadata:
  name: cluster1
spec:
  hubAcceptsClient: true
  taints:
    - effect: NoSelect
      key: cluster.open-cluster-management.io/unreachable
      timeAdded: '2022-02-21T08:11:06Z'
```

Tolerations of Placements

Tolerations are applied to Placements, and allow Placements to select ManagedClusters with matching taints. Refer to Placement Taints/Tolerations to see how it is used for cluster selection.

Cluster removal

A previously registered cluster can opt-out cutting off the connection from either hub cluster or managed cluster. This is helpful for tackling emergency problems in your OCM environment, e.g.:

When the hub cluster is overloaded, under emergency
When the managed cluster is intended to detach from OCM
When the hub cluster is found sending wrong orders to the managed cluster
When the managed cluster is spamming requests to the hub cluster

Unregister from hub cluster

A recommended way to unregister a managed cluster will flip the .spec.hubAcceptsClient bit back to false, which will be triggering the hub control plane to offload the managed cluster from effective management. Meanwhile, a permanent way to kick a managed cluster from the hub control plane is simply deleting its ManagedCluster resource.

$ kubectl delete managedcluster <cluster name>

This is also revoking the previously-granted RBAC permission for the managed cluster instantly in the background. If we hope to defer the rejection to the next time when the klusterlet agent is renewing its certificate, as a minimal operation we can remove the following RBAC rules from the cluster’s effective cluster role resource:

# ClusterRole: open-cluster-management:managedcluster:<cluster name>
# Removing the following RBAC rule to stop the certificate rotation.
- apiGroups:
    - register.open-cluster-management.io
  resources:
    - managedclusters/clientcertificates
  verbs:
    - renew

Unregister from the managed cluster

The admin of the managed cluster can disable the prescriptions from hub cluster by scaling the OCM klusterlet agents to 0. Or just permanently deleting the agent components from the managed cluster.

Managed Cluster’s certificate rotation

The certificates used by the agents from the managed cluster to talk to the hub control plane will be periodically rotated with an ephemeral and random identity. The following picture shows the automated certificate rotation works.

What’s next?

Furthermore, we can do advanced cluster matching/selecting within a managedclusterset using the placement module.

1.4 - ManagedClusterSet

API-CHANGE NOTE:

The ManagedClusterSet and ManagedClusterSetBinding API v1beta1 version will no longer be served in OCM v0.12.0.

Migrate manifests and API clients to use the ManagedClusterSet and ManagedClusterSetBinding API v1beta2 version, available since OCM v0.9.0.
All existing persisted objects are accessible via the new API.
Notable changes:
- The default cluster selector type will be ExclusiveClusterSetLabel in v1beta2, and type LegacyClusterSetLabel in v1beta1 is removed.

What is ManagedClusterSet?

ManagedClusterSet is a cluster-scoped API in the hub cluster for grouping a few managed clusters into a “set” so that hub admin can operate these clusters altogether in a higher level. The concept is inspired by the enhancement from the Kubernetes SIG-Multicluster. Member clusters in the set are supposed to have common/similar attributes e.g. purpose of use, deployed regions, etc.

ManagedClusterSetBinding is a namespace-scoped API in the hub cluster to project a ManagedClusterSet into a certain namespace. Each ManagedClusterSet can be managed/administrated by different hub admins, and their RBAC permissions can also be isolated by binding the ManagedClusterSet to a “workspace namespace” in the hub cluster via ManagedClusterSetBinding.

Note that ManagedClusterSet and “workspace namespace” has an M*N relationship:

Bind multiple cluster sets to one workspace namespace indicates that the admin of that namespace can operate the member clusters from both sets.
Bind one cluster set to multiple workspace namespace indicates that the cluster set can be operated from all the bound namespaces at the same time.

The cluster set admin can flexibly operate the member clusters in the workspace namespace using Placement API, etc.

The following picture shows the hierarchies of how the cluster set works:

Operates ManagedClusterSet using clusteradm

Creating a ManagedClusterSet

Running the following command to create an example cluster set:

$ clusteradm create clusterset example-clusterset
$ clusteradm get clustersets
<ManagedClusterSet>
└── <default>
│   ├── <BoundNamespace>
│   ├── <Status> No ManagedCluster selected
└── <example-clusterset>
│   ├── <BoundNamespace>
│   ├── <Status> No ManagedCluster selected
└── <global>
    └── <BoundNamespace>
    └── <Status> 1 ManagedClusters selected

The newly created cluster set will be empty by default, so we can move on adding member clusters to the set.

Adding a ManagedCluster to a ManagedClusterSet

Running the following command to add a cluster to the set:

$ clusteradm clusterset set example-clusterset --clusters managed1
$ clusteradm get clustersets
<ManagedClusterSet>
└── <default>
│   ├── <BoundNamespace>
│   ├── <Status> No ManagedCluster selected
└── <example-clusterset>
│   ├── <BoundNamespace>
│   ├── <Status> 1 ManagedClusters selected
└── <global>
    └── <BoundNamespace>
    └── <Status> 1 ManagedClusters selected

Note that adding a cluster to a cluster set will require the admin to have “managedclustersets/join” access in the hub cluster.

Now the cluster set contains 1 valid cluster, and in order to operate that cluster set we are supposed to bind it to an existing namespace to make it a “workspace namespace”.

Binding the ManagedClusterSet to a workspace namespace

Running the following command to bind the cluster set to a namespace. Note that the namespace SHALL NOT be an existing “cluster namespace” (i.e. the namespace has the same name of a registered managed cluster).

Note that binding a cluster set to a namespace means that granting access from that namespace to its member clusters. And the bind process requires “managedclustersets/bind” access in the hub cluster which is clarified below.

$ clusteradm clusterset bind example-clusterset --namespace default
$ clusteradm get clustersets
<ManagedClusterSet>
└── <default>
│   ├── <BoundNamespace>
│   ├── <Status> No ManagedCluster selected
└── <example-clusterset>
│   ├── <Status> 1 ManagedClusters selected
│   ├── <BoundNamespace> default
└── <global>
    └── <BoundNamespace>
    └── <Status> 1 ManagedClusters selected

So far we successfully created a new cluster set containing 1 cluster and bind it a “workspace namespace”.

A glance at the “ManagedClusterSet” API

The ManagedClusterSet is a vanilla Kubernetes custom resource which can be checked by the command kubectl get managedclusterset <cluster set name> -o yaml:

apiVersion: cluster.open-cluster-management.io/v1beta2
kind: ManagedClusterSet
metadata:
  name: example-clusterset
spec:
  clusterSelector:
    selectorType: ExclusiveClusterSetLabel
status:
  conditions:
  - lastTransitionTime: "2022-02-21T09:24:38Z"
    message: 1 ManagedClusters selected
    reason: ClustersSelected
    status: "False"
    type: ClusterSetEmpty

apiVersion: cluster.open-cluster-management.io/v1beta2
kind: ManagedClusterSet
metadata:
  name: example-openshift-clusterset
spec:
  clusterSelector:
    labelSelector:
      matchLabels:
        vendor: OpenShift
    selectorType: LabelSelector
status:
  conditions:
  - lastTransitionTime: "2022-06-20T08:23:28Z"
    message: 1 ManagedClusters selected
    reason: ClustersSelected
    status: "False"
    type: ClusterSetEmpty

The ManagedClusterSetBinding can also be checked by the command kubectl get managedclustersetbinding <cluster set name> -n <workspace-namespace> -oyaml:

apiVersion: cluster.open-cluster-management.io/v1beta2
kind: ManagedClusterSetBinding
metadata:
  name: example-clusterset
  namespace: default
spec:
  clusterSet: example-clusterset
status:
  conditions:
  - lastTransitionTime: "2022-12-19T09:55:10Z"
    message: ""
    reason: ClusterSetBound
    status: "True"
    type: Bound

Clusterset RBAC permission control

Adding member cluster to a clusterset

Adding a new member cluster to a clusterset requires RBAC permission of updating the managed cluster and managedclustersets/join subresource. We can manually apply the following clusterrole to allow a hub user to manipulate that clusterset:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata: ...
rules:
  - apiGroups:
      - cluster.open-cluster-management.io
    resources:
      - managedclusters
    verbs:
      - update
  - apiGroups:
      - cluster.open-cluster-management.io
    resources:
      - managedclustersets/join
    verbs:
      - create

Binding a clusterset to a namespace

The “binding” process of a cluster set is policed by a validating webhook that checks whether the requester has sufficient RBAC access to the managedclustersets/bind subresource. We can also manually apply the following clusterrole to grant a hub user the permission to bind cluster sets:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata: ...
rules:
  - apiGroups:
      - cluster.open-cluster-management.io
    resources:
      - managedclustersets/bind
    verbs:
      - create

Default ManagedClusterSet

For easier management, we introduce a ManagedClusterSet called default. A default ManagedClusterSet will be automatically created initially. Any clusters not specifying a ManagedClusterSet will be added into the default. The user can move the cluster from the default clusterset to another clusterset using the command:

clusteradm clusterset set target-clusterset --clusters cluster-name

default clusterset is an alpha feature that can be disabled by disabling the feature gate in registration controller as: - "--feature-gates=DefaultClusterSet=false"

Global ManagedClusterSet

For easier management, we also introduce a ManagedClusterSet called global. A global ManagedClusterSet will be automatically created initially. The global ManagedClusterSet include all ManagedClusters.

global clusterset is an alpha feature that can be disabled by disabling the feature gate in registration controller as: - "--feature-gates=DefaultClusterSet=false"

global ManagedClusterSet detail:

apiVersion: cluster.open-cluster-management.io/v1beta2
kind: ManagedClusterSet
metadata:
  name: global
spec:
  clusterSelector:
    labelSelector: {}
    selectorType: LabelSelector
status:
  conditions:
  - lastTransitionTime: "2022-06-20T08:23:28Z"
    message: 1 ManagedClusters selected
    reason: ClustersSelected
    status: "False"
    type: ClusterSetEmpty

1.5 - Placement

CHANGE NOTE:

The Placement and PlacementDecision API v1alpha1 version will no longer be served in OCM v0.9.0.
- Migrate manifests and API clients to use the Placement and PlacementDecision API v1beta1 version, available since OCM v0.7.0.
- All existing persisted objects are accessible via the new API.
- Notable changes:
  - The field spec.prioritizerPolicy.configurations.name in Placement API v1alpha1 is removed and replaced by spec.prioritizerPolicy.configurations.scoreCoordinate.builtIn in v1beta1.
Clusters in terminating state will not be selected by placements from OCM v0.14.0.

Overall

Placement concept is used to dynamically select a set of managedClusters in one or multiple ManagedClusterSet so that higher level users can either replicate Kubernetes resources to the member clusters or run their advanced workload i.e. multi-cluster scheduling.

The “input” and “output” of the scheduling process are decoupled into two separated Kubernetes API Placement and PlacementDecision. As is shown in the following picture, we prescribe the scheduling policy in the spec of Placement API and the placement controller in the hub will help us to dynamically select a slice of managed clusters from the given cluster sets. The selected clusters will be listed in PlacementDecision.

Following the architecture of Kubernetes’ original scheduling framework, the multi-cluster scheduling is logically divided into two phases internally:

Predicate: Hard requirements for the selected clusters.
Prioritize: Rank the clusters by the soft requirements and select a subset among them.

Select clusters in ManagedClusterSet

By following the previous section about ManagedClusterSet, now we’re supposed to have one or multiple valid cluster sets in the hub clusters. Then we can move on and create a placement in the “workspace namespace” by specifying predicates and prioritizers in the Placement API to define our own multi-cluster scheduling policy.

Notes:

Clusters in terminating state will not be selected by placements.

Predicates

Label/Claim selection

In the predicates section, you can select clusters by labels or clusterClaims. For instance, you can select 3 clusters with label purpose=test and clusterClaim platform.open-cluster-management.io=aws as seen in the following examples:

apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Placement
metadata:
  name: placement1
  namespace: default
spec:
  numberOfClusters: 3
  clusterSets:
    - prod
  predicates:
    - requiredClusterSelector:
        labelSelector:
          matchLabels:
            purpose: test
        claimSelector:
          matchExpressions:
            - key: platform.open-cluster-management.io
              operator: In
              values:
                - aws

Note that the distinction between label-selecting and claim-selecting is elaborated in this page about how to extend attributes for the managed clusters.

Taints/Tolerations

In OCM, Taints and Tolerations work together to allow users to control the selection of managed clusters more flexibly.

Taints are properties of ManagedClusters, they allow a Placement to repel a set of ManagedClusters in predicates stage.

Tolerations are applied to Placements, and allow Placements to select ManagedClusters with matching taints.

The following example shows how to tolerate clusters with taints.

Tolerate clusters with taint

Suppose your managed cluster has taint added as below.

apiVersion: cluster.open-cluster-management.io/v1
kind: ManagedCluster
metadata:
  name: cluster1
spec:
  hubAcceptsClient: true
  taints:
    - effect: NoSelect
      key: gpu
      value: "true"
      timeAdded: '2022-02-21T08:11:06Z'

By default, the placement won’t select this cluster unless you define tolerations.

apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Placement
metadata:
  name: placement1
  namespace: ns1
spec:
  tolerations:
    - key: gpu
      value: "true"
      operator: Equal

With the above tolerations defined, cluster1 could be selected by placement because of the key: gpu and value: "true" match.

Tolerate clusters with taint for a period of time

TolerationSeconds represents the period of time the toleration tolerates the taint. It could be used for the case like, when a managed cluster gets offline, users can make applications deployed on this cluster to be transferred to another available managed cluster after a tolerated time.

apiVersion: cluster.open-cluster-management.io/v1
kind: ManagedCluster
metadata:
  name: cluster1
spec:
  hubAcceptsClient: true
  taints:
    - effect: NoSelect
      key: cluster.open-cluster-management.io/unreachable
      timeAdded: '2022-02-21T08:11:06Z'

If define a placement with TolerationSeconds as below, then the workload will be transferred to another available managed cluster after 5 minutes.

apiVersion: cluster.open-cluster-management.io/v1alpha1
kind: Placement
metadata:
  name: placement1
  namespace: ns1
spec:
  tolerations:
    - key: cluster.open-cluster-management.io/unreachable
      operator: Exists
      tolerationSeconds: 300

In tolerations section, it includes the following fields:

Key (optional). Key is the taint key that the toleration applies to.
Value (optional). Value is the taint value the toleration matches to.
Operator (optional). Operator represents a key’s relationship to the value. Valid operators are Exists and Equal. Defaults to Equal. A toleration “matches” a taint if the keys are the same and the effects are the same, and the operator is:
- Equal. The operator is Equal and the values are equal.
- Exists. Exists is equivalent to wildcard for value, so that a placement can tolerate all taints of a particular category.
Effect (optional). Effect indicates the taint effect to match. Empty means match all taint effects. When specified, allowed values are NoSelect, PreferNoSelect and NoSelectIfNew. (PreferNoSelect is not implemented yet, currently clusters with effect PreferNoSelect will always be selected.)
TolerationSeconds (optional). TolerationSeconds represents the period of time the toleration (which must be of effect NoSelect/PreferNoSelect, otherwise this field is ignored) tolerates the taint. The default value is nil, which indicates it tolerates the taint forever. The start time of counting the TolerationSeconds should be the TimeAdded in Taint, not the cluster scheduled time or TolerationSeconds added time.

Prioritizers

Score-based prioritizer

In prioritizerPolicy section, you can define the policy of prioritizers.

The following example shows how to select clusters with prioritizers.

Select a cluster with the largest allocatable memory.
```
apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Placement
metadata:
  name: placement1
  namespace: ns1
spec:
  numberOfClusters: 1
  prioritizerPolicy:
    configurations:
      - scoreCoordinate:
          builtIn: ResourceAllocatableMemory
```
The prioritizer policy has default mode additive and default prioritizers Steady and Balance.
In the above example, the prioritizers actually come into effect are Steady, Balance and ResourceAllocatableMemory.
And the end of this section has more description about the prioritizer policy mode and default prioritizers.
Select a cluster with the largest allocatable CPU and memory, and make placement sensitive to resource changes.
```
apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Placement
metadata:
  name: placement1
  namespace: ns1
spec:
  numberOfClusters: 1
  prioritizerPolicy:
    configurations:
      - scoreCoordinate:
          builtIn: ResourceAllocatableCPU
        weight: 2
      - scoreCoordinate:
          builtIn: ResourceAllocatableMemory
        weight: 2
```
The prioritizer policy has default mode additive and default prioritizers Steady and Balance, and their default weight is 1.
In the above example, the prioritizers actually come into effect are Steady with weight 1, Balance with weight 1, ResourceAllocatableCPU with weight 2 and ResourceAllocatableMemory with weight 2. The cluster score will be a combination of the 4 prioritizers score. Since ResourceAllocatableCPU and ResourceAllocatableMemory have higher weight, they will be weighted more in the results, and make placement sensitive to resource changes.
And the end of this section has more description about the prioritizer weight and how the final score is calculated.

Select two clusters with the largest addon score CPU ratio, and pin the placement decisions.

apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Placement
metadata:
  name: placement1
  namespace: ns1
spec:
  numberOfClusters: 2
  prioritizerPolicy:
    mode: Exact
    configurations:
      - scoreCoordinate:
          builtIn: Steady
        weight: 3
      - scoreCoordinate:
          type: AddOn
          addOn:
            resourceName: default
            scoreName: cpuratio

In the above example, explicitly define the mode as exact. The prioritizers actually come into effect are Steady with weight 3 and addon score cpuratio with weight 1. Go into the Extensible scheduling section to learn more about addon score.

In prioritizerPolicy section, it includes the following fields:

mode is either Exact, Additive or "", where "" is Additive by default.
- In Additive mode, any prioritizer not explicitly enumerated is enabled in its default Configurations, in which Steady and Balance prioritizers have the weight of 1 while other prioritizers have the weight of 0. Additive doesn’t require configuring all prioritizers. The default Configurations may change in the future, and additional prioritization will happen.
- In Exact mode, any prioritizer not explicitly enumerated is weighted as zero. Exact requires knowing the full set of prioritizers you want, but avoids behavior changes between releases.
configurations represents the configuration of prioritizers.
- scoreCoordinate represents the configuration of the prioritizer and score source.
  - type defines the type of the prioritizer score. Type is either BuiltIn, AddOn or “”, where "" is BuiltIn by default. When the type is BuiltIn, a BuiltIn prioritizer name must be specified. When the type is AddOn, need to configure the score source in AddOn.
    - builtIn defines the name of a BuiltIn prioritizer. Below are the valid BuiltIn prioritizer names.
      - Balance: balance the decisions among the clusters.
      - Steady: ensure the existing decision is stabilized.
      - ResourceAllocatableCPU: sort clusters based on the allocatable CPU.
      - ResourceAllocatableMemory: sort clusters based on the allocatable memory.
    - addOn defines the resource name and score name. AddOnPlacementScore is introduced to describe addon scores, go into the Extensible scheduling section to learn more about it.
      - resourceName defines the resource name of the AddOnPlacementScore. The placement prioritizer selects AddOnPlacementScore CR by this name.
      - scoreName defines the score name inside AddOnPlacementScore. AddOnPlacementScore contains a list of score name and score value, scoreName specifies the score to be used by the prioritizer.
- weight defines the weight of the prioritizer. The value must be ranged in [-10,10]. Each prioritizer will calculate an integer score of a cluster in the range of [-100, 100]. The final score of a cluster will be sum(weight * prioritizer_score). A higher weight indicates that the prioritizer weights more in the cluster selection, while 0 weight indicates that the prioritizer is disabled. A negative weight indicates wanting to select the last ones.

Extensible scheduling

In placement resource based scheduling, in some cases the prioritizer needs extra data (more than the default value provided by ManagedCluster) to calculate the score of the managed cluster. For example, schedule the clusters based on cpu or memory usage data of the clusters fetched from a monitoring system.

So we provide a new API AddOnPlacementScore to support a more extensible way to schedule based on customized scores.

As a user, as mentioned in the above section, can specify the score in placement yaml to select clusters.
As a score provider, a 3rd party controller could run on either hub or managed cluster, to maintain the lifecycle of AddOnPlacementScore and update score into it.

Extend the multi-cluster scheduling capabilities with placement introduces how to implement a customized score provider.

Refer to the enhancements to learn more.

PlacementDecisions

A slice of PlacementDecision will be created by placement controller in the same namespace, each with a label of cluster.open-cluster-management.io/placement={placement name}. PlacementDecision contains the results of the cluster selection as seen in the following examples.

apiVersion: cluster.open-cluster-management.io/v1beta1
kind: PlacementDecision
metadata:
  labels:
    cluster.open-cluster-management.io/placement: placement1
  name: placement1-decision-1
  namespace: default
status:
  decisions:
    - clusterName: cluster1
    - clusterName: cluster2
    - clusterName: cluster3

The status.decisions lists the top N clusters with the highest score and ordered by names. The status.decisions changes over time, the scheduling result update based on what endpoints exist.

The scheduling result in the PlacementDecision API is designed to be paginated with its page index as the name’s suffix to avoid “too large object” issue from the underlying Kubernetes API framework.

PlacementDecision can be consumed by another operand to decide how the workload should be placed in multiple clusters.

Decision strategy

The decisionStrategy section of Placement can be used to divide the created PlacementDecision into groups and define the number of clusters per decision group.

Assume an environment has 310 clusters, 10 of which have the label prod-canary-west and 10 have the label prod-canary-east. The following example demonstrates how to group the clusters with the labels prod-canary-west and prod-canary-east into 2 groups, and group the remaining clusters into groups with a maximum of 150 clusters each.

apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Placement
metadata:
  name: placement1
  namespace: default
spec:
  clusterSets:
    - global
  decisionStrategy:
    groupStrategy:
      clustersPerDecisionGroup: 150
      decisionGroups:
      - groupName: prod-canary-west
        groupClusterSelector:
          labelSelector:
            matchExpressions:
              - key: prod-canary-west
                operator: Exists
      - groupName: prod-canary-east
        groupClusterSelector:
          labelSelector:
            matchExpressions:
              - key: prod-canary-east
                operator: Exists

The decisionStrategy section includes the following fields:

decisionGroups: Represents a list of predefined groups to put decision results. Decision groups will be constructed based on the decisionGroups field at first. The clusters not included in the decisionGroups will be divided to other decision groups afterwards. Each decision group should not have the number of clusters larger than the clustersPerDecisionGroup.
- groupName: Represents the name to be added as the value of label key cluster.open-cluster-management.io/decision-group-name of created PlacementDecisions.
- groupClusterSelector: Defines the label selector to select clusters subset by label.
clustersPerDecisionGroup: A specific number or percentage of the total selected clusters. The specific number will divide the placementDecisions to decisionGroups, the max number of clusters in each group equal to that specific number.

With this decision strategy defined, the placement status will list the group result, including the decision group name and index, the cluster count, and the corresponding PlacementDecision names.

status:
...
  decisionGroups:
  - clusterCount: 10
    decisionGroupIndex: 0
    decisionGroupName: prod-canary-west
    decisions:
    - placement1-decision-1
  - clusterCount: 10
    decisionGroupIndex: 1
    decisionGroupName: prod-canary-east
    decisions:
    - placement1-decision-2
  - clusterCount: 150
    decisionGroupIndex: 2
    decisionGroupName: ""
    decisions:
    - placement1-decision-3
    - placement1-decision-4
  - clusterCount: 140
    decisionGroupIndex: 3
    decisionGroupName: ""
    decisions:
    - placement1-decision-5
    - placement1-decision-6
  numberOfSelectedClusters: 310

The PlacementDecision will have labels cluster.open-cluster-management.io/decision-group-name and cluster.open-cluster-management.io/decision-group-index to indicate which group name and group index it belongs to.

apiVersion: cluster.open-cluster-management.io/v1beta1
kind: PlacementDecision
metadata:
  labels:
    cluster.open-cluster-management.io/placement: placement1
    cluster.open-cluster-management.io/decision-group-index: "0"
    cluster.open-cluster-management.io/decision-group-name: prod-canary-west
  name: placement1-decision-1
  namespace: default
...

Rollout Strategy

Rollout Strategy API facilitate the use of placement decision strategy with OCM workload applier APIs such as Policy, Addon and ManifestWorkReplicaSet to apply workloads.

    placements:
    - name: placement-example
      rolloutStrategy:
        type: Progressive
        progressive:
          mandatoryDecisionGroups:
          - groupName: "prod-canary-west"
          - groupName: "prod-canary-east"
          maxConcurrency: 25%
          minSuccessTime: 5m
          progressDeadline: 10m
          maxFailures: 2

The Rollout Strategy API provides three rollout types;

All: means apply the workload to all clusters in the decision groups at once.
Progressive: means apply the workload to the selected clusters progressively per cluster. The workload will not be applied to the next cluster unless one of the current applied clusters reach the successful state and haven’t breached the MaxFailures configuration.
ProgressivePerGroup: means apply the workload to decisionGroup clusters progressively per group. The workload will not be applied to the next decisionGroup unless all clusters in the current group reach the successful state and haven’t breached the MaxFailures configuration.

The RollOut Strategy API also provides rollOut config to fine-tune the workload apply progress based on the use-case requirements;

MinSuccessTime: defined in seconds/minutes/hours for how long workload applier controller will wait from the beginning of the rollout to proceed with the next rollout, assuming a successful state had been reached and MaxFailures hasn’t been breached. Default is 0 meaning the workload applier proceeds immediately after a successful state is reached.
ProgressDeadline: defined in seconds/minutes/hours for how long workload applier controller will wait until the workload reaches a successful state in the spoke cluster. If the workload does not reach a successful state after ProgressDeadline, the controller will stop waiting and workload will be treated as “timeout” and be counted into MaxFailures. Once the MaxFailures is breached, the rollout will stop. Default value is “None”, meaning the workload applier will wait for a successful state indefinitely.
MaxFailures: defined as the maximum percentage of or number of clusters that can fail in order to proceed with the rollout. Fail means the cluster has a failed status or timeout status (does not reach successful status after ProgressDeadline). Once the MaxFailures is breached, the rollout will stop. Default is 0 means that no failures are tolerated.
MaxConcurrency: is the max number of clusters to deploy workload concurrently. The MaxConcurrency can be defined only in case rollout type is progressive.
MandatoryDecisionGroups: is a list of decision groups to apply the workload first. If mandatoryDecisionGroups not defined the decision group index is considered to apply the workload in groups by order. The MandatoryDecisionGroups can be defined only in case rollout type is progressive or progressivePerGroup.

Troubleshooting

If no PlacementDecision generated after you creating Placement, you can run below commands to troubleshoot.

Check the `Placement` conditions

For example:

$ kubectl describe placement <placement-name>
Name:         demo-placement
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  cluster.open-cluster-management.io/v1beta1
Kind:         Placement
...
Status:
  Conditions:
    Last Transition Time:       2022-09-30T07:39:45Z
    Message:                    Placement configurations check pass
    Reason:                     Succeedconfigured
    Status:                     False
    Type:                       PlacementMisconfigured
    Last Transition Time:       2022-09-30T07:39:45Z
    Message:                    No valid ManagedClusterSetBindings found in placement namespace
    Reason:                     NoManagedClusterSetBindings
    Status:                     False
    Type:                       PlacementSatisfied
  Number Of Selected Clusters:  0
...

The Placement has 2 types of condition, PlacementMisconfigured and PlacementSatisfied.

If the condition PlacementMisconfigured is true, means your placement has configuration errors, the message tells you more details about the failure.
If the condition PlacementSatisfied is false, means no ManagedCluster satisfy this placement, the message tells you more details about the failure. In this example, it is because no ManagedClusterSetBindings found in placement namespace.

Check the `Placement` events

For example:

$ kubectl describe placement <placement-name>
Name:         demo-placement
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  cluster.open-cluster-management.io/v1beta1
Kind:         Placement
...
Events:
  Type    Reason          Age   From                 Message
  ----    ------          ----  ----                 -------
  Normal  DecisionCreate  2m10s   placementController  Decision demo-placement-decision-1 is created with placement demo-placement in namespace default
  Normal  DecisionUpdate  2m10s   placementController  Decision demo-placement-decision-1 is updated with placement demo-placement in namespace default
  Normal  ScoreUpdate     2m10s   placementController  cluster1:0 cluster2:100 cluster3:200
  Normal  DecisionUpdate  3s      placementController  Decision demo-placement-decision-1 is updated with placement demo-placement in namespace default
  Normal  ScoreUpdate     3s      placementController  cluster1:200 cluster2:145 cluster3:189 cluster4:200

The placement controller will give a score to each filtered ManagedCluster and generate an event for it. When the cluster score changes, a new event will generate. You can check the score of each cluster in the Placment events, to know why some clusters with lower score are not selected.

Debug

If you want to know more defails of how clusters are selected in each step, can following below step to access the debug endpoint.

Create clusterrole “debugger” to access debug path and bind this to anonymous user.

kubectl create clusterrole "debugger" --verb=get --non-resource-url="/debug/*"
kubectl create clusterrolebinding debugger --clusterrole=debugger --user=system:anonymous

Export placement 8443 port to local.

kubectl port-forward -n open-cluster-management-hub deploy/cluster-manager-placement-controller 8443:8443

Curl below url to debug one specific placement.

curl -k  https://127.0.0.1:8443/debug/placements/<namespace>/<name>

For example, the environment has a Placement named placement1 in default namespace, which selects 2 ManagedClusters, the output would be like:

$ curl -k  https://127.0.0.1:8443/debug/placements/default/placement1
{"filteredPiplieResults":[{"name":"Predicate","filteredClusters":["cluster1","cluster2"]},{"name":"Predicate,TaintToleration","filteredClusters":["cluster1","cluster2"]}],"prioritizeResults":[{"name":"Balance","weight":1,"scores":{"cluster1":100,"cluster2":100}},{"name":"Steady","weight":1,"scores":{"cluster1":100,"cluster2":100}}]}

Future work

In addition to selecting cluster by predicates, we are still working on other advanced features including

Various workload spread policies.

1.6 - ManifestWork

What is `ManifestWork`

ManifestWork is used to define a group of Kubernetes resources on the hub to be applied to the managed cluster. In the open-cluster-management project, a ManifestWork resource must be created in the cluster namespace. A work agent implemented in work project is run on the managed cluster and monitors the ManifestWork resource in the cluster namespace on the hub cluster.

An example of ManifestWork to deploy a deployment to the managed cluster is shown in the following example.

apiVersion: work.open-cluster-management.io/v1
kind: ManifestWork
metadata:
  namespace: <target managed cluster>
  name: hello-work-demo
spec:
  workload:
    manifests:
      - apiVersion: apps/v1
        kind: Deployment
        metadata:
          name: hello
          namespace: default
        spec:
          selector:
            matchLabels:
              app: hello
          template:
            metadata:
              labels:
                app: hello
            spec:
              containers:
                - name: hello
                  image: quay.io/asmacdo/busybox
                  command:
                    ["sh", "-c", 'echo "Hello, Kubernetes!" && sleep 3600']

Status tracking

Work agent will track all the resources defined in ManifestWork and update its status. There are two types of status in manifestwork. The resourceStatus tracks the status of each manifest in the ManifestWork and conditions reflects the overall status of the ManifestWork. Work agent currently checks whether a resource is Available, meaning the resource exists on the managed cluster, and Applied means the resource defined in ManifestWork has been applied to the managed cluster.

Here is an example.

apiVersion: work.open-cluster-management.io/v1
kind: ManifestWork
metadata: ...
spec: ...
status:
  conditions:
    - lastTransitionTime: "2021-06-15T02:26:02Z"
      message: Apply manifest work complete
      reason: AppliedManifestWorkComplete
      status: "True"
      type: Applied
    - lastTransitionTime: "2021-06-15T02:26:02Z"
      message: All resources are available
      reason: ResourcesAvailable
      status: "True"
      type: Available
  resourceStatus:
    manifests:
      - conditions:
          - lastTransitionTime: "2021-06-15T02:26:02Z"
            message: Apply manifest complete
            reason: AppliedManifestComplete
            status: "True"
            type: Applied
          - lastTransitionTime: "2021-06-15T02:26:02Z"
            message: Resource is available
            reason: ResourceAvailable
            status: "True"
            type: Available
        resourceMeta:
          group: apps
          kind: Deployment
          name: hello
          namespace: default
          ordinal: 0
          resource: deployments
          version: v1

Fine-grained field values tracking

Optionally, we can let the work agent aggregate and report certain fields from the distributed resources to the hub clusters by setting FeedbackRule for the ManifestWork:

apiVersion: work.open-cluster-management.io/v1
kind: ManifestWork
metadata: ...
spec:
  workload: ...
  manifestConfigs:
    - resourceIdentifier:
        group: apps
        resource: deployments
        namespace: default
        name: hello
      feedbackRules:
        - type: WellKnownStatus
        - type: JSONPaths
          jsonPaths:
            - name: isAvailable
              path: '.status.conditions[?(@.type=="Available")].status'

The feedback rules prescribe the work agent to periodically get the latest states of the resources, and scrape merely those expected fields from them, which is helpful for trimming the payload size of the status. Note that the collected feedback values on the ManifestWork will not be updated unless the latest value is changed/different from the previous recorded value. Currently, it supports two kinds of FeedbackRule:

WellKnownStatus: Using the pre-built template of feedback values for those well-known kubernetes resources.
JSONPaths: A valid Kubernetes JSON-Path that selects a scalar field from the resource. Currently supported types are Integer, String, Boolean and JsonRaw. JsonRaw returns only when you have enabled the RawFeedbackJsonString feature gate on the agent. The agent will return the whole structure as a JSON string.

The default feedback value scraping interval is 30 second, and we can override it by setting --status-sync-interval on your work agent. Too short period can cause excessive burden to the control plane of the managed cluster, so generally a recommended lower bound for the interval is 5 second.

In the end, the scraped values from feedback rules will be shown in the status:

apiVersion: work.open-cluster-management.io/v1
kind: ManifestWork
metadata: ...
spec: ...
status:
  resourceStatus:
    manifests:
    - conditions: ...
      resourceMeta: ...
      statusFeedback:
        values:
        - fieldValue:
            integer: 1
            type: Integer
          name: ReadyReplicas
        - fieldValue:
            integer: 1
            type: Integer
          name: Replicas
        - fieldValue:
            integer: 1
            type: Integer
          name: AvailableReplicas
        - fieldValue:
            string: "True"
            type: String
          name: isAvailable

Garbage collection

To ensure the resources applied by ManifestWork are reliably recorded, the work agent creates an AppliedManifestWork on the managed cluster for each ManifestWork as an anchor for resources relating to ManifestWork. When ManifestWork is deleted, work agent runs a Foreground deletion, that ManifestWork will stay in deleting state until all its related resources has been fully cleaned in the managed cluster.

Delete options

User can explicitly choose not to garbage collect the applied resources when a ManifestWork is deleted. The user should specify the deleteOption in the ManifestWork. By default, deleteOption is set as Foreground which means the applied resources on the spoke will be deleted with the removal of ManifestWork. User can set it to Orphan so the applied resources will not be deleted. Here is an example:

apiVersion: work.open-cluster-management.io/v1
kind: ManifestWork
metadata: ...
spec:
  workload: ...
  deleteOption:
    propagationPolicy: Orphan

Alternatively, user can also specify a certain resource defined in the ManifestWork to be orphaned by setting the deleteOption to be SelectivelyOrphan. Here is an example with SelectivelyOrphan specified. It ensures the removal of deployment resource specified in the ManifestWork while the service resource is kept.

apiVersion: work.open-cluster-management.io/v1
kind: ManifestWork
metadata:
  name: selective-delete-work
spec:
  workload: ...
  deleteOption:
    propagationPolicy: SelectivelyOrphan
    selectivelyOrphans:
      orphaningRules:
      - group: ""
        resource: services
        namespace: default
        name: helloworld

Resource Race and Adoption

It is possible to create two ManifestWorks for the same cluster with the same resource defined. For example, the user can create two Manifestworks on cluster1, and both Manifestworks have the deployment resource hello in default namespace. If the content of the resource is different, the two ManifestWorks will fight, and it is desired since each ManifestWork is treated as equal and each ManifestWork is declaring the ownership of the resource. If there is another controller on the managed cluster that tries to manipulate the resource applied by a ManifestWork, this controller will also fight with work agent.

When one of the ManifestWork is deleted, the applied resource will not be removed no matter DeleteOption is set or not. The remaining ManifestWork will still keep the ownership of the resource.

To resolve such conflict, user can choose a different update strategy to alleviate the resource conflict.

CreateOnly: with this strategy, the work-agent will only ensure creation of the certain manifest if the resource does not exist. work-agent will not update the resource, hence the ownership of the whole resource can be taken over by another ManifestWork or controller.
ServerSideApply: with this strategy, the work-agent will run server side apply for the certain manifest. The default field manager is work-agent, and can be customized. If another ManifestWork or controller takes the ownership of a certain field in the manifest, the original ManifestWork will report conflict. User can prune the original ManifestWork so only field that it will own maintains.
ReadOnly: with this strategy, the work-agent will not apply manifests onto the cluster, but it still can read resource fields and return results when feedback rules are defined. Only metadata of the manifest is required to be defined in the spec of the ManifestWork with this strategy.

An example of using ServerSideApply strategy as following:

User creates a ManifestWork with ServerSideApply specified:

apiVersion: work.open-cluster-management.io/v1
kind: ManifestWork
metadata:
  namespace: <target managed cluster>
  name: hello-work-demo
spec:
  workload: ...
  manifestConfigs:
    - resourceIdentifier:
        group: apps
        resource: deployments
        namespace: default
        name: hello
      updateStrategy:
        type: ServerSideApply

User creates another ManifestWork with ServerSideApply but with different field manager.

apiVersion: work.open-cluster-management.io/v1
kind: ManifestWork
metadata:
  namespace: <target managed cluster>
  name: hello-work-replica-patch
spec:
  workload:
    manifests:
      - apiVersion: apps/v1
        kind: Deployment
        metadata:
          name: hello
          namespace: default
        spec:
          replicas: 3
  manifestConfigs:
    - resourceIdentifier:
        group: apps
        resource: deployments
        namespace: default
        name: hello
      updateStrategy:
        type: ServerSideApply
        serverSideApply:
          force: true
          fieldManager: work-agent-another

The second ManifestWork only defines replicas in the manifest, so it takes the ownership of replicas. If the first ManifestWork is updated to add replicas field with different value, it will get conflict condition and manifest will not be updated by it.

Instead of create the second ManifestWork, user can also set HPA for this deployment. HPA will also take the ownership of replicas, and the update of replicas field in the first ManifestWork will return conflict condition.

Permission setting for work agent

All workload manifests are applied to the managed cluster by the work agent, and by default the work agent has the following permission for the managed cluster:

clusterRole admin(instead of the cluster-admin) to apply kubernetes common resources
managing customresourcedefinitions, but can not manage a specific custom resource instance
managing clusterrolebindings, rolebindings, clusterroles, roles, including the bind and escalate permission, this is why we can grant work-agent service account extra permissions using ManifestWork

So if the workload manifests to be applied on the managed cluster exceeds the above permission, for example some Customer Resource instances, there will be an error ... is forbidden: User "system:serviceaccount:open-cluster-management-agent:klusterlet-work-sa" cannot get resource ... reflected on the ManifestWork status.

To prevent this, the service account klusterlet-work-sa used by the work-agent needs to be given the corresponding permissions. There are several ways:

add permission on the managed cluster directly, we can
- aggregate the new clusterRole for your to-be-applied resources to the existing admin clusterRole
- OR create role/clusterRole roleBinding/clusterRoleBinding for the klusterlet-work-sa service account
add permission on the hub cluster by another ManifestWork, the ManifestWork includes
- an clusterRole with label "open-cluster-management.io/aggregate-to-work": "true" for your to-be-applied resources, the rules defined in the clusterRole will be aggregated to the work agent(OCM version >= v0.12.0)
- OR role/clusterRole roleBinding/clusterRoleBinding for the klusterlet-work-sa service account

Below is an example use ManifestWork to give klusterlet-work-sa permission for resource machines.cluster.x-k8s.io

Option 1: Use aggregated clusterRole

apiVersion: work.open-cluster-management.io/v1
kind: ManifestWork
metadata:
  namespace: cluster1
  name: permission-set
spec:
  workload:
    manifests:
      - apiVersion: rbac.authorization.k8s.io/v1
        kind: ClusterRole
        metadata:
          name: open-cluster-management:klusterlet-work:my-role
          labels:
            open-cluster-management.io/aggregate-to-work: "true"  # with this label, the clusterRole will be selected to aggregate
        rules:
          # Allow agent to managed machines
          - apiGroups: ["cluster.x-k8s.io"]
            resources: ["machines"]
            verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

Option 2: Use clusterRole and clusterRoleBinding

apiVersion: work.open-cluster-management.io/v1
kind: ManifestWork
metadata:
  namespace: cluster1
  name: permission-set
spec:
  workload:
    manifests:
      - apiVersion: rbac.authorization.k8s.io/v1
        kind: ClusterRole
        metadata:
          name: open-cluster-management:klusterlet-work:my-role
        rules:
          # Allow agent to managed machines
          - apiGroups: ["cluster.x-k8s.io"]
            resources: ["machines"]
            verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
      - apiVersion: rbac.authorization.k8s.io/v1
        kind: ClusterRoleBinding
        metadata:
          name: open-cluster-management:klusterlet-work:my-binding
        roleRef:
          apiGroup: rbac.authorization.k8s.io
          kind: ClusterRole
          name: open-cluster-management:klusterlet-work:my-role
        subjects:
          - kind: ServiceAccount
            name: klusterlet-work-sa
            namespace: open-cluster-management-agent

Treating defaulting/immutable fields in API

The kube-apiserver sets the defaulting/immutable fields for some APIs if the user does not set them. And it may fail to deploy these APIs using ManifestWork. Because in the reconcile loop, the work agent will try to update the immutable or default field after comparing the desired manifest in the ManifestWork and existing resource in the cluster, and the update will fail or not take effect.

Let’s use Job as an example. The kube-apiserver will set a default selector and label on the Pod of Job if the user does not set spec.Selector in the Job. The fields are immutable, so the ManifestWork will report AppliedManifestFailed when we apply a Job without spec.Selector using ManifestWork.

apiVersion: work.open-cluster-management.io/v1
kind: ManifestWork
metadata:
  namespace: cluster1
  name: exmaple-job
spec:
  workload:
    manifests:
      - apiVersion: batch/v1
        kind: Job
        metadata:
          name: pi
          namespace: default
        spec:
          template:
            spec:
              containers:
              - name: pi
                image: perl:5.34.0
                command: ["perl",  "-Mbignum=bpi", "-wle", "print bpi(2000)"]
              restartPolicy: Never
          backoffLimit: 4

There are 2 options to fix this issue.

Specify the fields manually if they are configurable. For example, set spec.manualSelector=true and your own labels in the spec.selector of the Job, and set the same labels for the containers.

apiVersion: work.open-cluster-management.io/v1
kind: ManifestWork
metadata:
  namespace: cluster1
  name: exmaple-job-1
spec:
  workload:
    manifests:
      - apiVersion: batch/v1
        kind: Job
        metadata:
          name: pi
          namespace: default
        spec:
          manualSelector: true
          selector:
            matchLabels:
              job: pi
          template:
            metadata:
              labels:
                job: pi
            spec:
              containers:
              - name: pi
                image: perl:5.34.0
                command: ["perl",  "-Mbignum=bpi", "-wle", "print bpi(2000)"]
              restartPolicy: Never
          backoffLimit: 4

Set the updateStrategy ServerSideApply in the ManifestWork for the API.

apiVersion: work.open-cluster-management.io/v1
kind: ManifestWork
metadata:
  namespace: cluster1
  name: exmaple-job
spec:
  manifestConfigs:
    - resourceIdentifier:
        group: batch
        resource: jobs
        namespace: default
        name: pi
      updateStrategy:
        type: ServerSideApply
  workload:
    manifests:
      - apiVersion: batch/v1
        kind: Job
        metadata:
          name: pi
          namespace: default
        spec:
          template:
            spec:
              containers:
              - name: pi
                image: perl:5.34.0
                command: ["perl",  "-Mbignum=bpi", "-wle", "print bpi(2000)"]
              restartPolicy: Never
          backoffLimit: 4

Dynamic identity authorization

All manifests in ManifestWork are applied by the work-agent using the mounted service account to raise requests against the managed cluster by default. And the work agent has very high permission to access the managed cluster which means that any hub user with write access to the ManifestWork resources will be able to dispatch any resources that the work-agent can manipulate to the managed cluster.

The executor subject feature(introduced in release 0.9.0) provides a way to clarify the owner identity(executor) of the ManifestWork before it takes effect so that we can explicitly check whether the executor has sufficient permission in the managed cluster.

The following example clarifies the owner “executor1” of the ManifestWork, so before the work-agent applies the “default/test” ConfigMap to the managed cluster, it will first check whether the ServiceAccount “default/executor” has the permission to apply this ConfigMap

apiVersion: work.open-cluster-management.io/v1
kind: ManifestWork
metadata:
  namespace: cluster1
  name: example-manifestwork
spec:
  executor:
    subject:
      type: ServiceAccount
      serviceAccount:
        namespace: default
        name: executor1
  workload:
    manifests:
      - apiVersion: v1
        data:
          a: b
        kind: ConfigMap
        metadata:
          namespace: default
          name: test

Not any hub user can specify any executor at will. Hub users can only use the executor for which they have an execute-as(virtual verb) permission. For example, hub users bound to the following Role can use the “executor1” ServiceAccount in the “default” namespace on the managed cluster.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: cluster1-executor1
  namespace: cluster1
rules:
- apiGroups:
  - work.open-cluster-management.io
  resources:
  - manifestworks
  verbs:
  - execute-as
  resourceNames:
  - system:serviceaccount:default:executor1

For backward compatibility, if the executor is absent, the work agent will keep using the mounted service account to apply resources. But using the executor is encouraged, so we have a feature gate NilExecutorValidating to control whether any hub user is allowed to not set the executor. It is disabled by default, we can use the following configuration to the ClusterManager to enable it. When it is enabled, not setting executor will be regarded as using the “/klusterlet-work-sa” (namespace is empty, name is klusterlet-work-sa) virtual service account on the managed cluster for permission verification, which means only hub users with “execute-as” permissions on the “system:serviceaccount::klusterlet-work-sa” ManifestWork are allowed not to set the executor.

spec:
  workConfiguration:
    featureGates:
    - feature: NilExecutorValidating
      mode: Enable

Work-agent uses the SubjectAccessReview API to check whether an executor has permission to the manifest resources, which will cause a large number of SAR requests to the managed cluster API-server, so we provided a new feature gate ExecutorValidatingCaches(in release 0.10.0) to cache the result of the executor’s permission to the manifest resource, it is only works when the managed cluster uses RBAC mode authorization, and is disabled by default as well, but can be enabled by using the following configuration for Klusterlet:

spec:
  workConfiguration:
    featureGates:
    - feature: ExecutorValidatingCaches
      mode: Enable

Enhancement proposal: Work Executor Group

1.7 - ManifestWorkReplicaSet

What is `ManifestWorkReplicaSet`

ManifestWorkReplicaSet is an aggregator API that uses Manifestwork and Placement to create manifestwork for the placement-selected clusters.

View an example of ManifestWorkReplicaSet to deploy a CronJob and Namespace for a group of clusters selected by placements.

apiVersion: work.open-cluster-management.io/v1alpha1
kind: ManifestWorkReplicaSet
metadata:
  name: mwrset-cronjob
  namespace: ocm-ns
spec:
  placementRefs:
    - name: placement-rollout-all # Name of a created Placement
      rolloutStrategy:
        rolloutType: All
    - name: placement-rollout-progressive # Name of a created Placement
      rolloutStrategy:
        rolloutType: Progressive
        progressive:
          minSuccessTime: 5m
          progressDeadline: 10m
          maxFailures: 5%
          mandatoryDecisionGroups:
          - groupName: "prod-canary-west"
          - groupName: "prod-canary-east"
    - name: placement-rollout-progressive-per-group # Name of a created Placement
      rolloutStrategy:
        rolloutType: ProgressivePerGroup
        progressivePerGroup:
          progressDeadline: 10m
          maxFailures: 2
  manifestWorkTemplate:
    deleteOption:
      propagationPolicy: SelectivelyOrphan
      selectivelyOrphans:
        orphaningRules:
          - group: ''
            name: ocm-ns
            namespace: ''
            resource: Namespace
    manifestConfigs:
      - feedbackRules:
          - jsonPaths:
              - name: lastScheduleTime
                path: .status.lastScheduleTime
              - name: lastSuccessfulTime
                path: .status.lastSuccessfulTime
            type: JSONPaths
        resourceIdentifier:
          group: batch
          name: sync-cronjob
          namespace: ocm-ns
          resource: cronjobs
    workload:
      manifests:
        - kind: Namespace
          apiVersion: v1
          metadata:
            name: ocm-ns
        - kind: CronJob
          apiVersion: batch/v1
          metadata:
            name: sync-cronjob
            namespace: ocm-ns
          spec:
            schedule: '* * * * *'
            concurrencyPolicy: Allow
            suspend: false
            jobTemplate:
              spec:
                backoffLimit: 2
                template:
                  spec:
                    containers:
                      - name: hello
                        image: 'quay.io/prometheus/busybox:latest'
                        args:
                          - /bin/sh
                          - '-c'
                          - date; echo Hello from the Kubernetes cluster

The PlacementRefs uses the Rollout Strategy API to apply the manifestWork to the selected clusters. In the example above; the placementRefs refers to three placements; placement-rollout-all, placement-rollout-progressive and placement-rollout-progressive-per-group. For more info regards the rollout strategies check the Rollout Strategy section at the placement document. Note: The placement reference must be in the same namespace as the manifestWorkReplicaSet.

Status tracking

The ManifestWorkReplicaSet example above refers to three placements each one will have its placementSummary in ManifestWorkReplicaSet status. The PlacementSummary shows the number of manifestWorks applied to the placement’s clusters based on the placementRef’s rolloutStrategy and total number of clusters. The manifestWorkReplicaSet Summary aggregate the placementSummaries showing the total number of applied manifestWorks to all clusters.

The manifestWorkReplicaSet has three status conditions;

PlacementVerified verify the placementRefs status; not exist or empty cluster selection.
PlacementRolledOut verify the rollout strategy status; progressing or complete.
ManifestWorkApplied verify the created manifestWork status; applied, progressing, degraded or available.

The manifestWorkReplicaSet determine the ManifestWorkApplied condition status based on the resource state (applied or available) of each manifestWork.

Here is an example.

apiVersion: work.open-cluster-management.io/v1alpha1
kind: ManifestWorkReplicaSet
metadata:
  name: mwrset-cronjob
  namespace: ocm-ns
spec:
  placementRefs:
    - name: placement-rollout-all
      ...
    - name: placement-rollout-progressive
      ...
    - name: placement-rollout-progressive-per-group
      ...
  manifestWorkTemplate:
     ...
status:
 conditions:
   - lastTransitionTime: '2023-04-27T02:30:54Z'
     message: ''
     reason: AsExpected
     status: 'True'
     type: PlacementVerified
   - lastTransitionTime: '2023-04-27T02:30:54Z'
     message: ''
     reason: Progressing
     status: 'False'
     type: PlacementRolledOut
   - lastTransitionTime: '2023-04-27T02:30:54Z'
     message: ''
     reason: AsExpected
     status: 'True'
     type: ManifestworkApplied
 placementSummary:
 - name: placement-rollout-all
   availableDecisionGroups: 1 (10 / 10 clusters applied)
   summary:
     applied: 10
     available: 10
     progressing: 0
     degraded: 0
     total: 10
 - name: placement-rollout-progressive
   availableDecisionGroups: 3 (20 / 30 clusters applied)
   summary:
     applied: 20
     available: 20
     progressing: 0
     degraded: 0
     total: 20
 - name: placement-rollout-progressive-per-group
   availableDecisionGroups: 4 (15 / 20 clusters applied)
   summary:
     applied: 15
     available: 15
     progressing: 0
     degraded: 0
     total: 15
 summary:
   applied: 45
   available: 45
   progressing: 0
   degraded: 0
   total: 45

Release and Enable Feature

ManifestWorkReplicaSet is in alpha release and it is not enabled by default. In order to enable the ManifestWorkReplicaSet feature, it has to be enabled in the cluster-manager instance in the hub. Use the following command to edit the cluster-manager CR (custom resource) in the hub cluster.

$ oc edit ClusterManager cluster-manager

Add the workConfiguration field to the cluster-manager CR as below and save.

kind: ClusterManager
metadata:
  name: cluster-manager
spec:
   ...
  workConfiguration:
    featureGates:
    - feature: ManifestWorkReplicaSet
      mode: Enable

In order to assure the ManifestWorkReplicaSet has been enabled successfully check the cluster-manager using the command below

$ oc get ClusterManager cluster-manager -o yml

You should find under the status->generation the cluster-manager-work-controller deployment has been added as below

kind: ClusterManager
metadata:
  name: cluster-manager
spec:
   ...
status:
   ...
  generations:
    ...
  - group: apps
    lastGeneration: 2
    name: cluster-manager-work-webhook
    namespace: open-cluster-management-hub
    resource: deployments
    version: v1
  - group: apps
    lastGeneration: 1
    name: cluster-manager-work-controller
    namespace: open-cluster-management-hub
    resource: deployments
    version: v1

1.8 - Add-ons

What is an add-on?

Open-cluster-management has a built-in mechanism named addon-framework to help developers to develop an extension based on the foundation components for the purpose of working with multiple clusters in custom cases. A typical addon should consist of two kinds of components:

Addon Agent: A kubernetes controller in the managed cluster that manages the managed cluster for the hub admins. A typical addon agent is expected to be working by subscribing the prescriptions (e.g. in forms of CustomResources) from the hub cluster and then consistently reconcile the state of the managed cluster like an ordinary kubernetes operator does.
Addon Manager: A kubernetes controller in the hub cluster that applies manifests to the managed clusters via the ManifestWork api. In addition to resource dispatching, the manager can optionally manage the lifecycle of CSRs for the addon agents or even the RBAC permission bond to the CSRs’ requesting identity.

In general, if a management tool working inside the managed cluster needs to discriminate configuration for each managed cluster, it will be helpful to model its implementation as a working addon agent. The configurations for each agent are supposed to be persisted in the hub cluster, so the hub admin will be able to prescribe the agent to do its job in a declarative way. In abstraction, via the addon we will be decoupling a multi-cluster control plane into (1) strategy dispatching and (2) execution. The addon manager doesn’t actually apply any changes directly to the managed cluster, instead it just places its prescription to a dedicated namespace allocated for the accepted managed cluster. Then the addon agent pulls the prescriptions consistently and does the execution.

In addition to dispatching configurations before the agents, the addon manager will be automatically doing some fiddly preparation before the agent bootstraps, such as:

CSR applying, approving and signing.
Injecting and managing client credentials used by agents to access the hub cluster.
The RBAC permission for the agents both in the hub cluster or the managed cluster.
Installing strategy.

Architecture

The following architecture graph shows how the coordination between addon manager and addon agent works.

Add-on enablement

From a user’s perspective, to install the addon to the hub cluster the hub admin should register a globally-unique ClusterManagementAddon resource as a singleton placeholder in the hub cluster. For instance, the helloworld add-on can be registered to the hub cluster by creating:

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ClusterManagementAddOn
metadata:
  name: helloworld
spec:
  addOnMeta:
    displayName: helloworld

Enable the add-on manually

The addon manager running on the hub is taking responsibility of configuring the installation of addon agents for each managed cluster. When a user wants to enable the add-on for a certain managed cluster, the user should create a ManagedClusterAddOn resource on the cluster namespace. The name of the ManagedClusterAddOn should be the same name of the corresponding ClusterManagementAddon. For instance, the following example enables helloworld add-on in “cluster1”:

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ManagedClusterAddOn
metadata:
  name: helloworld
  namespace: cluster1
spec:
  installNamespace: helloworld

Enable the add-on automatically

If the addon is developed with automatic installation, which support auto-install by cluster discovery, then the ManagedClusterAddOn will be created for all managed cluster namespaces automatically, or be created for the selected managed cluster namespaces automatically.

Enable the add-on by install strategy

If the addon is developed following the guidelines mentioned in managing the add-on agent lifecycle by addon-manager, the user can define an installStrategy in the ClusterManagementAddOn to specify on which clusters the ManagedClusterAddOn should be enabled. Details see install strategy.

Add-on healthiness

The healthiness of the addon instances are visible when we list the addons via kubectl:

$ kubectl get managedclusteraddon -A
NAMESPACE   NAME                     AVAILABLE   DEGRADED   PROGRESSING
<cluster>   <addon>                  True

The addon agent are expected to report its healthiness periodically as long as it’s running. Also the versioning of the addon agent can be reflected in the resources optionally so that we can control the upgrading the agents progressively.

Clean the add-ons

Last but not least, a neat uninstallation of the addon is also supported by simply deleting the corresponding ClusterManagementAddon resource from the hub cluster which is the “root” of the whole addon. The OCM platform will automatically sanitize the hub cluster for you after the uninstalling by removing all the components either in the hub cluster or in the manage clusters.

Add-on lifecycle management

Install strategy

InstallStrategy represents that related ManagedClusterAddOns should be installed on certain clusters. For example, the following example enables the helloworld add-on on clusters with the aws label.

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ClusterManagementAddOn
metadata:
  name: helloworld
  annotations:
    addon.open-cluster-management.io/lifecycle: "addon-manager"
spec:
  addOnMeta:
    displayName: helloworld
  installStrategy:
    type: Placements
    placements:
    - name: placement-aws
      namespace: default

apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Placement
metadata:
  name: placement-aws
  namespace: default
spec:
  predicates:
    - requiredClusterSelector:
        claimSelector:
          matchExpressions:
            - key: platform.open-cluster-management.io
              operator: In
              values:
                - aws

Rollout strategy

With the rollout strategy defined in the ClusterManagementAddOn API, users can control the upgrade behavior of the addon when there are changes in the configurations.

For example, if the add-on user updates the “deploy-config” and wants to apply the change to the add-ons to a “canary” decision group first. If all the add-on upgrade successfully, then upgrade the rest of clusters progressively per cluster at a rate of 25%. The rollout strategy can be defined as follows:

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ClusterManagementAddOn
metadata:
  name: helloworld
  annotations:
    addon.open-cluster-management.io/lifecycle: "addon-manager"
spec:
  addOnMeta:
    displayName: helloworld
  installStrategy:
    type: Placements
    placements:
    - name: placement-aws
      namespace: default
      configs:
      - group: addon.open-cluster-management.io
        resource: addondeploymentconfigs
        name: deploy-config
        namespace: open-cluster-management
      rolloutStrategy:
        type: Progressive
        progressive:
          mandatoryDecisionGroups:
          - groupName: "prod-canary-west"
          - groupName: "prod-canary-east"
          maxConcurrency: 25%
          minSuccessTime: 5m
          progressDeadline: 10m
          maxFailures: 2

In the above example with type Progressive, once user updates the “deploy-config”, controller will rollout on the clusters in mandatoryDecisionGroups first, then rollout on the other clusters with the rate defined in maxConcurrency.

minSuccessTime is a “soak” time, means the controller will wait for 5 minutes when a cluster reach a successful state and maxFailures isn’t breached. If, after this 5 minutes interval, the workload status remains successful, the rollout progresses to the next.
progressDeadline means the controller will wait for a maximum of 10 minutes for the workload to reach a successful state. If, the workload fails to achieve success within 10 minutes, the controller stops waiting, marking the workload as “timeout,” and includes it in the count of maxFailures.
maxFailures means the controller can tolerate update to 2 clusters with failed status, once maxFailures is breached, the rollout will stop.

Currently add-on supports 3 types of rolloutStrategy, they are All, Progressive and ProgressivePerGroup, for more info regards the rollout strategies check the Rollout Strategy document.

Add-on configurations

Default configurations

In ClusterManagementAddOn, spec.supportedConfigs is a list of configuration types supported by add-on. defaultConfig represents the namespace and name of the default add-on configuration. In scenario where all add-ons have a same configuration.

In the below example, add-ons on all the clusters will use “default-deploy-config”.

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ClusterManagementAddOn
metadata:
  name: helloworld
  annotations:
    addon.open-cluster-management.io/lifecycle: "addon-manager"
spec:
  addOnMeta:
    displayName: helloworld
  supportedConfigs:
  - defaultConfig:
      name: default-deploy-config
      namespace: open-cluster-management
    group: addon.open-cluster-management.io
    resource: addondeploymentconfigs

Configurations per install strategy

In ClusterManagementAddOn, spec.installStrategy.placements[].configs list the configuration of ManagedClusterAddon during installation. It will override the Default configurations on certain clusters.

In the below example, add-ons on clusters selected by Placement placement-aws will use “deploy-config” and all the other add-ons still use “default-deploy-config”.

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ClusterManagementAddOn
metadata:
  name: helloworld
  annotations:
    addon.open-cluster-management.io/lifecycle: "addon-manager"
spec:
  addOnMeta:
    displayName: helloworld
  supportedConfigs:
  - defaultConfig:
      name: default-deploy-config
      namespace: open-cluster-management
    group: addon.open-cluster-management.io
    resource: addondeploymentconfigs
  installStrategy:
    type: Placements
    placements:
    - name: placement-aws
      namespace: default
      configs:
      - group: addon.open-cluster-management.io
        resource: addondeploymentconfigs
        name: deploy-config
        namespace: open-cluster-management

Configurations per cluster

In ManagedClusterAddOn, spec.configs is a list of add-on configurations. In scenario where the current add-on has its own configurations. It will override the Default configurations and Configurations per install strategy defined in ClusterManagementAddOn.

In the below example, add-on on cluster1 will use “cluster1-deploy-config”.

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ManagedClusterAddOn
metadata:
  name: helloworld
  namespace: cluster1
spec:
  configs:
  - group: addon.open-cluster-management.io
    resource: addondeploymentconfigs
    name: cluster1-deploy-config
    namespace: open-cluster-management

Supported configurations

Supported configurations is a list of configuration types that are allowed to override the add-on configurations defined in ClusterManagementAddOn spec. They are listed in the ManagedClusterAddon status.supportedConfigs, for example:

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ManagedClusterAddOn
metadata:
  name: helloworld
  namespace: cluster1
spec:
...
status:
...
  supportedConfigs:
  - group: addon.open-cluster-management.io
    resource: addondeploymentconfigs

Effective configurations

As the above described, there are 3 places to define the add-on configurations, they have an override order and eventually only one takes effect. The final effective configurations are listed in the ManagedClusterAddOn status.configReferences.

desiredConfig record the desired config and it’s spec hash.
lastAppliedConfig record the config when the corresponding ManifestWork is applied successfully.

For example:

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ManagedClusterAddOn
metadata:
  name: helloworld
  namespace: cluster1
...
status:
...
  configReferences:
  - desiredConfig:
      name: cluster1-deploy-config
      namespace: open-cluster-management
      specHash: dcf88f5b11bd191ed2f886675f967684da8b5bcbe6902458f672277d469e2044
    group: addon.open-cluster-management.io
    lastAppliedConfig:
      name: cluster1-deploy-config
      namespace: open-cluster-management
      specHash: dcf88f5b11bd191ed2f886675f967684da8b5bcbe6902458f672277d469e2044
    lastObservedGeneration: 1
    name: cluster1-deploy-config
    resource: addondeploymentconfigs

Examples

Here’s a few examples of cases where we will need add-ons:

A tool to collect alert events in the managed cluster, and send to the hub cluster.
A network solution that uses the hub to share the network info and establish connection among managed clusters. See cluster-proxy
A tool to spread security policies to multiple clusters.

Add-on framework

Add-on framework provides a library for developers to develop an add-ons in open-cluster-management more easily. Take a look at the helloworld example to understand how the add-on framework can be used.

Custom signers

The original Kubernetes CSR api only supports three built-in signers:

“kubernetes.io/kube-apiserver-client”
“kubernetes.io/kube-apiserver-client-kubelet”
“kubernetes.io/kubelet-serving”

However in some cases, we need to sign additional custom certificates for the addon agents which is not used for connecting any kube-apiserver. The addon manager can be serving as a custom CSR signer controller based on the addon-framework’s extensibility by implementing the signing logic. Note that after successfully signing the certificates, the framework will also keep rotating the certificates automatically for the addon.

Hub credential injection

The addon manager developed base on addon-framework will automatically persist the signed certificates as secret resource to the managed clusters after signed by either original Kubernetes CSR controller or custom signers. The injected secrets will be:

For “kubernetes.io/kube-apiserver-client” signer, the name will be “
-hub-kubeconfig” with properties:
- “kubeconfig”: a kubeconfig file for accessing hub cluster with the addon’s identity.
- “tls.crt”: the signed certificate.
- “tls.key”: the private key.
For custom signer, the name will be “
-
-client-cert” with properties:
- “tls.crt”: the signed certificate.
- “tls.key”: the private key.

Auto-install by cluster discovery

The addon manager can automatically install an addon to the managed clusters upon discovering new clusters by setting the InstallStrategy from the addon-framework. On the other hand, the admin can also manually install the addon for the clusters by applying ManagedClusterAddOn into their cluster namespace.

1.9 - Policy

Overview

Note: this is also covered in the Open Cluster Management - Configuring Your Kubernetes Fleet With the Policy Addon video.

The policy framework has the following API concepts:

Policy Templates are the policies that perform a desired check or action. For example, ConfigurationPolicy objects are embedded in Policy objects under the policy-templates array.
A Policy is a grouping mechanism for Policy Templates and is the smallest deployable unit on the hub cluster. Embedded Policy Templates are distributed to applicable managed clusters and acted upon by the appropriate policy controller.
A PolicySet is a grouping mechanism of Policy objects. Compliance of all grouped Policy objects is summarized in the PolicySet. A PolicySet is a deployable unit and its distribution is controlled by a Placement.
A PlacementBinding binds a Placement to a Policy or PolicySet.

The second half of the KubeCon NA 2022 - OCM Multicluster App & Config Management also covers an overview of the Policy addon.

Policy

A Policy is a grouping mechanism for Policy Templates and is the smallest deployable unit on the hub cluster. Embedded Policy Templates are distributed to applicable managed clusters and acted upon by the appropriate policy controller. The compliance state and status of a Policy represents all embedded Policy Templates in the Policy. The distribution of Policy objects is controlled by a Placement.

View a simple example of a Policy that embeds a ConfigurationPolicy policy template to manage a namespace called “prod”.

apiVersion: policy.open-cluster-management.io/v1
kind: Policy
metadata:
  name: policy-namespace
  namespace: policies
  annotations:
    policy.open-cluster-management.io/standards: NIST SP 800-53
    policy.open-cluster-management.io/categories: CM Configuration Management
    policy.open-cluster-management.io/controls: CM-2 Baseline Configuration
spec:
  remediationAction: enforce
  disabled: false
  policy-templates:
    - objectDefinition:
        apiVersion: policy.open-cluster-management.io/v1
        kind: ConfigurationPolicy
        metadata:
          name: policy-namespace-example
        spec:
          remediationAction: inform
          severity: low
          object-templates:
            - complianceType: musthave
              objectDefinition:
                kind: Namespace # must have namespace 'prod'
                apiVersion: v1
                metadata:
                  name: prod

The annotations are standard annotations for informational purposes and can be used by user interfaces, custom report scripts, or components that integrate with OCM.

The optional spec.remediationAction field dictates whether the policy controller should inform or enforce when violations are found and overrides the remediationAction field on each policy template. When set to inform, the Policy will become noncompliant if the underlying policy templates detect that the desired state is not met. When set to enforce, the policy controller applies the desired state when necessary and feasible.

The policy-templates array contains an array of Policy Templates. Here a single ConfigurationPolicy called policy-namespace-example defines a Namespace manifest to compare with objects on the cluster. It has the remediationAction set to inform but it is overridden by the optional global spec.remediationAction. The severity is for informational purposes similar to the annotations.

Inside of the embedded ConfigurationPolicy, the object-templates section describes the prod Namespace object that the ConfigurationPolicy applies to. The action that the ConfigurationPolicy will take is determined by the complianceType. In this case, it is set to musthave which means the prod Namespace object will be created if it doesn’t exist. Other compliance types include mustnothave and mustonlyhave. mustnothave would delete the prod Namespace object. mustonlyhave would ensure the prod Namespace object only exists with the fields defined in the ConfigurationPolicy. See the ConfigurationPolicy page for more information or see the templating in configuration policies topic for advanced templating use cases with ConfigurationPolicy.

When the Policy is bound to a Placement using a PlacementBinding, the Policy status will report on each cluster that matches the bound Placement:

status:
  compliant: Compliant
  placement:
    - placement: placement-hub-cluster
      placementBinding: binding-policy-namespace
  status:
    - clustername: local-cluster
      clusternamespace: local-cluster
      compliant: Compliant

To fully explore the Policy API, run the following command:

kubectl get crd policies.policy.open-cluster-management.io -o yaml

To fully explore the ConfigurationPolicy API, run the following command:

kubectl get crd configurationpolicies.policy.open-cluster-management.io -o yaml

PlacementBinding

A PlacementBinding binds a Placement to a Policy or PolicySet.

Below is an example of a PlacementBinding that binds the policy-namespace Policy to the placement-hub-cluster Placement.

apiVersion: policy.open-cluster-management.io/v1
kind: PlacementBinding
metadata:
  name: binding-policy-namespace
  namespace: policies
placementRef:
  apiGroup: cluster.open-cluster-management.io
  kind: Placement
  name: placement-hub-cluster
subjects:
  - apiGroup: policy.open-cluster-management.io
    kind: Policy
    name: policy-namespace

Once the Policy is bound, it will be distributed to and acted upon by the managed clusters that match the Placement.

PolicySet

A PolicySet is a grouping mechanism of Policy objects. Compliance of all grouped Policy objects is summarized in the PolicySet. A PolicySet is a deployable unit and its distribution is controlled by a Placement when bound through a PlacementBinding.

This enables a workflow where subject matter experts write Policy objects and then an IT administrator creates a PolicySet that groups the previously written Policy objects and binds the PolicySet to a Placement that deploys the PolicySet.

An example of a PolicySet is shown below.

apiVersion: policy.open-cluster-management.io/v1beta1
kind: PolicySet
metadata:
  name: ocm-hardening
  namespace: policies
spec:
  description: Apply standard best practices for hardening your Open Cluster Management installation.
  policies:
    - policy-check-backups
    - policy-managedclusteraddon-available
    - policy-subscriptions

Managed cluster policy controllers

The Policy on the hub delivers the policies defined in spec.policy-templates to the managed clusters via the policy framework controllers. Once on the managed cluster, these Policy Templates are acted upon by the associated controller on the managed cluster. The policy framework supports delivering the Policy Template kinds listed here:

Configuration policy
The ConfigurationPolicy is provided by OCM and defines Kubernetes manifests to compare with objects that currently exist on the cluster. The action that the ConfigurationPolicy will take is determined by its complianceType. Compliance types include musthave, mustnothave, and mustonlyhave. musthave means the object should have the listed keys and values as a subset of the larger object. mustnothave means an object matching the listed keys and values should not exist. mustonlyhave ensures objects only exist with the keys and values exactly as defined. See the page on Configuration Policy for more information.
Open Policy Agent Gatekeeper
Gatekeeper is a validating webhook with auditing capabilities that can enforce custom resource definition-based policies that are run with the Open Policy Agent (OPA). Gatekeeper ConstraintTemplates and constraints can be provided in an OCM Policy to sync to managed clusters that have Gatekeeper installed on them. See the page on Gatekeeper integration for more information.

Templating in configuration policies

Configuration policies support the inclusion of Golang text templates in the object definitions. These templates are resolved at runtime either on the hub cluster or the target managed cluster using configurations related to that cluster. This gives you the ability to define configuration policies with dynamic content and to inform or enforce Kubernetes resources that are customized to the target cluster.

The template syntax must follow the Golang template language specification, and the resource definition generated from the resolved template must be a valid YAML. (See the Golang documentation about package templates for more information.) Any errors in template validation appear as policy violations. When you use a custom template function, the values are replaced at runtime.

Template functions, such as resource-specific and generic lookup template functions, are available for referencing Kubernetes resources on the hub cluster (using the {{hub ... hub}} delimiters), or managed cluster (using the {{ ... }} delimiters). See the Hub cluster templates section for more details. The resource-specific functions are used for convenience and makes content of the resources more accessible. If you use the generic function, lookup, which is more advanced, it is best to be familiar with the YAML structure of the resource that is being looked up. In addition to these functions, utility functions like base64encode, base64decode, indent, autoindent, toInt, and toBool are also available.

To conform templates with YAML syntax, templates must be set in the policy resource as strings using quotes or a block character (| or >). This causes the resolved template value to also be a string. To override this, consider using toInt or toBool as the final function in the template to initiate further processing that forces the value to be interpreted as an integer or boolean respectively.

To bypass template processing you can either:

Override a single template by wrapping the template in additional braces. For example, the template {{ template content }} would become {{ '{{ template content }}' }}.
Override all templates in a ConfigurationPolicy by adding the policy.open-cluster-management.io/disable-templates: "true" annotation in the ConfigurationPolicy section of your Policy. Template processing will be bypassed for that ConfigurationPolicy.

Hub cluster templating in configuration policies

Hub cluster templates are used to define configuration policies that are dynamically customized to the target cluster. This reduces the need to create separate policies for each target cluster or hardcode configuration values in the policy definitions.

Hub cluster templates are based on Golang text template specifications, and the {{hub … hub}} delimiter indicates a hub cluster template in a configuration policy.

A configuration policy definition can contain both hub cluster and managed cluster templates. Hub cluster templates are processed first on the hub cluster, then the policy definition with resolved hub cluster templates is propagated to the target clusters. On the managed cluster, the Configuration Policy controller processes any managed cluster templates in the policy definition and then enforces or verifies the fully resolved object definition.

In OCM versions 0.9.x and older, policies are processed on the hub cluster only upon creation or after an update. Therefore, hub cluster templates are only resolved to the data in the referenced resources upon policy creation or update. Any changes to the referenced resources are not automatically synced to the policies.

A special annotation, policy.open-cluster-management.io/trigger-update can be used to indicate changes to the data referenced by the templates. Any change to the special annotation value initiates template processing, and the latest contents of the referenced resource are read and updated in the policy definition that is the propagator for processing on managed clusters. A typical way to use this annotation is to increment the value by one each time.

Templating value encryption

The encryption algorithm uses AES-CBC with 256-bit keys. Each encryption key is unique per managed cluster and is automatically rotated every 30 days. This ensures that your decrypted value is never stored in the policy on the managed cluster.

To force an immediate encryption key rotation, delete the policy.open-cluster-management.io/last-rotated annotation on the policy-encryption-key Secret in the managed cluster namespace on the hub cluster. Policies are then reprocessed to use the new encryption key.

Templating functions

Function	Description	Sample
`fromSecret`	Returns the value of the given data key in the secret.	`PASSWORD: '{{ fromSecret "default" "localsecret" "PASSWORD" }}'`
`fromConfigmap`	Returns the value of the given data key in the ConfigMap.	`log-file: '{{ fromConfigMap "default" "logs-config" "log-file" }}'`
`fromClusterClaim`	Returns the value of `spec.value` in the `ClusterClaim` resource.	`platform: '{{ fromClusterClaim "platform.open-cluster-management.io" }}'`
`lookup`	Returns the Kubernetes resource as a JSON compatible map. Note that if the requested resource does not exist, an empty map is returned.	`metrics-url: \|` `http://{{ (lookup "v1" "Service" "default" "metrics").spec.clusterIP }}:8080`
`base64enc`	Returns a `base64` encoded value of the input string.	`USER_NAME: '{{ fromConfigMap "default" "myconfigmap" "admin-user" \| base64enc }}'`
`base64dec`	Returns a `base64` decoded value of the input string.	`app-name: \|` `"{{ ( lookup "v1" "Secret" "testns" "mytestsecret") .data.appname ) \| base64dec }}"`
`indent`	Returns the input string indented by the given number of spaces.	`Ca-cert: \|` `{{ ( index ( lookup "v1" "Secret" "default" "mycert-tls" ).data "ca.pem" ) \| base64dec \| indent 4 }}`
`autoindent`	Acts like the `indent` function but automatically determines the number of leading spaces needed based on the number of spaces before the template.	`Ca-cert: \|` `{{ ( index ( lookup "v1" "Secret" "default" "mycert-tls" ).data "ca.pem" ) \| base64dec \| autoindent }}`
`toInt`	Returns the integer value of the string and ensures that the value is interpreted as an integer in the YAML.	`vlanid: \|` `{{ (fromConfigMap "site-config" "site1" "vlan") \| toInt }}`
`toBool`	Returns the boolean value of the input string and ensures that the value is interpreted as a boolean in the YAML.	`enabled: \|` `{{ (fromConfigMap "site-config" "site1" "enabled") \| toBool }}`
`protect`	Encrypts the input string. It is decrypted when the policy is evaluated. On the replicated policy in the managed cluster namespace, the resulting value resembles the following: `$ocm_encrypted:<encrypted-value>`	`enabled: \|` `{{hub "(lookup "route.openshift.io/v1" "Route" "openshift-authentication" "oauth-openshift").spec.host \| protect hub}}`

Additionally, OCM supports the following template functions that are included from the sprig open source project:

cat
contains
default
empty
fromJson
hasPrefix
hasSuffix
join
list
lower
mustFromJson
quote
replace
semver
semverCompare
split
splitn
ternary
trim
until
untilStep
upper

See the Sprig documentation for more details.

1.10 - Multicluster Control Plane

What is `Multicluster Control Plane`

The multicluster control plane is a lightweight Open Cluster Manager (OCM) control plane that is easy to install and has a small footprint. It can be running anywhere with or without a Kubernetes environment to serve the OCM control plane capabilities.

Why use `Multicluster Control Plane`

Some Kubernetes environments do not have CSR (e.g., EKS) so that the standard OCM control plane cannot be installed. The multicluster control plane can be able to install in these environments and expose the OCM control plane API via loadbalancer.
Some users may want to run multiple OCM control planes to isolate the data. The typical case is that the user wants to run one OCM control plane for production and another OCM control plane for development. The multicluster control plane is able to be installed in different namespaces in a single cluster. Each multicluster control plane is running independently and serving the OCM control plane capabilities.
Some users may want to run the OCM control plane without a Kubernetes environment. The multicluster control plane can run in a standalone mode, for example, running in a VM. Expose the control plane API to the outside so the managed clusters can register to it.

How to use `Multicluster Control Plane`

Start the standalone multicluster control plane

You need build multicluster-controlplane in your local host. Follow the below steps to build the binary and start the multicluster control plane.

git clone https://github.com/open-cluster-management-io/multicluster-controlplane.git
cd multicluster-controlplane
make run

Once the control plane is running, you can access the control plane by using kubectl --kubeconfig=./_output/controlplane/.ocm/cert/kube-aggregator.kubeconfig.

You can customize the control plane configurations by creating a config file and using the environment variable CONFIG_DIR to specify your config file directory. Please check the repository documentation for details.

Install via clusteradm

Install clusteradm CLI tool

It’s recommended to run the following command to download and install the latest release of the clusteradm command-line tool:

curl -L https://raw.githubusercontent.com/open-cluster-management-io/clusteradm/main/install.sh | bash

Install multicluster control plane

You can use clusteradm init to deploy the multicluster control plane in your Kubernetes environment.

Set the environment variable KUBECONFIG to your cluster kubeconfig path. For instance, create a new KinD cluster and deploy multicluster control plane in it.

export KUBECONFIG=/tmp/kind-controlplane.kubeconfig
kind create cluster --name multicluster-controlplane
export mc_cp_node_ip=$(kubectl get nodes -o=jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}')

Run following command to deploy a control plane

clusteradm init --singleton=true --set route.enabled=false --set nodeport.enabled=true --set nodeport.port=30443 --set apiserver.externalHostname=$mc_cp_node_ip --set apiserver.externalPort=30443 --singleton-name multicluster-controlplane

Refer to the repository documentation for how to customize the control plane configurations.

Get the control plane kubeconfig by running the following command:

kubectl -n multicluster-controlplane get secrets multicluster-controlplane-kubeconfig -ojsonpath='{.data.kubeconfig}' | base64 -d > /tmp/multicluster-controlplane.kubeconfig

Join a cluster to the multicluster control plane

You can use clusteradm to join a cluster. For instance, take the KinD cluster as an example, run the following command to join the cluster to the control plane:

kind create cluster --name cluster1 --kubeconfig /tmp/kind-cluster1.kubeconfig
clusteradm --kubeconfig=/tmp/multicluster-controlplane.kubeconfig get token --use-bootstrap-token
clusteradm --singleton=true --kubeconfig /tmp/kind-cluster1.kubeconfig join --hub-token <controlplane token> --hub-apiserver https://$mc_cp_node_ip:30443/ --cluster-name cluster1
clusteradm --kubeconfig=/tmp/multicluster-controlplane.kubeconfig accept --clusters cluster1

Verify the cluster join

Run this command to verify the cluster join:

kubectl --kubeconfig=/tmp/multicluster-controlplane.kubeconfig get managedcluster
NAME       HUB ACCEPTED   MANAGED CLUSTER URLS                  JOINED   AVAILABLE   AGE
cluster1   true           https://cluster1-control-plane:6443   True     True        5m25s

You should see the managedcluster joins to the multicluster control plane. Congratulations!

2 - Getting Started

2.1 - Quick Start

Prerequisites

Ensure kubectl and kustomize are installed.
Ensure kind(greater than v0.9.0+, or the latest version is preferred) is installed.

Install clusteradm CLI tool

Run the following command to download and install the latest clusteradm command-line tool:

curl -L https://raw.githubusercontent.com/open-cluster-management-io/clusteradm/main/install.sh | bash

Setup hub and managed cluster

Run the following command to quickly setup a hub cluster and 2 managed clusters by kind.

curl -L https://raw.githubusercontent.com/open-cluster-management-io/OCM/main/solutions/setup-dev-environment/local-up.sh | bash

If you want to setup OCM in a production environment or on a different kubernetes distribution, please refer to the Start the control plane and Register a cluster guides.

What is next

Now you have the OCM control plane with 2 managed clusters connected! Let’s start your OCM journey.

Deploy kubernetes resources onto a managed cluster
Visit kubernetes apiserver of managedcluster from cluster-proxy
Visit integration to check if any certain OCM addon will meet your use cases.
- Deploy Policies onto a managed cluster
Use the OCM VScode Extension to easily generate OCM related Kubernetes resources and track your cluster

2.2 - Installation

Install the core control plane that includes cluster registration and manifests distribution on the hub cluster.

Install the klusterlet agent on the managed cluster so that it can be registered and managed by the hub cluster.

2.2.1 - Start the control plane

Prerequisite

The hub cluster should be v1.19+. (To run on hub cluster version between [v1.16, v1.18], please manually enable feature gate “V1beta1CSRAPICompatibility”).
Currently the bootstrap process relies on client authentication via CSR. Therefore, if your Kubernetes distributions (like EKS) don’t support it, you can choose the multicluster-controlplane as the hub controlplane.
Ensure kubectl and kustomize are installed.

Network requirements

Configure your network settings for the hub cluster to allow the following connections.

Direction	Endpoint	Protocol	Purpose	Used by
Inbound	https://{hub-api-server-url}:{port}	TCP	Kubernetes API server of the hub cluster	OCM agents, including the add-on agents, running on the managed clusters

Install clusteradm CLI tool

It’s recommended to run the following command to download and install the latest release of the clusteradm command-line tool:

curl -L https://raw.githubusercontent.com/open-cluster-management-io/clusteradm/main/install.sh | bash

You can also install the latest development version (main branch) by running:

# Installing clusteradm to $GOPATH/bin/
GO111MODULE=off go get -u open-cluster-management.io/clusteradm/...

Bootstrap a cluster manager

Before actually installing the OCM components into your clusters, export the following environment variables in your terminal before running our command-line tool clusteradm so that it can correctly discriminate the hub cluster.

# The context name of the clusters in your kubeconfig
export CTX_HUB_CLUSTER=<your hub cluster context>

Call clusteradm init:

 # By default, it installs the latest release of the OCM components.
 # Use e.g. "--bundle-version=latest" to install latest development builds.
 # NOTE: For hub cluster version between v1.16 to v1.19 use the parameter: --use-bootstrap-token
 clusteradm init --wait --context ${CTX_HUB_CLUSTER}

The clusteradm init command installs the registration-operator on the hub cluster, which is responsible for consistently installing and upgrading a few core components for the OCM environment.

After the init command completes, a generated command is output on the console to register your managed clusters. An example of the generated command is shown below.

clusteradm join \
    --hub-token <your token data> \
    --hub-apiserver <your hub kube-apiserver endpoint> \
    --wait \
    --cluster-name <cluster_name>

It’s recommended to save the command somewhere secure for future use. If it’s lost, you can use clusteradm get token to get the generated command again.

Check out the running instances of the control plane

kubectl -n open-cluster-management get pod --context ${CTX_HUB_CLUSTER}
NAME                               READY   STATUS    RESTARTS   AGE
cluster-manager-695d945d4d-5dn8k   1/1     Running   0          19d

Additionally, to check out the instances of OCM’s hub control plane, run the following command:

kubectl -n open-cluster-management-hub get pod --context ${CTX_HUB_CLUSTER}
NAME                               READY   STATUS    RESTARTS   AGE
cluster-manager-placement-controller-857f8f7654-x7sfz      1/1     Running   0          19d
cluster-manager-registration-controller-85b6bd784f-jbg8s   1/1     Running   0          19d
cluster-manager-registration-webhook-59c9b89499-n7m2x      1/1     Running   0          19d
cluster-manager-work-webhook-59cf7dc855-shq5p              1/1     Running   0          19d
...

The overall installation information is visible on the clustermanager custom resource:

kubectl get clustermanager cluster-manager -o yaml --context ${CTX_HUB_CLUSTER}

Uninstall the OCM from the control plane

Before uninstalling the OCM components from your clusters, please detach the managed cluster from the control plane.

clusteradm clean --context ${CTX_HUB_CLUSTER}

Check the instances of OCM’s hub control plane are removed.

kubectl -n open-cluster-management-hub get pod --context ${CTX_HUB_CLUSTER}
No resources found in open-cluster-management-hub namespace.

kubectl -n open-cluster-management get pod --context ${CTX_HUB_CLUSTER}
No resources found in open-cluster-management namespace.

Check the clustermanager resource is removed from the control plane.

kubectl get clustermanager --context ${CTX_HUB_CLUSTER}
error: the server doesn't have a resource type "clustermanager"

2.2.2 - Register a cluster

After the cluster manager is installed on the hub cluster, you need to install the klusterlet agent on another cluster so that it can be registered and managed by the hub cluster.

Prerequisite

The managed clusters should be v1.11+.
Ensure kubectl and kustomize are installed.

Network requirements

Configure your network settings for the managed clusters to allow the following connections.

Direction	Endpoint	Protocol	Purpose	Used by
Outbound	https://{hub-api-server-url}:{port}	TCP	Kubernetes API server of the hub cluster	OCM agents, including the add-on agents, running on the managed clusters

To use a proxy, please make sure the proxy server is well configured to allow the above connections and the proxy server is reachable for the managed clusters. See Register a cluster to hub through proxy server for more details.

Install clusteradm CLI tool

It’s recommended to run the following command to download and install the latest release of the clusteradm command-line tool:

curl -L https://raw.githubusercontent.com/open-cluster-management-io/clusteradm/main/install.sh | bash

You can also install the latest development version (main branch) by running:

# Installing clusteradm to $GOPATH/bin/
GO111MODULE=off go get -u open-cluster-management.io/clusteradm/...

Bootstrap a klusterlet

# The context name of the clusters in your kubeconfig
export CTX_HUB_CLUSTER=<your hub cluster context>
export CTX_MANAGED_CLUSTER=<your managed cluster context>

Copy the previously generated command – clusteradm join, and add the arguments respectively based on the different distribution.

NOTE: If there is no configmap kube-root-ca.crt in kube-public namespace of the hub cluster, the flag –ca-file should be set to provide a valid hub ca file to help set up the external client.

# NOTE: For KinD clusters use the parameter: --force-internal-endpoint-lookup
clusteradm join \
    --hub-token <your token data> \
    --hub-apiserver <your hub cluster endpoint> \
    --wait \
    --cluster-name "cluster1" \    # Or other arbitrary unique name
    --force-internal-endpoint-lookup \
    --context ${CTX_MANAGED_CLUSTER}

clusteradm join \
    --hub-token <your token data> \
    --hub-apiserver <your hub cluster endpoint> \
    --wait \
    --cluster-name "cluster1" \   # Or other arbitrary unique name
    --context ${CTX_MANAGED_CLUSTER}

Bootstrap a klusterlet in hosted mode(Optional)

Using the above command, the klusterlet components(registration-agent and work-agent) will be deployed on the managed cluster, it is mandatory to expose the hub cluster to the managed cluster. We provide an option for running the klusterlet components outside the managed cluster, for example, on the hub cluster(hosted mode).

The hosted mode deploying is till in experimental stage, consider to use it only when:

want to reduce the footprints of the managed cluster.
do not want to expose the hub cluster to the managed cluster directly

In hosted mode, the cluster where the klusterlet is running is called the hosting cluster. Running the following command to the hosting cluster to register the managed cluster to the hub.

<div class="highlight"><pre tabindex="0" style="background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span><span style="color:#8f5902;font-style:italic"># NOTE for KinD clusters:</span>
</span></span><span style="display:flex;"><span><span style="color:#8f5902;font-style:italic">#  1. hub is KinD, use the parameter: --force-internal-endpoint-lookup</span>
</span></span><span style="display:flex;"><span><span style="color:#8f5902;font-style:italic">#  2. managed is Kind, --managed-cluster-kubeconfig should be internal: `kind get kubeconfig --name managed --internal`</span>
</span></span><span style="display:flex;"><span>clusteradm join <span style="color:#4e9a06">\
</span></span></span><span style="display:flex;"><span><span style="color:#4e9a06"></span>    --hub-token &lt;your token data&gt; <span style="color:#4e9a06">\
</span></span></span><span style="display:flex;"><span><span style="color:#4e9a06"></span>    --hub-apiserver &lt;your hub cluster endpoint&gt; <span style="color:#4e9a06">\
</span></span></span><span style="display:flex;"><span><span style="color:#4e9a06"></span>    --wait <span style="color:#4e9a06">\
</span></span></span><span style="display:flex;"><span><span style="color:#4e9a06"></span>    --cluster-name <span style="color:#4e9a06">&#34;cluster1&#34;</span> <span style="color:#4e9a06">\ </span>   <span style="color:#8f5902;font-style:italic"># Or other arbitrary unique name</span>
</span></span><span style="display:flex;"><span>    --mode hosted <span style="color:#4e9a06">\
</span></span></span><span style="display:flex;"><span><span style="color:#4e9a06"></span>    --managed-cluster-kubeconfig &lt;your managed cluster kubeconfig&gt; <span style="color:#4e9a06">\ </span>   <span style="color:#8f5902;font-style:italic"># Should be an internal kubeconfig</span>
</span></span><span style="display:flex;"><span>    --force-internal-endpoint-lookup <span style="color:#4e9a06">\
</span></span></span><span style="display:flex;"><span><span style="color:#4e9a06"></span>    --context &lt;your hosting cluster context&gt;
</span></span></code></pre></div>

<div class="highlight"><pre tabindex="0" style="background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>clusteradm join <span style="color:#4e9a06">\
</span></span></span><span style="display:flex;"><span><span style="color:#4e9a06"></span>    --hub-token &lt;your token data&gt; <span style="color:#4e9a06">\
</span></span></span><span style="display:flex;"><span><span style="color:#4e9a06"></span>    --hub-apiserver &lt;your hub cluster endpoint&gt; <span style="color:#4e9a06">\
</span></span></span><span style="display:flex;"><span><span style="color:#4e9a06"></span>    --wait <span style="color:#4e9a06">\
</span></span></span><span style="display:flex;"><span><span style="color:#4e9a06"></span>    --cluster-name <span style="color:#4e9a06">&#34;cluster1&#34;</span> <span style="color:#4e9a06">\ </span>   <span style="color:#8f5902;font-style:italic"># Or other arbitrary unique name</span>
</span></span><span style="display:flex;"><span>    --mode hosted <span style="color:#4e9a06">\
</span></span></span><span style="display:flex;"><span><span style="color:#4e9a06"></span>    --managed-cluster-kubeconfig &lt;your managed cluster kubeconfig&gt; <span style="color:#4e9a06">\
</span></span></span><span style="display:flex;"><span><span style="color:#4e9a06"></span>    --context &lt;your hosting cluster context&gt;
</span></span></code></pre></div>

Bootstrap a klusterlet in singleton mode

To reduce the footprint of agent in the managed cluster, singleton mode is introduced since v0.12.0. In the singleton mode, the work and registration agent will be run as a single pod in the managed cluster.

Note: to run klusterlet in singleton mode, you must have a clusteradm version equal or higher than v0.12.0

<div class="highlight"><pre tabindex="0" style="background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span><span style="color:#8f5902;font-style:italic"># NOTE: For KinD clusters use the parameter: --force-internal-endpoint-lookup</span>
</span></span><span style="display:flex;"><span>clusteradm join <span style="color:#4e9a06">\
</span></span></span><span style="display:flex;"><span><span style="color:#4e9a06"></span>    --hub-token &lt;your token data&gt; <span style="color:#4e9a06">\
</span></span></span><span style="display:flex;"><span><span style="color:#4e9a06"></span>    --hub-apiserver &lt;your hub cluster endpoint&gt; <span style="color:#4e9a06">\
</span></span></span><span style="display:flex;"><span><span style="color:#4e9a06"></span>    --wait <span style="color:#4e9a06">\
</span></span></span><span style="display:flex;"><span><span style="color:#4e9a06"></span>    --cluster-name <span style="color:#4e9a06">&#34;cluster1&#34;</span> <span style="color:#4e9a06">\ </span>   <span style="color:#8f5902;font-style:italic"># Or other arbitrary unique name</span>
</span></span><span style="display:flex;"><span>    --singleton <span style="color:#4e9a06">\
</span></span></span><span style="display:flex;"><span><span style="color:#4e9a06"></span>    --force-internal-endpoint-lookup <span style="color:#4e9a06">\
</span></span></span><span style="display:flex;"><span><span style="color:#4e9a06"></span>    --context <span style="color:#4e9a06">${</span><span style="color:#000">CTX_MANAGED_CLUSTER</span><span style="color:#4e9a06">}</span>
</span></span></code></pre></div>

<div class="highlight"><pre tabindex="0" style="background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>clusteradm join <span style="color:#4e9a06">\
</span></span></span><span style="display:flex;"><span><span style="color:#4e9a06"></span>    --hub-token &lt;your token data&gt; <span style="color:#4e9a06">\
</span></span></span><span style="display:flex;"><span><span style="color:#4e9a06"></span>    --hub-apiserver &lt;your hub cluster endpoint&gt; <span style="color:#4e9a06">\
</span></span></span><span style="display:flex;"><span><span style="color:#4e9a06"></span>    --wait <span style="color:#4e9a06">\
</span></span></span><span style="display:flex;"><span><span style="color:#4e9a06"></span>    --cluster-name <span style="color:#4e9a06">&#34;cluster1&#34;</span> <span style="color:#4e9a06">\ </span>  <span style="color:#8f5902;font-style:italic"># Or other arbitrary unique name</span>
</span></span><span style="display:flex;"><span>    --singleton <span style="color:#4e9a06">\
</span></span></span><span style="display:flex;"><span><span style="color:#4e9a06"></span>    --context <span style="color:#4e9a06">${</span><span style="color:#000">CTX_MANAGED_CLUSTER</span><span style="color:#4e9a06">}</span>
</span></span></code></pre></div>

Accept the join request and verify

After the OCM agent is running on your managed cluster, it will be sending a “handshake” to your hub cluster and waiting for an approval from the hub cluster admin. In this section, we will walk through accepting the registration requests from the perspective of an OCM’s hub admin.

Wait for the creation of the CSR object which will be created by your managed clusters’ OCM agents on the hub cluster:

kubectl get csr -w --context ${CTX_HUB_CLUSTER} | grep cluster1  # or the previously chosen cluster name

An example of a pending CSR request is shown below:

cluster1-tqcjj   33s   kubernetes.io/kube-apiserver-client   system:serviceaccount:open-cluster-management:cluster-bootstrap   Pending

Accept the join request using the clusteradm tool:
```
clusteradm accept --clusters cluster1 --context ${CTX_HUB_CLUSTER}
```
After running the accept command, the CSR from your managed cluster named “cluster1” will be approved. Additionally, it will instruct the OCM hub control plane to setup related objects (such as a namespace named “cluster1” in the hub cluster) and RBAC permissions automatically.

Verify the installation of the OCM agents on your managed cluster by running:

kubectl -n open-cluster-management-agent get pod --context ${CTX_MANAGED_CLUSTER}
NAME                                             READY   STATUS    RESTARTS   AGE
klusterlet-registration-agent-598fd79988-jxx7n   1/1     Running   0          19d
klusterlet-work-agent-7d47f4b5c5-dnkqw           1/1     Running   0          19d

Verify that the cluster1 ManagedCluster object was created successfully by running:

kubectl get managedcluster --context ${CTX_HUB_CLUSTER}

Then you should get a result that resembles the following:

NAME       HUB ACCEPTED   MANAGED CLUSTER URLS      JOINED   AVAILABLE   AGE
cluster1   true           <your endpoint>           True     True        5m23s

If the managed cluster status is not true, refer to Troubleshooting to debug on your cluster.

Apply a Manifestwork

After the managed cluster is registered, test that you can deploy a pod to the managed cluster from the hub cluster. Create a manifest-work.yaml as shown in this example:

apiVersion: work.open-cluster-management.io/v1
kind: ManifestWork
metadata:
  name: mw-01
  namespace: ${MANAGED_CLUSTER_NAME}
spec:
  workload:
    manifests:
      - apiVersion: v1
        kind: Pod
        metadata:
          name: hello
          namespace: default
        spec:
          containers:
            - name: hello
              image: busybox
              command: ["sh", "-c", 'echo "Hello, Kubernetes!" && sleep 3600']
          restartPolicy: OnFailure

Apply the yaml file to the hub cluster.

kubectl apply -f manifest-work.yaml --context ${CTX_HUB_CLUSTER}

Verify that the manifestwork resource was applied to the hub.

kubectl -n ${MANAGED_CLUSTER_NAME} get manifestwork/mw-01 --context ${CTX_HUB_CLUSTER} -o yaml

Check on the managed cluster and see the hello Pod has been deployed from the hub cluster.

$ kubectl -n default get pod --context ${CTX_MANAGED_CLUSTER}
NAME    READY   STATUS    RESTARTS   AGE
hello   1/1     Running   0          108s

Troubleshooting

If the managed cluster status is not true.

For example, the result below is shown when checking managedcluster.

$ kubectl get managedcluster --context ${CTX_HUB_CLUSTER}
NAME                   HUB ACCEPTED   MANAGED CLUSTER URLS   JOINED   AVAILABLE   AGE
${MANAGED_CLUSTER_NAME} true           https://localhost               Unknown     46m

There are many reasons for this problem. You can use the commands below to get more debug info. If the provided info doesn’t help, please log an issue to us.

On the hub cluster, check the managedcluster status.

kubectl get managedcluster ${MANAGED_CLUSTER_NAME} --context ${CTX_HUB_CLUSTER} -o yaml

On the hub cluster, check the lease status.

kubectl get lease -n ${MANAGED_CLUSTER_NAME} --context ${CTX_HUB_CLUSTER}

On the managed cluster, check the klusterlet status.

kubectl get klusterlet -o yaml --context ${CTX_MANAGED_CLUSTER}

Detach the cluster from hub

Remove the resources generated when registering with the hub cluster.

clusteradm unjoin --cluster-name "cluster1" --context ${CTX_MANAGED_CLUSTER}

Check the installation of the OCM agent is removed from the managed cluster.

kubectl -n open-cluster-management-agent get pod --context ${CTX_MANAGED_CLUSTER}
No resources found in open-cluster-management-agent namespace.

Check the klusterlet is removed from the managed cluster.

kubectl get klusterlet --context ${CTX_MANAGED_CLUSTER}
error: the server doesn't have a resource type "klusterlet

2.3 - Add-ons and Integrations

Enhance the open-cluster-management core control plane with optional add-ons and integrations.

2.3.1 - Application lifecycle management

After the cluster manager is installed, you could install the application management components to the hub cluster.

Architecture

For more details, visit the multicloud-operators-subscription GitHub page.

Prerequisite

You must meet the following prerequisites to install the application lifecycle management add-on:

Ensure kubectl and kustomize are installed.
Ensure the open-cluster-management cluster manager is installed. See Start the control plane for more information.
Ensure the open-cluster-management klusterlet is installed. See Register a cluster for more information.

Install via Clusteradm CLI tool

Ensure clusteradm CLI tool is installed. Download and extract the clusteradm binary. For more details see the clusteradm GitHub page.

$ clusteradm
Usage:
  clusteradm [command]
...

Deploy the subscription operators to the hub cluster.

$ kubectl config use-context ${CTX_HUB_CLUSTER}
$ clusteradm install hub-addon --names application-manager
Installing built-in application-manager add-on to the Hub cluster...
$ kubectl -n open-cluster-management get deploy multicluster-operators-subscription --context ${CTX_HUB_CLUSTER}
NAME                                READY   UP-TO-DATE   AVAILABLE   AGE
multicluster-operators-subscription   1/1     1            1           25s

Create the open-cluster-management-agent-addon namespace on the managed cluster.

$ kubectl create ns open-cluster-management-agent-addon --context ${CTX_MANAGED_CLUSTER}
namespace/open-cluster-management-agent-addon created

Deploy the subscription add-on in corresponding managed cluster namespace on the hub cluster.

$ kubectl config use-context ${CTX_HUB_CLUSTER}
$ clusteradm addon enable --names application-manager --clusters ${MANAGED_CLUSTER_NAME}
Deploying application-manager add-on to managed cluster: <managed_cluster_name>.
$ kubectl -n ${MANAGED_CLUSTER_NAME} get managedclusteraddon # kubectl -n cluster1 get managedclusteraddon
NAME                  AVAILABLE   DEGRADED   PROGRESSING
application-manager   True

Check the the subscription add-on deployment on the managed cluster.

$ kubectl -n open-cluster-management-agent-addon get deploy --context ${CTX_MANAGED_CLUSTER}
NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
application-manager   1/1     1            1           103s

Install from source

Clone the multicloud-operators-subscription repository.

git clone https://github.com/open-cluster-management-io/multicloud-operators-subscription
cd multicloud-operators-subscription

Deploy the subscription operators to the hub cluster.

$ kubectl config use-context ${CTX_HUB_CLUSTER}
$ make deploy-hub
$ kubectl -n open-cluster-management get deploy multicluster-operators-subscription --context ${CTX_HUB_CLUSTER}
NAME                                READY   UP-TO-DATE   AVAILABLE   AGE
multicluster-operators-subscription   1/1     1            1           25s

Create the open-cluster-management-agent-addon namespace on the managed cluster and it’s optional if clusteradm is used which create the ns during join action.

$ kubectl create ns open-cluster-management-agent-addon --context ${CTX_MANAGED_CLUSTER}
namespace/open-cluster-management-agent-addon created

Deploy the subscription add-on in corresponding managed cluster namespace on the hub cluster.

$ kubectl config use-context ${CTX_HUB_CLUSTER}
$ make deploy-addon
$ kubectl -n ${MANAGED_CLUSTER_NAME} get managedclusteraddon # kubectl -n cluster1 get managedclusteraddon
NAME                  AVAILABLE   DEGRADED   PROGRESSING
application-manager   True

Check the the subscription add-on deployment on the managed cluster.

$ kubectl -n open-cluster-management-agent-addon get deploy --context ${CTX_MANAGED_CLUSTER}
NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
application-manager   1/1     1            1           103s

What is next

After a successful deployment, test the subscription operator with a helm subscription. Run the following command where the examples/helmrepo-hub-channel locates at here:

kubectl apply -f examples/helmrepo-hub-channel --context ${CTX_HUB_CLUSTER}

After a while, you should see the subscription is propagated to the managed cluster and the Helm app is installed. By default, when a subscribed applications is deployed to the target clusters, the applications are installed in the coresponding subscription namespace. To confirm, run the following command:

$ kubectl get subscriptions.apps --context ${CTX_MANAGED_CLUSTER}
NAME        STATUS       AGE    LOCAL PLACEMENT   TIME WINDOW
nginx-sub   Subscribed   107m   true
$ kubectl get pod --context ${CTX_MANAGED_CLUSTER}
NAME                                                   READY   STATUS      RESTARTS   AGE
nginx-ingress-47f79-controller-6f495bb5f9-lpv7z        1/1     Running     0          108m
nginx-ingress-47f79-default-backend-7559599b64-rhwgm   1/1     Running     0          108m

Try this out

Let VScode Extension help you out!

Create a Bootstrap Project specifically tailored to your channel type, with all the Custom Resource (CR) templates you will need already auto-generated to get you started!

2.3.2 - Cluster proxy

Cluster proxy is an OCM addon providing L4 network connectivity from hub cluster to the managed clusters without any additional requirement to the managed cluster’s network infrastructure by leveraging the Kubernetes official SIG sub-project apiserver-network-proxy.

Background

The original architecture of OCM allows a cluster from anywhere to be registered and managed by OCM’s control plane (i.e. the hub cluster) as long as a klusterlet agent can reach hub cluster’s endpoint. So the minimal requirement for the managed cluster’s network infrastructure in OCM is “klusterlet -> hub” connectivity. However, there are still some cases where the components in the hub cluster hope to proactively dail/request the services in the managed clusters which will need the “hub -> klusterlet” connectivity on the other hand. In addition to that, the cases can be even more complex when each of the managed clusters are not in the same network.

Cluster proxy is aiming at seamlessly delivering the outbound L4 requests to the services in the managed cluster’s network without any assumptions upon the infrastructure as long as the clusters are successfully registered. Basically the connectivity provided by cluster proxy is working over the secured reserve proxy tunnels established by the apiserver-network-proxy.

About apiserver-network-proxy

Apiserver-network-proxy is the underlying technique of a Kubernetes' feature called konnectivity egress-selector which is majorly for setting up a TCP-level proxy for kube-apiserver to get access to the node/cluster network. Here are a few terms we need to clarify before we elaborate on how the cluster proxy resolve multi-cluster control plane network connectivity for us:

Proxy Tunnel: A Grpc long connection that multiplexes and transmits TCP-level traffic from the proxy servers to the proxy agents. Note that there will be only one tunnel instance between each pair of server and agent.
Proxy Server: An mTLS Grpc server opened for establishing tunnels which is the traffic ingress of proxy tunnel.
Proxy Agent: A mTLS Grpc agent that maintains the tunnel between the server and is also the egress of the proxy tunnel.
Konnectivity Client: The SDK library for talking through the tunnel. Applicable to any Golang client of which the Dialer is overridable. Note that for non-golang clients, the proxy server also supports HTTP-Connect based proxying as alternative.

Architecture

Cluster proxy runs inside OCM’s hub cluster as an addon manager which is developed based on the Addon-Framework. The addon manager of cluster proxy will be responsible for:

Managing the installation of proxy servers in the hub cluster.
Managing the installation of proxy agents in the managed cluster.
Collecting healthiness and the other stats consistently in the hub cluster.

The following picture shows the overall architecture of cluster proxy:

Note that the green lines in the picture above is the active proxy tunnels between proxy servers and agents, and HA setup is natively supported by apiserver-network-proxy both for the servers and the agents. The orange dash line started by the konnectivity client is the path of how the traffic flows from the hub cluster to arbitrary managed clusters. Meanwhile the core components including registration and work will help us manage the lifecycle of all the components distributed in the multiple managed clusters, so the hub admin won’t need to directly operate the managed clusters to install or configure the proxy agents no more.

Prerequisite

You must meet the following prerequisites to install the cluster-proxy:

Ensure your open-cluster-management release is greater than v0.5.0.
Ensure kubectl is installed.
Ensure helm is installed.

Installation

To install the cluster proxy addon to the OCM control plane, run:

$ helm repo add ocm https://open-cluster-management.io/helm-charts
$ helm repo update
$ helm search repo ocm
NAME                              	CHART VERSION	APP VERSION	DESCRIPTION
ocm/cluster-proxy                 	v0.1.1       	1.0.0      	A Helm chart for Cluster-Proxy
...

Then run the following helm command to install the cluster-proxy addon:

$ helm install -n open-cluster-management-addon --create-namespace \
    cluster-proxy ocm/cluster-proxy
$ kubectl -n open-cluster-management-addon get deploy
NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
cluster-proxy                          3/3     3            3           24h
cluster-proxy-addon-manager            1/1     1            1           24h
...

Then the addon manager of cluster-proxy will be created into the hub cluster in the form of a deployment named cluster-proxy-addon-manager. As is also shown above, the proxy servers will also be created as deployment resource called cluster-proxy.

By default, the addon manager will be automatically discovering the addition or removal the managed clusters and installs the proxy agents into them on the fly. To check out the healthiness status of the proxy agents, we can run:

$  kubectl get managedclusteraddon -A
NAMESPACE     NAME                     AVAILABLE   DEGRADED   PROGRESSING
<cluster#1>   cluster-proxy            True
<cluster#2>   cluster-proxy            True

The proxy agent distributed in the managed cluster will be periodically renewing the lease lock of the addon instance.

Usage

Command-line tools

Using the clusteradm to check the status of the cluster-proxy addon:

$ clusteradm proxy health
CLUSTER NAME    INSTALLED    AVAILABLE    PROBED HEALTH    LATENCY
<cluster#1>     True         True         True             67.595144ms
<cluster#2>     True         True         True             85.418368ms

Example code

An example client in the cluster proxy repo shows us how to dynamically talk to the kube-apiserver of a managed cluster from the hub cluster by simply prescribing the name of the target cluster. Here’s also a TL;DR code snippet:

// 1. instantiate a dialing tunnel instance.
// NOTE: recommended to be a singleton in your golang program.
tunnel, err := konnectivity.CreateSingleUseGrpcTunnel(
    context.TODO(),
    <your proxy server endpoint>,
    grpc.WithTransportCredentials(grpccredentials.NewTLS(<your proxy server TLS config>)),
)
if err != nil {
    panic(err)
}
...
// 2. Overriding the Dialer to tunnel. Dialer is a common abstraction
// in Golang SDK.
cfg.Dial = tunnel.DialContext

Another example will be cluster-gateway which is an aggregated apiserver optionally working over cluster-proxy for routing traffic to the managed clusters dynamically in HTTPs protocol.

Note that by default the client credential for konnectivity client will be persisted as secrets resources under the namespace where the addon-manager is running. With that being said, to mount the secret to the systems in the other namespaces, the users are expected to copy the secret on their own manually.

More insights

Troubleshooting

The installation of proxy servers and agents are prescribed by the custom resource called “managedproxyconfiguration”. We can check it out by the following commands:

$ kubectl get managedproxyconfiguration cluster-proxy -o yaml
apiVersion: proxy.open-cluster-management.io/v1alpha1
kind: ManagedProxyConfiguration
metadata: ...
spec:
  proxyAgent:
    image: <expected image of the proxy agents>
    replicas: <expected replicas of proxy agents>
  proxyServer:
    entrypoint:
      loadBalancerService:
        name: proxy-agent-entrypoint
      type: LoadBalancerService # Or "Hostname" to set a fixed address
                                # for establishing proxy tunnels.
    image: <expected image of the proxy servers>
    inClusterServiceName: proxy-entrypoint
    namespace: <target namespace to install proxy server>
    replicas: <expected replicas of proxy servers>
  authentication: # Customize authentication between proxy server/agent
status:
  conditions: ...

See the original design proposal for reference.

2.3.3 - Managed service account

Managed Service Account is an OCM addon enabling a hub cluster admin to manage service account across multiple clusters on ease. By controlling the creation and removal of the service account, the addon agent will project and rotate the corresponding token back to the hub cluster which is very useful for the Kube API client from the hub cluster to request against the managed clusters.

Background

Normally there are two major approaches for a Kube API client to authenticate and access a Kubernetes cluster:

Valid X.509 certificate-key pair
Service account bearer token

The service account token will be automatically persisted as a secret resource inside the hosting Kubernetes clusters upon creation, which is commonly used for the “in-cluster” client. However, in terms of OCM, the hub cluster is completely an external system to the managed clusters, so we will need a local agent in each managed cluster to reflect the tokens consistently to the hub cluster so that the Kube API client from hub cluster can “push” the requests directly against the managed cluster. By delegating the multi-cluster service account management to this addon, we can:

Project the service account token from the managed clusters to the hub cluster with custom API audience.
Rotate the service account tokens dynamically.
Homogenize the client identities so that we can easily write a static RBAC policy that applies to multiple managed clusters.

Prerequisite

You must meet the following prerequisites to install the managed service account:

Ensure your open-cluster-management release is greater than v0.5.0.
Ensure kubectl is installed.
Ensure helm is installed.

Installation

To install the managed service account addon to the OCM control plane, run:

$ helm repo add ocm https://open-cluster-management.io/helm-charts
$ helm repo update
$ helm search repo ocm
NAME                              	CHART VERSION	APP VERSION	DESCRIPTION
ocm/managed-serviceaccount          <...>           1.0.0       A Helm chart for Managed ServiceAccount Addon
...

Then run the following helm command to continue the installation:

$ helm install -n open-cluster-management-addon --create-namespace \
    managed-serviceaccount  ocm/managed-serviceaccount
$ kubectl -n open-cluster-management-addon get pod
NAME                                                    READY   STATUS    RESTARTS   AGE
managed-serviceaccount-addon-manager-5m9c95b7d8-xsb94   1/1     Running   1          4d4h
...

By default, the addon manager will be automatically discovering the addition or removal the managed clusters and installs the managed serviceaccount agents into them on the fly. To check out the healthiness status of the managed serviceaccount agents, we can run:

$ kubectl get managedclusteraddon -A
NAMESPACE         NAME                     AVAILABLE   DEGRADED   PROGRESSING
<cluster name>    managed-serviceaccount   True

Usage

To exercise the new ManagedServiceAccount API introduced by this addon, we can start by applying the following sample resource:

$ export CLUSTER_NAME=<cluster name>
$ kubectl create -f - <<EOF
apiVersion: authentication.open-cluster-management.io/v1alpha1
kind: ManagedServiceAccount
metadata:
  name: my-sample
  namespace: ${CLUSTER_NAME}
spec:
  rotation: {}
EOF

Then the addon agent in each of the managed cluster is responsible for executing and refreshing the status of the ManagedServiceAccount, e.g.:

$ kubectl describe ManagedServiceAccount -n cluster1
...
  status:
    conditions:
    - lastTransitionTime: "2021-12-09T09:08:15Z"
      message: ""
      reason: TokenReported
      status: "True"
      type: TokenReported
    - lastTransitionTime: "2021-12-09T09:08:15Z"
      message: ""
      reason: SecretCreated
      status: "True"
      type: SecretCreated
    expirationTimestamp: "2022-12-04T09:08:15Z"
    tokenSecretRef:
      lastRefreshTimestamp: "2021-12-09T09:08:15Z"
      name: my-sample

The service account will be created in the managed cluster (assume the name is cluster1):

$ kubectl get sa my-sample -n open-cluster-management-managed-serviceaccount --context kind-cluster1
NAME        SECRETS   AGE
my-sample   1         9m57s

The corresponding secret will also be created in the hub cluster, which is visible via:

$ kubectl -n <your cluster> get secret my-sample
NAME        TYPE     DATA   AGE
my-sample   Opaque   2      2m23s

Repo: https://github.com/open-cluster-management-io/managed-serviceaccount

See the design proposal at: https://github.com/open-cluster-management-io/enhancements/tree/main/enhancements/sig-architecture/19-projected-serviceaccount-token

2.3.4 - Policy controllers

The Policy API on the hub delivers the policies defined in spec.policy-templates to the managed clusters via the policy framework controllers. Once on the managed cluster, these Policy Templates are acted upon by the associated controller on the managed cluster. The policy framework supports delivering the Policy Template kinds listed.

Configuration policy

The ConfigurationPolicy is provided by OCM and defines Kubernetes manifests to compare with objects that currently exist on the cluster. The action that the ConfigurationPolicy will take is determined by its complianceType. Compliance types include musthave, mustnothave, and mustonlyhave. musthave means the object should have the listed keys and values as a subset of the larger object. mustnothave means an object matching the listed keys and values should not exist. mustonlyhave ensures objects only exist with the keys and values exactly as defined.

Open Policy Agent Gatekeeper

Gatekeeper is a validating webhook with auditing capabilities that can enforce custom resource definition-based policies that are run with the Open Policy Agent (OPA). Gatekeeper ConstraintTemplates and constraints can be provided in an OCM Policy to sync to managed clusters that have Gatekeeper installed on them.

2.3.4.1 - Configuration Policy

The ConfigurationPolicy defines Kubernetes manifests to compare with objects that currently exist on the cluster. The Configuration policy controller is provided by Open Cluster Management and runs on managed clusters.

Prerequisites

You must meet the following prerequisites to install the configuration policy controller:

Ensure kubectl and kustomize are installed.
Ensure Golang is installed, if you are planning to install from the source.
Ensure the open-cluster-management policy framework is installed. See Policy Framework for more information.

Installing the configuration policy controller

Deploy via Clusteradm CLI

Ensure clusteradm CLI is installed and is newer than v0.3.0. Download and extract the clusteradm binary. For more details see the clusteradm GitHub page.

Deploy the configuration policy controller to the managed clusters (this command is the same for a self-managed hub):

# Deploy the configuration policy controller
clusteradm addon enable addon --names config-policy-controller --clusters <cluster_name> --context ${CTX_HUB_CLUSTER}

Ensure the pod is running on the managed cluster with the following command:

$ kubectl get pods -n open-cluster-management-agent-addon
NAME                                               READY   STATUS    RESTARTS   AGE
config-policy-controller-7f8fb64d8c-pmfx4          1/1     Running   0          44s

Deploy from source

Deploy the config-policy-controller to the managed cluster with the following commands:

# The context name of the clusters in your kubeconfig
# If the clusters are created by KinD, then the context name will the follow the pattern "kind-<cluster name>".
export CTX_HUB_CLUSTER=<your hub cluster context>           # export CTX_HUB_CLUSTER=kind-hub
export CTX_MANAGED_CLUSTER=<your managed cluster context>   # export CTX_MANAGED_CLUSTER=kind-cluster1

# Configure kubectl to point to the managed cluster
kubectl config use-context ${CTX_MANAGED_CLUSTER}

# Create the namespace for the controller
export MANAGED_NAMESPACE="open-cluster-management-agent-addon"
kubectl create ns ${MANAGED_NAMESPACE}

# Apply the CRD
export COMPONENT="config-policy-controller"
export GIT_PATH="https://raw.githubusercontent.com/open-cluster-management-io/${COMPONENT}/v0.12.0/deploy"
kubectl apply -f ${GIT_PATH}/crds/policy.open-cluster-management.io_configurationpolicies.yaml
kubectl apply -f ${GIT_PATH}/crds/policy.open-cluster-management.io_operatorpolicies.yaml

# Set the managed cluster name
export MANAGED_CLUSTER_NAME=<your managed cluster name>  # export MANAGED_CLUSTER_NAME=cluster1

# Deploy the controller
kubectl apply -f ${GIT_PATH}/operator.yaml -n ${MANAGED_NAMESPACE}
kubectl set env deployment/${COMPONENT} -n ${MANAGED_NAMESPACE} --containers=${COMPONENT} WATCH_NAMESPACE=${MANAGED_CLUSTER_NAME}

See config-policy-controller for more information.

Ensure the pod is running on the managed cluster with the following command:

$ kubectl get pods -n ${MANAGED_NAMESPACE}
NAME                                               READY   STATUS    RESTARTS   AGE
config-policy-controller-7f8fb64d8c-pmfx4          1/1     Running   0          44s

Sample configuration policy

After a successful deployment, test the policy framework and configuration policy controller with a sample policy. You can use a policy that includes a Placement mapping or if you installed Application management’s PlacementRule support you can use either placement implementation. Perform the steps in the Placement API or the Placement Rule API section based on which placement API you desire to use.

For more information on how to use a ConfigurationPolicy, read the Policy API concept section.

Placement API

Run the following command to create a policy on the hub that uses Placement:

# Configure kubectl to point to the hub cluster
kubectl config use-context ${CTX_HUB_CLUSTER}

# Apply the example policy and placement
kubectl apply -n default -f https://raw.githubusercontent.com/open-cluster-management-io/policy-collection/main/community/CM-Configuration-Management/policy-pod-placement.yaml

Update the Placement to distribute the policy to the managed cluster with the following command (this clusterSelector will deploy the policy to all managed clusters):

kubectl patch -n default placement.cluster.open-cluster-management.io/placement-policy-pod --type=merge -p "{\"spec\":{\"predicates\":[{\"requiredClusterSelector\":{\"labelSelector\":{\"matchExpressions\":[]}}}]}}"

Make sure the default namespace has a ManagedClusterSetBinding for a ManagedClusterSet with at least one managed cluster resource in the ManagedClusterSet. See Bind ManagedClusterSet to a namespace for more information on this.

To confirm that the managed cluster is selected by the Placement, run the following command:

$ kubectl get -n default placementdecision.cluster.open-cluster-management.io/placement-policy-pod-decision-1 -o yaml
...
status:
  decisions:
  - clusterName: <managed cluster name>
    reason: ""
...

Placement Rule API

NOTE: Skip this section if you applied the Placement API policy manifests.

Run the following command to create a policy on the hub that uses PlacementRule:

# Configure kubectl to point to the hub cluster
kubectl config use-context ${CTX_HUB_CLUSTER}

# Apply the example policy and placement rule
kubectl apply -n default -f https://raw.githubusercontent.com/open-cluster-management-io/policy-collection/main/stable/CM-Configuration-Management/policy-pod.yaml

Update the PlacementRule to distribute the policy to the managed cluster with the following command (this clusterSelector will deploy the policy to all managed clusters):

$ kubectl patch -n default placementrule.apps.open-cluster-management.io/placement-policy-pod --type=merge -p "{\"spec\":{\"clusterSelector\":{\"matchExpressions\":[]}}}"
placementrule.apps.open-cluster-management.io/placement-policy-pod patched

To confirm that the managed cluster is selected by the PlacementRule, run the following command:

$ kubectl get -n default placementrule.apps.open-cluster-management.io/placement-policy-pod -o yaml
...
status:
  decisions:
  - clusterName: ${MANAGED_CLUSTER_NAME}
    clusterNamespace: ${MANAGED_CLUSTER_NAME}
...

Final steps to apply the policy

Perform the following steps to continue working with the policy to test the policy framework now that a placement method has been selected between Placement or PlacementRule.

Enforce the policy to make the configuration policy automatically correct any misconfigurations on the managed cluster:

$ kubectl patch -n default policy.policy.open-cluster-management.io/policy-pod --type=merge -p "{\"spec\":{\"remediationAction\": \"enforce\"}}"
policy.policy.open-cluster-management.io/policy-pod patched

After a few seconds, your policy is propagated to the managed cluster. To confirm, run the following command:

$ kubectl config use-context ${CTX_MANAGED_CLUSTER}
$ kubectl get policy -A
NAMESPACE   NAME                 REMEDIATION ACTION   COMPLIANCE STATE   AGE
cluster1    default.policy-pod   enforce              Compliant          4m32s

The missing pod is created by the policy on the managed cluster. To confirm, run the following command on the managed cluster:

$ kubectl get pod -n default
NAME               READY   STATUS    RESTARTS   AGE
sample-nginx-pod   1/1     Running   0          23s

2.3.4.2 - Open Policy Agent Gatekeeper

Gatekeeper is a validating webhook with auditing capabilities that can enforce custom resource definition-based policies that are run with the Open Policy Agent (OPA). Gatekeeper constraints can be used to evaluate Kubernetes resource compliance. You can leverage OPA as the policy engine, and use Rego as the policy language.

Installing Gatekeeper

See the Gatekeeper documentation to install the desired version of Gatekeeper to the managed cluster.

Sample Gatekeeper policy

Gatekeeper policies are written using constraint templates and constraints. View the following YAML examples that use Gatekeeper constraints in an OCM Policy:

ConstraintTemplates and constraints: Use the Gatekeeper integration feature by using OCM policies for multicluster distribution of Gatekeeper constraints and Gatekeeper audit results aggregation on the hub cluster. The following example defines a Gatekeeper ConstraintTemplate and constraint (K8sRequiredLabels) to ensure the “gatekeeper” label is set on all namespaces:

apiVersion: policy.open-cluster-management.io/v1
kind: Policy
metadata:
  name: require-gatekeeper-labels-on-ns
spec:
  remediationAction: inform # (1)
  disabled: false
  policy-templates:
    - objectDefinition:
        apiVersion: templates.gatekeeper.sh/v1beta1
        kind: ConstraintTemplate
        metadata:
          name: k8srequiredlabels
        spec:
          crd:
            spec:
              names:
                kind: K8sRequiredLabels
              validation:
                openAPIV3Schema:
                  properties:
                    labels:
                      type: array
                      items: string
          targets:
            - target: admission.k8s.gatekeeper.sh
              rego: |
                package k8srequiredlabels
                violation[{"msg": msg, "details": {"missing_labels": missing}}] {
                  provided := {label | input.review.object.metadata.labels[label]}
                  required := {label | label := input.parameters.labels[_]}
                  missing := required - provided
                  count(missing) > 0
                  msg := sprintf("you must provide labels: %v", [missing])
                }                
    - objectDefinition:
        apiVersion: constraints.gatekeeper.sh/v1beta1
        kind: K8sRequiredLabels
        metadata:
          name: ns-must-have-gk
        spec:
          enforcementAction: dryrun
          match:
            kinds:
              - apiGroups: [""]
                kinds: ["Namespace"]
          parameters:
            labels: ["gatekeeper"]

Since the remediationAction is set to “inform”, the enforcementAction field of the Gatekeeper constraint is overridden to “warn”. This means that Gatekeeper detects and warns you about creating or updating a namespace that is missing the “gatekeeper” label. If the policy remediationAction is set to “enforce”, the Gatekeeper constraint enforcementAction field is overridden to “deny”. In this context, this configuration prevents any user from creating or updating a namespace that is missing the gatekeeper label.

With the previous policy, you might receive the following policy status message:

warn - you must provide labels: {“gatekeeper”} (on Namespace default); warn - you must provide labels: {“gatekeeper”} (on Namespace gatekeeper-system).

Once a policy containing Gatekeeper constraints or ConstraintTemplates is deleted, the constraints and ConstraintTemplates are also deleted from the managed cluster.

Notes:

The Gatekeeper audit functionality runs every minute by default. Audit results are sent back to the hub cluster to be viewed in the OCM policy status of the managed cluster.

Auditing Gatekeeper events: The following example uses an OCM configuration policy within an OCM policy to check for Kubernetes API requests denied by the Gatekeeper admission webhook:

apiVersion: policy.open-cluster-management.io/v1
kind: Policy
metadata:
  name: policy-gatekeeper-admission
spec:
  remediationAction: inform
  disabled: false
  policy-templates:
    - objectDefinition:
      apiVersion: policy.open-cluster-management.io/v1
      kind: ConfigurationPolicy
      metadata:
        name: policy-gatekeeper-admission
      spec:
        remediationAction: inform # will be overridden by remediationAction in parent policy
        severity: low
        object-templates:
          - complianceType: mustnothave
            objectDefinition:
              apiVersion: v1
              kind: Event
              metadata:
                namespace: gatekeeper-system # set it to the actual namespace where gatekeeper is running if different
                annotations:
                  constraint_action: deny
                  constraint_kind: K8sRequiredLabels
                  constraint_name: ns-must-have-gk
                  event_type: violation

2.3.5 - Policy framework

The policy framework provides governance capabilities to OCM managed Kubernetes clusters. Policies provide visibility and drive remediation for various security and configuration aspects to help IT administrators meet their requirements.

API Concepts

View the Policy API page for additional details about the Policy API managed by the Policy Framework components, including:

Architecture

The governance policy framework distributes policies to managed clusters and collects results to send back to the hub cluster.

Note that in OCM versions newer than 0.8.x, the following controllers were consolidated into the policy framework addon.

Prerequisite

You must meet the following prerequisites to install the policy framework:

Ensure kubectl and kustomize are installed.
Ensure Golang is installed, if you are planning to install from the source.
Ensure the open-cluster-management cluster manager is installed. See Start the control plane for more information.
Ensure the open-cluster-management klusterlet is installed. See Register a cluster for more information.
If you are using PlacementRules with your policies, ensure the open-cluster-management application is installed . See Application management for more information. If you are using the default Placement API, you can skip the Application management installation, but you do need to install the PlacementRule CRD with this command:
```
kubectl apply -f https://raw.githubusercontent.com/open-cluster-management-io/multicloud-operators-subscription/main/deploy/hub-common/apps.open-cluster-management.io_placementrules_crd.yaml
```

Install the governance-policy-framework hub components

Install via Clusteradm CLI

Ensure clusteradm CLI is installed and is at least v0.3.0. Download and extract the clusteradm binary. For more details see the clusteradm GitHub page.

Deploy the policy framework controllers to the hub cluster:

# The context name of the clusters in your kubeconfig
# If the clusters are created by KinD, then the context name will the follow the pattern "kind-<cluster name>".
export CTX_HUB_CLUSTER=<your hub cluster context>           # export CTX_HUB_CLUSTER=kind-hub
export CTX_MANAGED_CLUSTER=<your managed cluster context>   # export CTX_MANAGED_CLUSTER=kind-cluster1

# Set the deployment namespace
export HUB_NAMESPACE="open-cluster-management"

# Deploy the policy framework hub controllers
clusteradm install hub-addon --names governance-policy-framework --context ${CTX_HUB_CLUSTER}

Ensure the pods are running on the hub with the following command:

$ kubectl get pods -n ${HUB_NAMESPACE}
NAME                                                       READY   STATUS    RESTARTS   AGE
governance-policy-addon-controller-bc78cbcb4-529c2         1/1     Running   0          94s
governance-policy-propagator-8c77f7f5f-kthvh               1/1     Running   0          94s

See more about the governance-policy-framework components:
- policy-propagator
- policy-addon-controller

Install from source

Deploy the policy Custom Resource Definitions (CRD) and policy propagator component to the open-cluster-management namespace on the hub cluster with the following commands:

# Configure kubectl to point to the hub cluster
kubectl config use-context ${CTX_HUB_CLUSTER}

# Create the namespace
export HUB_NAMESPACE="open-cluster-management"
kubectl create ns ${HUB_NAMESPACE}

# Set the hub cluster name
export HUB_CLUSTER_NAME="hub"

# Set the hub kubeconfig file
export HUB_KUBECONFIG="hub-kubeconfig"

# Apply the CRDs
export GIT_PATH="https://raw.githubusercontent.com/open-cluster-management-io/governance-policy-propagator/v0.12.0/deploy"
kubectl apply -f ${GIT_PATH}/crds/policy.open-cluster-management.io_policies.yaml
kubectl apply -f ${GIT_PATH}/crds/policy.open-cluster-management.io_placementbindings.yaml
kubectl apply -f ${GIT_PATH}/crds/policy.open-cluster-management.io_policyautomations.yaml
kubectl apply -f ${GIT_PATH}/crds/policy.open-cluster-management.io_policysets.yaml

# Deploy the policy-propagator
kubectl apply -f ${GIT_PATH}/operator.yaml -n ${HUB_NAMESPACE}

The policy propagator manages a webhook that requires a certificate. You can either disable the webhook or deploy cert-manager alongside the webhook resources to ensure the policy propagator runs properly:

Optional 1: Enable the webhook

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.12.0/cert-manager.yaml
kubectl apply -f ${GIT_PATH}/webhook.yaml -n ${HUB_NAMESPACE}

Optional 2: Disable the webhook with the --enable-webhooks=false argument

kubectl patch deployment governance-policy-propagator --type='json' -p='[
     {"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value": "--enable-webhooks=false"}
   ]' -n ${HUB_NAMESPACE}
kubectl patch deployment governance-policy-propagator --type='json' -p='[
     {"op": "remove", "path": "/spec/template/spec/containers/0/volumeMounts/0"},
     {"op": "remove", "path": "/spec/template/spec/volumes/0"}
   ]' -n ${HUB_NAMESPACE}

Ensure the pods are running on the hub with the following command:

$ kubectl get pods -n ${HUB_NAMESPACE}
NAME                                                       READY   STATUS    RESTARTS   AGE
governance-policy-propagator-8c77f7f5f-kthvh               1/1     Running   0          94s

See more about the governance-policy-framework components:
- policy-propagator
- policy-addon-controller

Deploy the synchronization components to the managed cluster(s)

Deploy via Clusteradm CLI

To deploy the synchronization components to a self-managed hub cluster:

clusteradm addon enable --names governance-policy-framework --clusters <managed_hub_cluster_name> --annotate addon.open-cluster-management.io/on-multicluster-hub=true --context ${CTX_HUB_CLUSTER}

To deploy the synchronization components to a managed cluster:

clusteradm addon enable --names governance-policy-framework --clusters <cluster_name> --context ${CTX_HUB_CLUSTER}

Verify that the governance-policy-framework-addon controller pod is running on the managed cluster with the following command:
```
$ kubectl get pods -n open-cluster-management-agent-addon
NAME                                                     READY   STATUS    RESTARTS   AGE
governance-policy-framework-addon-57579b7c-652zj         1/1     Running   0          87s
```
NOTE: If you are using clusteradm v0.3.x or older, the pod will be called governance-policy-framework and have a container per synchronization component (2 on a self-managed Hub, or 3 on a managed cluster).

Deploy from source

Export the hub cluster kubeconfig with the following command:

For kind cluster:

kind get kubeconfig --name ${HUB_CLUSTER_NAME} --internal > ${HUB_KUBECONFIG}

For non-kind clusters:

kubectl config view --context=${CTX_HUB_CLUSTER} --minify --flatten > ${HUB_KUBECONFIG}

Deploy the policy synchronization component to each managed cluster. Run the following commands:

NOTE: The --disable-spec-sync flag should be set to true in the governance-policy-framework-addon container arguments when deploying the synchronization component to a hub that is managing itself.

# Set whether or not this is being deployed on the Hub
export DEPLOY_ON_HUB=false

# Configure kubectl to point to the managed cluster
kubectl config use-context ${CTX_MANAGED_CLUSTER}

# Create the namespace for the synchronization components
export MANAGED_NAMESPACE="open-cluster-management-agent-addon"
kubectl create ns ${MANAGED_NAMESPACE}

# Create the secret to authenticate with the hub
kubectl -n ${MANAGED_NAMESPACE} create secret generic hub-kubeconfig --from-file=kubeconfig=${HUB_KUBECONFIG}

# Apply the policy CRD
export GIT_PATH="https://raw.githubusercontent.com/open-cluster-management-io"
kubectl apply -f ${GIT_PATH}/governance-policy-propagator/v0.12.0/deploy/crds/policy.open-cluster-management.io_policies.yaml

# Set the managed cluster name and create the namespace
export MANAGED_CLUSTER_NAME=<your managed cluster name>  # export MANAGED_CLUSTER_NAME=cluster1
kubectl create ns ${MANAGED_CLUSTER_NAME}

# Deploy the synchronization component
export COMPONENT="governance-policy-framework-addon"
kubectl apply -f ${GIT_PATH}/${COMPONENT}/v0.12.0/deploy/operator.yaml -n ${MANAGED_NAMESPACE}
kubectl patch deployment governance-policy-framework-addon -n ${MANAGED_NAMESPACE} \
  -p "{\"spec\":{\"template\":{\"spec\":{\"containers\":[{\"name\":\"governance-policy-framework-addon\",\"args\":[\"--hub-cluster-configfile=/var/run/klusterlet/kubeconfig\", \"--cluster-namespace=${MANAGED_CLUSTER_NAME}\", \"--enable-lease=true\", \"--log-level=2\", \"--disable-spec-sync=${DEPLOY_ON_HUB}\"]}]}}}}"

Verify that the pods are running on the managed cluster with the following command:

$ kubectl get pods -n ${MANAGED_NAMESPACE}
NAME                                                READY   STATUS    RESTARTS   AGE
governance-policy-framework-addon-6474b6d898-tmkw6  1/1     Running   0          2m14s

What is next

Install the policy controllers to the managed clusters.

2.4 - Administration

A few general guide about operating the open-cluster-management’s control plane and the managed clusters.

2.4.1 - Monitoring OCM using Prometheus-Operator

In this page, we provide a way to monitor your OCM environment using Prometheus-Operator.

Before you get started

You must have a OCM environment setuped. You can also follow our recommended quick start guide to set up a playgroud OCM environment.

And then please install the Prometheus-Operator in your hub cluster. You can also run the following commands copied from the official doc:

git clone https://github.com/prometheus-operator/kube-prometheus.git
cd kube-prometheus

# Create the namespace and CRDs, and then wait for them to be availble before creating the remaining resources
kubectl create -f manifests/setup

# Wait until the "servicemonitors" CRD is created. The message "No resources found" means success in this context.
until kubectl get servicemonitors --all-namespaces ; do date; sleep 1; echo ""; done

kubectl create -f manifests/

Monitoring the control-plane resource usage.

You can use kubectl proxy to open prometheus UI in your browser on localhost:9090:

kubectl --namespace monitoring port-forward svc/prometheus-k8s 9090

The following queries are to monitor the control-plane pods’ cpu usage, memory usage and apirequestcount for critical CRs:

rate(container_cpu_usage_seconds_total{namespace=~"open-cluster-management.*"}[3m])

container_memory_working_set_bytes{namespace=~"open-cluster-management.*"}

rate(apiserver_request_total{resource=~"managedclusters|managedclusteraddons|managedclustersetbindings|managedclustersets|addonplacementscores|placementdecisions|placements|manifestworks|manifestworkreplicasets"}[1m])

Visualized with Grafana

We provide a intial grafana dashboard for you to visualize the metrics. But you can also customize your own dashboard.

First, use the following command to proxy grafana service:

kubectl --namespace monitoring port-forward svc/grafana 3000

Next, open the grafana UI in your browser on localhost:3000.

Click the “Import Dashboard” and run the following command to copy a sample dashboard and paste it to the grafana:

curl https://raw.githubusercontent.com/open-cluster-management-io/open-cluster-management-io.github.io/main/content/en/getting-started/administration/assets/grafana-sample.json | pbcopy

Then, you will get a sample grafana dashboard that you can fine-tune further:

grafana

2.4.2 - Upgrading your OCM environment

This page provides the suggested steps to upgrade your OCM environment including both the hub cluster and the managed clusters. Overall the major steps you should follow are:

Read the release notes to confirm the latest OCM release version. (Note that some add-ons’ version might be different from OCM’s overall release version.)
Upgrade your command line tools clusteradm

Before you begin

You must have an existing OCM environment and there’s supposed to be registration-operator running in your clusters. The registration-operators is supposed to be installed if you’re previously following our recommended quick start guide to set up your OCM. The operator is responsible for helping you upgrade the other components on ease.

Upgrade command-line tool

In order to retrieve the latest version of OCM’s command-line tool clusteradm, run the following one-liner command:

$ curl -L https://raw.githubusercontent.com/open-cluster-management-io/clusteradm/main/install.sh | bash

Then you’re supposed to see the following outputs:

Getting the latest clusteradm CLI...
Your system is darwin_amd64

clusteradm CLI is detected:
Reinstalling clusteradm CLI - /usr/local/bin/clusteradm...

Installing v0.1.0 OCM clusteradm CLI...
Downloading https://github.com/open-cluster-management-io/clusteradm/releases/download/v0.1.0/clusteradm_darwin_amd64.tar.gz ...
clusteradm installed into /usr/local/bin successfully.

To get started with clusteradm, please visit https://open-cluster-management.io/getting-started/

Also, your can confirm the installed cli version by running:

$ clusteradm version
client		    version	:v0.1.0
server release	version	: ...

Upgrade OCM Components via Command-line tool

Hub Cluster

For example, to upgrade OCM components in the hub cluster, run the following command:

$ clusteradm upgrade clustermanager --bundle-version=0.7.0

Then clusteradm will make sure everything in the hub cluster is upgraded to the expected version. To check the latest status after the upgrade, continue to run the following command:

$ clusteradm get hub-info

Managed Clusters

To upgrade the OCM components in the managed clusters, switch the client context e.g. overriding KUBECONFIG environment variable, then simply run the following command:

$ clusteradm upgrade klusterlet --bundle-version=0.7.0

To check the status after the upgrade, continue running this command against the managed cluster:

$ clusteradm get klusterlet-info

Upgrade OCM Components via Manual Edit

Hub Cluster

Upgrading the registration-operator

Navigate into the namespace where you installed registration-operator (named “open-cluster-management” by default) and edit the image version of its deployment resource:

$ kubectl -n open-cluster-management edit deployment cluster-manager

Then update the image tag version to your target release version, which is exactly the OCM’s overall release version.

--- image: quay.io/open-cluster-management/registration-operator:<old release>
+++ image: quay.io/open-cluster-management/registration-operator:<new release>

Upgrading the core components

After the upgrading of registration-operator is done, it’s about time to surge the working modules of OCM. Go on and edit the clustermanager custom resource to prescribe the registration-operator to perform the automated upgrading:

$ kubectl edit clustermanager cluster-manager

In the content of clustermanager resource, you’re supposed to see a few images listed in its spec:

apiVersion: operator.open-cluster-management.io/v1
kind: ClusterManager
metadata: ...
spec:
  registrationImagePullSpec: quay.io/open-cluster-management/registration:<target release>
  workImagePullSpec: quay.io/open-cluster-management/work:<target release>
  # NOTE: Placement release versioning differs from the OCM root version, please refer to the release note.
  placementImagePullSpec: quay.io/open-cluster-management/placement:<target release>

Replacing the old release version to the latest and commit the changes will trigger the process of background upgrading. Note that the status of upgrade can be actively tracked via the status of clustermanager, so if anything goes wrong during the upgrade it should also be reflected in that status.

Managed Clusters

Upgrading the registration-operator

Similar to the process of upgrading hub’s registration-operator, the only difference you’re supposed to notice when upgrading the managed cluster is the name of deployment. Note that before running the following command, you are expected to switch the context to access the managed clusters not the hub.

$ kubectl -n open-cluster-management edit deployment klusterlet

Then repeatedly, update the image tag version to your target release version and commit the changes will upgrade the registration-operator.

Upgrading the agent components

After the registration-operator is upgraded, move on and edit the corresponding klusterlet custom resource to trigger the upgrading process in your managed cluster:

$ kubectl edit klusterlet klusterlet

In the spec of klusterlet, what is expected to be updated is also its image list:

apiVersion: operator.open-cluster-management.io/v1
kind: Klusterlet
metadata: ...
spec:
  ...
  registrationImagePullSpec: quay.io/open-cluster-management/registration:<target release>
  workImagePullSpec: quay.io/open-cluster-management/work:<target release>

After committing the updates, actively checking the status of the klusterlet to confirm whether everything is correctly upgraded. And repeat the above steps to each of the managed clusters to perform a cluster-wise progressive upgrade.

Confirm the upgrade

Getting the overall status of the managed cluster will help you to detect the availability in case any of the managed clusters are running into failure:

$ kubectl get managedclusters

And the upgrading is all set if all the steps above is succeeded.

3 - Developer Guides

3.1 - Add-on Developer Guide

This page is a developer guide about how to build an OCM add-on using addon-framework.

Supported version

The OCM v0.14.0 requires addon-framework v0.8.1 and above versions.

And notice there’s breaking changes in automatic installation in addon-framework version v0.10.0.

Overview

Add-on is an extension which can work with multiple clusters based on the foundation components in open-cluster-management. Add-ons are Open Cluster Management-based extensions that can be used to work with multiple clusters. Add-ons can support different configurations for different managed clusters, and can also be used to read data from the hub cluster. For example, you might use the managed-serviceaccount add-on to collect the tokens from managed cluster back to the hub cluster, use the cluster-proxy addon to establish a reverse proxy tunnels from the managed cluster to the hub cluster, etc.

A typical add-on should consist of two kinds of components:

Add-on agent: The components running in the managed clusters which can be any kubernetes resources, for example it might be a container with permissions to access the hub cluster, an Operator, or an instance of Operator, etc.

Add-on manager: A kubernetes controller in the hub cluster that generates and applies the add-on agent manifests to the managed clusters via the ManifestWork API. The manager also can optionally manage the lifecycle of add-on.

There are 2 API resources for add-on in the OCM hub cluster:

ClusterManagementAddOn: This is a cluster-scoped resource which allows the user to discover which add-on is available for the cluster manager and also provides metadata information about the add-on such as display name and description information. The name of the ClusterManagementAddOn resource will be used for the namespace-scoped ManagedClusterAddOn resource.

ManagedClusterAddOn: This is a namespace-scoped resource which is used to trigger the add-on agent to be installed on the managed cluster, and should be created in the ManagedCluster namespace of the hub cluster. ManagedClusterAddOn also holds the current state of an add-on.

There is a library named addon-framework which provides some simple user interfaces for developers to build their add-on managers easily.

We have some available add-ons in the OCM community:

Write your first add-on

Let’s implement a simple add-on manager using addon-framework, which deploys a busybox deployment in the managed cluster. You can find the example in here.

Implement the addon manager

First, create your Go project, and the project should contain a main.go file and a folder manifests. The folder name can be customized, the example uses manifests as the folder name. main.go contains the Go code of the addon manager. manifests contains the addon agent’s manifest files to be deployed on the managed cluster.

The main.go file is like this:

package main

import (
  "context"
  "embed"
  "os"
  restclient "k8s.io/client-go/rest"
  "k8s.io/klog/v2"
  "open-cluster-management.io/addon-framework/pkg/addonfactory"
  "open-cluster-management.io/addon-framework/pkg/addonmanager"
)

//go:embed manifests
var FS embed.FS

const (
  addonName = "busybox-addon"
)

func main() {
  kubeConfig, err := restclient.InClusterConfig()
  if err != nil {
     os.Exit(1)
  }
  addonMgr, err := addonmanager.New(kubeConfig)
  if err != nil {
     klog.Errorf("unable to setup addon manager: %v", err)
     os.Exit(1)
  }

  agentAddon, err := addonfactory.NewAgentAddonFactory(addonName, FS, "manifests").BuildTemplateAgentAddon()
  if err != nil {
     klog.Errorf("failed to build agent addon %v", err)
     os.Exit(1)
  }

  err = addonMgr.AddAgent(agentAddon)
  if err != nil {
     klog.Errorf("failed to add addon agent: %v", err)
     os.Exit(1)
  }

  ctx := context.Background()
  go addonMgr.Start(ctx)

  <-ctx.Done()
}

You need to define an embed.FS to embed the files in manifests folder.

And then you need to build an agentAddon using the agentAddonFactory, and tell the agentAddonFactory the name of the add-on and the agent manifests.

Finally, you just add the agentAddon to the addonManager and start the addonManager.

With above code, the addon manager is implemented. Next is to implement the addon agent part. In this example, the add-on agent manifest to be deployed on managed cluster is a busybox deployment.

Create file deployment.yaml in manifests folder, the deployment.yaml is like this:

kind: Deployment
apiVersion: apps/v1
metadata:
 name: busybox
 namespace: open-cluster-management-agent-addon
spec:
 replicas: 1
 selector:
   matchLabels:
     addon: busybox
 template:
   metadata:
     labels:
       addon: busybox
   spec:
     containers:
       - name: busybox
         image: busybox
         imagePullPolicy: IfNotPresent
         args:
           - "sleep"
           - "3600"

Then you can follow next section to deploy the add-on manager on your hub cluster. The add-on manager will watch the ManagedClusterAddOn, and deploy the add-on agent manifests to the targeted managed cluster via ManifestWork.

Deploy the add-on manager on your hub cluster

Now you can build your add-on manager as an image and deploy it on the hub cluster.

Following below steps to build the image for the example. This image contains several example addon managers, including the busybox example.

git clone https://github.com/open-cluster-management-io/addon-framework.git
cd addon-framework
make images

In addition to the deployment definition, there are also some additional resources to be deployed on the hub cluster. An example of the deployment manifests for the add-on manager is here. Following below steps to deploy the add-on manager.

make deploy-busybox

With the add-on manager deployed, you can see the busybox-addon-controller running in namespace open-cluster-management on the hub cluster.

$ oc get pods -n open-cluster-management
NAME                                       READY   STATUS    RESTARTS   AGE
busybox-addon-controller-d977665d5-x28qc   1/1     Running   0          27m

RBAC of the addon manager

There are some minimum required permissions for the addon manager controller to run on the hub cluster. It needs to:

get/list/watch/update the ManagedCluster.
get/list/watch/create/update/patch/delete the ManagedClusterAddOn and ManifestWork.
get/list/watch the ClusterManagementAddOn.

ClusterManagementAddOn

From a user’s perspective, to install the addon to the hub cluster the hub admin should register a globally-unique ClusterManagementAddOn resource as a singleton placeholder in the hub cluster. For instance, the ClusterManagementAddOn for the busybox-addon:

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ClusterManagementAddOn
metadata:
 name: busybox-addon
spec:
 addOnMeta:
   displayName: Busybox Addon
   description: "busybox-addon is an example addon to deploy busybox pod on the managed cluster"

Enable the add-on for a managed cluster.

Now your addon-manager is running on the hub cluster. To deploy the busybox add-on agent to a certain managed cluster, you need to create a ManagedClusterAddOn in the managed cluster namespace of hub cluster.

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ManagedClusterAddOn
metadata:
 name: busybox-addon
 namespace: cluster1
spec:
 installNamespace: open-cluster-management-agent-addon

You can set any existing namespace in the managed cluster as the installNamespace here, and the add-on manager will deploy the add-on agent manifests in this namespace of the managed cluster.

Note: open-cluster-management-agent-addon is our default namespace to install the add-on agent manifests in the managed cluster.

You can also use the clusteradm command to enable the busybox-addon for the managed cluster.

$ clusteradm addon enable --names busybox-addon --namespace open-cluster-management-agent-addon --clusters cluster1

After enabling the add-on for the managed cluster, you can find a ManifestWork named addon-busybox-addon-deploy is deploying on the managed cluster namespace of the hub cluster.

$ kubectl get manifestworks.work.open-cluster-management.io -n cluster1
NAME                         AGE
addon-busybox-addon-deploy   2m

And the busybox deployment is deployed on the managed cluster too.

$ kubectl get deployment -n open-cluster-management-agent-addon
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
busybox                   1/1     1            1           2m

Disable the add-on for a managed cluster

You can delete the ManagedClusterAddOn CR in the managed cluster namespace of the hub cluster to disable the add-on for the managed cluster.The created ManifestWork will be deleted and the add-on agent manifests will be removed from the managed cluster too.

You also can use the clusteradm command to disable the add-on for a managed cluster.

$ clusteradm addon disable --names busybox-addon --clusters cluster1

You can also use the clusteradm command to disable the add-ons for all managed clusters.

$ clusteradm addon disable --names busybox-addon --all-clusters true

If you delete the ClusterManagementAddOn on the hub cluster, the ManagedClusterAddOn CRs in all managed cluster namespaces will be deleted too.

What’s the next

However, this add-on just ensures a pod to run on the managed cluster, and you cannot see the status of the addon, and there are not any functionality to manage the clusters. The addon-framework also provides other configurations for add-on developers.

$ kubectl get managedclusteraddons.addon.open-cluster-management.io -n cluster1
NAME                AVAILABLE   DEGRADED   PROGRESSING
busybox-addon      Unknown

Next, you need to configure the addon.

Add-on agent configurations

Monitor addon healthiness

In the busybox example above, we found the AVAILABLE status of the ManagedClusterAddOn is always Unknown.

That’s because the add-on manager did not monitor the status of the add-on agent from the hub cluster.

We support 3 kinds of health prober types to monitor the healthiness of add-on agents.

Lease
The add-on agent maintains a Lease in its installation namespace with its status, the registration agent will check this Lease to maintain the AVAILABLE status of the ManagedClusterAddOn.
The addon-framework provides a leaseUpdater interface which can make it easier.
```
leaseUpdater := lease.NewLeaseUpdater(spokeKubeClient, addonName, installNamespace)
go leaseUpdater.Start(context.Background())
```
Lease is the default prober type for add-on, there is nothing to configure for the add-on manager.
Work
Work health prober indicates the healthiness of the add-on is equal to the overall dispatching status of the corresponding ManifestWork resources. It’s applicable to those add-ons that don’t have a container agent in the managed clusters or don’t expect to add Lease for the agent container. The add-on manager will check if the work is Available on the managed clusters. In addition, the user can define a HealthCheck prober function to check more detailed status based on status feedback from the ManifestWork.
It is required to define a HealthProber instance first. Here is an example to check if the availableReplicas of add-on agent deployment is more than 1. If yes, it will set the AVAILABLE status of ManagedClusterAddOn to true. Otherwise, the AVAILABLE status of ManagedClusterAddOn will be false.
```
healthProber := utils.NewDeploymentProber(types.NamespacedName{Name: "workprober-addon-agent", Namespace: "open-cluster-management-agent-addon"})
```
And then you can configure the HealthProber to the agentAddon.
```
agentAddon, err := addonfactory.NewAgentAddonFactory(addonName, FS, "manifests").
                    WithAgentHealthProber(healthProber).
                    BuildTemplateAgentAddon()
```
DeploymentAvailability
DeploymentAvailability health prober indicates the healthiness of the add-on is connected to the availability of the corresponding agent deployment resources on the managed cluster. It’s applicable to those add-ons that running Deployment type workload on the managed cluster. The add-on manager will check if the availableReplicas of the add-on agent deployment is more than 1 to set the addon Status.
Set the type of healthProber to DeploymentAvailability to enable this prober.
```
healthProber := &agent.HealthProber{
    Type: agent.HealthProberTypeDeploymentAvailability,
}
```
None
If you want to check and maintain the AVAILABLE status of ManagedClusterAddOn by yourself, set the type of healthProber to None.
```
healthProber := &agent.HealthProber{
    Type: agent.HealthProberTypeNone,
}
```

Automatic installation

NOTE:

The automatic installation is no longer supported since addon-framework v0.10.0. Please use the InstallStrategy in Managing the add-on agent lifecycle by addon-manager section instead.
The automatic installation is still avaliable in addon-framework version v0.8.1 and v0.9.3, which is also the minimal supported addon-framework version in OCM v0.14.0. Using the addon-framework v0.8.0 and previous version will have install conficts.

In the busybox add-on example, you need to create a ManagedClusterAddOn CR to enable the add-on manually. The addon-framework also provides a configuration called InstallStrategy to support installing addon automatically.

Currently, the addon-framework supports InstallAllStrategy and InstallByLabelStrategy strategies.

InstallAllStrategy will create ManagedClusterAddOn for all managed cluster namespaces automatically.

installStrategy := agent.InstallAllStrategy("open-cluster-management-agent-addon")

InstallByLabelStrategy will create ManagedClusterAddOn for the selected managed cluster namespaces automatically.

installStrategy := &agent.InstallStrategy{
	Type:             agent.InstallByLabel,
    InstallNamespace: "open-cluster-management-agent-addon",
    LabelSelector:    &metav1.LabelSelector{...},
}

Configure the InstallStrategy to the agentAddon:

agentAddon, err := addonfactory.NewAgentAddonFactory(addonName, FS, "manifests").
                    WithInstallStrategy(installStrategy).
                    BuildTemplateAgentAddon()

Addtionally, if you are using addon-framework v0.8.1 or higher, need to grant a patch permission on ClusterManagementAddon to your addon manager.

kind: ClusterRole
  apiVersion: rbac.authorization.k8s.io/v1
  metadata:
    name: helloworld-addon
  rules:
...
    - apiGroups: ["addon.open-cluster-management.io"]
      resources: ["clustermanagementaddons"]
      verbs: ["get", "list", "watch", "patch"]

The below annotation will be added automatically to claim the ManagedClusterAddon lifecycle is managed by the addon itself.

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ClusterManagementAddOn
metadata:
  annotations:
    addon.open-cluster-management.io/lifecycle: "self"
  name: helloworld
spec:
  installStrategy:
    type: Manual

Register your add-on

In most cases, the add-ons have requirements to access the hub cluster or other central service endpoint with TLS authentication. For example, an add-on agent needs to get a resource in its cluster namespace of the hub cluster, or the add-on agent needs to access the exposed service on the hub cluster.

The addon-framework supports a solution that the addon can access the kube-apiserver with a kube style API or other endpoints on the hub cluster with client certificate authentication after it is registered using CSR.

The addon-framework provides an interface to help add-on manager to save the add-on configuration information to its corresponding ManagedClusterAddOns.

On the managed cluster, the registration agent watches ManagedClusterAddOns on the hub cluster. The registration agent follows next steps to register an add-on:

The registration agent creates a CSR request with its own hub kubeConfig to register the add-on to the hub cluster.
On the hub cluster, the add-on manager approves the CSR request. The addon-framework also provides an interface which the add-on manager can implement it to approve its CSR automatically.
After the CSR request is approved on the hub cluster, the registration agent gets the certificate from the CSR request and saves the client certificate to a secret in the add-on agent install namespace. If the SignerName is kubernetes.io/kube-apiserver-client, the secret name will be {addon name}-hub-kubeconfig. Otherwise, the secret name will be {addon name}-{signer name}-client-cert.
The add-on agent can mount the secret to get the client certificate to connect with the hub cluster or the custom service endpoint.
When the certificate of managed cluster addon is about to expire, the registration agent will send a request to rotate the certificate on the hub cluster, the addon manager will approve the certificate rotation request.

Now we build another add-on that is going to sync configmap from the hub cluster to the managed cluster. The add-on code can be found here .

Specifically, since the addon agent needs to read configmap from the hub, we need to define the registration option for this addon.

func NewRegistrationOption(kubeConfig *rest.Config, addonName, agentName string) *agent.RegistrationOption {
    return &agent.RegistrationOption{
        CSRConfigurations: agent.KubeClientSignerConfigurations(addonName, agentName),
        CSRApproveCheck:   utils.DefaultCSRApprover(agentName),
        PermissionConfig:  rbac.AddonRBAC(kubeConfig),
    }
}

CSRConfigurations returns a list of CSR configuration for the addd-on agent in a managed cluster. The CSR will be created from the managed cluster for add-on agent with each CSRConfiguration.

func KubeClientSignerConfigurations(addonName, agentName string) func(cluster *clusterv1.ManagedCluster) []addonapiv1alpha1.RegistrationConfig {
	return func(cluster *clusterv1.ManagedCluster) []addonapiv1alpha1.RegistrationConfig {
        return []addonapiv1alpha1.RegistrationConfig{
            {
                SignerName: certificatesv1.KubeAPIServerClientSignerName,
                Subject: addonapiv1alpha1.Subject{
                    User:   DefaultUser(cluster.Name, addonName, agentName),
                    Groups: DefaultGroups(cluster.Name, addonName),
                },
            },
        }
	}
}

The original Kubernetes CSR API only supports three built-in signers:

kubernetes.io/kube-apiserver-client
kubernetes.io/kube-apiserver-client-kubelet
kubernetes.io/kubelet-serving

However, in some cases, we need to sign additional custom certificates for the add-on agents which are not used for connecting any kube-apiserver. The add-on manager can be serving as a custom CSR signer controller based on the addon-framework’s extensibility by implementing the signing logic. The addon-framework will also keep rotating the certificates automatically for the add-on after successfully signing the certificates.

CSRApproveCheck checks whether the add-on agent registration should be approved by the add-on manager. The utils.DefaultCSRApprover is implemented to auto-approve all the CSRs. A better CSR check is recommended to include:

The validity of the requester’s requesting identity.
The other request payload such as key-usages.

If the function is not set, the registration and certificate renewal of the add-on agent needs to be approved manually on the hub cluster.

PermissionConfig defines a function for an add-on to set up RBAC permissions on the hub cluster after the CSR is approved. In this example, it will create a role in the managed cluster namespace with get/list/watch configmaps permissions, and bind the role to the group defined in CSRConfigurations.

Configure the registrationOption to the agentAddon.

agentAddon, err := addonfactory.NewAgentAddonFactory(helloworld.AddonName, helloworld.FS, "manifests/templates").
                    WithGetValuesFuncs(helloworld.GetValues, addonfactory.GetValuesFromAddonAnnotation).
                    WithAgentRegistrationOption(registrationOption).
                    WithInstallStrategy(addonagent.InstallAllStrategy(agent.HelloworldAgentInstallationNamespace)).
                    BuildTemplateAgentAddon()

After deploying the example add-on, you can find the registration configuration in the ManagedClusterAddOn status.

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ManagedClusterAddOn
metadata:
  name: helloworld
  namespace: cluster1
  ownerReferences:
  - apiVersion: addon.open-cluster-management.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: ClusterManagementAddOn
    name: helloworld
spec:
  installNamespace: default
status:
  registrations:
  - signerName: kubernetes.io/kube-apiserver-client
    subject:
      groups:
      - system:open-cluster-management:cluster:cluster1:addon:helloworld
      - system:open-cluster-management:addon:helloworld
      - system:authenticated
      user: system:open-cluster-management:cluster:cluster1:addon:helloworld:agent:2rn8d

In this example, the addon requires a CSR access hub kube-api (with singer name kubernetes.io/kube-apiserver-client). After the CSR is created on the hub cluster, the add-on manager will check the signer, group and subject of the CSRs to verify whether the CSR is valid. If all fields are valid, the add-on manager will approve the CSR.

apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  labels:
    open-cluster-management.io/addon-name: helloworld
    open-cluster-management.io/cluster-name: cluster1
  name: addon-cluster1-helloworld-lb7cb
spec:
  groups:
  - system:open-cluster-management:cluster1
  - system:open-cluster-management:managed-clusters
  - system:authenticated
  request: xxx
  signerName: kubernetes.io/kube-apiserver-client
  usages:
  - digital signature
  - key encipherment
  - client auth
  username: system:open-cluster-management:cluster1:9bkfw

After the CSR is approved, the add-on controller creates the Role and Rolebinding in the cluster namespace.

$ kubectl get role -n cluster1
NAME                                       CREATED AT
open-cluster-management:helloworld:agent   2022-07-10T10:08:37Z
$ kubectl get rolebinding -n cluster1
NAME                                                           ROLE                                                              AGE
open-cluster-management:helloworld:agent                       Role/open-cluster-management:helloworld:agent                     13m

The Rolebinding binds the Role to the Group system:open-cluster-management:cluster:cluster1:addon:helloworld.

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: open-cluster-management:helloworld:agent
  namespace: cluster1
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: open-cluster-management:helloworld:agent
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:open-cluster-management:cluster:cluster1:addon:helloworld

The registration agent will create a kubeConfig secret named <add-on name>-hub-kubeconfig in the addonInstallNamesapce. The addon agent can mount the secret to get the hub kubeConfig to connect with the hub cluster to get/list/watch the Configmaps.

$ kubectl get secret -n default
NAME                              TYPE                                  DATA   AGE
helloworld-hub-kubeconfig         Opaque                                3      9m52s

Add your add-on agent supported configurations

For some cases, you want to specify the configurations for your add-on agent, for example, you may want to use a configuration to configure your add-on agent image or use a configuration to configure your add-on agent node Selector and tolerations to make the agent to run on specific nodes.

The addon-framework supports re-rendering the add-on agent deployment when the add-on agent configurations are changed.

You can choose the AddOnDeploymentConfig API as the configuration for your add-on agent, it supports setting customized variables and node placement for your add-on agent deployment, and meanwhile you can also choose your own configuration.

You can do the following steps to reference your configurations in your add-on APIs with add-on framework

Add the supported configuration types in your add-on ClusterManagementAddOn, we support to add multiple different configuration types in the ClusterManagementAddOn, for example

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ClusterManagementAddOn
metadata:
  name: helloworldhelm
spec:
  # the add-on supported configurations
  supportedConfigs:
  - group: addon.open-cluster-management.io
    resource: addondeploymentconfigs
  - resource: configmaps

In this example, the helloworldhelm add-on supports using AddOnDeploymentConfig and ConfigMap as its configuration, and you can specify one default configuration for one configuration type, for example

 apiVersion: addon.open-cluster-management.io/v1alpha1
 kind: ClusterManagementAddOn
 metadata:
   name: helloworldhelm
 spec:
   # the add-on supported configurations
   supportedConfigs:
   - group: addon.open-cluster-management.io
     resource: addondeploymentconfigs
     # the default config for helloworldhelm
     defaultConfig:
       name: deploy-config
       namespace: open-cluster-management
   - resource: configmaps

Thus, all helloworldhelm add-ons on each managed cluster have one same default configuration open-cluster-management/deploy-config

Implement a GetValuesFunc to transform the configuration to addon-framework Values object and add the GetValuesFunc to the AgentAddonFactory, for example

agentAddon, err := addonfactory.NewAgentAddonFactory("helloworldhelm", helloworld_helm.FS, "manifests/charts/helloworld").
    // register the supported configuration types
    WithConfigGVRs(
      schema.GroupVersionResource{Version: "v1", Resource: "configmaps"},
      schema.GroupVersionResource{Group: "addon.open-cluster-management.io", Version: "v1alpha1", Resource: "addondeploymentconfigs"},
    ).
    WithGetValuesFuncs(
      // get the AddOnDeloymentConfig object and transform it to Values object
      addonfactory.GetAddOnDeloymentConfigValues(
        addonfactory.NewAddOnDeloymentConfigGetter(addonClient),
        addonfactory.ToAddOnNodePlacementValues,
      ),
      // get the ConfigMap object and transform it to Values object
      helloworld_helm.GetImageValues(kubeClient),
    ).WithAgentRegistrationOption(registrationOption).
    BuildHelmAgentAddon()

In this example, we register the ConfigMap and AddOnDeploymentConfig as the helloworldhelm add-on configuration. We use add-on framework help function GetAddOnDeloymentConfigValues to transform the AddOnDeploymentConfig, and we implemented the GetImageValues function to transform the ConfigMap, you can find more details for add-on framework Values from the Values definition part.

Add the get, list and watch permissions to an add-on clusterrole, for example, the clusterrole of helloworldhelm should have the following permissions

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: helloworldhelm-addon
rules:
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["addon.open-cluster-management.io"]
    resources: ["addondeploymentconfigs"]
    verbs: ["get", "list", "watch"]

To configure add-on, the add-on user need reference their configuration objects in ManagedClusterAddOn, for example

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ManagedClusterAddOn
metadata:
  name: helloworldhelm
  namespace: cluster1
spec:
  installNamespace: open-cluster-management-agent-addon
  configs:
  - group: addon.open-cluster-management.io
    resource: addondeploymentconfigs
    name: deploy-config
    namespace: cluster1
  - resource: configmaps
    name: image-config
    namespace: cluster1

In this example, the add-on user reference the configuration cluster1/deploy-config and cluster1/image-config for helloworldhelm on cluster1. When the configuration references are added to an add-on, the add-on framework will show them in the status of ManagedClusterAddOn and render the add-on once, during the rendering process, the add-on framework will callback the GetValuesFuncs to transform the add-on configuraton object to add-on framework Values object and use Values object to render the add-on agent deployment resources. If the add-on configuration objects are updated, the add-on framework will render the add-on again.

Build an addon using helm charts or raw manifests.

Building steps

The addon-framework supports helm charts or raw manifests as the add-on agent manifests. The building steps are the same:

Copy the helm chart or raw manifests files into the add-on manager project. And define an embed.FS to embed the files into your Go program.
The example using helm chart is helloworld_helm addon, and the example using raw manifests is helloworld addon.

Build different agentAddons using the agentAddonFactory instance with BuildHelmAgentAddon or BuildTemplateAgentAddon.

For helm chart building:

agentAddon, err := addonfactory.NewAgentAddonFactory(helloworld_helm.AddonName, helloworld_helm.FS, "manifests/charts/helloworld").
                    WithGetValuesFuncs(helloworld_helm.GetValues, addonfactory.GetValuesFromAddonAnnotation).
                    WithAgentRegistrationOption(registrationOption).
                    BuildHelmAgentAddon()

For raw manifests building:

agentAddon, err := addonfactory.NewAgentAddonFactory(helloworld.AddonName, helloworld.FS, "manifests/templates").
                    WithGetValuesFuncs(helloworld.GetValues, addonfactory.GetValuesFromAddonAnnotation).
                    WithAgentRegistrationOption(registrationOption).
                    WithInstallStrategy(addonagent.InstallAllStrategy(agent.HelloworldAgentInstallationNamespace)).
                    BuildTemplateAgentAddon()

Add the agentAddon to the addon manager.
Start the addon manager.

Values definition

The addon-framework supports 3 add-on built-in values and 3 helm chart built-in values for helm chart add-on manifests.

Value.clusterName
Value.addonInstallNamespace
Value.hubKubeConfigSecret (used when the add-on is needed to register to the hub cluster)
Capabilities.KubeVersion is the ManagedCluster.Status.Version.Kubernetes.
Release.Name is the add-on name.
Release.Namespace is the addonInstallNamespace.

The addon-framework supports 3 add-on built-in values in the config of templates for the raw manifests add-on.

ClusterName
AddonInstallNamespace
HubKubeConfigSecret (used when the AddOn is needed to register to the hub cluster)

In the list of GetValuesFuncs, the values from the big index Func will override the one from low index Func.

The built-in values will override the values obtained from the list of GetValuesFuncs.

The Variable names in Values should begin with lowercase. So the best practice is to define a json struct for the values, and convert it to Values using the JsonStructToValues.

Values from annotation of ManagedClusterAddOn

The addon-framework supports a helper GetValuesFunc named GetValuesFromAddonAnnotation which can get values from the annotations of ManagedClusterAddOn.

The key of the Helm Chart values in annotation is addon.open-cluster-management.io/values, and the value should be a valid json string which has key-value format.

Hosted mode

The addon-framework supports add-on in Hosted mode, that the agent manifests will be deployed outside the managed cluster. We can choose to run add-on in Hosted mode or Default mode if the managed cluster is imported to the hub in Hosted mode. By default, the add-on agent will run on the managed cluster(Default mode). We can add an annotation addon.open-cluster-management.io/hosting-cluster-name for the ManagedClusterAddon, so that the add-on agent will be deployed on the certain hosting cluster(Hosted mode), the value of the annotation is the hosting cluster which should:

be a managed cluster of the hub as well.
be the same cluster where the managed cluster klusterlet(registration-agent & work-agent) runs.

We defined a label addon.open-cluster-management.io/hosted-manifest-location to indicate which cluster the add-on agent manifests should be deployed.

No matter what the value is, all manifests will be deployed on the managed cluster in Default mode.
When the label does not exist or the value is managed: the manifest will be deployed on the managed cluster in Hosted mode.
When the value is hosting: the manifest will be deployed on the hosting cluster in Hosted mode.
When the value is none: the manifest will not be deployed in Hosted mode.

More details you can find in the design, and we have an example in here.

Pre-delete hook

The addon-framework provides a hook manifest before delete the add-on. The hook manifest supports Jobs or Pods to do some cleanup work before the add-on agent is deleted on the managed cluster.

You need only add the label open-cluster-management.io/addon-pre-delete to the Jobs or Podsin the add-on manifests. The Jobs or Pods will not be applied until the ManagedClusterAddOn is deleted. And the Jobs or Pods will be applied on the managed cluster by applying the manifestWork named addon-<addon name>-pre-delete when the ManagedClusterAddOn is under deleting. After the Jobs are Completed or Pods are in the Succeeded phase, all the deployed ManifestWorks will be deleted.

You can find the example from here.

What happened under the scene

This architecture graph shows how the coordination between add-on manager and add-on agent works.

The registration agent creates a CSR request with its own hub kubeConfig to register the add-on to the hub cluster.
On the hub cluster, the add-on manager approves the CSR request.
After the CSR request is approved on the hub cluster, the registration agent gets the certificate from the CSR request to establish the hub kubeConfig and save the hub kubeConfig to a secret in the managed cluster addon namespace.
The add-on manager is watching the ManagedClusterAddOn for all managed cluster namespaces. And will create an add-on deploy ManifestWork in the managed cluster namespace once the ManagedClusterAddOn is created in this managed cluster namespace.
The work agent will apply the manifests in the ManifestWork on the managed cluster.
The add-on agent will mount the secret created by the registration agent to get the hub kubeConfig to connect with the hub cluster.

Managing the add-on agent lifecycle by addon-manager

The add-on agent lifecycle can now be managed by the general addon-manager starting from OCM v0.11.0. This is achieved through enhancements to the ClusterManagementAddOn and ManagedClusterAddOn APIs.

Install strategy

With the install strategy defined in the ClusterManagementAddOn API, users can configure which clusters the related ManagedClusterAddon should be enabled by referencing the Placement. For example, enabling the helloworld add-on on clusters labeled with aws.

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ClusterManagementAddOn
metadata:
  name: helloworld
  annotations:
    addon.open-cluster-management.io/lifecycle: "addon-manager"
spec:
  addOnMeta:
    displayName: helloworld
  installStrategy:
    type: Placements
    placements:
    - name: placement-aws
      namespace: default

apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Placement
metadata:
  name: placement-aws
  namespace: default
spec:
  predicates:
    - requiredClusterSelector:
        claimSelector:
          matchExpressions:
            - key: platform.open-cluster-management.io
              operator: In
              values:
                - aws

Rollout strategy

With the rollout strategy defined in the ClusterManagementAddOn API, users can control the upgrade behavior of the add-on when there are changes in the supported configurations.

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ClusterManagementAddOn
metadata:
  name: helloworld
  annotations:
    addon.open-cluster-management.io/lifecycle: "addon-manager"
spec:
  addOnMeta:
    displayName: helloworld
  installStrategy:
    type: Placements
    placements:
    - name: placement-aws
      namespace: default
      configs:
      - group: addon.open-cluster-management.io
        resource: addondeploymentconfigs
        name: deploy-config
        namespace: open-cluster-management
      rolloutStrategy:
        type: Progressive
        progressive:
          mandatoryDecisionGroups:
          - groupName: "canary"
          maxConcurrency: 25%

Add-on developers can use addon-framework v0.8.0 and the above versions to support the scenarios mentioned above.

Modify the go.mod file to use the latest addon-framework and API versions.

open-cluster-management.io/addon-framework v0.9.3 // // or latest
open-cluster-management.io/api v0.13.0 // or latest

Remove the WithInstallStrategy() function described in the automatic installation section since it conflicts with the install strategy defined in the ClusterManagementAddOn API level.
Claim that the addon is managed by the general addon-manager by adding the annotation addon.open-cluster-management.io/lifecycle: "addon-manager" explicitly in the ClusterManagementAddOn.

Skip this step for OCM v0.14.0 and later version. The annotation is automatically added by the general addon manager.

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ClusterManagementAddOn
metadata:
  name: helloworld
  annotations:
    addon.open-cluster-management.io/lifecycle: "addon-manager"
...

Define the installStrategy and rolloutStrategy in the ClusterManagementAddOn as shown in the example above. Note that the rollout strategy is triggered by changes in configurations, so if the addon does not have supported cofingurations, the rollout strategy will not take effect.
If you do not want the automatic addon installation, set the install strategy type to Manual.

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ClusterManagementAddOn
metadata:
  annotations:
    addon.open-cluster-management.io/lifecycle: "addon-manager"
  name: helloworld
spec:
  installStrategy:
    type: Manual

Build an addon with addon template

Using the addon-framework to develop an addon requires developers to implement the interface defined in the addon-framework via code and deploy a dedicated addon manager deployment on the hub cluster. But if the addon you are trying to develop:

not going to support hosted mode
the crucial agent workloads that need to be deployed to the managed cluster are Deployments and/or DaemonSets
no other customized API is needed to configure the addon besides the AddOnDeploymentConfig
no need to run anything on the hub cluster other than managing the addon agent

you can have a try with the new API AddOnTemplate introduced from OCM v0.12.0 to build the addon, which can get rid of coding, and only need to define some yaml files to build an addon.

Using AddOnTemplate to build an addon, the AddonManagement feature gate must not be disabled in ClusterManager.spec.addOnManagerConfiguration and Klusterlet.spec.registrationConfiguration

Enhancement proposal: Add-on Template

Note: The DaemonSet type workload is supported in the addon template(injecting environment variables, injecting volumes, health probe for daemonsets) from OCM v0.14.0.

Steps to build an addon with addon template

Create an AddOnTemplate object to define the addon: The AddOnTemplate API provides two parts of information to build an addon:

manifests: what resources will be deployed to the managed cluster
registration: how to register the addon to the hub cluster

For example, the following yaml file defines the hello-template addon, which will:

deploy a Deployment, a ServiceAccount, and a ClusterRoleBinding to the managed cluster
register the addon to the hub cluster, and make the addon agent(Deployment hello-template-agent):
- have the permission to access resources defined in the cm-admin clusterRole in the namespace on the hub cluster(KubeClient type registration, CurrentCluster)
- have the permission to access resources defined in the cm-reader Role in the open-cluster-management namespace on the hub cluster(KubeClient type registration, SingleNamespace)
- have the credential to access the customized endpoint(CustomSigner type registration)

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: AddOnTemplate
metadata:
  name: hello-template
spec:
  addonName: hello-template
  agentSpec: # required
      workload:
        manifests:
          - kind: Deployment
            apiVersion: apps/v1
            metadata:
              name: hello-template-agent
              namespace: open-cluster-management-agent-addon
              labels:
                app: hello-template-agent
            spec:
              replicas: 1
              selector:
                matchLabels:
                  app: hello-template-agent
              template:
                metadata:
                  labels:
                    app: hello-template-agent
                spec:
                  serviceAccountName: hello-template-agent-sa
                  containers:
                    - name: helloworld-agent
                      image: quay.io/open-cluster-management/addon-examples:latest
                      imagePullPolicy: IfNotPresent
                      args:
                        - "/helloworld"
                        - "agent"
                        - "--cluster-name={{CLUSTER_NAME}}"
                        - "--addon-namespace=open-cluster-management-agent-addon"
                        - "--addon-name=hello-template"
                        - "--hub-kubeconfig={{HUB_KUBECONFIG}}"
                        - "--v={{LOG_LEVEL}}" # addonDeploymentConfig variables
          - kind: ServiceAccount
            apiVersion: v1
            metadata:
              name: hello-template-agent-sa
              namespace: open-cluster-management-agent-addon
          - kind: ClusterRoleBinding
            apiVersion: rbac.authorization.k8s.io/v1
            metadata:
              name: hello-template-agent
            roleRef:
              apiGroup: rbac.authorization.k8s.io
              kind: ClusterRole
              name: cluster-admin
            subjects:
              - kind: ServiceAccount
                name: hello-template-agent-sa
                namespace: open-cluster-management-agent-addon
  registration: # optional
    # kubeClient or custom signer, if kubeClient, user and group is in a certain format.
    # user is "system:open-cluster-management:cluster:{clusterName}:addon:{addonName}:agent:{agentName}"
    # group is ["system:open-cluster-management:cluster:{clusterName}:addon:{addonName}",
    #           "system:open-cluster-management:addon:{addonName}", "system:authenticated"]
    - type: KubeClient
      kubeClient:
        hubPermissions:
          - type: CurrentCluster
            currentCluster:
              clusterRoleName: cm-admin # should be created by user
          - type: SingleNamespace
            singleNamespace:
              namespace: open-cluster-management
              roleRef:
                apiGroup: rbac.authorization.k8s.io
                kind: Role
                # should be created by user; the addon manager will grant the permission to the agent, so if the
                # role/clusterRole contains some permissions that the addon manager doesn't have, user needs to grant
                # the permission to the addon-manager (service account open-cluster-management-hub/addon-manager-controller-sa),
                # otherwise the addon manager will fail to grant the permission to the agent
                name: cm-reader
    - type: CustomSigner
      # addon-manager only generates the credential for the agent to authenticate to the hub cluster, not responsible
      # for the authroization which should be taken care of by the user
      customSigner:
        signerName: example.com/signer-test
        subject:
          user: user-test
          groups:
            - group-test
          organizationUnit:
            - organization-test
        signingCA:
          # type is "kubernetes.io/tls", namespace is "open-cluster-management-agent-addon", user needs to grant the
          # permission to the addon-manager (service account open-cluster-management-hub/addon-manager-controller-sa)
          # to access the secret
          name: ca-secret

Notes:

The permission related resources(i.e. RoleBinding ClusterRoleBinding) for the addon agent access the local managed cluster defined in the addonTemplate.agentSpec.workload.manifests will be created on the managed cluster by the work-agent, but the work-agent may not have permission to create these resources, users should refer to permission-setting-for-work-agent to grant the work-agnet permissions to address the permission issue on the managed cluster side.
Permissions for the addon agent access the hub cluster defined in addonTemplate.registration[*].kubeClient.hubPermissions, users should ensure:
1. the referenced clusterrole/role(.hubPermissions.currentCluster.clusterRoleName .hubPermissions.singleNamespace.roleRef.name, cm-admin and cm-reader in the above example) exists on the hub cluster
2. the addon-manager has permission to create rolebinding to bind these (cluster)role for the addon-agent. For example: users can create a clusterrolebinding to grant the permission to the addon-manager (service account open-cluster-management-hub/addon-manager-controller-sa) to address the permission issue on the hub cluster side. For the above example, if the addon-manager doesn’t have the permission to create the RoleBinding to bind the cm-admin role, users can grant the permission to the addon-manager by creating a ClusterRoleBinding like below:
```
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: addon-manager-cm-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cm-admin
subjects:
  - kind: ServiceAccount
    name: addon-manager-controller-sa
    namespace: open-cluster-management-hub
```

Create a ClusterManagementAddOn to declare this is template type addon which should be managed by the addon-manager:

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ClusterManagementAddOn
metadata:
  name: hello-template
  annotations:
    addon.open-cluster-management.io/lifecycle: "addon-manager"
spec:
  addOnMeta:
    description: hello-template is a addon built with addon template
    displayName: hello-template
  supportedConfigs: # declare it is a template type addon
  - group: addon.open-cluster-management.io
    resource: addontemplates
    defaultConfig:
      name: hello-template

Create a ManagedClusterAddOn to enable the addon on cluster1

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ManagedClusterAddOn
metadata:
  name: hello-template
  namespace: cluster1
spec:
  installNamespace: open-cluster-management-agent-addon

Use variables in the addon template

Users can use variables in the addonTemplate.agentSpec.workload.manifests field in the form of {{VARIABLE_NAME}}, it is similar to go template syntax but not identical, only String value is supported. And there are two types of variables:

built-in variables;
- constant parameters(can not be overridden by user’s variables):
  - CLUSTER_NAME: name of the managed cluster(e.g cluster1)
- default parameters(can be overridden by user’s variables)
  - HUB_KUBECONFIG: path of the kubeconfig to access the hub cluster, default value is /managed/hub-kubeconfig/kubeconfig
Customize variables; Variables defines in addonDeploymentConfig.customizedVariables can be used.

Using kubeconfig/certificates in the addon agent Deployment

The addon manager will inject volumes into the addon agent deployments and daemonsets based on the addonTemplate.spec.registration field.

If there is a KubeClient type registration, the hub kubeconfig will be injected to the deployments defined in the addon template, so users can use the hub kubeconfig located at /managed/hub-kubeconfig/kubeconfig to access the hub

...
spec:
  containers:
    - name: addon-agent
      ...
      volumeMounts:
        - mountPath: /managed/hub-kubeconfig
          name: hub-kubeconfig
  volumes:
    - name: hub-kubeconfig
      secret:
        defaultMode: 420
        secretName: <addon-name>-hub-kubeconfig
...

If there is a CustomSigner type registration, the secret signed via the custom signer defined in the CustomSignerRegistrationConfig will be injected to the deployments and daemonsets defined in the addon template, so users can use the certificate located at /managed/<signer-name>/tls.crt and /managed/<signer-name>/tls.key

...
spec:
  containers:
    - name: addon-agent
      ...
      volumeMounts:
        - mountPath: /managed/<signer-name> # if the signer name contains "/", it will be replaced by "-"
          name: cert-<signer-name>
  volumes:
    - name: cert-<signer-name> # if the signer name contains "/", it will be replaced by "-"
      secret:
        defaultMode: 420
        secretName: <addon-name>-<signer-name>-client-cert # if the signer name contains "/", it will be replaced by "-"

health probe of the template type addon

Since we only support the Deployment and DaemonSet resource as the crucial agent runtime workload, the addon-manager will check if the deployment and daemonsets are available, if not, the addon will be considered as unhealthy.

3.2 - VScode Extension

The OCM VScode Extension is a UI tool for OCM related Kubernetes resources. The extension has been built upon Visual Studio Code and offers additional OCM administrative and monitoring features in order to improve operational efficiency and accelerate development within engineering teams. The OCM VScode Extension provides tons of useful features, including easy to generate OCM related Kubernetes resources, automated local OCM environment creation and a simple and convenient monitoring view for cluster resources. Having an effective UI tool for managing your cluster is key to a healthy environment. With the right user interface, you can administer changes to the cluster, track resources, and troubleshoot issues more efficiently, without having to use numerous additional CLI tools.

Prerequisite

You must meet the following prerequisites to install the VScode Extension:

Ensure your Visual Studio Code release is at least v1.71.0.

Some VScode Extension features require a few additional prerequisites:

To Manage Existing Resources ensure kubectl is installed.
To Create Local Environment ensure kind, kubectl and clusteradm are all installed.
To load Resources Snippets or create a Bootstrap Project no additional prerequisites are needed

Installation

Install Latest version

In your VScode instance, go to the Extensions Marketplace (by clicking on the Extensions icon in the composite bar on the left hand side or simply typing Ctrl+Shift+X).

Search for OCM Extension and click install.

Install specific version

To download a specific version of VScode Extention visit the releases page, expand the Assets section of the target version, and download the version for your operating system.

In your VScode instance, go to the Extensions Marketplace (by clicking on the Extensions icon in the composite bar on the left hand side or simply typing Ctrl+Shift+X).

Click the … at the top right corner of the palette, select the Install from VSIX… action, and browes for the specific VSIX file you downloaded.

Select a VSIX file and hit install.

Usage

Resources Snippets

The VScode Extension allows users to load Custom Resource (CR) snippets from the command palette.

The resources available as snipperts are:

Subscription
Placement
Channel
ManagedClusterSet
ManagedClusterSetBinding

Subscription snippets are available specifically tailored to either Git, Helm or ObjectBucket.

To exercise this feature you must take the following steps:

Create a new yaml file in a desired path
Open the file and press Ctrl+Shift+P on your keyboard
In the search container at the top of your screen choose Snippets: Insert Snippet
Choose a desired snippet from list

Bootstrap Project

Another Valuable feature of the VScode Extension is to create a Bootstrap project for the various channel types.

The Bootstrap project is basically a collection of snippets, customized to fit one of three channel types - Git, Helm or ObjectBucket, that form a project template for you to build upon.

To exercise this feature you must take the following steps:

Press Ctrl+Shift+P on your keyboard
In the search container at the top of your screen choose OCM: Create an application-manager project
Choose a desired channel type
Type in a project name and hit enter

Create Local Environment

The Create Local Environment feature allows you to create a quick OCM multi-cluster control plane on a local kind environment with only a click of a button.

The feature does it all for you:

Verifies the existence of the required tools.
Creates as many kind clusters as you need (customizable).
Initializes the hub cluster by installing a Cluster Manager using clusteradm, and kubectl.
Installs a Klusterlet agent on the managed clusters and sends join requests to the hub cluster using clusteradm, and kubectl.
Accepts the join requests from the hub cluster using clusteradm, and kubectl.

To exercise this feature you must take the following steps:

Click on the OCM icon in the composite bar on the left hand side to open the VScode Extension control pane
Locate the Create Local Environment button in the Developers Tools section and give it a click
Notice that at the top of your screen an action container will prompt you to choose between default or custom configuration
After configuration was specified, a progress bar at the botom right of your screen will provide feadback on the build process

Manage Existing Resources

The Manage Existing Resources feature provides a wide and detailed overview of all the resources residing in any one cluster.

To exercise this feature you must take the following steps:

Access the Cluster Details tab either by: * Pressing Ctrl+Shift+P on your keyboard and choosing OCM-View: Cluster Details in the search container at the top of your screen * Clicking on the OCM icon in the composite bar on the left hand side to open the VScode Extension control pane, locating the Connected Clusters section and selecting any cluster in the list
Inside the Cluster Details tab, use the dropdown to select any cluster from the list of availabe clusters in order to view the resources within it

Visit the VScode Extension on Github.

4 - User Scenarios

4.1 - Deploy Kubernetes resources to the managed clusters

After bootstrapping an OCM environment of at least one managed clusters, now it’s time to begin your first journey of deploying Kubernetes resources into your managed clusters with OCM’s ManifestWork API.

Prerequisites

Before we get start with the following tutorial, let’s clarify a few terms we’re going to use in the context.

Cluster namespace: After a managed cluster is successfully registered into the hub. The hub registration controller will be automatically provisioning a cluster namespace dedicated for the cluster of which the name will be same as the managed cluster. The cluster namespace is used for storing any custom resources/configurations that effectively belongs to the managed cluster.
ManifestWork: A custom resource in the hub cluster that groups a list of kubernetes resources together and meant for dispatching them into the managed cluster if the ManifestWork is created in a valid cluster namespace, see details in this page.

Deploy the resource to a target cluster

Now you can deploy a set of kubernetes resources defined in files to any clusters managed by the hub cluster.

Connect to your hub cluster and you have 2 options to create a ManifestWork:

Use clusteradm command

clusteradm create work my-first-work -f <kubernetes yaml file or directory> --clusters <cluster name>

where kubernetes yaml file should be kubernetes definitions, a sample:

apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: default
  name: my-sa
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: default
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      serviceAccountName: my-sa
      containers:
        - name: nginx
          image: nginx:1.14.2
          ports:
            - containerPort: 80

Use kubectl to create the ManifestWork

kubectl apply -f <kubernetes yaml file or directory>

where kubernetes yaml file should be kubernetes definitions wrapped by ManifestWork, a sample:

apiVersion: work.open-cluster-management.io/v1
kind: ManifestWork
metadata:
  namespace: cluster1
  name: my-first-work
spec:
  workload:
    manifests:
      - apiVersion: v1
        kind: ServiceAccount
        metadata:
          namespace: default
          name: my-sa
      - apiVersion: apps/v1
        kind: Deployment
        metadata:
          namespace: default
          name: nginx-deployment
          labels:
            app: nginx
        spec:
          replicas: 3
          selector:
            matchLabels:
              app: nginx
          template:
            metadata:
              labels:
                app: nginx
            spec:
              serviceAccountName: my-sa
              containers:
                - name: nginx
                  image: nginx:1.14.2
                  ports:
                    - containerPort: 80

The above command should create a ManifestWork in cluster namespace of your hub cluster. To see the detailed status of this ManifestWork, you can run:

clusteradm get works my-first-work --cluster <cluster name>

If you have some change on the manifest files, you can apply the change to the targeted cluster by running:

clusteradm create work my-first-work -f <kubernetes yaml file or directory> --clusters <cluster name> --overwrite

To remove the resources deployed on the targeted cluster, run:

kubectl delete manifestwork my-first-work -n <cluster name>

What happens behind the scene

Say we would like to deploy a nginx together with a service account into “cluster1”. A ManifestWork can be defined as follows:

apiVersion: work.open-cluster-management.io/v1
kind: ManifestWork
metadata:
  namespace: cluster1
  name: my-first-work
spec:
  workload:
    manifests:
      - apiVersion: v1
        kind: ServiceAccount
        metadata:
          namespace: default
          name: my-sa
      - apiVersion: apps/v1
        kind: Deployment
        metadata:
          namespace: default
          name: nginx-deployment
          labels:
            app: nginx
        spec:
          replicas: 3
          selector:
            matchLabels:
              app: nginx
          template:
            metadata:
              labels:
                app: nginx
            spec:
              serviceAccountName: my-sa
              containers:
                - name: nginx
                  image: nginx:1.14.2
                  ports:
                    - containerPort: 80

In this example:

A ManifestWork named “my-first-work” will be created into a “cluster namespace” named “cluster1”.

$ kubectl get manifestwork -A --context kind-hub
NAMESPACE   NAME            AGE
cluster1    my-first-work   2m59s

The resources in the ManifestWork including a service-account, a deployment will be created to the cluster “cluster1”.

$ kubectl get deployment --context kind-cluster1
NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   3/3     3            3           4m10s

$ kubectl get sa my-sa --context kind-cluster1
NAME    SECRETS   AGE
my-sa   1         4m23s

In the status of ManifestWork we can check out the aggregated status indicating whether the prescribed resources are successfully deployed by the conditions in the field .status.conditions[*]:
- Applied: Whether all the resources from the spec are successfully applied since the last observed generation of ManifestWork.
- Available: Whether all the resources from the spec are existing in the target managed cluster.
Beside the aggregated status, the ManifestWork is also tracking the per-resource status under .status.resourceStatus[*] where we can discriminate different resource types via the .status.resourceStatus[*].resourceMeta field. e.g.:

resourceStatus:
  manifests:
  - conditions:
    - lastTransitionTime: "2021-11-25T10:17:43Z"
      message: Apply manifest complete
      reason: AppliedManifestComplete
      status: "True"
      type: Applied
    - lastTransitionTime: "2021-11-25T10:17:43Z"
      message: Resource is available
      reason: ResourceAvailable
      status: "True"
      type: Available
    resourceMeta:
      group: apps
      kind: Deployment
      name: nginx-deployment
      namespace: default
      ordinal: 1
      resource: deployments
      version: v1
  ...

If possible, you can also switch the context of your kubeconfig to “cluster1” to check out the new resources delivered by ManifestWork:

$ kubectl --context kind-cluster1 get pod
NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-556c5468f7-d5h2m   1/1     Running   0          33m
nginx-deployment-556c5468f7-gf574   1/1     Running   0          33m
nginx-deployment-556c5468f7-hhmjf   1/1     Running   0          33m

Updating the ManifestWork

Any updates applied to the ManifestWork are expected to take effect immediately as long as the work agent deployed in the managed cluster are healthy and actively in touch with the hub cluster.

The work agent will be dynamically computing a hash from the prescribed resources, and a corresponding AppliedManifestWork of which the name contains the hash value will be persisted to the managed cluster and replacing the previously persisted AppliedManifestWork connected to the same ManifestWork after the latest resources are applied.

$ kubectl --context kind-cluster1 get appliedmanifestwork
NAME                                                                                                   AGE
ed59251487ad4e4465fa2990b36a1cc398b83e63b59fa16b83591f5afdc3dd6d-my-first-work                  59m

Note that if the work agent was disconnected from the hub control plane for a period of time and missed the new updates upon ManifestWork. The work agent will be catching up the latest state of ManifestWork as soon as it re-connects.

Deleting the ManifestWork

The local resources deployed in the managed cluster should be cleaned up upon receiving the deletion event from the corresponding ManifestWork. The resource ManifestWork in the hub cluster will be protected by the finalizer named:

“cluster.open-cluster-management.io/manifest-work-cleanup”

It will be removed if the corresponding AppliedManifestWork is gracefully removed from the managed cluster. Meanwhile, the AppliedManifestWork resource is also protected by another finalizer named:

“cluster.open-cluster-management.io/applied-manifest-work-cleanup”

This finalizer is supposed to be detached after the deployed local resources are completely removed from the manged cluster. With that being said, if any deployed local resources are holding at the “Terminating” due to graceful deletion. Both of its ManifestWork and AppliedManifestWork should stay undeleted.

Troubleshoot

In case of run into any unexpected failures, you can make sure your environment by checking the following conditions:

The CRD ManifestWork is installed in the hub cluster:

$ kubectl get crd manifestworks.work.open-cluster-management.io

The CRD AppliedManifestWork is installed in the managed cluster:

$ kubectl get crd appliedmanifestworks.work.open-cluster-management.io

The work agent is successfully running in the managed cluster:

$ kubectl -n open-cluster-management-agent get pod
NAME                                             READY   STATUS    RESTARTS   AGE
klusterlet-registration-agent-598fd79988-jxx7n   1/1     Running   0          20d
klusterlet-work-agent-7d47f4b5c5-dnkqw           1/1     Running   0          20d

4.2 - Distribute workload with placement selected managed clusters

The Placement API is used to dynamically select a set of ManagedCluster in one or multiple ManagedClusterSets so that the workloads can be deployed to these clusters.

If you define a valid Placement, the placement controller generates a corresponding PlacementDecision with the selected clusters listed in the status. As an end-user, you can parse the selected clusters and then operate on the target clusters. You can also integrate a high-level workload orchestrator with the PlacementDecision to leverage its scheduling capabilities.

For example, with OCM addon policy installed, a Policy that includes a Placement mapping can distribute the Policy to the managed clusters. For details see this example.

Some popular open source projects also integrate with the Placement API. For example Argo CD, it can leverage the generated PlacementDecision to drive the assignment of Argo CD Applications to a desired set of clusters, details see this example. And KubeVela, as an implementation of the open application model, also will take advantage of the Placement API for workload scheduling.

And in this article, we want to show you how to use clusteradm to deploy ManifestWork to Placement selected clusters.

Prerequisites

Before starting with the following steps, suggest you understand the content below.

Placement: The Placement API is used to dynamically select a set of ManagedCluster in one or multiple ManagedClusterSets so that higher-level users can either replicate Kubernetes resources to the member clusters or run their advanced workload i.e. multi-cluster scheduling.
ManifestWork: A custom resource in the hub cluster that groups a list of Kubernetes resources together and is meant for dispatching them into the managed cluster if the ManifestWork is created in a valid cluster namespace.

Deploy manifestwork to placement selected managed clusters

In deploy Kubernetes resources to the managed clusters, it shows you how to use clusteradm to create a ManifestWork and deploy it onto a specific managed clusters. As Placement can dynamically select a set of ManagedCluster, the next steps will show you how clusteradm leverages placement scheduling ability and dynamically deploy ManifestWork to a set of managed clusters.

Following setup dev environment by kind to prepare an environment.

curl -sSL https://raw.githubusercontent.com/open-cluster-management-io/OCM/main/solutions/setup-dev-environment/local-up.sh | bash

Confirm there are 2 ManagedCluster and a default ManagedClusterSet created.

$ clusteradm get clusters
NAME       ACCEPTED   AVAILABLE   CLUSTERSET   CPU   MEMORY       KUBERNETES VERSION
cluster1   true       True        default      24    49265496Ki   v1.23.4
cluster2   true       True        default      24    49265496Ki   v1.23.4

$ clusteradm get clustersets
NAME      BOUND NAMESPACES   STATUS
default                      2 ManagedClusters selected

Bind the default ManagedClusterSet to default Namespace.

clusteradm clusterset bind default --namespace default

$ clusteradm get clustersets
NAME      BOUND NAMESPACES   STATUS
default   default            2 ManagedClusters selected

Note: click here to see more details about how to operate ManagedClusterSet using clusteradm.

Create a Placement placement1 to select the two clusters in default ManagedClusterSet.

cat << EOF | kubectl apply -f -
apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Placement
metadata:
  name: placement1
  namespace: default
spec:
  numberOfClusters: 2
  clusterSets:
    - default
EOF

Use clusteradm command to create ManifestWork my-first-work with Placement placement1.

clusteradm create work my-first-work -f work.yaml --placement default/placement1

The work.yaml contains kubernetes resource definitions, for sample:

$ cat work.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: default
  name: my-sa
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: default
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      serviceAccountName: my-sa
      containers:
        - name: nginx
          image: nginx:1.14.2
          ports:
            - containerPort: 80

Check the ManifestWork, it should be distributed to both cluster1 and cluster2.

$ kubectl get manifestwork -A
NAMESPACE   NAME            AGE
cluster1    my-first-work   28s
cluster2    my-first-work   28s

Update the Placement placement1 to select only one managed cluster.

kubectl patch placement placement1 --patch '{"spec": {"clusterSets": ["default"],"numberOfClusters": 1}}' --type=merge

As the placement decision changes, running below command to reschedule ManifestWork my-first-work to the newly selected cluster.
```
clusteradm create work my-first-work -f work.yaml --placement default/placement1 --overwrite
```

Check the ManifestWork again, now it’s only deployed to cluster1.

$ kubectl get manifestwork -A
NAMESPACE   NAME            AGE
cluster1    my-first-work   18m

What happens behind the scene

The main idea is that clusteradm parse the selected clusters generated by Placement, and fill in that as ManifestWork namespace. Then create the ManifestWork and it would be distributed to a set of clusters. Let’s see more details.

Placement placement1 generates a PlacementDecision placement1-decision-1.

$ kubectl get placementdecision -n default -l cluster.open-cluster-management.io/placement=placement1 -oyaml
apiVersion: v1
items:
- apiVersion: cluster.open-cluster-management.io/v1beta1
  kind: PlacementDecision
  metadata:
    creationTimestamp: "2022-07-06T15:03:12Z"
    generation: 1
    labels:
      cluster.open-cluster-management.io/placement: placement1
    name: placement1-decision-1
    namespace: default
    ownerReferences:
    - apiVersion: cluster.open-cluster-management.io/v1beta1
      blockOwnerDeletion: true
      controller: true
      kind: Placement
      name: placement1
      uid: aa339f57-0eb7-4260-8d4d-f30c1379fd35
    resourceVersion: "47679"
    uid: 9f948619-1647-429d-894d-81e11dd8bcf1
  status:
    decisions:
    - clusterName: cluster1
      reason: ""
    - clusterName: cluster2
      reason: ""
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

clusteradm get the PlacementDecision generated by Placement placement1 with label cluster.open-cluster-management.io/placement: placement1, reference code. Then parse the clusterName cluster1 and cluster2, fill in that as ManifestWork namespace, reference code. Then installs ManifestWork to namespace cluster1 and cluster2, which will finally be distributed to the two clusters.
```
$ kubectl get manifestwork -A
NAMESPACE   NAME            AGE
cluster1    my-first-work   28s
cluster2    my-first-work   28s
```

4.3 - Extend the multicluster scheduling capabilities with placement

The Placement API is used to dynamically select a set of ManagedCluster in one or multiple ManagedClusterSets so that the workloads can be deployed to these clusters. You can use placement to filter clusters by label or claim selector, also placement provides some default prioritizers which can be used to sort and select the most suitable clusters.

One of the default prioritizers are ResourceAllocatableCPU and ResourceAllocatableMemory. They provide the capability to sort clusters based on the allocatable CPU and memory. However, when considering the resource based scheduling, there’s a gap that the cluster’s “AllocatableCPU” and “AllocatableMemory” are static values that won’t change even if “the cluster is running out of resources”. And in some cases, the prioritizer needs more extra data to calculate the score of the managed cluster. For example, there is a requirement to schedule based on resource monitoring data from the cluster. For this reason, we need a more extensible way to support scheduling based on customized scores.

What is Placement extensible scheduling?

OCM placement introduces an API AddOnPlacementScore to support scheduling based on customized scores. This API supports storing the customized scores and being used by placement. Details of the API’s definition refer to types_addonplacementscore.go. An example of AddOnPlacementScore is as below.

apiVersion: cluster.open-cluster-management.io/v1alpha1
kind: AddOnPlacementScore
metadata:
  name: default
  namespace: cluster1
status:
  conditions:
  - lastTransitionTime: "2021-10-28T08:31:39Z"
    message: AddOnPlacementScore updated successfully
    reason: AddOnPlacementScoreUpdated
    status: "True"
    type: AddOnPlacementScoreUpdated
  validUntil: "2021-10-29T18:31:39Z"
  scores:
  - name: "cpuAvailable"
    value: 66
  - name: "memAvailable"
    value: 55

conditions. Conditions contain the different condition statuses for this AddOnPlacementScore.
validUntil. ValidUntil defines the valid time of the scores. After this time, the scores are considered to be invalid by placement. nil means never expire. The controller owning this resource should keep the scores up-to-date.
scores. Scores contain a list of score names and values of this managed cluster. In the above example, the API contains a list of customized scores: cpuAvailable and memAvailable.

All the customized scores information is stored in status, as we don’t expect end users to update it.

As a score provider, a 3rd party controller could run on either hub or managed cluster, to maintain the lifecycle of AddOnPlacementScore and update the score into the status.

As an end user, you need to know the resource name “default” and customized score name “cpuAvailable"and “memAvailable” , so you can specify the name in placement yaml to select clusters. For example, the below placement wants to select the top 3 clusters with the highest cpuAvailable score.

apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Placement
metadata:
  name: placement
  namespace: ns1
spec:
  numberOfClusters: 3
  prioritizerPolicy:
    mode: Exact
    configurations:
      - scoreCoordinate:
          type: AddOn
          addOn:
            resourceName: default
            scoreName: cpuAvailable
        weight: 1

In placement, if the end-user defines the scoreCoordinate type as AddOn, the placement controller will get the AddOnPlacementScore resource with the name “default” in each cluster’s namespace, read score “cpuAvailable” in the score list, and use that score to sort clusters.

You can refer to the enhancements to learn more details about the design. In the design, how to maintain the lifecycle (create/update/delete) of the AddOnPlacementScore CRs is not covered, as we expect the customized score provider itself to manage it. In this article, we will use an example to show you how to implement a 3rd part controller to update your own scores and extend the multiple clusters scheduling capability with your own scores.

How to implement a customized score provider

The example code is in GitHub repo resource-usage-collect-addon. It provides the score of the cluster’s available CPU and available memory, which can reflect the cluster’s real-time resource utilization. It is developed with OCM addon-framework and can be installed as an addon plugin to update customized scores into AddOnPlacementScore. (This article won’t talk many details about addon-framework, referring to Add-on Developer Guide to learn how to develop an addon.)

The resource-usage-collect addon follows the hub-agent architecture as below.

The resource-usage-collect addon contains a controller and an agent.

On the hub cluster, the resource-usage-collect-controller is running. It is responsible for creating the ManifestWork for resource-usage-collect-agent in each cluster namespace.
On each managed cluster, the work agent watches the ManifestWork and installs the resource-usage-collect-agent on each cluster. The resource-usage-collect-agent is the core part of this addon, it creates the AddonPlacementScore for each cluster on the Hub cluster, and refreshes the scores and validUntil every 60 seconds.

When the AddonPlacementScore is ready, the end user can specify the customized core in a placement to select clusters.

The working flow and logic of resource-usage-collect addon are quite easy to understand. Now let’s follow the below steps to get started!

Prepare an OCM environment with 2 ManagedClusters.

Following setup dev environment by kind to prepare an environment.

curl -sSL https://raw.githubusercontent.com/open-cluster-management-io/OCM/main/solutions/setup-dev-environment/local-up.sh | bash

Confirm there are 2 ManagedCluster and a default ManagedClusterSet created.

$ clusteradm get clusters
NAME       ACCEPTED   AVAILABLE   CLUSTERSET   CPU   MEMORY       KUBERNETES VERSION
cluster1   true       True        default      24    49265496Ki   v1.23.4
cluster2   true       True        default      24    49265496Ki   v1.23.4

$ clusteradm get clustersets
NAME      BOUND NAMESPACES   STATUS
default                      2 ManagedClusters selected

Bind the default ManagedClusterSet to default Namespace.

clusteradm clusterset bind default --namespace default

$ clusteradm get clustersets
NAME      BOUND NAMESPACES   STATUS
default   default            2 ManagedClusters selected

Install the resource-usage-collect addon.

Git clone the source code.

git clone git@github.com:open-cluster-management-io/addon-contrib.git
cd addon-contrib/resource-usage-collect-addon

Prepare the image.

# Set image name, this is an optional step.
export IMAGE_NAME=quay.io/haoqing/resource-usage-collect-addon:latest
# Build image
make images

If your are using kind, load image into kind cluster.

kind load docker-image $IMAGE_NAME --name <cluster_name> # kind load docker-image $IMAGE_NAME --name hub

Deploy the resource-usage-collect addon.

make deploy

Verify the installation.

On the hub cluster, verify the resource-usage-collect-controller pod is running.

$ kubectl get pods -n open-cluster-management | grep resource-usage-collect-controller
resource-usage-collect-controller-55c58bbc5-t45dh   1/1     Running   0          71s

On the hub cluster, verify the AddonPlacementScore is generated for each managed cluster.

$ kubectl get addonplacementscore -A
NAMESPACE   NAME                   AGE
cluster1    resource-usage-score   3m23s
cluster2    resource-usage-score   3m24s

The AddonPlacementScore status should contain a list of scores as below.

$ kubectl get addonplacementscore -n cluster1 resource-usage-score -oyaml
apiVersion: cluster.open-cluster-management.io/v1alpha1
kind: AddOnPlacementScore
metadata:
  creationTimestamp: "2022-08-08T06:46:04Z"
  generation: 1
  name: resource-usage-score
  namespace: cluster1
  resourceVersion: "3907"
  uid: 6c4280e4-38be-4d45-9c73-c18c84799781
status:
  scores:
  - name: cpuAvailable
    value: 12
  - name: memAvailable
    value: 4

If AddonPlacementScore is not created or there are no scores in the status, go into the managed cluster, and check if the resource-usage-collect-agent pod is running well.

$ kubectl get pods -n default | grep resource-usage-collect-agent
resource-usage-collect-agent-5b85cbf848-g5kqm   1/1     Running   0          2m

Select clusters with the customized scores.

If everything is running well, now you can try to create placement and select clusters with the customized scores.

Create a placement to select 1 cluster with the highest cpuAvailable score.

cat << EOF | kubectl apply -f -
apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Placement
metadata:
  name: placement1
  namespace: default
spec:
  numberOfClusters: 1
  clusterSets:
    - default
  prioritizerPolicy:
    mode: Exact
    configurations:
      - scoreCoordinate:
          type: AddOn
          addOn:
            resourceName: resource-usage-score
            scoreName: cpuAvailable
        weight: 1
EOF

Verify the placement decision.

$ kubectl describe placementdecision -n default | grep Status -A 3
Status:
  Decisions:
    Cluster Name:  cluster1
    Reason:

Cluster1 is selected by PlacementDecision.

Running below command to get the customized score in AddonPlacementScore and the cluster score set by Placement. You can see that the “cpuAvailable” score is 12 in AddonPlacementScore, and this value is also the cluster score in Placement events, this indicates that placement is using the customized score to select clusters.

$ kubectl get addonplacementscore -A -o=jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.status.scores}{"\n"}{end}'
cluster1        [{"name":"cpuAvailable","value":12},{"name":"memAvailable","value":4}]
cluster2        [{"name":"cpuAvailable","value":12},{"name":"memAvailable","value":4}]

$ kubectl describe placement -n default placement1 | grep Events -A 10
Events:
  Type    Reason          Age   From                 Message
  ----    ------          ----  ----                 -------
  Normal  DecisionCreate  50s   placementController  Decision placement1-decision-1 is created with placement placement1 in namespace default
  Normal  DecisionUpdate  50s   placementController  Decision placement1-decision-1 is updated with placement placement1 in namespace default
<

Documentation

1 - Concepts

1.1 - Architecture

Overview

“Hub-spoke” architecture

Modularity and extensibility

Concepts

Cluster registering: “double opt-in handshaking”

Cluster registration security model

Cluster namespace

Addons

Building blocks

Registration

Work

Placement

Application lifecycle

Governance and risk

Registration operator

1.2 - ClusterClaim

What is ClusterClaim?

Usage

Example

1.3 - ManagedCluster

What is ManagedCluster?

Cluster registration and acceptance

Bootstrapping registration

Approving registration

Cluster heartbeats and status

Cluster taints and tolerations

Taints of ManagedClusters

Tolerations of Placements

Cluster removal

Unregister from hub cluster

Unregister from the managed cluster

Managed Cluster’s certificate rotation

What’s next?

1.4 - ManagedClusterSet

What is ManagedClusterSet?

Operates ManagedClusterSet using clusteradm

Creating a ManagedClusterSet

Adding a ManagedCluster to a ManagedClusterSet

Binding the ManagedClusterSet to a workspace namespace

A glance at the “ManagedClusterSet” API

Clusterset RBAC permission control

Adding member cluster to a clusterset

Binding a clusterset to a namespace

Default ManagedClusterSet

Global ManagedClusterSet

1.5 - Placement

Overall

Select clusters in ManagedClusterSet

Predicates

Label/Claim selection

Taints/Tolerations

Prioritizers

Score-based prioritizer

Extensible scheduling

PlacementDecisions

Decision strategy

Rollout Strategy

Troubleshooting

Check the Placement conditions

Check the Placement events

Debug

Future work

1.6 - ManifestWork

What is ManifestWork

Status tracking

Fine-grained field values tracking

Garbage collection

Delete options

Resource Race and Adoption

Permission setting for work agent

Treating defaulting/immutable fields in API

Dynamic identity authorization

1.7 - ManifestWorkReplicaSet

What is ManifestWorkReplicaSet

Status tracking

Release and Enable Feature

1.8 - Add-ons

Check the `Placement` conditions

Check the `Placement` events

What is `ManifestWork`

What is `ManifestWorkReplicaSet`

What is `Multicluster Control Plane`

Why use `Multicluster Control Plane`

How to use `Multicluster Control Plane`