ManifestWork is used to define a group of Kubernetes resources on the hub to be applied to the managed cluster. In the open-cluster-management project, a
ManifestWork resource must be created in the cluster namespace. A work agent implemented in work project is run on the managed cluster and monitors the
ManifestWork resource in the cluster namespace on the hub cluster.
An example of
ManifestWork to deploy a deployment to the managed cluster is shown in the following example.
apiVersion: work.open-cluster-management.io/v1 kind: ManifestWork metadata: namespace: <target managed cluster> name: hello-work-demo spec: workload: manifests: - apiVersion: apps/v1 kind: Deployment metadata: name: hello namespace: default spec: selector: matchLabels: app: hello template: metadata: labels: app: hello spec: containers: - name: hello image: quay.io/asmacdo/busybox command: ["sh", "-c", 'echo "Hello, Kubernetes!" && sleep 3600']
Work agent will track all the resources defined in
ManifestWork and update its status. There are two types of status in manifestwork. The
resourceStatus tracks the status of each manifest in the
conditions reflects the overall status of the
ManifestWork. Work agent currently checks whether a resource is
Available, meaning the resource exists on the managed cluster, and
Applied means the resource defined in
ManifestWork has been applied to the managed cluster.
Here is an example.
apiVersion: work.open-cluster-management.io/v1 kind: ManifestWork metadata: ... spec: ... status: conditions: - lastTransitionTime: "2021-06-15T02:26:02Z" message: Apply manifest work complete reason: AppliedManifestWorkComplete status: "True" type: Applied - lastTransitionTime: "2021-06-15T02:26:02Z" message: All resources are available reason: ResourcesAvailable status: "True" type: Available resourceStatus: manifests: - conditions: - lastTransitionTime: "2021-06-15T02:26:02Z" message: Apply manifest complete reason: AppliedManifestComplete status: "True" type: Applied - lastTransitionTime: "2021-06-15T02:26:02Z" message: Resource is available reason: ResourceAvailable status: "True" type: Available resourceMeta: group: apps kind: Deployment name: hello namespace: default ordinal: 0 resource: deployments version: v1
Fine-grained field values tracking
Optionally, we can let the work agent aggregate and report certain fields from
the distributed resources to the hub clusters by setting
apiVersion: work.open-cluster-management.io/v1 kind: ManifestWork metadata: ... spec: workload: ... manifestConfigs: - resourceIdentifier: group: apps resource: deployments namespace: default name: hello feedbackRules: - type: WellKnownStatus - type: JSONPaths jsonPaths: - name: isAvailable path: '.status.conditions[?(@.type=="Available")].status'
The feedback rules prescribe the work agent to periodically get the latest
states of the resources, and scrape merely those expected fields from them,
which is helpful for trimming the payload size of the status. Note that the
collected feedback values on the
ManifestWork will not be updated unless
the latest value is changed/different from the previous recorded value.
Currently, it supports two kinds of
WellKnownStatus: Using the pre-built template of feedback values for those well-known kubernetes resources.
JSONPaths: A valid Kubernetes JSON-Path that selects a scalar field from the resource. Currently supported types are Integer, String and Boolean.
The default feedback value scraping interval is 30 second, and we can override
it by setting
--status-sync-interval on your work agent. Too short period can
cause excessive burden to the control plane of the managed cluster, so generally
a recommended lower bound for the interval is 5 second.
In the end, the scraped values from feedback rules will be shown in the status:
apiVersion: work.open-cluster-management.io/v1 kind: ManifestWork metadata: ... spec: ... status: resourceStatus: manifests: - conditions: ... resourceMeta: ... statusFeedback: values: - fieldValue: integer: 1 type: Integer name: ReadyReplicas - fieldValue: integer: 1 type: Integer name: Replicas - fieldValue: integer: 1 type: Integer name: AvailableReplicas - fieldValue: string: "True" type: String name: isAvailable
To ensure the resources applied by
ManifestWork are reliably recorded, the work agent creates an
AppliedManifestWork on the managed cluster for each
ManifestWork as an anchor for resources relating to
ManifestWork is deleted, work agent runs a
Foreground deletion, that
ManifestWork will stay in deleting state until all its related resources has been fully cleaned in the managed cluster.
User can explicitly choose not to garbage collect the applied resources when a
ManifestWork is deleted. the user should specify the
deleteOption in the
ManifestWork. By default,
deleteOption is set as
which means the applied resources on the spoke will be deleted with the removal of
ManifestWork. User can set it to
Orphan so the applied resources will not be deleted. Here is an example:
apiVersion: work.open-cluster-management.io/v1 kind: ManifestWork metadata: ... spec: workload: ... deleteOption: propagationPolicy: Orphan
Alternatively, user can also specify a certain resource defined in the
ManifestWork to be orphaned by setting the
deleteOption to be
SelectivelyOrphan. Here is an example with
SelectivelyOrphan specified. It ensures the removal of deployment resource specified in the
ManifestWork while the service resource is kept.
apiVersion: work.open-cluster-management.io/v1 kind: ManifestWork metadata: name: selective-delete-work spec: workload: ... deleteOption: propagationPolicy: SelectivelyOrphan selectivelyOrphans: orphaningRules: - group: "" resource: services namespace: default name: helloworld
Resource Race and Adoption
It is possible to create two
ManifestWorks for the same cluster with the same resource defined.
For example, the user can create two
Manifestworks on cluster1, and both
Manifestworks have the
hello in default namespace. If the content of the resource is different, the
ManifestWorks will fight and it is desired since each
ManifestWork is treated as equal and
ManifestWork is declaring the ownership of the resource.
When one of the
ManifestWork is deleted, the applied resource will not be removed no matter
DeleteOption is set or not. The remaining
ManifestWork will still keep the ownership of the resource.