Distribute workload with placement selected managed clusters

The Placement API is used to dynamically select a set of ManagedCluster in one or multiple ManagedClusterSets so that the workloads can be deployed to these clusters.

If you define a valid Placement, the placement controller generates a corresponding PlacementDecision with the selected clusters listed in the status. As an end-user, you can parse the selected clusters and then operate on the target clusters. You can also integrate a high-level workload orchestrator with the PlacementDecision to leverage its scheduling capabilities.

For example, with OCM addon policy installed, a Policy that includes a Placement mapping can distribute the Policy to the managed clusters. For details see this example.

Some popular open source projects also integrate with the Placement API. For example Argo CD, it can leverage the generated PlacementDecision to drive the assignment of Argo CD Applications to a desired set of clusters, details see this example. And KubeVela, as an implementation of the open application model, also will take advantage of the Placement API for workload scheduling.

And in this article, we want to show you how to use clusteradm to deploy ManifestWork to Placement selected clusters.

Prerequisites

Before starting with the following steps, suggest you understand the content below.

• Placement: The Placement API is used to dynamically select a set of ManagedCluster in one or multiple ManagedClusterSets so that higher-level users can either replicate Kubernetes resources to the member clusters or run their advanced workload i.e. multi-cluster scheduling.

• ManifestWork: A custom resource in the hub cluster that groups a list of Kubernetes resources together and is meant for dispatching them into the managed cluster if the ManifestWork is created in a valid cluster namespace.

Deploy manifestwork to placement selected managed clusters

In deploy Kubernetes resources to the managed clusters, it shows you how to use clusteradm to create a ManifestWork and deploy it onto a specific managed clusters. As Placement can dynamically select a set of ManagedCluster, the next steps will show you how clusteradm leverages placement scheduling ability and dynamically deploy ManifestWork to a set of managed clusters.

1. Following setup dev environment by kind to prepare an environment.

   curl -sSL https://raw.githubusercontent.com/open-cluster-management-io/OCM/main/solutions/setup-dev-environment/local-up.sh | bash
   

2. Confirm there are 2 ManagedCluster and a default ManagedClusterSet created.

   $ clusteradm get clusters
   NAME       ACCEPTED   AVAILABLE   CLUSTERSET   CPU   MEMORY       KUBERNETES VERSION
   cluster1   true       True        default      24    49265496Ki   v1.23.4
   cluster2   true       True        default      24    49265496Ki   v1.23.4
   
   $ clusteradm get clustersets
   NAME      BOUND NAMESPACES   STATUS
   default                      2 ManagedClusters selected
   

3. Bind the default ManagedClusterSet to default Namespace.

   clusteradm clusterset bind default --namespace default
   

   $ clusteradm get clustersets
   NAME      BOUND NAMESPACES   STATUS
   default   default            2 ManagedClusters selected
   

   Note: click here to see more details about how to operate ManagedClusterSet using clusteradm.

4. Create a Placement placement1 to select the two clusters in default ManagedClusterSet.

   cat << EOF | kubectl apply -f -
   apiVersion: cluster.open-cluster-management.io/v1beta1
   kind: Placement
   metadata:
     name: placement1
     namespace: default
   spec:
     numberOfClusters: 2
     clusterSets:
       - default
   EOF
   

5. Use clusteradm command to create ManifestWork my-first-work with Placement placement1.

   clusteradm create work my-first-work -f work.yaml --placement default/placement1
   

   The work.yaml contains kubernetes resource definitions, for sample:

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     namespace: default
     name: my-sa
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     namespace: default
     name: nginx-deployment
     labels:
       app: nginx
   spec:
     replicas: 3
     selector:
       matchLabels:
         app: nginx
     template:
       metadata:
         labels:
           app: nginx
       spec:
         serviceAccountName: my-sa
         containers:
           - name: nginx
             image: nginx:1.14.2
             ports:
               - containerPort: 80
   

6. Check the ManifestWork, it should be distributed to both cluster1 and cluster2.

   $ kubectl get manifestwork -A
   NAMESPACE   NAME            AGE
   cluster1    my-first-work   28s
   cluster2    my-first-work   28s
   

7. Update the Placement placement1 to select only one managed cluster.

   kubectl patch placement placement1 --patch '{"spec": {"clusterSets": ["default"],"numberOfClusters": 1}}' --type=merge
   

8. As the placement decision changes, running below command to reschedule ManifestWork my-first-work to the newly selected cluster.

   clusteradm create work my-first-work -f work.yaml --placement default/placement1 --overwrite
   

9. Check the ManifestWork again, now it’s only deployed to cluster1.

   $ kubectl get manifestwork -A
   NAMESPACE   NAME            AGE
   cluster1    my-first-work   18m
   

What happens behind the scene

The main idea is that clusteradm parse the selected clusters generated by Placement, and fill in that as ManifestWork namespace. Then create the ManifestWork and it would be distributed to a set of clusters. Let’s see more details.

1. Placement placement1 generates a PlacementDecision placement1-decision-1.

   $ kubectl get placementdecision -n default -l cluster.open-cluster-management.io/placement=placement1 -oyaml
   apiVersion: v1
   items:
   - apiVersion: cluster.open-cluster-management.io/v1beta1
     kind: PlacementDecision
     metadata:
       creationTimestamp: "2022-07-06T15:03:12Z"
       generation: 1
       labels:
         cluster.open-cluster-management.io/placement: placement1
       name: placement1-decision-1
       namespace: default
       ownerReferences:
       - apiVersion: cluster.open-cluster-management.io/v1beta1
         blockOwnerDeletion: true
         controller: true
         kind: Placement
         name: placement1
         uid: aa339f57-0eb7-4260-8d4d-f30c1379fd35
       resourceVersion: "47679"
       uid: 9f948619-1647-429d-894d-81e11dd8bcf1
     status:
       decisions:
       - clusterName: cluster1
         reason: ""
       - clusterName: cluster2
         reason: ""
   kind: List
   metadata:
     resourceVersion: ""
     selfLink: ""
   

2. clusteradm get the PlacementDecision generated by Placement placement1 with label cluster.open-cluster-management.io/placement: placement1, reference code. Then parse the clusterName cluster1 and cluster2, fill in that as ManifestWork namespace, reference code. Then installs ManifestWork to namespace cluster1 and cluster2, which will finally be distributed to the two clusters.

   $ kubectl get manifestwork -A
   NAMESPACE   NAME            AGE
   cluster1    my-first-work   28s
   cluster2    my-first-work   28s