Provision CosmosDB in Azure using CRDs in Kubernetes (k8s)

6 min readMay 3, 2020

Recently I have created a post about how to provision CosmosDB using Terraform, here is the link for your reference, this is kind of extension to that. https://medium.com/@prashix/provision-cosmosdb-in-azure-using-terraform-54179c720872

Introduction

I want to explore further into this solution using Terraform but with more control over who creates these instances and whys/whats/hows etc., One of the solutions is to provision any cloud resource using an existing Kubernetes platform and utilize all the excellent and efficient goodies that come along with it like, CLI, API services, RBAC, security etc.,

For example, how about “kubectl apply -f cosmosdb.yaml” and “kubectl delete -f cosmosdb.yaml” sounds ? Fairly simple to [de]provision, right ?. Well, but CosmosDB isn’t one of the Kubernetes native object, to control and maintain it, similar to like that of a POD. So, how do we achieve this? …… hmm, ever heard of Custom Resource Definition (CRD) ? Well that comes to our rescue today.

Google cloud does already provides CRDs for few of GCP resources, full list and for further read on this → https://thenewstack.io/how-google-turned-kubernetes-into-a-control-plane-to-manage-gcp-resources/. Azure might follow the same suit, as this is gaining popularity in the open source community.

So, what is a Custom Resource Definition ?

It’s a feature that was added into Kubernetes which helps users, to add their own custom objects (resources) to the Kubernetes cluster and treat them like any other Kubernetes native objects. This is a very useful and powerful feature that can come very handy by applying CI/CD pipelines & IaC (infrastructure as code).

We need controller to control these custom objects which are to be created using CRDs. There are few articles to start and do everything by yourself from scratch, one of them is here → https://medium.com/@trstringer/create-kubernetes-controllers-for-core-and-custom-resources-62fc35ad64a3.

Kubeform

I generally prefer not to start working from scratch; rather to use/extend already existing work, if it’s already done, open source/reusability you get my point. Well, luckily when we were exploring we found an existing product similar to our thought process. Kubeform ( https://kubeform.com/ )

Kubeform basically provides us with various CRDs to provision cloud resources on AWS, Azure, GCP, using terraform internally. Here is their blog on how Kubeform works → https://blog.byte.builders/post/introducing-kubeform/. The picture below from that blog, well, summarizes what it does basically.

Demo

Today, I will show you how to use kubeform to provision our CosmosDbAccount in Azure using Kubernetes on GCP, GKE (Google Kubernetes Engine). Start by provisioning a simple basic Kubernetes cluster on GKE. Login to your gcp cloud using your shell (install google cloud sdk) and just run

gcloud container clusters create cluster-1 --zone us-central1-a

This will provision a simple k8s cluster with 3 nodes (n1-standard-1). Please note this will cost you, although not much, use a free trial if available.

Once the cluster comes up, run the following command to get the kube config on to your local to communicate with you new cluster. You need to give your user cluster admin role (optional, if you are using same user).

gcloud container clusters get-credentials cluster-1 --zone us-central1-a(optional)
kubectl create clusterrolebinding "cluster-admin-sample" \
--clusterrole=cluster-admin --user="<your gcp email id>"

We need to first install Kubeform controller. We can do this using Helm chart. Install Helm → https://helm.sh/docs/intro/install/

helm repo add appscode https://charts.appscode.com/stable/helm repo updatehelm install kfc appscode/kubeform --version v0.1.0 --namespace kube-system

This will create kfc (kubeform controller) in the kube-system (optional) namespace. In the image the controller starts as ‘kfc-kubeform-*’

Now, let’s apply the required CRD for our cosmosDbAccount creation. Its available in kubeform Github (https://github.com/kubeform/kubeform).

kubectl apply -f https://raw.githubusercontent.com/kubeform/kubeform/master/api/crds/azurerm.kubeform.com_cosmosdbaccounts.yaml

Time to provide our Azure information to be used by terraform; to provision our cosmosDb. We need to provide this information using a secret. Please find the template below.

apiVersion: v1
kind: Secret
metadata:
  name: azure
type: "kubeform.com/azurerm"
data:
  version: PTEuNDMuMAo= # =1.43.0
  subscription_id: <Base64 encoded value of Azure Subscription ID>
  client_id: <Base64 encoded value of Azure Client ID>
  client_secret: <Base64 encoded value of Azure Client Secret>
  tenant_id: <Base64 encoded value of Azure Tenant ID>

Use base64 (https://www.base64encode.org/) to encode your subscription_id and tenant_id, I have mentioned on how to get this information from previous blog, link provided in the beginning. Now, to get your client_id and client_secret, follow this blog. https://community.microfocus.com/t5/Identity-Manager-Tips/Creating-the-application-Client-ID-and-Client-Secret-from/ta-p/1776619

Once you create the secret.yaml like the one above, please apply it to your new cluster. Please note the important key ‘version’, which tells the terraform which azure provider version to use. I tested with 1.43 as its recommended by Kubeform person.

kubectl apply -f secret.yaml

Now, time to create our cosmosDbAccount.yaml file. I have created one, so you can borrow this template.

The important key here is the providerRef: azure, this basically tells the controller to get the details from the secret named “azure”.

Second one, is resourceGroupName: dev, please create this resource group manually from portal.azure.com. Or if you are feeling like exploring you have a CRD for creating one :). https://github.com/kubeform/kubeform/blob/master/api/crds/azurerm.kubeform.com_resourcegroups.yaml

Rest are standard keys useful for cosmosDbAccount creation using terraform module.

Example: https://www.terraform.io/docs/providers/azurerm/r/cosmosdb_account.html

apiVersion: azurerm.kubeform.com/v1alpha1
kind: CosmosdbAccount
metadata:
  name: db-cosmos
spec:
  name: db-cosmos-sample-prashix
  resourceGroupName: dev
  location: East US
  offerType: Standard
  kind: GlobalDocumentDB
  consistencyPolicy:
  - consistencyLevel: Session
  geoLocation:
  - location: East US
    failoverPriority: 0
  providerRef:
    name: azure

After applying this yaml we can look at the logs from the controller to debug or get the information about our account information. Run this for logs.

kubectl apply -f cosmosDbAccount.yamlkubectl logs -f <kfc-kubeform.... pod name> -n kube-system

Conclusion

In short, it’s a very simple process, yet, look at the power, we just created a NoSql DB resource from one Cloud provider (GCP) to another (Azure) using a Kubernetes platform (GKE).

This methodology comes handy for many companies, who might be using different cloud providers or in the middle of a transition. Overall, now we can govern many security policies on how/who can create any resources on any cloud provider as long as you are using Kubernetes as a platform.

So, in this solution, we are basically using the efficiency of Terraform combined with yet another powerful tool Kubernetes.

Next steps, would be to lock the features of Terraform and pick only what we need along with implementing a security policy on the CRD resource manifest like what params can be set/allowed and what aren’t.

Oops, almost forget, please delete your cluster (GCP) and also resource group (Azure) from GUI or command line. You can do this gracefully by first deleting the resource, or just go nuts :).

kubectl delete -f <cosmosDbaccount.yaml> and then once deleted confirm from logs, gcloud container clusters delete cluster-1 --zone us-central1-agcloud container clusters list (to confirm you have zero clusters)

Hope this solution helps you, let me know what you think and if there are any other tools like Kubeform which does implement similar solution, please let me know. Thank you.