Grafana CloudWatch Datasource IRSA for Kubernetes using Terraform

This tutorial will show you how to configure a AWS IAM Role Service Account for your Kubernetes Cluster on AWS EKS, in order for your Grafana Deployment to access AWS CloudWatch using its Service Account and we will be deploying it with Terraform so that there are no manual changes.

Assumptions

I already have a EKS Cluster provisioned through Terraform, which looks more or less like this:

main.tf

module "eks" {
  source       = "terraform-aws-modules/eks/aws"
  version      = "~> v20.0.0"
  cluster_name = "test-eks-cluster"
  ...
}

I have deployed a kube-prometheus-stack helm chart using the helm terraform provider (you can use whatever implementation you like), which looks like this:

main.tf

resource "helm_release" "prometheus" {
    name       = "prometheus"
    repository = "https://prometheus-community.github.io/helm-charts"
    chart      = "kube-prometheus-stack"
    version    = "56.9.0"
    namespace  = "prometheus"

    values = [templatefile("${path.module}/values.yaml.tpl", {
      service_account_name = "prometheus-grafana"
    })]

}

Then in my values.yaml.tpl:

values.yaml.tpl

grafana:
  enabled: true
  serviceAccount:
    create: true
    name: "${service_account_name}"

Implement IAM Role Service Account

First we need to define the IAM Role, define Trust Relationship and associate the IAM Permission Policy:

main.tf

data "aws_caller_identity" "current" {}

locals {
  oidc_id = element(split("/", module.eks.oidc_provider_arn), length(split("/", module.eks.oidc_provider_arn)) - 1)
}

resource "aws_iam_role" "grafana_cloudwatch_iam_role" {
  name = var.grafana_cloudwatch_role_name
  
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = "sts:AssumeRoleWithWebIdentity"
        Principal = {
          Federated = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/oidc.eks.eu-west-1.amazonaws.com/id/${local.oidc_id}"
        }
        Condition = {
          StringEquals = {
            "oidc.eks.eu-west-1.amazonaws.com/id/${local.oidc_id}:sub" = "system:serviceaccount:prometheus:prometheus-grafana",
            "oidc.eks.eu-west-1.amazonaws.com/id/${local.oidc_id}:aud" = "sts.amazonaws.com"
          }
        }
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "grafana_cloudwatch_permissions" {
  role       = aws_iam_role.grafana_cloudwatch_iam_role.name
  policy_arn = "arn:aws:iam::aws:policy/CloudWatchReadOnlyAccess"
}

It's important to note that in our condition with system:serviceaccount:prometheus:prometheus-grafana, it breaks up into:

system:serviceaccount:<namespace>:<service-account-name>

It needs to match that exactly that namespace and service account.

Now to add the additions to our release and values template, we will define the value of aws_assume_role_arn so that it can be rendered in our values yaml:

main.tf

resource "helm_release" "prometheus" {
    name       = "prometheus"
    repository = "https://prometheus-community.github.io/helm-charts"
    chart      = "kube-prometheus-stack"
    version    = "56.9.0"
    namespace  = "prometheus"

    values = [templatefile("${path.module}/values.yaml.tpl", {
      service_account_name = "prometheus-grafana"
      aws_assume_role_arn  = aws_iam_role.grafana_cloudwatch_role.arn
    })]

}

The additions for our values.yaml.tpl:

values.yaml.tpl

grafana:
  enabled: true
  serviceAccount:
    create: true
    name: "${service_account_name}"
    annotations: 
      eks.amazonaws.com/role-arn: "${aws_assume_role_arn}"

  additionalDataSources:
    - name: CloudWatch
      type: cloudwatch
      jsonData:
        authType: default
        defaultRegion: eu-west-1

Once you have deployed the changes you can head over to your Grafana Datasources and test CloudWatch, and you should be able to communicate with the CloudWatch API's via your IAM Role Service Account from your Grafana Pods.

Thank You

Thanks for reading, if you like my content, feel free to check out my website, and subscrube to my newsletter or follow me at @ruanbekker on Twitter.

Linktree: https://go.ruan.dev/links
Patreon: https://go.ruan.dev/patreon