---
title: "Monitoring your EKS clusters audit logs"
linktitle: "Monitoring your EKS clusters audit logs"
date: 2023-05-30
author: Thomas Labarussias
slug: k8saudit-eks-plugin
tags: ["Falco Plugins","Cloud"]
---

{{% pageinfo color="info" %}}
This blog post is an update of a post of November 2022
{{% /pageinfo %}}

At the beginning of the year 2022, Falco introduced a game changing feature, the [Falco Plugins](https://org/blog/announcing-plugins/). They allow Falco to monitor and trigger alerts for any kind of event. Since the launch of the new plugin framework the Falco community has collaborated to create plugins for [Github](https://github.com/falcosecurity/plugins/tree/master/plugins/github), [AWS CloudTrail](https://github.com/falcosecurity/plugins/tree/master/plugins/cloudtrail) and [Okta](https://github.com/falcosecurity/plugins/tree/master/plugins/okta). A plugin has also replaced the way Falco consumes the Audit Logs generated by a K8s API server through a [dedicated plugin](https://github.com/falcosecurity/plugins/tree/master/plugins/k8saudit). With these plugins, Falco covers more in depth the aspects of your infrastructure and allows you to use a single syntax for rules.

For months (okay, maybe years...), our adopters have asked us for a way to monitor [K8s Audit Logs](https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/). The previous implementation used an internal web server to receive the logs from the Kubernetes API, although it was functional, it was a very manual process to install and manage clusters. This method didn't support clusters managed by cloud providers, such as EKS, AKS, or GKE as they had to capture the Audit Logs for their own usage and then add them to their log aggregators.

This situation is now solved thanks to the plugin framework and we're proud to announce the first release of the plugin for EKS Audit Logs: [`k8saudit-eks`](https://github.com/falcosecurity/plugins/tree/master/plugins/k8saudit-eks)!!!

## How it works

AWS captures the Audit Logs and exposes them in the [CloudWatch Logs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html) service. We have made available libs to create a clean session with the AWS API and pull logs from the relevant Cloudwatch Logs Stream. You can reuse these [libs](https://github.com/falcosecurity/plugins/tree/master/shared/go/aws) for any plugin you'd like to create for any Amazon service.

## Usage

The configuration for the usage of the plugin is:

```yaml
plugins:
  - name: k8saudit-eks
    library_path: libk8saudit-eks.so
    init_config:
      region: "{REGION}"
      profile: "{AWS_PROFILE}"
      shift: 10
      polling_interval: 10
      use_async: false
      buffer_size: 500
    open_params: "{CLUSTER_NAME}"
  - name: json
    library_path: libjson.so
    init_config: ""

load_plugins: [k8saudit-eks, json]
```

With:
* `init_config`:
  * **profile**: The Profile to use to create the session, env var AWS_PROFILE if present
  * **region**: The Region of your EKS cluster, env var AWS_REGION is used if present
  * **use_async**: If true then async extraction optimization is enabled (Default: true)
  * **polling_interval**: Polling Interval in seconds (default: 5s)
  * **shift**: Time shift in past in seconds (default: 1s)
  * **buffer_size**: Buffer Size (default: 200)
* `open_params`: A string which contains the name of your EKS Cluster (required).

If you run Falco inside an EKS cluster with a setup of an OIDC provider, the **profile** and **region** parameters can be omitted in favor of a `service account` + IAM Role (see [the official docs](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html)).

## IAM

Whatever the method you use for the authentication to the AWS API, you need to set up this minimal policy to your user/role:

```yaml
{
  "Version":"2012-10-17",
  "Statement":[
    {
      "Sid":"ReadAccessToCloudWatchLogs",
      "Effect":"Allow",
      "Action":[
        "logs:Describe*",
        "logs:FilterLogEvents",
        "logs:Get*",
        "logs:List*"
      ],
      "Resource":[
        "arn:aws:logs:${REGION}:${ACCOUNT_ID}:log-group:/aws/eks/${CLUSTER_NAME}/cluster:*"
      ]
    }
  ]
}
```

With:
* `REGION`: The Region of your EKS cluster
* `ACCOUNT_ID`: The ID of the account running the EKS cluster
* `CLUSTER_NAME`: The name of your cluster EKS cluster (same value than in your plugin configuration)

## Default Rules

A good thing about Kubernetes is that it brings standards into our industry. Despite a few differences, the cluster works in the same way and produces the same format of logs. This helps us enormously. By creating the [k8saudit plugin](https://github.com/falcosecurity/plugins/tree/master/plugins/k8saudit) we declared the fields to extract, as well as some default rules, which we can reuse for any plugin that consumes the same Audit Logs. It is a time saver for both, developers and adopters.

You can find the proposed default rules [here](https://github.com/falcosecurity/plugins/blob/master/plugins/k8saudit/rules/k8s_audit_rules.yaml).

## Installation

For the installation, we'll use [`falcoctl`](https://github.com/falcosecurity/falcoctl). We can use it as a CLI tool for a locale installation or as a sidecar for Kubernetes. See [this blog post](https://org/blog/falcoctl-install-manage-rules-plugins/) for more information.

### Local installation

In this example, we'll see how to install the `k8saudit-eks` in a local host.

First, add the index of artifacts for `falcoctl`:
```shell
sudo falcoctl index add falcosecurity https://falcosecurity.github.io/falcoctl/index.yaml
```
Install the latest version of the plugins:
```shell
sudo falcoctl artifact install k8saudit-eks
```
```shell
sudo falcoctl artifact install json
```
As the output notices, the plugins will be now available in `/usr/share/plugins`:
```shell
 INFO  Reading all configured index files from "/root/.config/falcoctl/indexes.yaml"
 INFO  Resolving dependencies ...
 INFO  Installing the following artifacts: [ghcr.io/falcosecurity/plugins/plugin/k8saudit-eks:latest]
 INFO  Preparing to pull "ghcr.io/falcosecurity/plugins/plugin/k8saudit-eks:latest"
 INFO  Pulling b2daf90a878e: ############################################# 100%
 INFO  Pulling 60b0846d1e18: ############################################# 100%
 INFO  Pulling 12c7f0f5f00e: ############################################# 100%
 INFO  Artifact successfully installed in "/usr/share/plugins"
```
```shell
 INFO  Reading all configured index files from "/root/.config/falcoctl/indexes.yaml"
 INFO  Resolving dependencies ...
 INFO  Installing the following artifacts: [ghcr.io/falcosecurity/plugins/plugin/json:latest]
 INFO  Preparing to pull "ghcr.io/falcosecurity/plugins/plugin/json:latest"
 INFO  Pulling 087dba2e76d6: ############################################# 100%
 INFO  Pulling c4cb35bb528a: ############################################# 100%
 INFO  Pulling 9b50ea237bc6: ############################################# 100%
 INFO  Artifact successfully installed in "/usr/share/plugins"
```

After installing the `k8saudit-eks` plugin, we also need to install the rules that it uses. (Those rules are the same as the `k8saudit` plugin rules.):
```shell
sudo falcoctl artifact install k8saudit-rules --resolve-deps=false
```
And we get the rules in `/etc/falco`:
```shell
 INFO  Reading all configured index files from "/root/.config/falcoctl/indexes.yaml"
 INFO  Installing the following artifacts: [ghcr.io/falcosecurity/plugins/ruleset/k8saudit:latest]
 INFO  Preparing to pull "ghcr.io/falcosecurity/plugins/ruleset/k8saudit:latest"
 INFO  Pulling b8c6b1def3eb: ############################################# 100%
 INFO  Pulling 6b6f3231f2b2: ############################################# 100%
 INFO  Pulling 91b2b7a9944d: ############################################# 100%
 INFO  Artifact successfully installed in "/etc/falco"
```

{{% pageinfo color="info" %}}
By default, the installation of a ruleset comes with the associated plugins. In this blog, we have installed them on our own, this is why `--resolve-deps` is set to false.
{{% /pageinfo %}}

We now have the plugins and rules in our system. The last step is to configure Falco to use them.
Modify your `/etc/yaml` to make it looks like the following (with the relevant values for `REGION`, `ACCOUNT_ID`, and `CLUSTER_NAME`): 

```yaml
rules_file:
  ...
  - /etc/k8s_audit_rules.yaml

plugins:
  - name: k8saudit-eks
    library_path: libk8saudit-eks.so
    init_config:
      region: "{REGION}"
      profile: "{AWS_PROFILE}"
      shift: 10
      polling_interval: 10
      use_async: false
      buffer_size: 500
    open_params: "{CLUSTER_NAME}"
  - name: json
    library_path: libjson.so
    init_config: ""

load_plugins: ["k8saudit-eks","json"]
...
```

You can now start Falco ( `sudo /usr/bin/falco --disable-source syscall`) and monitor the audit logs of your EKS cluster:
```shell
Wed May 10 14:07:18 2023: Falco version: 0.34.1 (x86_64)                                                                                                                                                    
Wed May 10 14:07:18 2023: Falco initialized with configuration file: /etc/yaml                                                                                                                  
Wed May 10 14:07:18 2023: Loading plugin 'k8saudit-eks' from file /usr/share/plugins/libk8saudit-eks.so                                                                                               
Wed May 10 14:07:18 2023: Loading plugin 'json' from file /usr/share/plugins/libjson.so                                                                                                               
Wed May 10 14:07:18 2023: Loading rules from file /etc/k8s_audit_rules.yaml                                                                                                                           
Wed May 10 14:07:18 2023: Loading rules from file /etc/rules.d/override-k8saudit.yaml                                                                                                                 
Wed May 10 14:07:18 2023: Starting health webserver with threadiness 4, listening on port 8765                                                                                                             
Wed May 10 14:07:18 2023: Enabled event sources: k8s_audit                                                                                                                                                  
Wed May 10 14:07:18 2023: Opening capture with plugin 'k8saudit-eks'
```

{{% pageinfo color="info" %}}
In this example, we also disable the collection of the syscalls with the option `--disable-source syscall`. The `k8saudit-eks` plugin works with a `pull` model, you should have only one Falco instance collecting the audit logs to avoid any duplication of alerts. We dedicate that instance to that role.
{{% /pageinfo %}}

And get alerts:
```shell
14:09:17.699479000: Informational K8s Serviceaccount Created (user=system:node:ip-192-168-34-135.ec2.internal serviceaccount=default ns=default resource=serviceaccounts resp=201 decision=allow reason=)
14:09:24.046826000: Notice Attach/Exec to pod (user=kubernetes-admin pod=kubecon resource=pods ns=default action=exec command=sh)
```

### Kubernetes installation

Next, we will see how to install the `k8saudit-eks` plugin with the `falcoctl` tool in a Kubernetes setup. We'll use the official Helm chart with the following `values.yaml` file:
```yaml
serviceAccount:
  create: true
  annotations:
    - eks.amazonaws.com/role-arn: {ROLE_ARN}

falco:
  rules_file:
    - /etc/falco/k8s_audit_rules.yaml
    - /etc/falco/rules.d
  plugins:
    - name: k8saudit-eks
      library_path: libk8saudit-eks.so
      init_config:
        shift: 10
        polling_interval: 10
        use_async: false
        buffer_size: 500
      open_params: "{CLUSTER_NAME}"
    - name: json
      library_path: libjson.so
      init_config: ""
  load_plugins: [k8saudit-eks, json]

falcosidekick:
  enabled: true

driver:
  enabled: false
collectors:
  enabled: false

controller:
  kind: deployment
  deployment:
    replicas: 1

falcoctl:
  indexes:
  - name: falcosecurity
    url: https://falcosecurity.github.io/falcoctl/index.yaml
  artifact:
    install:
      enabled: true
    follow:
      enabled: true
  config:
    artifact:
      allowedTypes:
        - plugin
        - rulesfile
      install:
        resolveDeps: false
        refs: [k8saudit-rules:0, k8saudit-eks:0, json:0]
      follow:
        refs: [k8saudit-rules:0]
```

Let's look into this configuration section by section:
```yaml
serviceAccount:
  create: true
  annotations:
    - eks.amazonaws.com/role-arn: {ROLE_ARN}
```
With EKS, we can use an OIDC provider and annotations on the service accounts to allow access to the AWS API ( see [the official docs](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html)). This section allows you to set which `ROLE_ARN` to use.

```yaml
falco:
  rules_file:
    - /etc/falco/k8s_audit_rules.yaml
    - /etc/falco/rules.d
  plugins:
    - name: k8saudit-eks
      library_path: libk8saudit-eks.so
      init_config:
        shift: 10
        polling_interval: 10
        use_async: false
        buffer_size: 500
      open_params: "{CLUSTER_NAME}"
    - name: json
      library_path: libjson.so
      init_config: ""
  load_plugins: [k8saudit-eks, json]
```

As the local installation, this section sets the used plugins.
```yaml
driver:
  enabled: false
collectors:
  enabled: false

controller:
  kind: deployment
  deployment:
    replicas: 1
```

{{% pageinfo color="warning" %}}
To collect and monitor all the syscalls, Falco must be deployed as a DaemontSet in your cluster, to have one instance per Kernel. In this example, we're deploying Falco with the `k8saudit-eks` plugin, which relies on a `pull` model. To avoid several Falco pods to collect the same audit logs and duplicate the alerts, we MUST install Falco with the `k8saudit-eks` plugin only once. The Helm chart allows to use a one replica deployment and to disable the syscalls and K8s metdadata collections, this is the purpose of this section.
{{% /pageinfo %}}

```yaml
falcoctl:
  indexes:
  - name: falcosecurity
    url: https://falcosecurity.github.io/falcoctl/index.yaml
  artifact:
    install:
      enabled: true
    follow:
      enabled: true
  config:
    artifact:
      allowedTypes:
        - plugin
        - rulesfile
      install:
        resolveDeps: false
        refs: [k8saudit-rules:0, k8saudit-eks:0, json:0]
      follow:
        refs: [k8saudit-rules:0]
```

The last section sets up `falcoctl`. It will install the requested plugins and rules and track new versions of the rules.


Now you can run your helm install command:
```shell
helm install falco-k8saudit-eks -n falco falcosecurity/falco -f ./values.yml
```

And see the result:
```shell
$ kubectl get pods -n falco

NAME                                      READY   STATUS    RESTARTS        AGE
falco-k8saudit-eks-96d7f6f99-vwjgc        2/2     Running   0               70m
```
```shell
$ kubectl logs falco-k8saudit-eks-96d7f6f99-vwjgc -n falco -c falco
Wed May 10 14:07:18 2023: Falco version: 0.34.1 (x86_64)
Wed May 10 14:07:18 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
Wed May 10 14:07:18 2023: Loading plugin 'k8saudit-eks' from file /usr/share/falco/plugins/libk8saudit-eks.so
Wed May 10 14:07:18 2023: Loading plugin 'json' from file /usr/share/falco/plugins/libjson.so
Wed May 10 14:07:18 2023: Loading rules from file /etc/falco/k8s_audit_rules.yaml
Wed May 10 14:07:18 2023: Loading rules from file /etc/falco/rules.d/override-k8saudit.yaml
Wed May 10 14:07:18 2023: Starting health webserver with threadiness 4, listening on port 8765
Wed May 10 14:07:18 2023: Enabled event sources: k8s_audit
Wed May 10 14:07:18 2023: Opening capture with plugin 'k8saudit-eks'
{"hostname":"falco-k8saudit-eks-96d7f6f99-vwjgc","output":"14:09:17.699479000: Informational K8s Serviceaccount Created (user=system:node:ip-192-168-34-135.ec2.internal serviceaccount=default ns=default resource=serviceaccounts resp=201 decision=allow reason=)","priority":"Informational","rule":"K8s Serviceaccount Created","source":"k8s_audit","tags":["k8s"],"time":"2023-05-10T14:09:17.699479000Z", "output_fields": {"evt.time":1683727757699479000,"ka.auth.decision":"allow","ka.auth.reason":"","ka.response.code":"201","ka.target.name":"default","ka.target.namespace":"default","ka.target.resource":"serviceaccounts","ka.user.name":"system:node:ip-192-168-34-135.ec2.internal"}}
{"hostname":"falco-k8saudit-eks-96d7f6f99-vwjgc","output":"14:09:24.046826000: Notice Attach/Exec to pod (user=kubernetes-admin pod=kubecon resource=pods ns=default action=exec command=sh)","priority":"Notice","rule":"Attach/Exec Pod","source":"k8s_audit","tags":["k8s"],"time":"2023-05-10T14:09:24.046826000Z", "output_fields": {"evt.time":1683727764046826000,"ka.target.name":"kubecon","ka.target.namespace":"default","ka.target.resource":"pods","ka.target.subresource":"exec","ka.uri.param[command]":"sh","ka.user.name":"kubernetes-admin"}}
```

## Disclaimer

Our tests noticed some latencies between the presence of the logs in the CloudWatch Logs Stream and their evaluation by Falco. This is more visible with highly requested API servers. The solution is to adapt the size of your nodes where Falco runs, considering a minimal size of `2xlarge` as a safe option.

## Conclusion

With this first Plugin for a managed K8s solution, we hope to open the door for new contributions from the community for other flavors like GKE and AKS. If you need to create a plugin for another AWS service, take also a look at the libs we created to help the developers.

You can find us in the [Falco community](https://github.com/falcosecurity/community). Please feel free to reach out to us for any questions, suggestions, or even for a friendly chat!

If you would like to find out more about Falco:

* Get started in [Falco.org](http://falco.org/)
* Check out the [Falco project in GitHub](https://github.com/falcosecurity/falco).
* Get involved in the [Falco community](https://falco.org/community).
* Meet the maintainers on the [Falco Slack](https://kubernetes.slack.com/?redir=%2Farchives%2FCMWH3EH32).
* Follow [@falco_org on Twitter](https://twitter.com/falco_org).
