Add blog post about recording seccomp profiles in edge scenarios
Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
This commit is contained in:
Sascha Grunert
parent
26f5f6b64e
commit
af4cec641c
|
|
@ -0,0 +1,282 @@
|
|||
---
|
||||
layout: blog
|
||||
title: "Having fun with seccomp profiles on the edge"
|
||||
date: 2023-05-18
|
||||
slug: seccomp-profiles-edge
|
||||
---
|
||||
|
||||
**Author**: Sascha Grunert
|
||||
|
||||
The [Security Profiles Operator (SPO)][spo] is a feature-rich
|
||||
[operator][operator] for Kubernetes to make managing seccomp, SELinux and
|
||||
AppArmor profiles easier than ever. Recording those profiles from scratch is one
|
||||
of the key features of this operator, which usually involves the integration
|
||||
into large CI/CD systems. Being able to test the recording capabilities of the
|
||||
operator in edge cases is one of the recent development efforts of the SPO and
|
||||
makes it excitingly easy to play around with seccomp profiles.
|
||||
|
||||
[spo]: https://github.com/kubernetes-sigs/security-profiles-operator
|
||||
[operator]: https://kubernetes.io/docs/concepts/extend-kubernetes/operator
|
||||
|
||||
## Recording seccomp profiles with `spoc record`
|
||||
|
||||
The [v0.8.0][spo-latest] release of the Security Profiles Operator shipped a new
|
||||
command line interface called `spoc`, a little helper tool for recording and
|
||||
replaying seccomp profiles among various other things that are out of scope of
|
||||
this blog post.
|
||||
|
||||
[spo-latest]: https://github.com/kubernetes-sigs/security-profiles-operator/releases/v0.8.0
|
||||
|
||||
Recording a seccomp profile requires a binary to be executed, which can be a
|
||||
simple golang application which just calls [`uname(2)`][uname]:
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"syscall"
|
||||
)
|
||||
|
||||
func main() {
|
||||
utsname := syscall.Utsname{}
|
||||
if err := syscall.Uname(&utsname); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
[uname]: https://man7.org/linux/man-pages/man2/uname.2.html
|
||||
|
||||
Building a binary from that code can be done by:
|
||||
|
||||
```console
|
||||
> go build -o main main.go
|
||||
> ldd ./main
|
||||
not a dynamic executable
|
||||
```
|
||||
|
||||
Now it's possible to download the latest binary of [`spoc` from
|
||||
GitHub][spoc-latest] and run the application on Linux with it:
|
||||
|
||||
[spoc-latest]: https://github.com/kubernetes-sigs/security-profiles-operator/releases/download/v0.8.0/spoc.amd64
|
||||
|
||||
```console
|
||||
> sudo ./spoc record ./main
|
||||
10:08:25.591945 Loading bpf module
|
||||
10:08:25.591958 Using system btf file
|
||||
libbpf: loading object 'recorder.bpf.o' from buffer
|
||||
…
|
||||
libbpf: prog 'sys_enter': relo #3: patched insn #22 (ALU/ALU64) imm 16 -> 16
|
||||
10:08:25.610767 Getting bpf program sys_enter
|
||||
10:08:25.610778 Attaching bpf tracepoint
|
||||
10:08:25.611574 Getting syscalls map
|
||||
10:08:25.611582 Getting pid_mntns map
|
||||
10:08:25.613097 Module successfully loaded
|
||||
10:08:25.613311 Processing events
|
||||
10:08:25.613693 Running command with PID: 336007
|
||||
10:08:25.613835 Received event: pid: 336007, mntns: 4026531841
|
||||
10:08:25.613951 No container ID found for PID (pid=336007, mntns=4026531841, err=unable to find container ID in cgroup path)
|
||||
10:08:25.614856 Processing recorded data
|
||||
10:08:25.614975 Found process mntns 4026531841 in bpf map
|
||||
10:08:25.615110 Got syscalls: read, close, mmap, rt_sigaction, rt_sigprocmask, madvise, nanosleep, clone, uname, sigaltstack, arch_prctl, gettid, futex, sched_getaffinity, exit_group, openat
|
||||
10:08:25.615195 Adding base syscalls: access, brk, capget, capset, chdir, chmod, chown, close_range, dup2, dup3, epoll_create1, epoll_ctl, epoll_pwait, execve, faccessat2, fchdir, fchmodat, fchown, fchownat, fcntl, fstat, fstatfs, getdents64, getegid, geteuid, getgid, getpid, getppid, getuid, ioctl, keyctl, lseek, mkdirat, mknodat, mount, mprotect, munmap, newfstatat, openat2, pipe2, pivot_root, prctl, pread64, pselect6, readlink, readlinkat, rt_sigreturn, sched_yield, seccomp, set_robust_list, set_tid_address, setgid, setgroups, sethostname, setns, setresgid, setresuid, setsid, setuid, statfs, statx, symlinkat, tgkill, umask, umount2, unlinkat, unshare, write
|
||||
10:08:25.616293 Wrote seccomp profile to: /tmp/profile.yaml
|
||||
10:08:25.616298 Unloading bpf module
|
||||
```
|
||||
|
||||
I have to execute `spoc` as root because it will internally run an [ebpf][ebpf]
|
||||
program by reusing the same code parts from the Security Profiles Operator
|
||||
itself. I can see that the bpf module got loaded successfully and `spoc`
|
||||
attached the required tracepoint to it. Then it will track the main application
|
||||
by using its [mount namespace][mntns] and process the recorded syscall data. The
|
||||
nature of ebpf programs is that they see the whole context of the Kernel, which
|
||||
means that `spoc` tracks all syscalls of the system, but does not interfere with
|
||||
their execution.
|
||||
|
||||
[ebpf]: https://ebpf.io
|
||||
[mntns]: https://man7.org/linux/man-pages/man7/mount_namespaces.7.html
|
||||
|
||||
The logs indicate that `spoc` found the syscalls `read`, `close`,
|
||||
`mmap` and so on, including `uname`. All other syscalls than `uname` are coming
|
||||
from the golang runtime and its garbage collection, which already adds overhead
|
||||
to a basic application like in our demo. I can also see from the log line
|
||||
`Adding base syscalls: …` that `spoc` adds a bunch of base syscalls to the
|
||||
resulting profile. Those are used by the OCI runtime (like [runc][runc] or
|
||||
[crun][crun]) in order to be able to run a container. This means that `spoc`
|
||||
can be used to record seccomp profiles which then can be containerized directly.
|
||||
This behavior can be disabled in `spoc` by using the `--no-base-syscalls`/`-n`
|
||||
or customized via the `--base-syscalls`/`-b` command line flags This can be
|
||||
helpful in cases where different OCI runtimes other than crun and runc are used,
|
||||
or if I just want to record the seccomp profile for the application and stack
|
||||
it with another [base profile][base].
|
||||
|
||||
[runc]: https://github.com/opencontainers/runc
|
||||
[crun]: https://github.com/containers/crun
|
||||
[base]: https://github.com/kubernetes-sigs/security-profiles-operator/blob/35ebdda/installation-usage.md#base-syscalls-for-a-container-runtime
|
||||
|
||||
The resulting profile is now available in `/tmp/profile.yaml`, but the default
|
||||
location can be changed using the `--output-file value`/`-o` flag:
|
||||
|
||||
```console
|
||||
> cat /tmp/profile.yaml
|
||||
```
|
||||
|
||||
```yaml
|
||||
apiVersion: security-profiles-operator.x-k8s.io/v1beta1
|
||||
kind: SeccompProfile
|
||||
metadata:
|
||||
creationTimestamp: null
|
||||
name: main
|
||||
spec:
|
||||
architectures:
|
||||
- SCMP_ARCH_X86_64
|
||||
defaultAction: SCMP_ACT_ERRNO
|
||||
syscalls:
|
||||
- action: SCMP_ACT_ALLOW
|
||||
names:
|
||||
- access
|
||||
- arch_prctl
|
||||
- brk
|
||||
- …
|
||||
- uname
|
||||
- …
|
||||
status: {}
|
||||
```
|
||||
|
||||
The seccomp profile Custom Resource Definition (CRD) can be directly used
|
||||
together with the Security Profiles Operator for managing it within Kubernetes.
|
||||
`spoc` is also capable of producing raw seccomp profiles (as JSON), by using the
|
||||
`--type`/`-t` `raw-seccomp` flag:
|
||||
|
||||
```console
|
||||
> sudo ./spoc record --type raw-seccomp ./main
|
||||
…
|
||||
52.628827 Wrote seccomp profile to: /tmp/profile.json
|
||||
```
|
||||
|
||||
```
|
||||
> jq . /tmp/profile.json
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"defaultAction": "SCMP_ACT_ERRNO",
|
||||
"architectures": ["SCMP_ARCH_X86_64"],
|
||||
"syscalls": [
|
||||
{
|
||||
"names": ["access", "…", "write"],
|
||||
"action": "SCMP_ACT_ALLOW"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
The utility `spoc record` allows us to record complex seccomp profiles directly
|
||||
from binary invocations in any Linux system which is capable of running the ebpf
|
||||
code within the Kernel. But it can do more: How about modifying the seccomp
|
||||
profile and then testing it by using `spoc run`.
|
||||
|
||||
## Running seccomp profiles with `spoc run`
|
||||
|
||||
`spoc` is also able to run binaries with applied seccomp profiles, making it
|
||||
easy to test any modification to it. To do that, just run:
|
||||
|
||||
```console
|
||||
> sudo ./spoc run ./main
|
||||
10:29:58.153263 Reading file /tmp/profile.yaml
|
||||
10:29:58.153311 Assuming YAML profile
|
||||
10:29:58.154138 Setting up seccomp
|
||||
10:29:58.154178 Load seccomp profile
|
||||
10:29:58.154189 Starting audit log enricher
|
||||
10:29:58.154224 Enricher reading from file /var/log/audit/audit.log
|
||||
10:29:58.155356 Running command with PID: 437880
|
||||
>
|
||||
```
|
||||
|
||||
It looks like that the application exited successfully, which is anticipated
|
||||
because I did not modify the previously recorded profile yet. I can also
|
||||
specify a custom location for the profile by using the `--profile`/`-p` flag,
|
||||
but this was not necessary because I did not modify the default output location
|
||||
from the record. `spoc` will automatically determine if it's a raw (JSON) or CRD
|
||||
(YAML) based seccomp profile and then apply it to the process.
|
||||
|
||||
The Security Profiles Operator supports a [log enricher feature][enricher],
|
||||
which provides additional seccomp related information by parsing the audit logs.
|
||||
`spoc run` uses the enricher in the same way to provide more data to the end
|
||||
users when it comes to debugging seccomp profiles.
|
||||
|
||||
[enricher]: https://github.com/kubernetes-sigs/security-profiles-operator/blob/35ebdda/installation-usage.md#using-the-log-enricher
|
||||
|
||||
Now I have to modify the profile to see anything valuable in the output. For
|
||||
example, I could remove the allowed `uname` syscall:
|
||||
|
||||
```console
|
||||
> jq 'del(.syscalls[0].names[] | select(. == "uname"))' /tmp/profile.json > /tmp/no-uname-profile.json
|
||||
```
|
||||
|
||||
And then try to run it again with the new profile `/tmp/no-uname-profile.json`:
|
||||
|
||||
```
|
||||
> sudo ./spoc run -p /tmp/no-uname-profile.json ./main
|
||||
10:39:12.707798 Reading file /tmp/no-uname-profile.json
|
||||
10:39:12.707892 Setting up seccomp
|
||||
10:39:12.707920 Load seccomp profile
|
||||
10:39:12.707982 Starting audit log enricher
|
||||
10:39:12.707998 Enricher reading from file /var/log/audit/audit.log
|
||||
10:39:12.709164 Running command with PID: 480512
|
||||
panic: operation not permitted
|
||||
|
||||
goroutine 1 [running]:
|
||||
main.main()
|
||||
/path/to/main.go:10 +0x85
|
||||
10:39:12.713035 Unable to run: launch runner: wait for command: exit status 2
|
||||
```
|
||||
|
||||
Alright, that was expected! The applied seccomp profile blocks the `uname`
|
||||
syscall, which results in an "operation not permitted" error. This error is
|
||||
pretty generic and does not provide any hint on what got blocked by seccomp.
|
||||
It is generally extremely difficult to predict how applications behave if single
|
||||
syscalls are forbidden by seccomp. It could be possible that the application
|
||||
terminates like in our simple demo, but it could also lead to a strange
|
||||
misbehavior and the application does not stop at all.
|
||||
|
||||
If I now change the default seccomp action of the profile from `SCMP_ACT_ERRNO`
|
||||
to `SCMP_ACT_LOG` like this:
|
||||
|
||||
```console
|
||||
> jq '.defaultAction = "SCMP_ACT_LOG"' /tmp/no-uname-profile.json > /tmp/no-uname-profile-log.json
|
||||
```
|
||||
|
||||
Then the log enricher will give us a hint that the `uname` syscall got blocked
|
||||
when using `spoc run`:
|
||||
|
||||
```
|
||||
> sudo ./spoc run -p /tmp/no-uname-profile-log.json ./main
|
||||
10:48:07.470126 Reading file /tmp/no-uname-profile-log.json
|
||||
10:48:07.470234 Setting up seccomp
|
||||
10:48:07.470245 Load seccomp profile
|
||||
10:48:07.470302 Starting audit log enricher
|
||||
10:48:07.470339 Enricher reading from file /var/log/audit/audit.log
|
||||
10:48:07.470889 Running command with PID: 522268
|
||||
10:48:07.472007 Seccomp: uname (63)
|
||||
```
|
||||
|
||||
The application will not terminate any more, but seccomp will log the behavior
|
||||
to `/var/log/audit/audit.log` and `spoc` will parse the data to correlate it
|
||||
directly to our program. Generating the log messages to the audit subsystem
|
||||
comes with a large performance overhead and should be handled with care in
|
||||
production systems. It also comes with a security risk when running untrusted
|
||||
apps in audit mode in production environments.
|
||||
|
||||
This demo should give you an impression how to debug seccomp profile issues with
|
||||
applications, probably by using our shiny new helper tool powered by the
|
||||
features of the Security Profiles Operator. `spoc` is a flexible and portable
|
||||
binary suitable for edge cases where resources are limited and even Kubernetes
|
||||
itself may not be available with its full capabilities.
|
||||
|
||||
Thank you for reading this blog post! If you're interested in more, providing
|
||||
feedback or asking for help, then feel free to get in touch with us directly via
|
||||
[Slack (#security-profiles-operator)][slack] or the [mailing list][mail].
|
||||
|
||||
[slack]: https://kubernetes.slack.com/messages/security-profiles-operator
|
||||
[mail]: https://groups.google.com/forum/#!forum/kubernetes-dev
|
||||
Loading…
Reference in New Issue