mirror of https://github.com/containers/podman.git
ebacfbd091
the exit file
If the container exit code needs to be retained, it cannot be retained
in tmpfs, because libpod runs in a memcg itself so it can't leave
traces with a daemon-less design.
This wasn't a memleak detectable by kmemleak for example. The kernel
never lost track of the memory and there was no erroneous refcounting
either. The reference count dependencies however are not easy to track
because when a refcount is increased, there's no way to tell who's
still holding the reference. In this case it was a single page of
tmpfs pagecache holding a refcount that kept pinned a whole hierarchy
of dying memcg, slab kmem, cgropups, unrechable kernfs nodes and the
respective dentries and inodes. Such a problem wouldn't happen if the
exit file was stored in a regular filesystem because the pagecache
could be reclaimed in such case under memory pressure. The tmpfs page
can be swapped out, but that's not enough to release the memcg with
CONFIG_MEMCG_SWAP_ENABLED=y.
No amount of more aggressive kernel slab shrinking could have solved
this. Not even assigning slab kmem of dying cgroups to alive cgroup
would fully solve this. The only way to free the memory of a dying
cgroup when a struct page still references it, would be to loop over
all "struct page" in the kernel to find which one is associated with
the dying cgroup which is a O(N) operation (where N is the number of
pages and can reach billions). Linking all the tmpfs pages to the
memcg would cost less during memcg offlining, but it would waste lots
of memory and CPU globally. So this can't be optimized in the kernel.
A cronjob running this command can act as workaround and will allow
all slab cache to be released, not just the single tmpfs pages.
rm -f /run/libpod/exits/*
This patch solved the memleak with a reproducer, booting with
cgroup.memory=nokmem and with selinux disabled. The reason memcg kmem
and selinux were disabled for testing of this fix, is because kmem
greatly decreases the kernel effectiveness in reusing partial slab
objects. cgroup.memory=nokmem is strongly recommended at least for
workstation usage. selinux needs to be further analyzed because it
causes further slab allocations.
The upstream podman commit used for testing is
|
||
|---|---|---|
| .. | ||
| common | ||
| define | ||
| driver | ||
| events | ||
| image | ||
| layers | ||
| lock | ||
| logs | ||
| boltdb_state.go | ||
| boltdb_state_internal.go | ||
| boltdb_state_linux.go | ||
| boltdb_state_unsupported.go | ||
| common_test.go | ||
| container.go | ||
| container.log.go | ||
| container_api.go | ||
| container_commit.go | ||
| container_graph.go | ||
| container_graph_test.go | ||
| container_inspect.go | ||
| container_internal.go | ||
| container_internal_linux.go | ||
| container_internal_test.go | ||
| container_internal_unsupported.go | ||
| container_linux.go | ||
| container_log_linux.go | ||
| container_log_unsupported.go | ||
| container_top_linux.go | ||
| container_top_unsupported.go | ||
| container_unsupported.go | ||
| diff.go | ||
| events.go | ||
| healthcheck.go | ||
| healthcheck_linux.go | ||
| healthcheck_unsupported.go | ||
| in_memory_state.go | ||
| info.go | ||
| kube.go | ||
| mounts_linux.go | ||
| networking_linux.go | ||
| networking_unsupported.go | ||
| oci.go | ||
| oci_attach_linux.go | ||
| oci_attach_linux_cgo.go | ||
| oci_attach_linux_nocgo.go | ||
| oci_attach_unsupported.go | ||
| oci_internal_linux.go | ||
| oci_linux.go | ||
| oci_unsupported.go | ||
| options.go | ||
| pod.go | ||
| pod_api.go | ||
| pod_internal.go | ||
| pod_top_linux.go | ||
| pod_top_unsupported.go | ||
| runtime.go | ||
| runtime_cstorage.go | ||
| runtime_ctr.go | ||
| runtime_img.go | ||
| runtime_img_test.go | ||
| runtime_migrate.go | ||
| runtime_migrate_unsupported.go | ||
| runtime_pod.go | ||
| runtime_pod_infra_linux.go | ||
| runtime_pod_linux.go | ||
| runtime_pod_unsupported.go | ||
| runtime_renumber.go | ||
| runtime_volume.go | ||
| runtime_volume_linux.go | ||
| runtime_volume_unsupported.go | ||
| state.go | ||
| state_test.go | ||
| stats.go | ||
| stats_config.go | ||
| stats_unsupported.go | ||
| storage.go | ||
| util.go | ||
| util_linux.go | ||
| util_test.go | ||
| util_unsupported.go | ||
| volume.go | ||
| volume_internal.go | ||