update(docs): bumped changelog for release.

Signed-off-by: Federico Di Pierro <nierro92@gmail.com>
chore(userspace/falco): make re2 patterns statically lived.
2025-05-28 10:40:16 +02:00 · 2025-05-28 10:40:16 +02:00 · 2025-05-28 10:40:16 +02:00 · 2025-05-28 10:40:16 +02:00 · 2025-05-28 10:40:16 +02:00 · 2025-05-28 10:40:16 +02:00
19 changed files with 585 additions and 341 deletions
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@ -1,5 +1,90 @@
 # Change Log

+## v0.41.0
+
+Released on 2025-05-29
+
+### Breaking Changes :warning:
+
+* cleanup(engine)!: only consider .yaml/.yml rule files [[#3551](https://github.com/falcosecurity/falco/pull/3551)] - [@LucaGuerra](https://github.com/LucaGuerra)
+* cleanup(userspace)!: deprecate print of `container.info` [[#3543](https://github.com/falcosecurity/falco/pull/3543)] - [@FedeDP](https://github.com/FedeDP)
+* cleanup(userspace/falco)!: drop deprecated in 0.40.0 CLI flags. [[#3496](https://github.com/falcosecurity/falco/pull/3496)] - [@FedeDP](https://github.com/FedeDP)
+
+
+### Major Changes
+
+* new(falco): add json_include_output_fields option [[#3527](https://github.com/falcosecurity/falco/pull/3527)] - [@LucaGuerra](https://github.com/LucaGuerra)
+* new(build,userspace): switch to use container plugin [[#3482](https://github.com/falcosecurity/falco/pull/3482)] - [@FedeDP](https://github.com/FedeDP)
+* new(docker,scripts,ci): use an override config file to enable ISO 8601 output timeformat on docker images [[#3488](https://github.com/falcosecurity/falco/pull/3488)] - [@FedeDP](https://github.com/FedeDP)
+
+
+### Minor Changes
+
+* chore(build): update falcoctl to v0.11.2, rules for artifact follow to v4 [[#3580](https://github.com/falcosecurity/falco/pull/3580)] - [@LucaGuerra](https://github.com/LucaGuerra)
+* update(cmake): bumped falcoctl to 0.11.1 and rules to 4.0.0. [[#3577](https://github.com/falcosecurity/falco/pull/3577)] - [@FedeDP](https://github.com/FedeDP)
+* update(containers): update opencontainers labels [[#3575](https://github.com/falcosecurity/falco/pull/3575)] - [@LucaGuerra](https://github.com/LucaGuerra)
+* update(metrics): improve restart/hot_reload conditions inspection [[#3562](https://github.com/falcosecurity/falco/pull/3562)] - [@incertum](https://github.com/incertum)
+* update: empty `values` in `exceptions` won't emit a warning anymore [[#3529](https://github.com/falcosecurity/falco/pull/3529)] - [@leogr](https://github.com/leogr)
+* chore(falco.yaml): enable libs_logger by default with info level [[#3507](https://github.com/falcosecurity/falco/pull/3507)] - [@FedeDP](https://github.com/FedeDP)
+
+
+### Bug Fixes
+
+* fix(metrics/prometheus): gracefully handle multiple event sources, avoid erroneous duplicate metrics [[#3563](https://github.com/falcosecurity/falco/pull/3563)] - [@incertum](https://github.com/incertum)
+* fix(ci): properly install rpm systemd-rpm-macro package on building packages pipeline [[#3521](https://github.com/falcosecurity/falco/pull/3521)] - [@FedeDP](https://github.com/FedeDP)
+* fix(userspace/falco): init cmdline options after loading all config files [[#3493](https://github.com/falcosecurity/falco/pull/3493)] - [@FedeDP](https://github.com/FedeDP)
+* fix(cmake): add support for 16K kernel page to jemalloc [[#3490](https://github.com/falcosecurity/falco/pull/3490)] - [@Darkness4](https://github.com/Darkness4)
+* fix(userspace/falco): fix jemalloc enabled in minimal build. [[#3478](https://github.com/falcosecurity/falco/pull/3478)] - [@FedeDP](https://github.com/FedeDP)
+
+
+
+### Non user-facing changes
+
+* chore(deps): Bump submodules/falcosecurity-rules from `4ccf111` to `cb17833` [[#3572](https://github.com/falcosecurity/falco/pull/3572)] - [@dependabot[bot]](https://github.com/apps/dependabot)
+* update(cmake/rules): bump to falco-rules-4.0.0-rc1 [[#3567](https://github.com/falcosecurity/falco/pull/3567)] - [@leogr](https://github.com/leogr)
+* cleanup(userspace/falco): drop unused `libs_metrics_collector` variable. [[#3566](https://github.com/falcosecurity/falco/pull/3566)] - [@FedeDP](https://github.com/FedeDP)
+* update(cmake): update libs and driver to latest master [[#3564](https://github.com/falcosecurity/falco/pull/3564)] - [@github-actions[bot]](https://github.com/apps/github-actions)
+* fix(build): fixed container custom_target `sed` command. [[#3556](https://github.com/falcosecurity/falco/pull/3556)] - [@FedeDP](https://github.com/FedeDP)
+* chore(deps): Bump submodules/falcosecurity-rules from `ae6ed41` to `4ccf111` [[#3555](https://github.com/falcosecurity/falco/pull/3555)] - [@dependabot[bot]](https://github.com/apps/dependabot)
+* fix(cmake): fix bundled c-ares cmake issue with e.g. SLES [[#3559](https://github.com/falcosecurity/falco/pull/3559)] - [@terror96](https://github.com/terror96)
+* chore(deps): Bump submodules/falcosecurity-rules from `1d2c6b1` to `ae6ed41` [[#3553](https://github.com/falcosecurity/falco/pull/3553)] - [@dependabot[bot]](https://github.com/apps/dependabot)
+* chore: revert "chore(deps): Bump submodules/falcosecurity-rules from `1d2c6b1` to `371e431`" [[#3552](https://github.com/falcosecurity/falco/pull/3552)] - [@FedeDP](https://github.com/FedeDP)
+* update(cmake): update libs and driver to latest master [[#3550](https://github.com/falcosecurity/falco/pull/3550)] - [@github-actions[bot]](https://github.com/apps/github-actions)
+* update(cmake): update libs and driver to latest master [[#3549](https://github.com/falcosecurity/falco/pull/3549)] - [@github-actions[bot]](https://github.com/apps/github-actions)
+* update(adopters): added SafeDep as adopter [[#3548](https://github.com/falcosecurity/falco/pull/3548)] - [@KunalSin9h](https://github.com/KunalSin9h)
+* update(cmake): update libs and driver to latest master [[#3547](https://github.com/falcosecurity/falco/pull/3547)] - [@github-actions[bot]](https://github.com/apps/github-actions)
+* update(cmake): update libs and driver to latest master [[#3541](https://github.com/falcosecurity/falco/pull/3541)] - [@github-actions[bot]](https://github.com/apps/github-actions)
+* fix(userspace): fixed engine `openssl` dep. [[#3535](https://github.com/falcosecurity/falco/pull/3535)] - [@FedeDP](https://github.com/FedeDP)
+* fix(userspace/falco): fix outputs_http timeout [[#3523](https://github.com/falcosecurity/falco/pull/3523)] - [@benierc](https://github.com/benierc)
+* fix(ci): use clang-19 to build modern_ebpf skeleton. [[#3537](https://github.com/falcosecurity/falco/pull/3537)] - [@FedeDP](https://github.com/FedeDP)
+* update(cmake): update libs and driver to latest master [[#3531](https://github.com/falcosecurity/falco/pull/3531)] - [@github-actions[bot]](https://github.com/apps/github-actions)
+* update(cmake): update libs and driver to latest master [[#3530](https://github.com/falcosecurity/falco/pull/3530)] - [@github-actions[bot]](https://github.com/apps/github-actions)
+* update(cmake): update libs and driver to latest master [[#3525](https://github.com/falcosecurity/falco/pull/3525)] - [@github-actions[bot]](https://github.com/apps/github-actions)
+* update(cmake): update libs and driver to latest master [[#3520](https://github.com/falcosecurity/falco/pull/3520)] - [@github-actions[bot]](https://github.com/apps/github-actions)
+* update(cmake): update libs and driver to latest master [[#3516](https://github.com/falcosecurity/falco/pull/3516)] - [@github-actions[bot]](https://github.com/apps/github-actions)
+* docs(README.md): cleanups and enhancements [[#3514](https://github.com/falcosecurity/falco/pull/3514)] - [@leogr](https://github.com/leogr)
+* update(cmake): update libs and driver to latest master [[#3511](https://github.com/falcosecurity/falco/pull/3511)] - [@github-actions[bot]](https://github.com/apps/github-actions)
+* chore(deps): Bump submodules/falcosecurity-rules from `1d2c6b1` to `371e431` [[#3510](https://github.com/falcosecurity/falco/pull/3510)] - [@dependabot[bot]](https://github.com/apps/dependabot)
+* update(cmake): update libs and driver to latest master [[#3508](https://github.com/falcosecurity/falco/pull/3508)] - [@github-actions[bot]](https://github.com/apps/github-actions)
+* update(cmake): update libs and driver to latest master [[#3506](https://github.com/falcosecurity/falco/pull/3506)] - [@github-actions[bot]](https://github.com/apps/github-actions)
+* fix(userspace/falco): when counting `-M` timeout, do not account for async events [[#3505](https://github.com/falcosecurity/falco/pull/3505)] - [@FedeDP](https://github.com/FedeDP)
+* chore(deps): Bump submodules/falcosecurity-rules from `d8415c1` to `1d2c6b1` [[#3504](https://github.com/falcosecurity/falco/pull/3504)] - [@dependabot[bot]](https://github.com/apps/dependabot)
+* docs(proposals): correct typo in example [[#3499](https://github.com/falcosecurity/falco/pull/3499)] - [@leogr](https://github.com/leogr)
+* fix(docker): fixed entrypoints paths with new docker context. [[#3492](https://github.com/falcosecurity/falco/pull/3492)] - [@FedeDP](https://github.com/FedeDP)
+* feat(falco/app): move actions not using config before `load_config` [[#3483](https://github.com/falcosecurity/falco/pull/3483)] - [@ekoops](https://github.com/ekoops)
+* refactor(falco/app): apply early return pattern in actions code [[#3484](https://github.com/falcosecurity/falco/pull/3484)] - [@ekoops](https://github.com/ekoops)
+* chore(deps): Bump submodules/falcosecurity-rules from `abf6637` to `d8415c1` [[#3489](https://github.com/falcosecurity/falco/pull/3489)] - [@dependabot[bot]](https://github.com/apps/dependabot)
+* Add NETWAYS Web Services to ADOPTERS.md [[#3487](https://github.com/falcosecurity/falco/pull/3487)] - [@mocdaniel](https://github.com/mocdaniel)
+* chore: add back Falco static package to the release template. [[#3472](https://github.com/falcosecurity/falco/pull/3472)] - [@FedeDP](https://github.com/FedeDP)
+
+### Statistics
+
+|   MERGED PRS    | NUMBER |
+|-----------------|--------|
+| Not user-facing |     36 |
+| Release note    |     17 |
+| Total           |     53 |
+
 ## v0.40.0

 Released on 2025-01-28
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@ -267,6 +267,12 @@ if(NOT WIN32
   AND NOT MUSL_OPTIMIZED_BUILD
 )
 	include(falcoctl)
+	set(CONTAINER_VERSION "0.2.4")
+	if(${CMAKE_HOST_SYSTEM_PROCESSOR} STREQUAL "x86_64")
+		set(CONTAINER_HASH "2b6cf7c014fa29dffbc063582343402b863581218e704ca8021bc971c3e029fc")
+	else() # arm64
+		set(CONTAINER_HASH "ad96c2baa299fa51b6be07a93b21dd03fe6e2a9bea44cc13ea50a346e5d22774")
+	endif()
 	include(container_plugin)

 	# Generate a binary_dir/falco.yaml that automatically enables the plugin to be used for local
--- a/cmake/modules/driver.cmake
+++ b/cmake/modules/driver.cmake
@ -35,9 +35,9 @@ else()
 	# FALCOSECURITY_LIBS_VERSION. In case you want to test against another driver version (or
 	# branch, or commit) just pass the variable - ie., `cmake -DDRIVER_VERSION=dev ..`
 	if(NOT DRIVER_VERSION)
-		set(DRIVER_VERSION "9c2734a64338abff04c4a8274d3770e40c964e21")
+		set(DRIVER_VERSION "8.1.0+driver")
 		set(DRIVER_CHECKSUM
-			"SHA256=8094cfb04c77b317a4e5a69cf8556dccb54067d1decf9ec920c7cc3fa1ea831a"
+			"SHA256=182e6787bf86249a846a3baeb4dcd31578b76d4a13efa16ce3f44d66b18a77a6"
 		)
 	endif()

--- a/cmake/modules/falcoctl.cmake
+++ b/cmake/modules/falcoctl.cmake
@ -20,16 +20,16 @@ option(ADD_FALCOCTL_DEPENDENCY "Add falcoctl dependency while building falco" ON
 if(ADD_FALCOCTL_DEPENDENCY)
 	string(TOLOWER ${CMAKE_HOST_SYSTEM_NAME} FALCOCTL_SYSTEM_NAME)

-	set(FALCOCTL_VERSION "0.11.0")
+	set(FALCOCTL_VERSION "0.11.2")

 	message(STATUS "Building with falcoctl: ${FALCOCTL_VERSION}")

 	if(${CMAKE_HOST_SYSTEM_PROCESSOR} STREQUAL "x86_64")
 		set(FALCOCTL_SYSTEM_PROC_GO "amd64")
-		set(FALCOCTL_HASH "b9d0e0f50813e7172a945f36f70c5c3c16a677ab4c85b35b6f7a155bc92768fc")
+		set(FALCOCTL_HASH "8d55818987c90e54f7406e1c1441a18df1f485db858bb0b3efda5db217be3b48")
 	else() # aarch64
 		set(FALCOCTL_SYSTEM_PROC_GO "arm64")
-		set(FALCOCTL_HASH "689c625d1d414cbf53d39ef94083a53dda3ea4ac4908799fb85f4519e21442e0")
+		set(FALCOCTL_HASH "7c36404b5b7a515df25e7dc6d827a74ebc8526b1b49850954bbdd40860961bc2")
 	endif()

 	ExternalProject_Add(
--- a/cmake/modules/falcosecurity-libs.cmake
+++ b/cmake/modules/falcosecurity-libs.cmake
@ -42,9 +42,9 @@ else()
 	# version (or branch, or commit) just pass the variable - ie., `cmake
 	# -DFALCOSECURITY_LIBS_VERSION=dev ..`
 	if(NOT FALCOSECURITY_LIBS_VERSION)
-		set(FALCOSECURITY_LIBS_VERSION "9c2734a64338abff04c4a8274d3770e40c964e21")
+		set(FALCOSECURITY_LIBS_VERSION "0.21.0")
 		set(FALCOSECURITY_LIBS_CHECKSUM
-			"SHA256=8094cfb04c77b317a4e5a69cf8556dccb54067d1decf9ec920c7cc3fa1ea831a"
+			"SHA256=9e977001dd42586df42a5dc7e7a948c297124865a233402e44bdec68839d322a"
 		)
 	endif()

--- a/cmake/modules/rules.cmake
+++ b/cmake/modules/rules.cmake
@ -1,6 +1,6 @@
 # SPDX-License-Identifier: Apache-2.0
 #
-# Copyright (C) 2024 The Falco Authors.
+# Copyright (C) 2025 The Falco Authors.
 #
 # Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
 # in compliance with the License. You may obtain a copy of the License at
@ -18,9 +18,9 @@ include(ExternalProject)

 if(NOT DEFINED FALCOSECURITY_RULES_FALCO_PATH)
 	# falco_rules.yaml
-	set(FALCOSECURITY_RULES_FALCO_VERSION "falco-rules-3.2.0")
+	set(FALCOSECURITY_RULES_FALCO_VERSION "falco-rules-4.0.0")
 	set(FALCOSECURITY_RULES_FALCO_CHECKSUM
-		"SHA256=b3990bf0209cfbf6a903b361e458a1f5851a9a5aeee808ad26a5ddbe1377157d"
+		"SHA256=132320ddbfa1e2580981ed1bdd3ee3d0128a1e2306b2bee8978d1f0a930d6127"
 	)
 	set(FALCOSECURITY_RULES_FALCO_PATH
 		"${PROJECT_BINARY_DIR}/falcosecurity-rules-falco-prefix/src/falcosecurity-rules-falco/falco_rules.yaml"
--- a/docker/driver-loader-buster/Dockerfile
+++ b/docker/driver-loader-buster/Dockerfile
@ -1,7 +1,11 @@
 FROM debian:buster

-LABEL maintainer="cncf-falco-dev@lists.cncf.io"
-LABEL org.opencontainers.image.source="https://github.com/falcosecurity/falco"
+LABEL org.opencontainers.image.authors='The Falco Authors https://falco.org' \
+      org.opencontainers.image.url='https://falco.org' \
+      org.opencontainers.image.source='https://github.com/falcosecurity/falco' \
+      org.opencontainers.image.vendor='Falco Organization' \
+      org.opencontainers.image.licenses='Apache-2.0' \
+      maintainer="cncf-falco-dev@lists.cncf.io"

 LABEL usage="docker run -i -t --privileged -v /root/.falco:/root/.falco -v /proc:/host/proc:ro -v /boot:/host/boot:ro -v /lib/modules:/host/lib/modules:ro -v /usr:/host/usr:ro -v /etc:/host/etc:ro falcosecurity/falco-driver-loader:latest-buster [driver] [options]"

--- a/docker/driver-loader/Dockerfile
+++ b/docker/driver-loader/Dockerfile
@ -1,8 +1,12 @@
 ARG FALCO_IMAGE_TAG=latest
 FROM docker.io/falcosecurity/falco:${FALCO_IMAGE_TAG}-debian

-LABEL maintainer="cncf-falco-dev@lists.cncf.io"
-LABEL org.opencontainers.image.source="https://github.com/falcosecurity/falco"
+LABEL org.opencontainers.image.authors='The Falco Authors https://falco.org' \
+      org.opencontainers.image.url='https://falco.org' \
+      org.opencontainers.image.source='https://github.com/falcosecurity/falco' \
+      org.opencontainers.image.vendor='Falco Organization' \
+      org.opencontainers.image.licenses='Apache-2.0' \
+      maintainer="cncf-falco-dev@lists.cncf.io"

 LABEL usage="docker run -i -t --privileged -v /root/.falco:/root/.falco -v /proc:/host/proc:ro -v /boot:/host/boot:ro -v /lib/modules:/host/lib/modules:ro -v /usr:/host/usr:ro -v /etc:/host/etc:ro falcosecurity/falco-driver-loader:latest [driver] [options]"

--- a/docker/falco-debian/Dockerfile
+++ b/docker/falco-debian/Dockerfile
@ -1,7 +1,11 @@
 FROM debian:12-slim

-LABEL maintainer="cncf-falco-dev@lists.cncf.io"
-LABEL org.opencontainers.image.source="https://github.com/falcosecurity/falco/docker/falco-debian"
+LABEL org.opencontainers.image.authors='The Falco Authors https://falco.org' \
+      org.opencontainers.image.url='https://falco.org' \
+      org.opencontainers.image.source='https://github.com/falcosecurity/falco' \
+      org.opencontainers.image.vendor='Falco Organization' \
+      org.opencontainers.image.licenses='Apache-2.0' \
+      maintainer="cncf-falco-dev@lists.cncf.io"

 LABEL usage="docker run -i -t --privileged -v /var/run/docker.sock:/host/var/run/docker.sock -v /proc:/host/proc:ro -v /etc:/host/etc:ro falcosecurity/falco:latest-debian"

--- a/docker/falco/Dockerfile
+++ b/docker/falco/Dockerfile
@ -1,7 +1,11 @@
 FROM cgr.dev/chainguard/wolfi-base

-LABEL maintainer="cncf-falco-dev@lists.cncf.io"
-LABEL org.opencontainers.image.source="https://github.com/falcosecurity/falco"
+LABEL org.opencontainers.image.authors='The Falco Authors https://falco.org' \
+      org.opencontainers.image.url='https://falco.org' \
+      org.opencontainers.image.source='https://github.com/falcosecurity/falco' \
+      org.opencontainers.image.vendor='Falco Organization' \
+      org.opencontainers.image.licenses='Apache-2.0' \
+      maintainer="cncf-falco-dev@lists.cncf.io"

 LABEL usage="docker run -i -t --privileged -v /var/run/docker.sock:/host/var/run/docker.sock -v /proc:/host/proc:ro  -v /etc:/host/etc:ro falcosecurity/falco:latest"
 # NOTE: for the "least privileged" use case, please refer to the official documentation
--- a/scripts/falcoctl/falcoctl.yaml.in
+++ b/scripts/falcoctl/falcoctl.yaml.in
@ -10,7 +10,7 @@ artifact:
    every: 6h0m0s
    falcoVersions: http://localhost:8765/versions
    refs:
-    - falco-rules:3
+    - falco-rules:4
 indexes:
 - name: falcosecurity
  url: https://falcosecurity.github.io/falcoctl/index.yaml
--- a/userspace/falco/app/actions/load_config.cpp
+++ b/userspace/falco/app/actions/load_config.cpp
@ -66,6 +66,10 @@ falco::app::run_result falco::app::actions::load_config(const falco::app::state&
 		}
 	}

+	s.config->m_falco_reload_ts = (int64_t)std::chrono::duration_cast<std::chrono::nanoseconds>(
+	                                      std::chrono::system_clock::now().time_since_epoch())
+	                                      .count();
+
 	s.config->m_buffered_outputs = !s.options.unbuffered_outputs;

 	return apply_deprecated_options(s);
--- a/userspace/falco/app/actions/process_events.cpp
+++ b/userspace/falco/app/actions/process_events.cpp
@ -235,6 +235,12 @@ static falco::app::run_result do_inspect(
 			}

 			// for capture mode, the source name can change at every event
+			// TODO: This may currently cause issues for multiple event sources. We are deferring
+			// the fix to Falco 0.42.0.
+			// For multiple event sources, it generates `n` metrics logs per source at a time, as
+			// expected, with the engine_name correctly reflected. However, the order may interfere,
+			// as the correct inspector for the syscalls event source seems to never get passed,
+			// resulting in most metrics being missing.
 			stats_collector.collect(inspector,
 			                        inspector->event_sources()[source_engine_idx],
 			                        num_evts);
--- a/userspace/falco/configuration.h
+++ b/userspace/falco/configuration.h
@ -214,7 +214,15 @@ public:
 	gvisor_config m_gvisor = {};

 	yaml_helper m_config;
+
+	//
+	// Runtime-Generated values (not user-configurable)
+	//
+
+	// JSON schema generated from a hardcoded string
 	nlohmann::json m_config_schema;
+	// Timestamp of most recent configuration reload
+	int64_t m_falco_reload_ts{0};

 private:
 	void merge_config_files(const std::string& config_name, config_loaded_res& res);
--- a/userspace/falco/falco_metrics.cpp
+++ b/userspace/falco/falco_metrics.cpp
@ -31,8 +31,36 @@ namespace fs = std::filesystem;

 /*!
    \class falco_metrics
-    \brief This class is used to convert the metrics provided by the application
-    and falco libs into a string to be return by the metrics endpoint.
+    \brief Converts metrics provided by the application and Falco libraries into a formatted string
+           for the metrics endpoint.
+
+    ## Metrics Overview
+    This section explains why looping over inspectors is necessary.
+    Falco utilizes multiple inspectors when loading plugins with an event source.
+    Most metrics should only be retrieved once, ideally by the syscalls inspector if applicable.
+    To maximize metrics retrieval and prevent duplicate data, the syscalls inspector is always
+    positioned at index 0 in the loop when it exists.
+
+    Wrapper fields: See https://falco.org/docs/concepts/metrics/
+    - `engine_name` and `event_source` are pushed for each inspector.
+    - All other wrapper fields are agnostic and should be retrieved once.
+
+    ## Metrics Collection Behavior
+    - `rules_counters_enabled` -> Agnostic; resides in falco; retrieved from the state, not an
+      inspector; only performed once.
+    - `resource_utilization_enabled` -> Agnostic; resides in libs; inspector is irrelevant;
+      only performed once.
+    - `state_counters_enabled` -> Semi-agnostic; resides in libs; must be retrieved by the syscalls
+      inspector if applicable.
+    - `kernel_event_counters_enabled` -> Resides in libs; must be retrieved by the syscalls
+   inspector; not available for other inspectors.
+    - `kernel_event_counters_per_cpu_enabled` -> Resides in libs; must be retrieved by the syscalls
+      inspector; not available for other inspectors.
+    - `libbpf_stats_enabled` -> Resides in libs; must be retrieved by the syscalls inspector;
+      not available for other inspectors.
+    - `plugins_metrics_enabled` -> Must be retrieved for each inspector.
+    - `jemalloc_stats_enabled` -> Agnostic; resides in falco; inspector is irrelevant;
+      only performed once.
 */

 /*!
@ -42,108 +70,354 @@ namespace fs = std::filesystem;

    https://prometheus.io/docs/instrumenting/exposition_formats/#text-based-format
 */
-const std::string falco_metrics::content_type = "text/plain; version=0.0.4";
+const std::string falco_metrics::content_type_prometheus = "text/plain; version=0.0.4";

-/*!
-    \brief this method takes an application \c state and returns a textual representation of
-    its configured metrics.
+std::string falco_metrics::falco_to_text_prometheus(
+        const falco::app::state& state,
+        libs::metrics::prometheus_metrics_converter& prometheus_metrics_converter,
+        std::vector<metrics_v2>& additional_wrapper_metrics) {
+	std::string prometheus_text;

-    The current implementation returns a Prometheus exposition formatted string.
-*/
-std::string falco_metrics::to_text(const falco::app::state& state) {
+	// # HELP falcosecurity_falco_version_info https://falco.org/docs/metrics/
+	// # TYPE falcosecurity_falco_version_info gauge
+	// falcosecurity_falco_version_info{version="0.41.0-100+334ca42"} 1
+	prometheus_text += prometheus_metrics_converter.convert_metric_to_text_prometheus(
+	        "version",
+	        "falcosecurity",
+	        "falco",
+	        {{"version", FALCO_VERSION}});
+
+#if defined(__linux__) and !defined(MINIMAL_BUILD) and !defined(__EMSCRIPTEN__)
+	// Note that the rule counter metrics are retrieved from the state, not from any inspector
+	// Distinguish between config and rules files using labels, following Prometheus best
+	// practices: https://prometheus.io/docs/practices/naming/#labels
+
+	// # HELP falcosecurity_falco_sha256_rules_files_info https://falco.org/docs/metrics/
+	// # TYPE falcosecurity_falco_sha256_rules_files_info gauge
+	// falcosecurity_falco_sha256_rules_files_info{file_name="falco_rules.yaml",sha256="6f0078862a26528cb50a860f9ebebbfbe3162e5009187089c73cb0cdf91d0b06"}
+	// 1
+	for(const auto& item : state.config.get()->m_loaded_rules_filenames_sha256sum) {
+		fs::path fs_path = item.first;
+		prometheus_text += prometheus_metrics_converter.convert_metric_to_text_prometheus(
+		        "sha256_rules_files",
+		        "falcosecurity",
+		        "falco",
+		        {{"file_name", fs_path.filename()}, {"sha256", item.second}});
+	}
+
+	// # HELP falcosecurity_falco_sha256_config_files_info https://falco.org/docs/metrics/
+	// # TYPE falcosecurity_falco_sha256_config_files_info gauge
+	// falcosecurity_falco_sha256_config_files_info{file_name="falco.yaml",sha256="f97de5fa6f513b5e07cd9f29ee9904ee4267cb120ef6501f8555543d5a98dd1c"}
+	// 1
+	for(const auto& item : state.config.get()->m_loaded_configs_filenames_sha256sum) {
+		fs::path fs_path = item.first;
+		prometheus_text += prometheus_metrics_converter.convert_metric_to_text_prometheus(
+		        "sha256_config_files",
+		        "falcosecurity",
+		        "falco",
+		        {{"file_name", fs_path.filename()}, {"sha256", item.second}});
+	}
+
+#endif
+	// # HELP falcosecurity_falco_outputs_queue_num_drops_total https://falco.org/docs/metrics/
+	// # TYPE falcosecurity_falco_outputs_queue_num_drops_total counter
+	// falcosecurity_falco_outputs_queue_num_drops_total 0
+	additional_wrapper_metrics.emplace_back(libs::metrics::libsinsp_metrics::new_metric(
+	        "outputs_queue_num_drops",
+	        METRICS_V2_MISC,
+	        METRIC_VALUE_TYPE_U64,
+	        METRIC_VALUE_UNIT_COUNT,
+	        METRIC_VALUE_METRIC_TYPE_MONOTONIC,
+	        state.outputs->get_outputs_queue_num_drops()));
+
+	// # HELP falcosecurity_falco_reload_timestamp_nanoseconds https://falco.org/docs/metrics/
+	// # TYPE falcosecurity_falco_reload_timestamp_nanoseconds gauge
+	// falcosecurity_falco_reload_timestamp_nanoseconds 1748338536592811359
+	additional_wrapper_metrics.emplace_back(libs::metrics::libsinsp_metrics::new_metric(
+	        "reload_ts",
+	        METRICS_V2_MISC,
+	        METRIC_VALUE_TYPE_S64,
+	        METRIC_VALUE_UNIT_TIME_TIMESTAMP_NS,
+	        METRIC_VALUE_METRIC_TYPE_NON_MONOTONIC_CURRENT,
+	        state.config->m_falco_reload_ts));
+
+	if(state.config->m_metrics_flags & METRICS_V2_RULE_COUNTERS) {
+		// rules_counters_enabled
+		const stats_manager& rule_stats_manager = state.engine->get_rule_stats_manager();
+		const indexed_vector<falco_rule>& rules = state.engine->get_rules();
+		const std::vector<std::unique_ptr<std::atomic<uint64_t>>>& rules_by_id =
+		        rule_stats_manager.get_by_rule_id();
+		// Distinguish between rules counters using labels, following Prometheus best
+		// practices: https://prometheus.io/docs/practices/naming/#labels
+		for(size_t i = 0; i < rules_by_id.size(); i++) {
+			auto rule = rules.at(i);
+			auto count = rules_by_id[i]->load();
+			if(count > 0) {
+				// # HELP falcosecurity_falco_rules_matches_total https://falco.org/docs/metrics/
+				// # TYPE falcosecurity_falco_rules_matches_total counter
+				// falcosecurity_falco_rules_matches_total{priority="4",rule_name="Read sensitive
+				// file
+				// untrusted",source="syscall",tag_T1555="true",tag_container="true",tag_filesystem="true",tag_host="true",tag_maturity_stable="true",tag_mitre_credential_access="true"}
+				// 32 # HELP falcosecurity_falco_rules_matches_total https://falco.org/docs/metrics/
+				// # TYPE falcosecurity_falco_rules_matches_total counter
+				// falcosecurity_falco_rules_matches_total{priority="5",rule_name="Terminal shell in
+				// container",source="syscall",tag_T1059="true",tag_container="true",tag_maturity_stable="true",tag_mitre_execution="true",tag_shell="true"}
+				// 1
+				auto metric = libs::metrics::libsinsp_metrics::new_metric(
+				        "rules_matches",
+				        METRICS_V2_RULE_COUNTERS,
+				        METRIC_VALUE_TYPE_U64,
+				        METRIC_VALUE_UNIT_COUNT,
+				        METRIC_VALUE_METRIC_TYPE_MONOTONIC,
+				        count);
+				prometheus_metrics_converter.convert_metric_to_unit_convention(metric);
+				std::map<std::string, std::string> const_labels = {
+				        {"rule_name", rule->name},
+				        {"priority", std::to_string(rule->priority)},
+				        {"source", rule->source},
+				};
+				std::for_each(rule->tags.cbegin(),
+				              rule->tags.cend(),
+				              [&const_labels](std::string const& tag) {
+					              const_labels.emplace(std::string{"tag_"} + tag, "true");
+				              });
+				prometheus_text += prometheus_metrics_converter.convert_metric_to_text_prometheus(
+				        metric,
+				        "falcosecurity",
+				        "falco",
+				        const_labels);
+			}
+		}
+	}
+#ifdef HAS_JEMALLOC
+	if(state.config->m_metrics_flags & METRICS_V2_JEMALLOC_STATS) {
+		// jemalloc_stats_enabled
+		nlohmann::json j;
+		malloc_stats_print(
+		        [](void* to, const char* from) {
+			        nlohmann::json* j = (nlohmann::json*)to;
+			        *j = nlohmann::json::parse(from);
+		        },
+		        &j,
+		        "Jmdablxeg");
+		const auto& j_stats = j["jemalloc"]["stats"];
+		for(auto it = j_stats.begin(); it != j_stats.end(); ++it) {
+			if(it.value().is_number_unsigned()) {
+				std::uint64_t val = it.value().template get<std::uint64_t>();
+				std::string key = "jemalloc." + it.key();
+				auto metric = libs::metrics::libsinsp_metrics::new_metric(
+				        key.c_str(),
+				        METRICS_V2_JEMALLOC_STATS,
+				        METRIC_VALUE_TYPE_U64,
+				        METRIC_VALUE_UNIT_MEMORY_BYTES,
+				        METRIC_VALUE_METRIC_TYPE_MONOTONIC,
+				        val);
+				prometheus_metrics_converter.convert_metric_to_unit_convention(metric);
+				prometheus_text += prometheus_metrics_converter.convert_metric_to_text_prometheus(
+				        metric,
+				        "falcosecurity",
+				        "falco");
+			}
+		}
+	}
+#endif
+	return prometheus_text;
+}
+
+std::string falco_metrics::sources_to_text_prometheus(
+        const falco::app::state& state,
+        libs::metrics::prometheus_metrics_converter& prometheus_metrics_converter,
+        std::vector<metrics_v2>& additional_wrapper_metrics) {
 	static const char* all_driver_engines[] = {BPF_ENGINE,
 	                                           KMOD_ENGINE,
 	                                           MODERN_BPF_ENGINE,
 	                                           SOURCE_PLUGIN_ENGINE,
 	                                           NODRIVER_ENGINE,
 	                                           GVISOR_ENGINE};
+	static re2::RE2 drops_buffer_pattern("n_drops_buffer_([^_]+(?:_[^_]+)*)_(enter|exit)$");
+	static re2::RE2 cpu_pattern("(\\d+)");

-	std::vector<std::shared_ptr<sinsp>> inspectors;
-	std::vector<libs::metrics::libs_metrics_collector> metrics_collectors;
+	std::string prometheus_text;
+	bool agent_info_written = false;
+	bool machine_info_written = false;

+	// Then, source-bound metrics
 	for(const auto& source : state.enabled_sources) {
 		auto source_info = state.source_infos.at(source);
 		auto source_inspector = source_info->inspector;
-		inspectors.emplace_back(source_inspector);
-		metrics_collectors.emplace_back(
-		        libs::metrics::libs_metrics_collector(source_inspector.get(),
-		                                              state.config->m_metrics_flags));
-	}
-	libs::metrics::prometheus_metrics_converter prometheus_metrics_converter;
-	std::string prometheus_text;

-	for(auto inspector : inspectors) {
-		// Falco wrapper metrics
-		//
-		for(size_t i = 0; i < sizeof(all_driver_engines) / sizeof(const char*); i++) {
-			if(inspector->check_current_engine(all_driver_engines[i])) {
+		// First thing: list of enabled engine names
+
+		// Falco wrapper metrics Part A: Repeated for each inspector, accounting for plugins w/
+		// event sources
+
+		/* Examples ...
+		    # HELP falcosecurity_scap_engine_name_info https://falco.org/docs/metrics/
+		    # TYPE falcosecurity_scap_engine_name_info gauge
+		    falcosecurity_scap_engine_name_info{engine_name="source_plugin",evt_source="dummy"} 1
+		    # HELP falcosecurity_scap_engine_name_info https://falco.org/docs/metrics/
+		    # TYPE falcosecurity_scap_engine_name_info gauge
+		    falcosecurity_scap_engine_name_info{engine_name="bpf",evt_source="syscall"} 1
+		*/
+
+		for(size_t j = 0; j < sizeof(all_driver_engines) / sizeof(const char*); j++) {
+			if(source_inspector->check_current_engine(all_driver_engines[j])) {
 				prometheus_text += prometheus_metrics_converter.convert_metric_to_text_prometheus(
 				        "engine_name",
 				        "falcosecurity",
 				        "scap",
-				        {{"engine_name", all_driver_engines[i]}});
+				        {{"engine_name", std::string(all_driver_engines[j])},
+				         {"evt_source", source}});
 				break;
 			}
 		}

-		const scap_agent_info* agent_info = inspector->get_agent_info();
-		const scap_machine_info* machine_info = inspector->get_machine_info();
-		libs::metrics::libs_metrics_collector libs_metrics_collector(inspector.get(), 0);
-		prometheus_text += prometheus_metrics_converter.convert_metric_to_text_prometheus(
-		        "version",
-		        "falcosecurity",
-		        "falco",
-		        {{"version", FALCO_VERSION}});
+		// Inspectors' metrics collectors
+		// Libs metrics categories
+		//
+		// resource_utilization_enabled
+		// state_counters_enabled
+		// kernel_event_counters_enabled
+		// kernel_event_counters_per_cpu_enabled
+		// libbpf_stats_enabled
+		auto metrics_collector =
+		        libs::metrics::libs_metrics_collector(source_inspector.get(),
+		                                              state.config->m_metrics_flags);
+		metrics_collector.snapshot();
+		auto metrics_snapshot = metrics_collector.get_metrics();
+
+		// Source plugin
+		if(source != falco_common::syscall_source) {
+			// Performed repeatedly for each inspectors' libs metrics collector
+			for(auto& metric : metrics_snapshot) {
+				if(metric.flags & METRICS_V2_PLUGINS) {
+					prometheus_metrics_converter.convert_metric_to_unit_convention(metric);
+					prometheus_text +=
+					        prometheus_metrics_converter.convert_metric_to_text_prometheus(
+					                metric,
+					                "falcosecurity",
+					                "plugins");
+				}
+			}
+		} else {
+			// Source syscall
+			for(auto& metric : metrics_snapshot) {
+				prometheus_metrics_converter.convert_metric_to_unit_convention(metric);
+				std::string prometheus_subsystem = "scap";
+
+				if(metric.flags & METRICS_V2_RESOURCE_UTILIZATION) {
+					prometheus_subsystem = "falco";
+				}
+
+				if(metric.flags & METRICS_V2_PLUGINS) {
+					prometheus_subsystem = "plugins";
+				}
+
+				// raw incoming in form of for example n_evts_cpu_15 or n_drops_cpu_15
+				if(strncmp(metric.name, "n_evts_cpu", 10) == 0 ||
+				   strncmp(metric.name, "n_drops_cpu", 11) == 0)  // prefix match
+				{
+					std::string name_str(metric.name);
+					std::string cpu_number;
+					if(re2::RE2::PartialMatch(name_str, cpu_pattern, &cpu_number)) {
+						re2::RE2::GlobalReplace(&name_str, cpu_pattern, "");
+						// possible double __ will be sanitized within libs
+						auto metric_new = libs::metrics::libsinsp_metrics::new_metric(
+						        name_str.c_str(),
+						        METRICS_V2_KERNEL_COUNTERS_PER_CPU,
+						        METRIC_VALUE_TYPE_U64,
+						        METRIC_VALUE_UNIT_COUNT,
+						        METRIC_VALUE_METRIC_TYPE_MONOTONIC,
+						        metric.value.u64);
+						const std::map<std::string, std::string>& const_labels = {
+						        {"cpu", cpu_number}};
+						/* Examples ...
+						    # HELP falcosecurity_scap_n_evts_cpu_total
+						   https://falco.org/docs/metrics/ # TYPE
+						   falcosecurity_scap_n_evts_cpu_total counter
+						    falcosecurity_scap_n_evts_cpu_total{cpu="7"} 237
+						    # HELP falcosecurity_scap_n_drops_cpu_total
+						   https://falco.org/docs/metrics/ # TYPE
+						   falcosecurity_scap_n_drops_cpu_total counter
+						    falcosecurity_scap_n_drops_cpu_total{cpu="7"} 0
+						*/
+						prometheus_text +=
+						        prometheus_metrics_converter.convert_metric_to_text_prometheus(
+						                metric_new,
+						                "falcosecurity",
+						                prometheus_subsystem,
+						                const_labels);
+					}
+				} else if(strcmp(metric.name, "n_drops_buffer_total") == 0) {
+					// Skip the libs aggregate metric since we distinguish between buffer drops
+					// using labels similar to the rules_matches
+					continue;
+				} else if(strncmp(metric.name, "n_drops_buffer", 14) == 0)  // prefix match
+				{
+					std::string drop;
+					std::string dir;
+					std::string name_str(metric.name);
+					if(re2::RE2::FullMatch(name_str, drops_buffer_pattern, &drop, &dir)) {
+						auto metric_new = libs::metrics::libsinsp_metrics::new_metric(
+						        "n_drops_buffer",
+						        METRICS_V2_KERNEL_COUNTERS,
+						        METRIC_VALUE_TYPE_U64,
+						        METRIC_VALUE_UNIT_COUNT,
+						        METRIC_VALUE_METRIC_TYPE_MONOTONIC,
+						        metric.value.u64);
+						const std::map<std::string, std::string>& const_labels = {{"drop", drop},
+						                                                          {"dir", dir}};
+						/* Examples ...
+						    # HELP falcosecurity_scap_n_drops_buffer_total
+						   https://falco.org/docs/metrics/ # TYPE
+						   falcosecurity_scap_n_drops_buffer_total counter
+						    falcosecurity_scap_n_drops_buffer_total{dir="enter",drop="clone_fork"} 0
+						    # HELP falcosecurity_scap_n_drops_buffer_total
+						   https://falco.org/docs/metrics/ # TYPE
+						   falcosecurity_scap_n_drops_buffer_total counter
+						    falcosecurity_scap_n_drops_buffer_total{dir="exit",drop="clone_fork"} 0
+						*/
+						prometheus_text +=
+						        prometheus_metrics_converter.convert_metric_to_text_prometheus(
+						                metric_new,
+						                "falcosecurity",
+						                prometheus_subsystem,
+						                const_labels);
+					}
+				} else {
+					prometheus_text +=
+					        prometheus_metrics_converter.convert_metric_to_text_prometheus(
+					                metric,
+					                "falcosecurity",
+					                prometheus_subsystem);
+				}
+			}
+		}
+
+		// Source wrapper metrics Part B: Agnostic, performed only once.
+		if(agent_info_written && machine_info_written) {
+			continue;
+		}
+
+		const scap_agent_info* agent_info = nullptr;
+		if(!agent_info_written) {
+			agent_info = source_inspector->get_agent_info();
+		}
+		const scap_machine_info* machine_info = nullptr;
+		if(!machine_info_written) {
+			machine_info = source_inspector->get_machine_info();
+		}

 		// Not all scap engines report agent and machine infos.
+		// However, recent lib refactors enable a linux lite platform, allowing non-syscall
+		// inspectors to retrieve these metrics if the syscall inspector is unavailable.
+		// We only push these info once.
 		if(agent_info) {
 			prometheus_text += prometheus_metrics_converter.convert_metric_to_text_prometheus(
 			        "kernel_release",
 			        "falcosecurity",
 			        "falco",
 			        {{"kernel_release", agent_info->uname_r}});
-		}
-		if(machine_info) {
-			prometheus_text += prometheus_metrics_converter.convert_metric_to_text_prometheus(
-			        "hostname",
-			        "falcosecurity",
-			        "evt",
-			        {{"hostname", machine_info->hostname}});
-		}
-
-#if defined(__linux__) and !defined(MINIMAL_BUILD) and !defined(__EMSCRIPTEN__)
-		// Distinguish between config and rules files using labels, following Prometheus best
-		// practices: https://prometheus.io/docs/practices/naming/#labels
-		for(const auto& item : state.config.get()->m_loaded_rules_filenames_sha256sum) {
-			fs::path fs_path = item.first;
-			prometheus_text += prometheus_metrics_converter.convert_metric_to_text_prometheus(
-			        "sha256_rules_files",
-			        "falcosecurity",
-			        "falco",
-			        {{"file_name", fs_path.filename()}, {"sha256", item.second}});
-		}
-
-		for(const auto& item : state.config.get()->m_loaded_configs_filenames_sha256sum) {
-			fs::path fs_path = item.first;
-			prometheus_text += prometheus_metrics_converter.convert_metric_to_text_prometheus(
-			        "sha256_config_files",
-			        "falcosecurity",
-			        "falco",
-			        {{"file_name", fs_path.filename()}, {"sha256", item.second}});
-		}
-
-#endif
-
-		for(const std::string& source : inspector->event_sources()) {
-			prometheus_text += prometheus_metrics_converter.convert_metric_to_text_prometheus(
-			        "evt_source",
-			        "falcosecurity",
-			        "falco",
-			        {{"evt_source", source}});
-		}
-		std::vector<metrics_v2> additional_wrapper_metrics;
-
-		if(agent_info) {
 			additional_wrapper_metrics.emplace_back(libs::metrics::libsinsp_metrics::new_metric(
 			        "start_ts",
 			        METRICS_V2_MISC,
@ -151,8 +425,25 @@ std::string falco_metrics::to_text(const falco::app::state& state) {
 			        METRIC_VALUE_UNIT_TIME_TIMESTAMP_NS,
 			        METRIC_VALUE_METRIC_TYPE_NON_MONOTONIC_CURRENT,
 			        agent_info->start_ts_epoch));
+			auto now = std::chrono::duration_cast<std::chrono::nanoseconds>(
+			                   std::chrono::system_clock::now().time_since_epoch())
+			                   .count();
+			additional_wrapper_metrics.emplace_back(libs::metrics::libsinsp_metrics::new_metric(
+			        "duration_sec",
+			        METRICS_V2_MISC,
+			        METRIC_VALUE_TYPE_U64,
+			        METRIC_VALUE_UNIT_TIME_S_COUNT,
+			        METRIC_VALUE_METRIC_TYPE_MONOTONIC,
+			        (uint64_t)((now - agent_info->start_ts_epoch) / ONE_SECOND_IN_NS)));
+			agent_info_written = true;
 		}
+
 		if(machine_info) {
+			prometheus_text += prometheus_metrics_converter.convert_metric_to_text_prometheus(
+			        "hostname",
+			        "falcosecurity",
+			        "evt",
+			        {{"hostname", machine_info->hostname}});
 			additional_wrapper_metrics.emplace_back(libs::metrics::libsinsp_metrics::new_metric(
 			        "host_boot_ts",
 			        METRICS_V2_MISC,
@ -167,225 +458,41 @@ std::string falco_metrics::to_text(const falco::app::state& state) {
 			        METRIC_VALUE_UNIT_COUNT,
 			        METRIC_VALUE_METRIC_TYPE_NON_MONOTONIC_CURRENT,
 			        machine_info->num_cpus));
+			machine_info_written = true;
 		}
-		additional_wrapper_metrics.emplace_back(libs::metrics::libsinsp_metrics::new_metric(
-		        "outputs_queue_num_drops",
-		        METRICS_V2_MISC,
-		        METRIC_VALUE_TYPE_U64,
-		        METRIC_VALUE_UNIT_COUNT,
-		        METRIC_VALUE_METRIC_TYPE_MONOTONIC,
-		        state.outputs->get_outputs_queue_num_drops()));
+	}  // End inspector loop
+
+	return prometheus_text;
+}
+
+/*!
+    \brief this method takes an application \c state and returns a textual representation of
+    its configured metrics.
+
+    The current implementation returns a Prometheus exposition formatted string.
+*/
+std::string falco_metrics::to_text_prometheus(const falco::app::state& state) {
+	libs::metrics::prometheus_metrics_converter prometheus_metrics_converter;
+	std::string prometheus_text;
+
+	std::vector<metrics_v2> additional_wrapper_metrics;
+
+	// Falco global metrics, once
+	prometheus_text += falco_to_text_prometheus(state,
+	                                            prometheus_metrics_converter,
+	                                            additional_wrapper_metrics);
+	// Metrics for each source
+	prometheus_text += sources_to_text_prometheus(state,
+	                                              prometheus_metrics_converter,
+	                                              additional_wrapper_metrics);
+
+	for(auto metric : additional_wrapper_metrics) {
+		prometheus_metrics_converter.convert_metric_to_unit_convention(metric);
+		prometheus_text +=
+		        prometheus_metrics_converter.convert_metric_to_text_prometheus(metric,
+		                                                                       "falcosecurity",
+		                                                                       "falco");
+	}

-		if(agent_info) {
-			auto now = std::chrono::duration_cast<std::chrono::nanoseconds>(
-			                   std::chrono::system_clock::now().time_since_epoch())
-			                   .count();
-			additional_wrapper_metrics.emplace_back(libs::metrics::libsinsp_metrics::new_metric(
-			        "duration_sec",
-			        METRICS_V2_MISC,
-			        METRIC_VALUE_TYPE_U64,
-			        METRIC_VALUE_UNIT_TIME_S_COUNT,
-			        METRIC_VALUE_METRIC_TYPE_MONOTONIC,
-			        (uint64_t)((now - agent_info->start_ts_epoch) / ONE_SECOND_IN_NS)));
-		}
-
-		for(auto metric : additional_wrapper_metrics) {
-			prometheus_metrics_converter.convert_metric_to_unit_convention(metric);
-			prometheus_text +=
-			        prometheus_metrics_converter.convert_metric_to_text_prometheus(metric,
-			                                                                       "falcosecurity",
-			                                                                       "falco");
-		}
-
-		// Falco metrics categories
-		//
-		// rules_counters_enabled
-		if(state.config->m_metrics_flags & METRICS_V2_RULE_COUNTERS) {
-			const stats_manager& rule_stats_manager = state.engine->get_rule_stats_manager();
-			const indexed_vector<falco_rule>& rules = state.engine->get_rules();
-			const std::vector<std::unique_ptr<std::atomic<uint64_t>>>& rules_by_id =
-			        rule_stats_manager.get_by_rule_id();
-			// Distinguish between rules counters using labels, following Prometheus best practices:
-			// https://prometheus.io/docs/practices/naming/#labels
-			for(size_t i = 0; i < rules_by_id.size(); i++) {
-				auto rule = rules.at(i);
-				auto count = rules_by_id[i]->load();
-				if(count > 0) {
-					/* Examples ...
-					    # HELP falcosecurity_falco_rules_matches_total
-					   https://falco.org/docs/metrics/ # TYPE
-					   falcosecurity_falco_rules_matches_total counter
-					    falcosecurity_falco_rules_matches_total{priority="4",rule_name="Read
-					   sensitive file
-					   untrusted",source="syscall",tag_T1555="true",tag_container="true",tag_filesystem="true",tag_host="true",tag_maturity_stable="true",tag_mitre_credential_access="true"}
-					   10 # HELP falcosecurity_falco_rules_matches_total
-					   https://falco.org/docs/metrics/ # TYPE
-					   falcosecurity_falco_rules_matches_total counter
-					    falcosecurity_falco_rules_matches_total{priority="5",rule_name="Unexpected
-					   UDP
-					   Traffic",source="syscall",tag_TA0011="true",tag_container="true",tag_host="true",tag_maturity_incubating="true",tag_mitre_exfiltration="true",tag_network="true"}
-					   1
-					*/
-					auto metric = libs::metrics::libsinsp_metrics::new_metric(
-					        "rules_matches",
-					        METRICS_V2_RULE_COUNTERS,
-					        METRIC_VALUE_TYPE_U64,
-					        METRIC_VALUE_UNIT_COUNT,
-					        METRIC_VALUE_METRIC_TYPE_MONOTONIC,
-					        rules_by_id[i]->load());
-					prometheus_metrics_converter.convert_metric_to_unit_convention(metric);
-					std::map<std::string, std::string> const_labels = {
-					        {"rule_name", rule->name},
-					        {"priority", std::to_string(rule->priority)},
-					        {"source", rule->source},
-					};
-					std::for_each(rule->tags.cbegin(),
-					              rule->tags.cend(),
-					              [&const_labels](std::string const& tag) {
-						              const_labels.emplace(std::string{"tag_"} + tag, "true");
-					              });
-					prometheus_text +=
-					        prometheus_metrics_converter.convert_metric_to_text_prometheus(
-					                metric,
-					                "falcosecurity",
-					                "falco",
-					                const_labels);
-				}
-			}
-		}
-#ifdef HAS_JEMALLOC
-		if(state.config->m_metrics_flags & METRICS_V2_JEMALLOC_STATS) {
-			nlohmann::json j;
-			malloc_stats_print(
-			        [](void* to, const char* from) {
-				        nlohmann::json* j = (nlohmann::json*)to;
-				        *j = nlohmann::json::parse(from);
-			        },
-			        &j,
-			        "Jmdablxeg");
-			const auto& j_stats = j["jemalloc"]["stats"];
-			for(auto it = j_stats.begin(); it != j_stats.end(); ++it) {
-				if(it.value().is_number_unsigned()) {
-					std::uint64_t val = it.value().template get<std::uint64_t>();
-					std::string key = "jemalloc." + it.key();
-					auto metric = libs::metrics::libsinsp_metrics::new_metric(
-					        key.c_str(),
-					        METRICS_V2_JEMALLOC_STATS,
-					        METRIC_VALUE_TYPE_U64,
-					        METRIC_VALUE_UNIT_MEMORY_BYTES,
-					        METRIC_VALUE_METRIC_TYPE_MONOTONIC,
-					        val);
-					prometheus_metrics_converter.convert_metric_to_unit_convention(metric);
-					prometheus_text +=
-					        prometheus_metrics_converter.convert_metric_to_text_prometheus(
-					                metric,
-					                "falcosecurity",
-					                "falco");
-				}
-			}
-		}
-#endif
-	}
-
-	// Libs metrics categories
-	//
-	// resource_utilization_enabled
-	// state_counters_enabled
-	// kernel_event_counters_enabled
-	// libbpf_stats_enabled
-	for(auto metrics_collector : metrics_collectors) {
-		metrics_collector.snapshot();
-		auto metrics_snapshot = metrics_collector.get_metrics();
-
-		for(auto& metric : metrics_snapshot) {
-			prometheus_metrics_converter.convert_metric_to_unit_convention(metric);
-			std::string prometheus_subsystem = "scap";
-
-			if(metric.flags & METRICS_V2_RESOURCE_UTILIZATION) {
-				prometheus_subsystem = "falco";
-			}
-
-			if(metric.flags & METRICS_V2_PLUGINS) {
-				prometheus_subsystem = "plugins";
-			}
-
-			// raw incoming in form of for example n_evts_cpu_15 or n_drops_cpu_15
-			if(strncmp(metric.name, "n_evts_cpu", 10) == 0 ||
-			   strncmp(metric.name, "n_drops_cpu", 11) == 0)  // prefix match
-			{
-				std::string name_str(metric.name);
-				re2::RE2 pattern("(\\d+)");
-				std::string cpu_number;
-				if(re2::RE2::PartialMatch(name_str, pattern, &cpu_number)) {
-					re2::RE2::GlobalReplace(&name_str, pattern, "");
-					// possible double __ will be sanitized within libs
-					auto metric_new = libs::metrics::libsinsp_metrics::new_metric(
-					        name_str.c_str(),
-					        METRICS_V2_KERNEL_COUNTERS_PER_CPU,
-					        METRIC_VALUE_TYPE_U64,
-					        METRIC_VALUE_UNIT_COUNT,
-					        METRIC_VALUE_METRIC_TYPE_MONOTONIC,
-					        metric.value.u64);
-					const std::map<std::string, std::string>& const_labels = {{"cpu", cpu_number}};
-					/* Examples ...
-					    # HELP falcosecurity_scap_n_evts_cpu_total https://falco.org/docs/metrics/
-					    # TYPE falcosecurity_scap_n_evts_cpu_total counter
-					    falcosecurity_scap_n_evts_cpu_total{cpu="7"} 237
-					    # HELP falcosecurity_scap_n_drops_cpu_total https://falco.org/docs/metrics/
-					    # TYPE falcosecurity_scap_n_drops_cpu_total counter
-					    falcosecurity_scap_n_drops_cpu_total{cpu="7"} 0
-					*/
-					prometheus_text +=
-					        prometheus_metrics_converter.convert_metric_to_text_prometheus(
-					                metric_new,
-					                "falcosecurity",
-					                prometheus_subsystem,
-					                const_labels);
-				}
-			} else if(strcmp(metric.name, "n_drops_buffer_total") == 0) {
-				// Skip the libs aggregate metric since we distinguish between buffer drops using
-				// labels similar to the rules_matches
-				continue;
-			} else if(strncmp(metric.name, "n_drops_buffer", 14) == 0)  // prefix match
-			{
-				re2::RE2 pattern("n_drops_buffer_([^_]+(?:_[^_]+)*)_(enter|exit)$");
-				std::string drop;
-				std::string dir;
-				std::string name_str(metric.name);
-				if(re2::RE2::FullMatch(name_str, pattern, &drop, &dir)) {
-					auto metric_new = libs::metrics::libsinsp_metrics::new_metric(
-					        "n_drops_buffer",
-					        METRICS_V2_KERNEL_COUNTERS,
-					        METRIC_VALUE_TYPE_U64,
-					        METRIC_VALUE_UNIT_COUNT,
-					        METRIC_VALUE_METRIC_TYPE_MONOTONIC,
-					        metric.value.u64);
-					const std::map<std::string, std::string>& const_labels = {{"drop", drop},
-					                                                          {"dir", dir}};
-					/* Examples ...
-					    # HELP falcosecurity_scap_n_drops_buffer_total
-					   https://falco.org/docs/metrics/ # TYPE
-					   falcosecurity_scap_n_drops_buffer_total counter
-					    falcosecurity_scap_n_drops_buffer_total{dir="enter",drop="clone_fork"} 0
-					    # HELP falcosecurity_scap_n_drops_buffer_total
-					   https://falco.org/docs/metrics/ # TYPE
-					   falcosecurity_scap_n_drops_buffer_total counter
-					    falcosecurity_scap_n_drops_buffer_total{dir="exit",drop="clone_fork"} 0
-					*/
-					prometheus_text +=
-					        prometheus_metrics_converter.convert_metric_to_text_prometheus(
-					                metric_new,
-					                "falcosecurity",
-					                prometheus_subsystem,
-					                const_labels);
-				}
-			} else {
-				prometheus_text += prometheus_metrics_converter.convert_metric_to_text_prometheus(
-				        metric,
-				        "falcosecurity",
-				        prometheus_subsystem);
-			}
-		}
-	}
 	return prometheus_text;
 }
--- a/userspace/falco/falco_metrics.h
+++ b/userspace/falco/falco_metrics.h
@ -26,6 +26,16 @@ struct state;

 class falco_metrics {
 public:
-	static const std::string content_type;
-	static std::string to_text(const falco::app::state& state);
+	static const std::string content_type_prometheus;
+	static std::string to_text_prometheus(const falco::app::state& state);
+
+private:
+	static std::string falco_to_text_prometheus(
+	        const falco::app::state& state,
+	        libs::metrics::prometheus_metrics_converter& prometheus_metrics_converter,
+	        std::vector<metrics_v2>& additional_wrapper_metrics);
+	static std::string sources_to_text_prometheus(
+	        const falco::app::state& state,
+	        libs::metrics::prometheus_metrics_converter& prometheus_metrics_converter,
+	        std::vector<metrics_v2>& additional_wrapper_metrics);
 };
--- a/userspace/falco/stats_writer.cpp
+++ b/userspace/falco/stats_writer.cpp
@ -233,7 +233,6 @@ void stats_writer::worker() noexcept {
 	bool use_file = !m_config->m_metrics_output_file.empty();
 	auto tick = stats_writer::get_ticker();
 	auto last_tick = tick;
-	auto first_tick = tick;

 	while(true) {
 // blocks until a message becomes availables
@ -244,35 +243,33 @@ void stats_writer::worker() noexcept {
 			return;
 		}

-		// this helps waiting for the first tick
 		tick = stats_writer::get_ticker();
-		if(first_tick != tick) {
-			if(last_tick != tick) {
-				m_total_samples++;
-			}
-			last_tick = tick;

-			try {
-				if(use_outputs) {
-					std::string rule = "Falco internal: metrics snapshot";
-					std::string msg = "Falco metrics snapshot";
-					m_outputs->handle_msg(m.ts,
-					                      falco_common::PRIORITY_INFORMATIONAL,
-					                      msg,
-					                      rule,
-					                      m.output_fields);
-				}
+		if(last_tick != tick) {
+			m_total_samples++;
+		}
+		last_tick = tick;

-				if(use_file) {
-					nlohmann::json jmsg;
-					jmsg["sample"] = m_total_samples;
-					jmsg["output_fields"] = m.output_fields;
-					m_file_output << jmsg.dump() << std::endl;
-				}
-			} catch(const std::exception& e) {
-				falco_logger::log(falco_logger::level::ERR,
-				                  "stats_writer (worker): " + std::string(e.what()) + "\n");
+		try {
+			if(use_outputs) {
+				std::string rule = "Falco internal: metrics snapshot";
+				std::string msg = "Falco metrics snapshot";
+				m_outputs->handle_msg(m.ts,
+				                      falco_common::PRIORITY_INFORMATIONAL,
+				                      msg,
+				                      rule,
+				                      m.output_fields);
 			}
+
+			if(use_file) {
+				nlohmann::json jmsg;
+				jmsg["sample"] = m_total_samples;
+				jmsg["output_fields"] = m.output_fields;
+				m_file_output << jmsg.dump() << std::endl;
+			}
+		} catch(const std::exception& e) {
+			falco_logger::log(falco_logger::level::ERR,
+			                  "stats_writer (worker): " + std::string(e.what()) + "\n");
 		}
 	}
 }
@ -353,6 +350,7 @@ void stats_writer::collector::get_metrics_output_fields_wrapper(
 	/* Wrapper fields useful for statistical analyses and attributions. Always enabled. */
 	output_fields["evt.time"] =
 	        now; /* Some ETLs may prefer a consistent timestamp within output_fields. */
+	output_fields["falco.reload_ts"] = m_writer->m_config->m_falco_reload_ts;
 	output_fields["falco.version"] = FALCO_VERSION;
 	if(agent_info) {
 		output_fields["falco.start_ts"] = agent_info->start_ts_epoch;
@ -659,6 +657,7 @@ void stats_writer::collector::collect(const std::shared_ptr<sinsp>& inspector,
 			                                  num_evts,
 			                                  now,
 			                                  stats_snapshot_time_delta_sec);
+
 			get_metrics_output_fields_additional(output_fields, stats_snapshot_time_delta_sec);

 			/* Send message in the queue */
--- a/userspace/falco/stats_writer.h
+++ b/userspace/falco/stats_writer.h
@ -82,7 +82,9 @@ public:
 		                                          double stats_snapshot_time_delta_sec);

 		std::shared_ptr<stats_writer> m_writer;
-		stats_writer::ticker_t m_last_tick = 0;
+		// Init m_last_tick w/ invalid value to enable metrics logging immediately after
+		// startup/reload
+		stats_writer::ticker_t m_last_tick = std::numeric_limits<ticker_t>::max();
 		uint64_t m_last_now = 0;
 		uint64_t m_last_n_evts = 0;
 		uint64_t m_last_n_drops = 0;
--- a/userspace/falco/webserver.cpp
+++ b/userspace/falco/webserver.cpp
@ -60,7 +60,8 @@ void falco_webserver::start(const falco::app::state &state,

 	if(state.config->m_metrics_enabled && webserver_config.m_prometheus_metrics_enabled) {
 		m_server->Get("/metrics", [&state](const httplib::Request &, httplib::Response &res) {
-			res.set_content(falco_metrics::to_text(state), falco_metrics::content_type);
+			res.set_content(falco_metrics::to_text_prometheus(state),
+			                falco_metrics::content_type_prometheus);
 		});
 	}
 	// run server in a separate thread
Author	SHA1	Message	Date
Federico Di Pierro	b94cda0b12	update(docs): bumped changelog for release. Signed-off-by: Federico Di Pierro <nierro92@gmail.com>	2025-05-28 10:40:16 +02:00
Federico Di Pierro	dfe08bdecb	chore(userspace/falco): make re2 patterns statically lived. Moreover, rename `falco_metrics::` methods to better expose they return prometheus metrics. Signed-off-by: Federico Di Pierro <nierro92@gmail.com> Co-authored-by: Samuel Gaist <samuel.gaist@idiap.ch>	2025-05-28 10:40:16 +02:00
Federico Di Pierro	79a5f9f3d9	cleanup(userspace/falco): only push metrics for enabled sources. Refactor `::to_text` a bit to be more clear. Also, we will push agent_info and machine_info only for the first inspector that exposes them, to avoid duplicated entries in the prometheus text. Signed-off-by: Federico Di Pierro <nierro92@gmail.com>	2025-05-28 10:40:16 +02:00
Melissa Kilby	466373359c	fix(metrics/prometheus): non-duplicate evt_source retrieval Signed-off-by: Melissa Kilby <melissa.kilby.oss@gmail.com>	2025-05-28 10:40:16 +02:00
Melissa Kilby	b46e4188d8	cleanup(metrics/prometheus): add detailed logic explanation wrt inspector loop Co-authored-by: Federico Di Pierro <nierro92@gmail.com> Signed-off-by: Melissa Kilby <melissa.kilby.oss@gmail.com>	2025-05-28 10:40:16 +02:00
Melissa Kilby	6bcef8f94e	update(metrics/prometheus): place syscalls inspector (if applicable) at index 0 of loop Signed-off-by: Melissa Kilby <melissa.kilby.oss@gmail.com>	2025-05-28 10:40:16 +02:00
Melissa Kilby	cf6857f13b	fix(metrics/prometheus): gracefully handle multiple event sources, avoid erroneous duplicate metrics Signed-off-by: Melissa Kilby <melissa.kilby.oss@gmail.com>	2025-05-28 10:40:16 +02:00
Luca Guerra	9c2cff370d	chore(build): update falcoctl, rules for artifact follow Signed-off-by: Luca Guerra <luca@guerra.sh>	2025-05-28 08:50:16 +02:00
Luca Guerra	1a4ff6781b	update(containers): add labels Signed-off-by: Luca Guerra <luca@guerra.sh>	2025-05-27 10:07:10 +02:00
Federico Di Pierro	f15dca4a3b	chore(userspace/falco): initialize `m_falco_reload_ts` to 0. Signed-off-by: Federico Di Pierro <nierro92@gmail.com> Co-authored-by: Melissa Kilby <melissa.kilby.oss@gmail.com>	2025-05-27 10:07:10 +02:00
Melissa Kilby	43122a21e5	cleanup: apply minor code pilot suggestions Signed-off-by: Melissa Kilby <melissa.kilby.oss@gmail.com>	2025-05-27 10:07:10 +02:00
Melissa Kilby	eb59f3603b	cleanup(configs): move runtime generated configs to section w/ clear comments Co-authored-by: Samuel Gaist <samuel.gaist@idiap.ch> Signed-off-by: Melissa Kilby <melissa.kilby.oss@gmail.com>	2025-05-27 10:07:10 +02:00
Melissa Kilby	72910f23e9	cleanup(metrics): simplify logic around immediate metrics logging after start/reload * For consistency don't make first run metrics log special * Remove firt tick variable altogether to enable metrics logging immediately after startup/reload Co-authored-by: Federico Di Pierro <nierro92@gmail.com> Signed-off-by: Melissa Kilby <melissa.kilby.oss@gmail.com>	2025-05-27 10:07:10 +02:00
Melissa Kilby	4c258afd9b	clenaup(metrics): rename new restart_ts to reload_ts to reflect hot relaod conditions Signed-off-by: Melissa Kilby <melissa.kilby.oss@gmail.com>	2025-05-27 10:07:10 +02:00
Melissa Kilby	aa67a0270a	update(metrics): introduce restart ts metric to statistically inspect restart/hot_reload conditions Signed-off-by: Melissa Kilby <melissa.kilby.oss@gmail.com>	2025-05-27 10:07:10 +02:00
Melissa Kilby	60721592e5	update(metrics): introduce immediate initial metrics msg (output_rule or output_file) upon start/restart/hot_reload Signed-off-by: Melissa Kilby <melissa.kilby.oss@gmail.com>	2025-05-27 10:07:10 +02:00
Federico Di Pierro	9e67d90e19	update(cmake): bumped falcoctl to 0.11.1 and rules to 4.0.0. Signed-off-by: Federico Di Pierro <nierro92@gmail.com>	2025-05-27 10:07:10 +02:00
Leonardo Grasso	afc4798d4c	update(cmake/rules): bump to falco-rules-4.0.0-rc1 Signed-off-by: Leonardo Grasso <me@leonardograsso.com>	2025-05-27 10:07:10 +02:00
Federico Di Pierro	6bcc73aeff	cleanup(userspace/falco): drop unused `libs_metrics_collector` variable. Signed-off-by: Federico Di Pierro <nierro92@gmail.com>	2025-05-27 10:07:10 +02:00
Luca Guerra	d4e7325c06	update(build): update container plugin to 0.2.4 Signed-off-by: Luca Guerra <luca@guerra.sh>	2025-05-26 13:06:04 +02:00
Federico Di Pierro	9fe7230d31	update(cmake): bump libs and driver to latest tagged releases. Signed-off-by: Federico Di Pierro <nierro92@gmail.com>	2025-05-19 16:07:02 +02:00