The kernel-collector currently fails to retrieve instance metadata on modern AWS EC2 instances where IMDSv1 is disabled by default. This makes the collector non-functional in a standard, secure AWS environment.
This fix introduces support for IMDSv2 while maintaining backward compatibility with IMDSv1.
The implementation follows this logic:
- A short-lived (2-second timeout) PUT request is made to the IMDSv2 token endpoint to fetch a session token.
- If a token is successfully retrieved, it is used in the 'X-aws-ec2-metadata-token' header for all subsequent metadata requests, which are directed to the IMDSv2 '/latest/' endpoints.
- If the token request fails (due to timeout, network error, or IMDSv1-only environment), the system gracefully falls back to the original IMDSv1 behavior, making requests without the token to the '/2016-09-02/' endpoints.
Fixes: #356
* add vmlinux.h submodule
* add libbpf cmake target and switch #includes to vmlinux.h
* fix static assert for clang / libbpf
* add missing include
* update the kernel version macros
* fix map definitions and their accessors
* fix perf ring event output in renderc
* fix include directory
* fix bpf_log line output
* refactor checking the error codes in bpf_map_lookup_elem to keep the same behavior
* remove old checking of delete unexpected return value
* migrate calls for ntohs to bpf_ntohs helper macros
* solve integer to non-integer cast warning with explicit cast
* migrate BPF_PROG_ARRAY to libbpf
* migrate tail calls
* add networking macros not in vmlinux.h
* add bpf_tracing.h helpers for PT_REGS_PARM
* solve unsigned-to-signed assignment warning
* migrate htons (we only changed ntohs earlier)
* port tcp processor's rings
* migrate DNS perf event output
* add another define for a macro missing from vmlinux.h
* port dns_message_array
* solve errors for missing define
* solve signed-unsigned conversion warnings
* add some netfilter structs missing from vmlinux.h
* move added kernel structs to vmlinux_extensions.h and added doc
* fix string_starts_with without strlen
* add structs from previous versions
* solve multiple kernel version compatibility and missing definition errors
* fix enum read from cgroup_subsys_id
* add enum declaration for cgroup_subsys_id
* convert another bcc map lookup to libbpf format
* remove preserve_access_index attribute from enum (not allowed)
* support configuration of constants using libbpf global variables. max_pid needs to be constant however
* fix string_starts_with
* fix number of parameters by inlining
* fix bpf_log exceeding number of function parameters by encoding fields onto stack to pass to bpf_log
* support functions with many parameters
* migrate userspace code to libbpf
* more migration work
Please enter the commit message for your changes. Lines starting
* use forward declaration for the skeleton where possible
* removed redundant includes of the skel file
* clang-format
* reorder #includes for correct ordering
* fix the too long prototype for tcp_recvmsg
* fix permission check to use libbpf
* run clang-format
* add SEC annotations to bpf programs
* remove kernel header fetching
* update kernel header test to just run the collector
* run clang format
* statically link against libbpf
* add libelf1 to kernel collector dockerfile
* run a reducer when testing the kernel collector
* remove kernel fetching from the kernel-collector-test, add libelf1
* do not cancel running workflows
* mount sysfs inside the kernel collector container, for BTF
* add host networking so kernel-collector components can reach the test reducer
* remove double "--network=host"
* extract kprobe parameters using REGS_PARM for taskstats_exit
* add license to bpf code
* change to kernel-compatible license
* try to fix load test checks
* parse kprobe parameters from ctx, first update
* parse kprobe parameters from ctx, second update
* run clang format
* fix conversion typo
* support pre-mounted /sys in kernel-collector entrypoint
* give verifier hint as to tgid range in END_SAVE_ARGS
* try using u64 for bpf_trace_printk in END_SAVE_ARGS
* output pid_tgid in bpf_trace_printk in END_SAVE_ARGS to appease the verifier
* allow mounted /sys in kernel-collector-test
* fix typo
* add &= to satisfy verifier
* make new unsigned variable for printing
* fix missing escape
* remove tgid printing in macro (cannot satisfty verifier)
* try to re-get pid_tgid
* different printk format string
* add explicit cast
* try signed printing
* print dummy
* give up on getting verifier to agree to trace in macro, count in global variables instead.
* make reads verifier safe and remove BPF_REGS_PARMS with KERNEL_VERSION (not well supported it seems)
* run clang-format
* use nc instead of reducer
* increase simple test time to 30 seconds
* switch some paths from kernel version to bpf_core_field_exists
* remove getting fd of bpf program
* remove kernel symbol resolution lookup
* remove unneeded fd tracking for kprobes
* install netcat for simple test
* add missing SEC markings
* fix program parameter parsing to use PT_REGS_PARM*
* run clang-format
* fix netcat fetching
* use CORE for msg_iter
* fix duplicate SEC markings
* fix probe naming
* annotate onret_cgroup_control with SEC
* add more missing SEC annotations
* fix PT_REGS_PARM and BPF_CORE_READ in new SEC markings
* clang format
* add BPF_CORE_READ where appropriate in udp_update_stats_impl()
* use the openbsd version of netcat
* move udp_update_stats to always inline
* simplify BPF_CORE_READ
* explicitly disable UDP tracing (was already disabled)
* add BPF_CORE_READ where appropriate
* enhance error reporting when loading tail calls
* mark tail calls with SEC("kprobe")
* add BPF_CORE_READ and PT_REGS_PARM to tail probes in render_bpf.c
* fix verifier in DNS reporting
* move reading of struct iov closer to its origin to help verifier
* fix verifier issues with string_starts_with
* run clang-format
* tackle verifier error on 5.10
* remove bpf-to-bpf calls in functions that also use tail calls so 5.4 kernels will verify
* enumerate existing cgroups using cgroup_get_from_fd
* run clang-format
* fail the simple tests if the word "error" appears in the kernel-collector output
* add __always_inline to avoid mixing bpf-to-bpf calls with tail calls (forbidden in older kernels)
* change failing BPF_CORE_READ of type to bpf_probe_read_kernel
* fix bpf_log variable
* add more __always_inline directives to allow tail calls
* run clang-format
* add __always_inline to another function where compiler complained
* add __always_inline to all functions with 6 or more parameters
* add explicit null checks after casting to pre 5.14 msg struct
* simplify verification for 5.10 kernel
* add __always_inline to functions that might be called from handle_receive_udp_skb to allow tail calls on 5.4 kernels
* simplify backwards compat of handle_kprobe__tcp_sendmsg
* run clang-format
* add more __always_inline to enable tail calls in continue_tcp_sendmsg on 5.4 kernels
* fix printouts during simple run
* remove debug print
* simplify backwards compat code in handle_kprobe__tcp_recvmsg to aid verifier on 5.10 kernels
* run clang-format
* add more __always_inline on functions we missed, for continue_tcp_sendmsg on 5.4 kernels
* fix the bpf_log in handle_kprobe__tcp_recvmsg
* remove the 4.19 kernel from the test matrix since it does not support BTF
* help verifier on 5.4 kernel limit iteration size in string_starts_with
* move bpf configuration to libbpf and remove the bpf code string handling
* fix kernel_collector_test unused variable
* remove dead code
* simplify function name and remove irrelevant docs
* remove `report_debug_events` global variable -- now unused
* remove unneded defines and pragmas
* remove old comment
* remove tcp_*_handler wrappers, unneeded
* remove redundant LINUX_KERNEL_VERSION externs
* Simplify parameter loading in tcp_recvmsg kprobe
* simplify parameter loading in tcp_sendmsg kprobe
* remove dead compat code
* remove bcc from makefiles
* remove compilation and header fetching errors from troubleshooting
* fix bcc-related code comments
* add deprecation warning to the bcc-based tcp-processor python script
* remove kprobe cleanup -- unnecessary with libbpf
* simplify flow
* remove entrypoint error reporting
* remove probe_handler class member -- unneeded
* run clang-format
* remove backwards compatibility artifact
* simplify cgroup selection
* move variables to right scope
* remove dead code
* remove dead code
* remove prefix __ from __onret_udp_get_port_impl
* remove unused parameters
* don't re-read sk twice in on_ip_send_skb
* fix wrong porting bcc->libbpf hidden behind #ifdef
* simplify on_skb_free_datagram_locked
* remove some pre 3.12 cgroup support
* run clang-format
* simplify backwards compatibility
* simplify get_css_id
* rename handle_cgroup_processing -> handle_existing_cgroup
* simplify backwards compatibility in onret_cgroup_control
* suppress libbpf prints
* remove entrypoint_error.h include in reducer
* run clang-format
* keep entrypoint_error message printing, but do not require the enum values
* clang format
* run kernel-collector tests on every gha run
* only check out submodules in ext/
* remove extraneous dependency of kct test on kernel-collector
* fix removal of change set
* Use CMAKE_HOST_SYSTEM_PROCESSOR instead of `uname -m`
* remove redundant cmd_args parameters in entrypoint_kct.sh
* write all libpf messages as debug in the BPF logging kind
* clang-format
* remove redundant __attribute__((preserve_access_index))
* write libbpf logs in default debug mode
* increase libbpf debug buffer size
* move LIBBPF_DEBUG messages to LOG::trace
* clang-format
* don't require HTTPS if pushing to localhost:5000
* fix variable scope
* add missing image pull from the temporary container registry
* clean up registry push script
* update go versions and vendor
* migrate ebpf tests to little-vm-helper
* fix actions versions
* speed up apt install
* build the container images for kernel collector, reducer and tests in separate jobs
* fix image references
* different way of speeding up apt installations
* add error handling
* get the container's internal error code, for the kernel-collector-test
* ci: remove snyk gha (#333)
* build: rename shell variable in clang-format scripts for better readability (#330)
found by @bjandras
* fix build for distro packages of curl, openssl, grpc, and abseil
* add header to include unique_ptr template
* fix by_key with initializer list
* add support for printing __int128 (for ipv6 addressess)
* add log formatter for protobuf
* fix constexpr enforcement in logging code
* fix protobuf dependencies so they won't recompile every time
* remove OpenSSL version check (we use the distro package)
* fix initializer list
* add function parameters required by new llvm version
* fix returning weak reference (a deleted function started getting enforced with llvm update)
* fix chrono logging support
* remove double template default parameter
* fix logging of optional<>
* move chrono support only to files requiring it
* update bcc for clang/llvm version 16
* use find_package for protobuf and grpc
* run clang-format with updated version (required for github actions to succeed)
* ci: disable fossa, ossf-scorecard, and trivy-scans on forks
* allow users to override their benv image
* use the same clang-format version as in benv for gha
* explicitly set clang-format binary with version
* run clang-format-16 on codebase
* use new mount structure in build-and-test.yaml
* update release directory structure
* reinstate linking workaround for zlib
* fix reference to protobuf
* fix protobuf generation dependency
* ensure protoc runs after all preparations
* add a protobuf build target
* fix overly restrictive casting in test
* remove counter_to_rate (dead code)
* removed mention of dead code
* fix debug build - requires chrono fmt
* update makefiles to use podman
* upgrade docker images to bookworm
* switch to install_packages (reduces apt state in container)
* update comment on protobuf
* add required libraries to containers
* add permissions for podman to make containers
* use podman when tagging docker images ahead of registry push
* migrate ebpf tests to little-vm-helper
* fix actions versions
* speed up apt install
* build the container images for kernel collector, reducer and tests in separate jobs
* fix image references
* different way of speeding up apt installations
* add error handling
* get the container's internal error code, for the kernel-collector-test
* build: update runners to ubuntu-24.04 from deprecated ubuntu-20.04
* update clang-format package installation
* use `clang-format` in checker script
* run clang-format
* add clang-format script
* run clang-format from ubuntu-24.04
* fix JSON request parsing in otlp grpc formatter test
* switch from camelCase to snake_case in otlp formatter verification
AGSaidi