semantic-conventions/docs/faas/aws-lambda.md

# Instrumenting AWS Lambda

**Status**: [Development][DocumentStatus]

This document defines how to apply semantic conventions when instrumenting an AWS Lambda request handler. AWS
Lambda largely follows the conventions for [FaaS][faas] while [HTTP](/docs/http/http-spans.md) conventions are also
applicable when handlers are for HTTP requests.

There are a variety of triggers for Lambda functions, and this document will grow over time to cover all the
use cases.

<!-- toc -->

- [All triggers](#all-triggers)
  - [AWS X-Ray active tracing considerations](#aws-x-ray-active-tracing-considerations)
    - [`xray-lambda` propagator functionality](#xray-lambda-propagator-functionality)
    - [`xray-lambda` Propagator Configuration](#xray-lambda-propagator-configuration)
- [API Gateway](#api-gateway)
- [SQS](#sqs)
  - [SQS event](#sqs-event)
  - [SQS message](#sqs-message)
- [Examples](#examples)
  - [API Gateway Request Proxy (Lambda tracing passive)](#api-gateway-request-proxy-lambda-tracing-passive)
  - [API Gateway Request Proxy (Lambda tracing active)](#api-gateway-request-proxy-lambda-tracing-active)
  - [SQS (Lambda tracing passive)](#sqs-lambda-tracing-passive)
  - [SQS (Lambda tracing active)](#sqs-lambda-tracing-active)
- [Resource Detector](#resource-detector)

<!-- tocstop -->

## All triggers

For all events, a span with kind `SERVER` SHOULD be created corresponding to the function invocation unless stated otherwise below.

The following attributes SHOULD be set:

- [`faas.invocation_id`][faas] - The value of the AWS Request ID, which is always available through an accessor on the Lambda `Context`.
- [`cloud.account.id`][cloud] - In some languages, this is available as an accessor on the Lambda `Context`. Otherwise, it can be parsed from the ARN as the fifth item when splitting on `:`

<!-- semconv span.aws.lambda.server -->
<!-- NOTE: THIS TEXT IS AUTOGENERATED. DO NOT EDIT BY HAND. -->
<!-- see templates/registry/markdown/snippet.md.j2 -->
<!-- prettier-ignore-start -->
<!-- markdownlint-capture -->
<!-- markdownlint-disable -->

**Status:** ![Development](https://img.shields.io/badge/-development-blue)

This span represents AWS Lambda invocation.

Consider setting other attributes of the [`faas` resource][faasres] and [trace][faas] conventions
and the [cloud resource conventions][cloud].

**Span name** MUST be set to the function name from the Lambda `Context`
unless stated otherwise.

**Span kind** MUST be set to `SERVER` unless stated otherwise.

**Span status** SHOULD follow the [Recording Errors](/docs/general/recording-errors.md) document.

| Attribute  | Type | Description  | Examples  | [Requirement Level](https://opentelemetry.io/docs/specs/semconv/general/attribute-requirement-level/) | Stability |
|---|---|---|---|---|---|
| [`aws.lambda.invoked_arn`](/docs/attributes-registry/aws.md) | string | The full invoked ARN as provided on the `Context` passed to the function (`Lambda-Runtime-Invoked-Function-Arn` header on the `/runtime/invocation/next` applicable). [1] | `arn:aws:lambda:us-east-1:123456:function:myfunction:myalias` | `Recommended` | ![Development](https://img.shields.io/badge/-development-blue) |

**[1] `aws.lambda.invoked_arn`:** This may be different from `cloud.resource_id` if an alias is involved.

<!-- markdownlint-restore -->
<!-- prettier-ignore-end -->
<!-- END AUTOGENERATED TEXT -->
<!-- endsemconv -->

[faas]: faas-spans.md (FaaS trace conventions)
[faasres]: /docs/resource/faas.md (FaaS resource conventions)
[cloud]: /docs/resource/cloud.md (Cloud resource conventions)

### AWS X-Ray active tracing considerations

When [AWS X-Ray Active Tracing](https://docs.aws.amazon.com/lambda/latest/dg/services-xray.html) is enabled for a Lambda,
the runtime will automatically generate a span based on configured sampling rates and propagate the span context
via the `_X_AMZN_TRACE_ID` environment variable (and the `com.amazonaws.xray.traceHeader` system property for Java Lambda functions).
This span context is encoded using the [X-Ray Tracing Header Format](https://docs.aws.amazon.com/xray/latest/devguide/xray-concepts.html#xray-concepts-tracingheader).

Users MUST be able to [configure the propagator](#xray-lambda-propagator-configuration) to prioritize propagating this X-Ray "Active Tracing" span context.
(FYI: Users probably want this enabled if OpenTelemetry is configured to report spans to AWS X-Ray so their trace is linked together properly.)

#### `xray-lambda` propagator functionality

SDK's that have instrumentation for AWS Lambda SHOULD provide an additional propagator alongside the X-Ray propagator
that can [be configured](#xray-lambda-propagator-configuration) via the `OTEL_PROPAGATORS` environment variable setting as `xray-lambda`.
This propagator is expected to replace the `xray` propagator in the `OTEL_PROPAGATORS` list. The behavior for this propagator is described in pseudo code below.

```
extract(context, carrier) {
    xrayContext = xrayPropagator.extract(context, carrier)

    // To avoid potential issues when extracting with an active span context (such as with a span link),
    // the `xray-lambda` propagator SHOULD check if the provided context already has an active span context.
    // If found, the propagator SHOULD just return the extract result of the `xray` propagator.
    if (Span.fromContext(context).getSpanContext().isValid())
      return xrayContext

    // If xray-lambda environment variable not set, return the xray extract result.
    traceHeader = getEnvironment("_X_AMZN_TRACE_ID")
    if (isEmptyOrNull(traceHeader))
      return xrayContext

    // Apply the xray propagator using the span context contained in the xray-lambda environment variable.
    return xrayPropagator.extract(xrayContext, ["X-Amzn-Trace-Id": traceHeader])
}
```

*Note:* Java implementations should use the system property value of the key `com.amazonaws.xray.traceHeader`
instead of the environment variable if the system property is not empty.

#### `xray-lambda` Propagator Configuration

**When reporting spans to AWS X-Ray** from AWS Lambda, the `xray-lambda` propagator SHOULD replace the `xray` propagator in the `OTEL_PROPAGATORS` configuration. Including both will prevent `xray-lambda` from functioning properly.

Example valid configuration when reporting spans to AWS X-Ray:

- `OTEL_PROPAGATORS=tracecontext,baggage,xray-lambda`

Example invalid configurations:

- `OTEL_PROPAGATORS=tracecontext,baggage,xray,xray-lambda`
- `OTEL_PROPAGATORS=tracecontext,baggage,xray-lambda,xray`

**When OpenTelemetry is reporting traces to another system besides AWS X-Ray**, users SHOULD NOT use `xray-lambda` or reported traces will be broken.

Example valid configuration when OpenTelemetry is reporting traces to another system besides AWS X-Ray:

- `OTEL_PROPAGATORS=tracecontext,baggage,xray`

## API Gateway

API Gateway allows a user to trigger a Lambda function in response to HTTP requests. It can be configured to be
a pure proxy, where the information about the original HTTP request is passed to the Lambda function, or as a
configuration for a REST API, in which case only a deserialized body payload is available.  In the case the API
gateway is configured to proxy to the Lambda function, the instrumented request handler will have access to all
the information about the HTTP request in the form of an API Gateway Proxy Request Event.

The Lambda span name and the [`http.route` span attribute](/docs/http/http-spans.md#http-server-span) SHOULD
be set to the [resource property][] from the proxy request event, which corresponds to the user configured HTTP
route instead of the function name.

[`faas.trigger`][faas] MUST be set to `http`. [HTTP attributes](/docs/http/http-spans.md) SHOULD be set based on the
available information in the Lambda event initiated by the proxy request. `http.scheme` is available as the
`x-forwarded-proto` header in the Lambda event. Refer to the [input event format][] for more details.

[resource property]: https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html#api-gateway-simple-proxy-for-lambda-input-format
[input event format]: https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html#api-gateway-simple-proxy-for-lambda-input-format

## SQS

Amazon Simple Queue Service (SQS) is a message queue that triggers a Lambda function with a batch of messages.
So we consider processing both of a batch and of each individual message. The function invocation span MUST
correspond to the SQS event, which is the batch of messages. For each message, an additional span SHOULD be
created to correspond with the handling of the SQS message. Because handling of a message will be inside user
business logic, not the Lambda framework, automatic instrumentation mechanisms without code change will often
not be able to instrument the processing of the individual messages. Instrumentation SHOULD provide utilities
for creating message processing spans within user code.

The span kind for both types of SQS spans SHOULD be `CONSUMER`.

### SQS event

For the SQS event span, if all the messages in the event have the same event source, the name of the span MUST
be `<event source> process`. If there are multiple sources in the batch, the name MUST be
`multiple_sources process`. The parent SHOULD be the `SERVER` span corresponding to the function invocation.

For every message in the event, the [message system attributes][] (not message attributes, which are provided by
the user) SHOULD be checked for the key `AWSTraceHeader`. If it is present, an OpenTelemetry `Context` SHOULD be
parsed from the value of the attribute using the [AWS X-Ray Propagator](https://github.com/open-telemetry/opentelemetry-specification/tree/v1.44.0/specification/context/api-propagators.md) and
added as a link to the span. This means the span may have as many links as messages in the batch.
See [compatibility](../non-normative/compatibility/aws.md#context-propagation) for more info.

- [`faas.trigger`][faas] MUST be set to `pubsub`.
- [`messaging.operation.type`](/docs/messaging/messaging-spans.md) MUST be set to `process`.
- [`messaging.system`](/docs/messaging/messaging-spans.md) MUST be set to `aws_sqs`.

### SQS message

For the SQS message span, the name MUST be `<event source> process`.  The parent MUST be the `CONSUMER` span
corresponding to the SQS event. The [message system attributes][] (not message attributes, which are provided by
the user) SHOULD be checked for the key `AWSTraceHeader`. If it is present, an OpenTelemetry `Context` SHOULD be
parsed from the value of the attribute using the [AWS X-Ray Propagator](https://github.com/open-telemetry/opentelemetry-specification/tree/v1.44.0/specification/context/api-propagators.md) and
added as a link to the span.
See [compatibility](../non-normative/compatibility/aws.md#context-propagation) for more info.

- [`faas.trigger`][faas] MUST be set to `pubsub`.
- [`messaging.operation.type`](/docs/messaging/messaging-spans.md#messaging-attributes) MUST be set to `process`.
- [`messaging.system`](/docs/messaging/messaging-spans.md#messaging-attributes) MUST be set to `aws_sqs`.

Other [Messaging attributes](/docs/messaging/messaging-spans.md#messaging-attributes) SHOULD be set based on the available information in the SQS message
event.

Note that `AWSTraceHeader` is the only supported mechanism for propagating `Context` in instrumentation for SQS
to prevent conflicts with other sources. Notably, message attributes (user-provided, not system) are not supported -
the linked contexts are always expected to have been sent as HTTP headers of the `SQS.SendMessage` request that
the message originated from. This is a function of AWS SDK instrumentation, not Lambda instrumentation.

Using the `AWSTraceHeader` ensures that propagation will work across AWS services that may be integrated to
Lambda via SQS, for example a flow that goes through S3 -> SNS -> SQS -> Lambda. `AWSTraceHeader` is only a means
of propagating context and not tied to any particular observability backend. Notably, using it does not imply
using AWS X-Ray - any observability backend will fully function using this propagation mechanism.

[message system attributes]: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-message-metadata.html#sqs-message-system-attributes

## Examples

### API Gateway Request Proxy (Lambda tracing passive)

Given a process C that sends an HTTP request to an API Gateway endpoint with path `/pets/{petId}` configured for
a Lambda function F:

```
Process C: | Span Client        |
--
Function F:    | Span Function |
```

| Field or Attribute | `Span Client` | `Span Function` |
|-|-|-|
| Span name | `HTTP GET` | `/pets/{petId}` |
| Parent |  | Span Client |
| SpanKind | `CLIENT` | `SERVER` |
| Status | `Ok` | `Ok` |
| `faas.invocation_id` | | `79104EXAMPLEB723` |
| `faas.trigger` | | `http` |
| `cloud.account.id` | | `12345678912` |
| `server.address` | `foo.execute-api.us-east-1.amazonaws.com` |  |
| `server.port` | `413` |  |
| `http.request.method` | `GET` | `GET` |
| `user_agent.original` | `okhttp 3.0` | `okhttp 3.0` |
| `url.scheme` | | `https` |
| `url.path` | | `/pets/10` |
| `http.route` | | `/pets/{petId}` |
| `http.response.status_code` | `200` | `200` |

### API Gateway Request Proxy (Lambda tracing active)

Active tracing in Lambda means an API Gateway span `Span APIGW` and a Lambda runtime invocation span `Span Lambda`
will be exported to AWS X-Ray by the infrastructure (not instrumentation). All attributes above are the same
except that in this case, the parent of `APIGW` is `Span Client` and the parent of `Span Function` is
`Span Lambda`. This means the hierarchy looks like:

```
Span Client --> Span APIGW --> Span Lambda --> Span Function
```

### SQS (Lambda tracing passive)

Given a process P, that sends two messages to a queue Q on SQS, and a Lambda function F, which processes both of them in one batch (Span ProcBatch) and
generates a processing span for each message separately (Spans Proc1 and Proc2).

```
Process P: | Span Prod1 | Span Prod2 |
--
Function F:                      | Span ProcBatch |
                                        | Span Proc1 |
                                               | Span Proc2 |
```

| Field or Attribute | Span Prod1 | Span Prod2 | Span ProcBatch | Span Proc1 | Span Proc2 |
|-|-|-|-|-|-|
| Span name | `send Q` | `send Q` | `process Q` | `process Q` | `process Q` |
| Parent |  |  |  | Span ProcBatch | Span ProcBatch |
| Links |  |  |  | Span Prod1 | Span Prod2 |
| SpanKind | `PRODUCER` | `PRODUCER` | `CONSUMER` | `CONSUMER` | `CONSUMER` |
| Status | `Ok` | `Ok` | `Ok` | `Ok` | `Ok` |
| `messaging.system` | `aws_sqs` | `aws_sqs` | `aws_sqs` | `aws_sqs` | `aws_sqs` |
| `messaging.destination.name` | `Q` | `Q` | `Q` | `Q` | `Q` |
| `messaging.operation.name` | `send` | `send` | `process` | `process` | `process` |
| `messaging.operation.type` | `publish` | `publish` | `process` | `process` | `process` |
| `messaging.message.id` | | | | `"a1"` | `"a2"` |

Note that if Span Prod1 and Span Prod2 were sent to different queues, Span ProcBatch would not have
`messaging.destination.name` set as it would correspond to multiple queues.

The above requires user code change to create `Span Proc1` and `Span Proc2`. In Java, the user would inherit from
[TracingSqsMessageHandler][] instead of Lambda's standard `RequestHandler` to enable them. Otherwise these two spans
would not exist.

[TracingSqsMessageHandler]: https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/v1.0.1/instrumentation/aws-lambda-1.0/library/src/main/java/io/opentelemetry/instrumentation/awslambda/v1_0/TracingSqsMessageHandler.java

### SQS (Lambda tracing active)

Active tracing in Lambda means a Lambda runtime invocation span `Span Lambda` will be exported to X-Ray by the
infrastructure (not instrumentation). In this case, all of the above is the same except `Span ProcBatch` will
have a parent of `Span Lambda`. This means the hierarchy looks like:

```
Span Lambda --> Span ProcBatch --> Span Proc1 (links to Span Prod1 and Span Prod2)
                               \-> Span Proc2 (links to Span Prod1 and Span Prod2)
```

## Resource Detector

AWS Lambda resource information is available as [environment variables][] provided by the runtime.

- [`cloud.provider`][cloud] MUST be set to `aws`
- [`cloud.region`][cloud] MUST be set to the value of the `AWS_REGION` environment variable
- [`faas.name`][faasres] MUST be set to the value of the `AWS_LAMBDA_FUNCTION_NAME` environment variable
- [`faas.version`][faasres] MUST be set to the value of the `AWS_LAMBDA_FUNCTION_VERSION` environment variable

Note that [`cloud.resource_id`][cloud] currently cannot be populated as a resource
because it is not available until function invocation.

[environment variables]: https://docs.aws.amazon.com/lambda/latest/dg/configuration-envvars.html#configuration-envvars-runtime

[DocumentStatus]: https://opentelemetry.io/docs/specs/otel/document-status