Update prom rw exporter (#1359)
This commit is contained in:
Youssef Gamal
GitHub
parent
26d3343428
commit
0dc16a4118
1
.flake8
1
.flake8
|
|
@ -16,6 +16,7 @@ exclude =
|
|||
target
|
||||
__pycache__
|
||||
exporter/opentelemetry-exporter-jaeger/src/opentelemetry/exporter/jaeger/gen/
|
||||
exporter/opentelemetry-exporter-prometheus-remote-write/src/opentelemetry/exporter/prometheus_remote_write/gen/
|
||||
exporter/opentelemetry-exporter-jaeger/build/*
|
||||
docs/examples/opentelemetry-example-app/src/opentelemetry_example_app/grpc/gen/
|
||||
docs/examples/opentelemetry-example-app/build/*
|
||||
|
|
|
|||
|
|
@ -31,6 +31,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||
([#1413](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/1413))
|
||||
- `opentelemetry-instrumentation-pyramid` Add support for regular expression matching and sanitization of HTTP headers.
|
||||
([#1414](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/1414))
|
||||
- Add metric exporter for Prometheus Remote Write
|
||||
([#1359](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/1359))
|
||||
|
||||
### Fixed
|
||||
|
||||
|
|
@ -62,6 +64,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||
- Add metric instrumentation in starlette
|
||||
([#1327](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/1327))
|
||||
|
||||
|
||||
### Fixed
|
||||
|
||||
- `opentelemetry-instrumentation-boto3sqs` Make propagation compatible with other SQS instrumentations, add 'messaging.url' span attribute, and fix missing package dependencies.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,29 @@
|
|||
OpenTelemetry Prometheus Remote Write Exporter
|
||||
==============================================
|
||||
|
||||
|pypi|
|
||||
|
||||
.. |pypi| image:: https://badge.fury.io/py/opentelemetry-exporter-prometheus-remote-write.svg
|
||||
:target: https://pypi.org/project/opentelemetry-exporter-prometheus-remote-write/
|
||||
|
||||
This package contains an exporter to send metrics from the OpenTelemetry Python SDK directly to a Prometheus Remote Write integrated backend
|
||||
(such as Cortex or Thanos) without having to run an instance of the Prometheus server.
|
||||
|
||||
|
||||
Installation
|
||||
------------
|
||||
|
||||
::
|
||||
|
||||
pip install opentelemetry-exporter-prometheus-remote-write
|
||||
|
||||
|
||||
.. _OpenTelemetry: https://github.com/open-telemetry/opentelemetry-python/
|
||||
.. _Prometheus Remote Write integrated backend: https://prometheus.io/docs/operating/integrations/
|
||||
|
||||
|
||||
References
|
||||
----------
|
||||
|
||||
* `OpenTelemetry Project <https://opentelemetry.io/>`_
|
||||
* `Prometheus Remote Write Integration <https://prometheus.io/docs/operating/integrations/>`_
|
||||
|
|
@ -0,0 +1,11 @@
|
|||
FROM python:3.8
|
||||
|
||||
RUN apt-get update -y && apt-get install libsnappy-dev -y
|
||||
|
||||
WORKDIR /code
|
||||
COPY . .
|
||||
|
||||
RUN pip install -e .
|
||||
RUN pip install -r ./examples/requirements.txt
|
||||
|
||||
CMD ["python", "./examples/sampleapp.py"]
|
||||
|
|
@ -0,0 +1,42 @@
|
|||
# Prometheus Remote Write Exporter Example
|
||||
This example uses [Docker Compose](https://docs.docker.com/compose/) to set up:
|
||||
|
||||
1. A Python program that creates 5 instruments with 5 unique
|
||||
aggregators and a randomized load generator
|
||||
2. An instance of [Cortex](https://cortexmetrics.io/) to receive the metrics
|
||||
data
|
||||
3. An instance of [Grafana](https://grafana.com/) to visualizse the exported
|
||||
data
|
||||
|
||||
## Requirements
|
||||
* Have Docker Compose [installed](https://docs.docker.com/compose/install/)
|
||||
|
||||
*Users do not need to install Python as the app will be run in the Docker Container*
|
||||
|
||||
## Instructions
|
||||
1. Run `docker-compose up -d` in the the `examples/` directory
|
||||
|
||||
The `-d` flag causes all services to run in detached mode and frees up your
|
||||
terminal session. This also causes no logs to show up. Users can attach themselves to the service's logs manually using `docker logs ${CONTAINER_ID} --follow`
|
||||
|
||||
2. Log into the Grafana instance at [http://localhost:3000](http://localhost:3000)
|
||||
* login credentials are `username: admin` and `password: admin`
|
||||
* There may be an additional screen on setting a new password. This can be skipped and is optional
|
||||
|
||||
3. Navigate to the `Data Sources` page
|
||||
* Look for a gear icon on the left sidebar and select `Data Sources`
|
||||
|
||||
4. Add a new Prometheus Data Source
|
||||
* Use `http://cortex:9009/api/prom` as the URL
|
||||
* (OPTIONAl) set the scrape interval to `2s` to make updates appear quickly
|
||||
* click `Save & Test`
|
||||
|
||||
5. Go to `Metrics Explore` to query metrics
|
||||
* Look for a compass icon on the left sidebar
|
||||
* click `Metrics` for a dropdown list of all the available metrics
|
||||
* (OPTIONAL) Adjust time range by clicking the `Last 6 hours` button on the upper right side of the graph
|
||||
* (OPTIONAL) Set up auto-refresh by selecting an option under the dropdown next to the refresh button on the upper right side of the graph
|
||||
* Click the refresh button and data should show up on the graph
|
||||
|
||||
6. Shutdown the services when finished
|
||||
* Run `docker-compose down` in the examples directory
|
||||
|
|
@ -0,0 +1,101 @@
|
|||
# This Cortex Config is copied from the Cortex Project documentation
|
||||
# Source: https://github.com/cortexproject/cortex/blob/master/docs/configuration/single-process-config.yaml
|
||||
|
||||
# Configuration for running Cortex in single-process mode.
|
||||
# This configuration should not be used in production.
|
||||
# It is only for getting started and development.
|
||||
|
||||
# Disable the requirement that every request to Cortex has a
|
||||
# X-Scope-OrgID header. `fake` will be substituted in instead.
|
||||
# pylint: skip-file
|
||||
auth_enabled: false
|
||||
|
||||
server:
|
||||
http_listen_port: 9009
|
||||
|
||||
# Configure the server to allow messages up to 100MB.
|
||||
grpc_server_max_recv_msg_size: 104857600
|
||||
grpc_server_max_send_msg_size: 104857600
|
||||
grpc_server_max_concurrent_streams: 1000
|
||||
|
||||
distributor:
|
||||
shard_by_all_labels: true
|
||||
pool:
|
||||
health_check_ingesters: true
|
||||
|
||||
ingester_client:
|
||||
grpc_client_config:
|
||||
# Configure the client to allow messages up to 100MB.
|
||||
max_recv_msg_size: 104857600
|
||||
max_send_msg_size: 104857600
|
||||
use_gzip_compression: true
|
||||
|
||||
ingester:
|
||||
# We want our ingesters to flush chunks at the same time to optimise
|
||||
# deduplication opportunities.
|
||||
spread_flushes: true
|
||||
chunk_age_jitter: 0
|
||||
|
||||
walconfig:
|
||||
wal_enabled: true
|
||||
recover_from_wal: true
|
||||
wal_dir: /tmp/cortex/wal
|
||||
|
||||
lifecycler:
|
||||
# The address to advertise for this ingester. Will be autodiscovered by
|
||||
# looking up address on eth0 or en0; can be specified if this fails.
|
||||
# address: 127.0.0.1
|
||||
|
||||
# We want to start immediately and flush on shutdown.
|
||||
join_after: 0
|
||||
min_ready_duration: 0s
|
||||
final_sleep: 0s
|
||||
num_tokens: 512
|
||||
tokens_file_path: /tmp/cortex/wal/tokens
|
||||
|
||||
# Use an in memory ring store, so we don't need to launch a Consul.
|
||||
ring:
|
||||
kvstore:
|
||||
store: inmemory
|
||||
replication_factor: 1
|
||||
|
||||
# Use local storage - BoltDB for the index, and the filesystem
|
||||
# for the chunks.
|
||||
schema:
|
||||
configs:
|
||||
- from: 2019-07-29
|
||||
store: boltdb
|
||||
object_store: filesystem
|
||||
schema: v10
|
||||
index:
|
||||
prefix: index_
|
||||
period: 1w
|
||||
|
||||
storage:
|
||||
boltdb:
|
||||
directory: /tmp/cortex/index
|
||||
|
||||
filesystem:
|
||||
directory: /tmp/cortex/chunks
|
||||
|
||||
delete_store:
|
||||
store: boltdb
|
||||
|
||||
purger:
|
||||
object_store_type: filesystem
|
||||
|
||||
frontend_worker:
|
||||
# Configure the frontend worker in the querier to match worker count
|
||||
# to max_concurrent on the queriers.
|
||||
match_max_concurrent: true
|
||||
|
||||
# Configure the ruler to scan the /tmp/cortex/rules directory for prometheus
|
||||
# rules: https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/#recording-rules
|
||||
ruler:
|
||||
enable_api: true
|
||||
enable_sharding: false
|
||||
storage:
|
||||
type: local
|
||||
local:
|
||||
directory: /tmp/cortex/rules
|
||||
|
||||
|
|
@ -0,0 +1,33 @@
|
|||
# Copyright The OpenTelemetry Authors
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
version: "3.8"
|
||||
|
||||
services:
|
||||
cortex:
|
||||
image: quay.io/cortexproject/cortex:v1.5.0
|
||||
command:
|
||||
- -config.file=./config/cortex-config.yml
|
||||
volumes:
|
||||
- ./cortex-config.yml:/config/cortex-config.yml:ro
|
||||
ports:
|
||||
- 9009:9009
|
||||
grafana:
|
||||
image: grafana/grafana:latest
|
||||
ports:
|
||||
- 3000:3000
|
||||
sample_app:
|
||||
build:
|
||||
context: ../
|
||||
dockerfile: ./examples/Dockerfile
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
psutil
|
||||
protobuf>=3.13.0
|
||||
requests>=2.25.0
|
||||
python-snappy>=0.5.4
|
||||
opentelemetry-api
|
||||
opentelemetry-sdk
|
||||
opentelemetry-proto
|
||||
|
|
@ -0,0 +1,114 @@
|
|||
# Copyright The OpenTelemetry Authors
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import logging
|
||||
import random
|
||||
import sys
|
||||
import time
|
||||
from logging import INFO
|
||||
|
||||
import psutil
|
||||
|
||||
from opentelemetry import metrics
|
||||
from opentelemetry.exporter.prometheus_remote_write import (
|
||||
PrometheusRemoteWriteMetricsExporter,
|
||||
)
|
||||
from opentelemetry.metrics import Observation
|
||||
from opentelemetry.sdk.metrics import MeterProvider
|
||||
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
|
||||
|
||||
logging.basicConfig(stream=sys.stdout, level=logging.INFO)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
testing_labels = {"environment": "testing"}
|
||||
|
||||
exporter = PrometheusRemoteWriteMetricsExporter(
|
||||
endpoint="http://cortex:9009/api/prom/push",
|
||||
headers={"X-Scope-Org-ID": "5"},
|
||||
)
|
||||
reader = PeriodicExportingMetricReader(exporter, 1000)
|
||||
provider = MeterProvider(metric_readers=[reader])
|
||||
metrics.set_meter_provider(provider)
|
||||
meter = metrics.get_meter(__name__)
|
||||
|
||||
|
||||
# Callback to gather cpu usage
|
||||
def get_cpu_usage_callback(observer):
|
||||
for (number, percent) in enumerate(psutil.cpu_percent(percpu=True)):
|
||||
labels = {"cpu_number": str(number)}
|
||||
yield Observation(percent, labels)
|
||||
|
||||
|
||||
# Callback to gather RAM usage
|
||||
def get_ram_usage_callback(observer):
|
||||
ram_percent = psutil.virtual_memory().percent
|
||||
yield Observation(ram_percent, {})
|
||||
|
||||
|
||||
requests_counter = meter.create_counter(
|
||||
name="requests",
|
||||
description="number of requests",
|
||||
unit="1",
|
||||
)
|
||||
|
||||
request_min_max = meter.create_counter(
|
||||
name="requests_min_max",
|
||||
description="min max sum count of requests",
|
||||
unit="1",
|
||||
)
|
||||
|
||||
request_last_value = meter.create_counter(
|
||||
name="requests_last_value",
|
||||
description="last value number of requests",
|
||||
unit="1",
|
||||
)
|
||||
|
||||
requests_active = meter.create_up_down_counter(
|
||||
name="requests_active",
|
||||
description="number of active requests",
|
||||
unit="1",
|
||||
)
|
||||
|
||||
meter.create_observable_counter(
|
||||
callbacks=[get_ram_usage_callback],
|
||||
name="ram_usage",
|
||||
description="ram usage",
|
||||
unit="1",
|
||||
)
|
||||
|
||||
meter.create_observable_up_down_counter(
|
||||
callbacks=[get_cpu_usage_callback],
|
||||
name="cpu_percent",
|
||||
description="per-cpu usage",
|
||||
unit="1",
|
||||
)
|
||||
|
||||
request_latency = meter.create_histogram("request_latency")
|
||||
|
||||
# Load generator
|
||||
num = random.randint(0, 1000)
|
||||
while True:
|
||||
# counters
|
||||
requests_counter.add(num % 131 + 200, testing_labels)
|
||||
request_min_max.add(num % 181 + 200, testing_labels)
|
||||
request_last_value.add(num % 101 + 200, testing_labels)
|
||||
|
||||
# updown counter
|
||||
requests_active.add(num % 7231 + 200, testing_labels)
|
||||
|
||||
request_latency.record(num % 92, testing_labels)
|
||||
logger.log(level=INFO, msg="completed metrics collection cycle")
|
||||
time.sleep(1)
|
||||
num += 9791
|
||||
|
|
@ -0,0 +1 @@
|
|||
opentelemetry
|
||||
|
|
@ -0,0 +1,3 @@
|
|||
## Instructions
|
||||
1. Install protobuf tools. Can use your package manager or download from [GitHub](https://github.com/protocolbuffers/protobuf/releases/tag/v21.7)
|
||||
2. Run `generate-proto-py.sh` from inside the `proto/` directory
|
||||
|
|
@ -0,0 +1,57 @@
|
|||
#!/bin/bash
|
||||
|
||||
PROM_VERSION=v2.39.0
|
||||
PROTO_VERSION=v1.3.2
|
||||
|
||||
# SRC_DIR is from protoc perspective. ie its the destination for our checkouts/clones
|
||||
SRC_DIR=opentelemetry/exporter/prometheus_remote_write/gen/
|
||||
DST_DIR=../src/opentelemetry/exporter/prometheus_remote_write/gen/
|
||||
|
||||
#TODO:
|
||||
# Check that black & protoc are installed properly
|
||||
echo "Creating our destination directory"
|
||||
mkdir -p ${SRC_DIR}/gogoproto
|
||||
|
||||
# Clone prometheus
|
||||
echo "Grabbing Prometheus protobuf files"
|
||||
git clone --filter=blob:none --sparse https://github.com/prometheus/prometheus.git
|
||||
cd prometheus
|
||||
git checkout ${PROM_VERSION}
|
||||
git sparse-checkout set prompb
|
||||
cd ..
|
||||
|
||||
|
||||
# We also need gogo.proto which is in the protobuf Repo
|
||||
# Could also try to pull this locally from the install location of protobuf
|
||||
# but that will be harder in a platform agnostic way.
|
||||
echo "Grabbing gogo.proto"
|
||||
git clone --filter=blob:none --sparse https://github.com/gogo/protobuf.git
|
||||
cd protobuf
|
||||
git checkout ${PROTO_VERSION}
|
||||
git sparse-checkout set /gogoproto/gogo.proto
|
||||
cd ..
|
||||
|
||||
# Move the proto files into our structure
|
||||
echo "Moving proto files to ${SRC_DIR}"
|
||||
cp prometheus/prompb/remote.proto prometheus/prompb/types.proto ${SRC_DIR}
|
||||
cp protobuf/gogoproto/gogo.proto ${SRC_DIR}/gogoproto/
|
||||
|
||||
|
||||
# A bit of a hack, but we need to fix the imports to fit the python structure.
|
||||
# using sed to find the 3 files and point them at each other using OUR structure
|
||||
echo "Fixing imports"
|
||||
sed -i 's/import "types.proto";/import "opentelemetry\/exporter\/prometheus_remote_write\/gen\/types.proto";/' ${SRC_DIR}/remote.proto
|
||||
sed -i 's/import "gogoproto\/gogo.proto";/import "opentelemetry\/exporter\/prometheus_remote_write\/gen\/gogoproto\/gogo.proto";/' ${SRC_DIR}/remote.proto
|
||||
sed -i 's/import "gogoproto\/gogo.proto";/import "opentelemetry\/exporter\/prometheus_remote_write\/gen\/gogoproto\/gogo.proto";/' ${SRC_DIR}/types.proto
|
||||
|
||||
|
||||
# Cleanup the repos
|
||||
echo "Removing clones..."
|
||||
rm -rf protobuf prometheus
|
||||
|
||||
# Used libprotoc 3.21.1 & protoc 21.7
|
||||
echo "Compiling proto files to Python"
|
||||
protoc -I . --python_out=../src ${SRC_DIR}/gogoproto/gogo.proto ${SRC_DIR}/remote.proto ${SRC_DIR}/types.proto
|
||||
|
||||
echo "Running formatting on the generated files"
|
||||
../../../scripts/eachdist.py format --path $PWD/..
|
||||
|
|
@ -0,0 +1,51 @@
|
|||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
|
||||
[project]
|
||||
name = "opentelemetry-exporter-prometheus-remote-write"
|
||||
dynamic = ["version"]
|
||||
description = "Prometheus Remote Write Metrics Exporter for OpenTelemetry"
|
||||
readme = "README.rst"
|
||||
license = "Apache-2.0"
|
||||
requires-python = ">=3.7"
|
||||
authors = [
|
||||
{ name = "OpenTelemetry Authors", email = "cncf-opentelemetry-contributors@lists.cncf.io" },
|
||||
]
|
||||
classifiers = [
|
||||
"Development Status :: 4 - Beta",
|
||||
"Intended Audience :: Developers",
|
||||
"License :: OSI Approved :: Apache Software License",
|
||||
"Programming Language :: Python",
|
||||
"Programming Language :: Python :: 3",
|
||||
"Programming Language :: Python :: 3.7",
|
||||
"Programming Language :: Python :: 3.8",
|
||||
"Programming Language :: Python :: 3.9",
|
||||
"Programming Language :: Python :: 3.10",
|
||||
]
|
||||
dependencies = [
|
||||
"protobuf ~= 4.21",
|
||||
"requests ~= 2.28",
|
||||
"opentelemetry-api ~= 1.12",
|
||||
"opentelemetry-sdk ~= 1.12",
|
||||
"python-snappy ~= 0.6",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
test = []
|
||||
|
||||
[project.urls]
|
||||
Homepage = "https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/exporter/opentelemetry-exporter-prometheus-remote-write"
|
||||
|
||||
[tool.hatch.version]
|
||||
path = "src/opentelemetry/exporter/prometheus_remote_write/version.py"
|
||||
|
||||
[tool.hatch.build.targets.sdist]
|
||||
include = [
|
||||
"/src",
|
||||
"/tests",
|
||||
]
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["src/opentelemetry"]
|
||||
|
|
@ -0,0 +1,414 @@
|
|||
# Copyright The OpenTelemetry Authors
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import logging
|
||||
import re
|
||||
from collections import defaultdict
|
||||
from itertools import chain
|
||||
from typing import Dict, Sequence
|
||||
|
||||
import requests
|
||||
import snappy
|
||||
|
||||
from opentelemetry.exporter.prometheus_remote_write.gen.remote_pb2 import ( # pylint: disable=no-name-in-module
|
||||
WriteRequest,
|
||||
)
|
||||
from opentelemetry.exporter.prometheus_remote_write.gen.types_pb2 import ( # pylint: disable=no-name-in-module
|
||||
Label,
|
||||
Sample,
|
||||
TimeSeries,
|
||||
)
|
||||
from opentelemetry.sdk.metrics import Counter
|
||||
from opentelemetry.sdk.metrics import Histogram as ClientHistogram
|
||||
from opentelemetry.sdk.metrics import (
|
||||
ObservableCounter,
|
||||
ObservableGauge,
|
||||
ObservableUpDownCounter,
|
||||
UpDownCounter,
|
||||
)
|
||||
from opentelemetry.sdk.metrics.export import (
|
||||
AggregationTemporality,
|
||||
Gauge,
|
||||
Histogram,
|
||||
Metric,
|
||||
MetricExporter,
|
||||
MetricExportResult,
|
||||
MetricsData,
|
||||
Sum,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
PROMETHEUS_NAME_REGEX = re.compile(r"^\d|[^\w:]")
|
||||
PROMETHEUS_LABEL_REGEX = re.compile(r"^\d|[^\w]")
|
||||
UNDERSCORE_REGEX = re.compile(r"_+")
|
||||
|
||||
|
||||
class PrometheusRemoteWriteMetricsExporter(MetricExporter):
|
||||
"""
|
||||
Prometheus remote write metric exporter for OpenTelemetry.
|
||||
|
||||
Args:
|
||||
endpoint: url where data will be sent (Required)
|
||||
basic_auth: username and password for authentication (Optional)
|
||||
headers: additional headers for remote write request (Optional)
|
||||
timeout: timeout for remote write requests in seconds, defaults to 30 (Optional)
|
||||
proxies: dict mapping request proxy protocols to proxy urls (Optional)
|
||||
tls_config: configuration for remote write TLS settings (Optional)
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
endpoint: str,
|
||||
basic_auth: Dict = None,
|
||||
headers: Dict = None,
|
||||
timeout: int = 30,
|
||||
tls_config: Dict = None,
|
||||
proxies: Dict = None,
|
||||
resources_as_labels: bool = True,
|
||||
preferred_temporality: Dict[type, AggregationTemporality] = None,
|
||||
preferred_aggregation: Dict = None,
|
||||
):
|
||||
self.endpoint = endpoint
|
||||
self.basic_auth = basic_auth
|
||||
self.headers = headers
|
||||
self.timeout = timeout
|
||||
self.tls_config = tls_config
|
||||
self.proxies = proxies
|
||||
self.resources_as_labels = resources_as_labels
|
||||
|
||||
if not preferred_temporality:
|
||||
preferred_temporality = {
|
||||
Counter: AggregationTemporality.CUMULATIVE,
|
||||
UpDownCounter: AggregationTemporality.CUMULATIVE,
|
||||
ClientHistogram: AggregationTemporality.CUMULATIVE,
|
||||
ObservableCounter: AggregationTemporality.CUMULATIVE,
|
||||
ObservableUpDownCounter: AggregationTemporality.CUMULATIVE,
|
||||
ObservableGauge: AggregationTemporality.CUMULATIVE,
|
||||
}
|
||||
|
||||
super().__init__(preferred_temporality, preferred_aggregation)
|
||||
|
||||
@property
|
||||
def endpoint(self):
|
||||
return self._endpoint
|
||||
|
||||
@endpoint.setter
|
||||
def endpoint(self, endpoint: str):
|
||||
if endpoint == "":
|
||||
raise ValueError("endpoint required")
|
||||
self._endpoint = endpoint
|
||||
|
||||
@property
|
||||
def basic_auth(self):
|
||||
return self._basic_auth
|
||||
|
||||
@basic_auth.setter
|
||||
def basic_auth(self, basic_auth: Dict):
|
||||
if basic_auth:
|
||||
if "username" not in basic_auth:
|
||||
raise ValueError("username required in basic_auth")
|
||||
if "password_file" in basic_auth:
|
||||
if "password" in basic_auth:
|
||||
raise ValueError(
|
||||
"basic_auth cannot contain password and password_file"
|
||||
)
|
||||
with open( # pylint: disable=unspecified-encoding
|
||||
basic_auth["password_file"]
|
||||
) as file:
|
||||
basic_auth["password"] = file.readline().strip()
|
||||
elif "password" not in basic_auth:
|
||||
raise ValueError("password required in basic_auth")
|
||||
self._basic_auth = basic_auth
|
||||
|
||||
@property
|
||||
def timeout(self):
|
||||
return self._timeout
|
||||
|
||||
@timeout.setter
|
||||
def timeout(self, timeout: int):
|
||||
if timeout <= 0:
|
||||
raise ValueError("timeout must be greater than 0")
|
||||
self._timeout = timeout
|
||||
|
||||
@property
|
||||
def tls_config(self):
|
||||
return self._tls_config
|
||||
|
||||
@tls_config.setter
|
||||
def tls_config(self, tls_config: Dict):
|
||||
if tls_config:
|
||||
new_config = {}
|
||||
if "ca_file" in tls_config:
|
||||
new_config["ca_file"] = tls_config["ca_file"]
|
||||
if "cert_file" in tls_config and "key_file" in tls_config:
|
||||
new_config["cert_file"] = tls_config["cert_file"]
|
||||
new_config["key_file"] = tls_config["key_file"]
|
||||
elif "cert_file" in tls_config or "key_file" in tls_config:
|
||||
raise ValueError(
|
||||
"tls_config requires both cert_file and key_file"
|
||||
)
|
||||
if "insecure_skip_verify" in tls_config:
|
||||
new_config["insecure_skip_verify"] = tls_config[
|
||||
"insecure_skip_verify"
|
||||
]
|
||||
self._tls_config = tls_config
|
||||
|
||||
@property
|
||||
def proxies(self):
|
||||
return self._proxies
|
||||
|
||||
@proxies.setter
|
||||
def proxies(self, proxies: Dict):
|
||||
self._proxies = proxies
|
||||
|
||||
@property
|
||||
def headers(self):
|
||||
return self._headers
|
||||
|
||||
@headers.setter
|
||||
def headers(self, headers: Dict):
|
||||
self._headers = headers
|
||||
|
||||
def export(
|
||||
self,
|
||||
metrics_data: MetricsData,
|
||||
timeout_millis: float = 10_000,
|
||||
**kwargs,
|
||||
) -> MetricExportResult:
|
||||
if not metrics_data:
|
||||
return MetricExportResult.SUCCESS
|
||||
timeseries = self._translate_data(metrics_data)
|
||||
if not timeseries:
|
||||
logger.error(
|
||||
"All records contain unsupported aggregators, export aborted"
|
||||
)
|
||||
return MetricExportResult.FAILURE
|
||||
message = self._build_message(timeseries)
|
||||
headers = self._build_headers()
|
||||
return self._send_message(message, headers)
|
||||
|
||||
def _translate_data(self, data: MetricsData) -> Sequence[TimeSeries]:
|
||||
rw_timeseries = []
|
||||
|
||||
for resource_metrics in data.resource_metrics:
|
||||
resource = resource_metrics.resource
|
||||
# OTLP Data model suggests combining some attrs into job/instance
|
||||
# Should we do that here?
|
||||
if self.resources_as_labels:
|
||||
resource_labels = [
|
||||
(n, str(v)) for n, v in resource.attributes.items()
|
||||
]
|
||||
else:
|
||||
resource_labels = []
|
||||
# Scope name/version probably not too useful from a labeling perspective
|
||||
for scope_metrics in resource_metrics.scope_metrics:
|
||||
for metric in scope_metrics.metrics:
|
||||
rw_timeseries.extend(
|
||||
self._parse_metric(metric, resource_labels)
|
||||
)
|
||||
return rw_timeseries
|
||||
|
||||
def _parse_metric(
|
||||
self, metric: Metric, resource_labels: Sequence
|
||||
) -> Sequence[TimeSeries]:
|
||||
"""
|
||||
Parses the Metric & lower objects, then converts the output into
|
||||
OM TimeSeries. Returns a List of TimeSeries objects based on one Metric
|
||||
"""
|
||||
|
||||
# Create the metric name, will be a label later
|
||||
if metric.unit:
|
||||
# Prom. naming guidelines add unit to the name
|
||||
name = f"{metric.name}_{metric.unit}"
|
||||
else:
|
||||
name = metric.name
|
||||
|
||||
# datapoints have attributes associated with them. these would be sent
|
||||
# to RW as different metrics: name & labels is a unique time series
|
||||
sample_sets = defaultdict(list)
|
||||
if isinstance(metric.data, (Gauge, Sum)):
|
||||
for dp in metric.data.data_points:
|
||||
attrs, sample = self._parse_data_point(dp, name)
|
||||
sample_sets[attrs].append(sample)
|
||||
elif isinstance(metric.data, Histogram):
|
||||
for dp in metric.data.data_points:
|
||||
dp_result = self._parse_histogram_data_point(dp, name)
|
||||
for attrs, sample in dp_result:
|
||||
sample_sets[attrs].append(sample)
|
||||
else:
|
||||
logger.warning("Unsupported Metric Type: %s", type(metric.data))
|
||||
return []
|
||||
return self._convert_to_timeseries(sample_sets, resource_labels)
|
||||
|
||||
def _convert_to_timeseries(
|
||||
self, sample_sets: Sequence[tuple], resource_labels: Sequence
|
||||
) -> Sequence[TimeSeries]:
|
||||
timeseries = []
|
||||
for labels, samples in sample_sets.items():
|
||||
ts = TimeSeries()
|
||||
for label_name, label_value in chain(resource_labels, labels):
|
||||
# Previous implementation did not str() the names...
|
||||
ts.labels.append(self._label(label_name, str(label_value)))
|
||||
for value, timestamp in samples:
|
||||
ts.samples.append(self._sample(value, timestamp))
|
||||
timeseries.append(ts)
|
||||
return timeseries
|
||||
|
||||
@staticmethod
|
||||
def _sample(value: int, timestamp: int) -> Sample:
|
||||
sample = Sample()
|
||||
sample.value = value
|
||||
sample.timestamp = timestamp
|
||||
return sample
|
||||
|
||||
def _label(self, name: str, value: str) -> Label:
|
||||
label = Label()
|
||||
label.name = self._sanitize_string(name, "label")
|
||||
label.value = value
|
||||
return label
|
||||
|
||||
@staticmethod
|
||||
def _sanitize_string(string: str, type_: str) -> str:
|
||||
# I Think Prometheus requires names to NOT start with a number this
|
||||
# would not catch that, but do cover the other cases. The naming rules
|
||||
# don't explicit say this, but the supplied regex implies it.
|
||||
# Got a little weird trying to do substitution with it, but can be
|
||||
# fixed if we allow numeric beginnings to metric names
|
||||
if type_ == "name":
|
||||
sanitized = PROMETHEUS_NAME_REGEX.sub("_", string)
|
||||
elif type_ == "label":
|
||||
sanitized = PROMETHEUS_LABEL_REGEX.sub("_", string)
|
||||
else:
|
||||
raise TypeError(f"Unsupported string type: {type_}")
|
||||
|
||||
# Remove consecutive underscores
|
||||
# TODO: Unfortunately this clobbbers __name__
|
||||
# sanitized = UNDERSCORE_REGEX.sub("_",sanitized)
|
||||
|
||||
return sanitized
|
||||
|
||||
def _parse_histogram_data_point(self, data_point, name):
|
||||
|
||||
sample_attr_pairs = []
|
||||
|
||||
base_attrs = list(data_point.attributes.items())
|
||||
timestamp = data_point.time_unix_nano // 1_000_000
|
||||
|
||||
def handle_bucket(value, bound=None, name_override=None):
|
||||
# Metric Level attributes + the bucket boundary attribute + name
|
||||
ts_attrs = base_attrs.copy()
|
||||
ts_attrs.append(
|
||||
(
|
||||
"__name__",
|
||||
self._sanitize_string(name_override or name, "name"),
|
||||
)
|
||||
)
|
||||
if bound:
|
||||
ts_attrs.append(("le", str(bound)))
|
||||
# Value is count of values in each bucket
|
||||
ts_sample = (value, timestamp)
|
||||
return tuple(ts_attrs), ts_sample
|
||||
|
||||
for bound_pos, bound in enumerate(data_point.explicit_bounds):
|
||||
sample_attr_pairs.append(
|
||||
handle_bucket(data_point.bucket_counts[bound_pos], bound)
|
||||
)
|
||||
|
||||
# Add the last label for implicit +inf bucket
|
||||
sample_attr_pairs.append(
|
||||
handle_bucket(data_point.bucket_counts[-1], bound="+Inf")
|
||||
)
|
||||
|
||||
# Lastly, add series for count & sum
|
||||
sample_attr_pairs.append(
|
||||
handle_bucket(data_point.sum, name_override=f"{name}_sum")
|
||||
)
|
||||
sample_attr_pairs.append(
|
||||
handle_bucket(data_point.count, name_override=f"{name}_count")
|
||||
)
|
||||
return sample_attr_pairs
|
||||
|
||||
def _parse_data_point(self, data_point, name=None):
|
||||
|
||||
attrs = tuple(data_point.attributes.items()) + (
|
||||
("__name__", self._sanitize_string(name, "name")),
|
||||
)
|
||||
sample = (data_point.value, (data_point.time_unix_nano // 1_000_000))
|
||||
return attrs, sample
|
||||
|
||||
@staticmethod
|
||||
def _build_message(timeseries: Sequence[TimeSeries]) -> bytes:
|
||||
write_request = WriteRequest()
|
||||
write_request.timeseries.extend(timeseries)
|
||||
serialized_message = write_request.SerializeToString()
|
||||
return snappy.compress(serialized_message)
|
||||
|
||||
def _build_headers(self) -> Dict:
|
||||
headers = {
|
||||
"Content-Encoding": "snappy",
|
||||
"Content-Type": "application/x-protobuf",
|
||||
"X-Prometheus-Remote-Write-Version": "0.1.0",
|
||||
}
|
||||
if self.headers:
|
||||
for header_name, header_value in self.headers.items():
|
||||
headers[header_name] = header_value
|
||||
return headers
|
||||
|
||||
def _send_message(
|
||||
self, message: bytes, headers: Dict
|
||||
) -> MetricExportResult:
|
||||
auth = None
|
||||
if self.basic_auth:
|
||||
auth = (self.basic_auth["username"], self.basic_auth["password"])
|
||||
|
||||
cert = None
|
||||
verify = True
|
||||
if self.tls_config:
|
||||
if "ca_file" in self.tls_config:
|
||||
verify = self.tls_config["ca_file"]
|
||||
elif "insecure_skip_verify" in self.tls_config:
|
||||
verify = self.tls_config["insecure_skip_verify"]
|
||||
|
||||
if (
|
||||
"cert_file" in self.tls_config
|
||||
and "key_file" in self.tls_config
|
||||
):
|
||||
cert = (
|
||||
self.tls_config["cert_file"],
|
||||
self.tls_config["key_file"],
|
||||
)
|
||||
try:
|
||||
response = requests.post(
|
||||
self.endpoint,
|
||||
data=message,
|
||||
headers=headers,
|
||||
auth=auth,
|
||||
timeout=self.timeout,
|
||||
proxies=self.proxies,
|
||||
cert=cert,
|
||||
verify=verify,
|
||||
)
|
||||
if not response.ok:
|
||||
response.raise_for_status()
|
||||
except requests.exceptions.RequestException as err:
|
||||
logger.error("Export POST request failed with reason: %s", err)
|
||||
return MetricExportResult.FAILURE
|
||||
return MetricExportResult.SUCCESS
|
||||
|
||||
def force_flush(self, timeout_millis: float = 10_000) -> bool:
|
||||
return True
|
||||
|
||||
def shutdown(self, timeout_millis: float = 30_000, **kwargs) -> None:
|
||||
pass
|
||||
File diff suppressed because one or more lines are too long
|
|
@ -0,0 +1,59 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
# Generated by the protocol buffer compiler. DO NOT EDIT!
|
||||
# source: opentelemetry/exporter/prometheus_remote_write/gen/remote.proto
|
||||
"""Generated protocol buffer code."""
|
||||
from google.protobuf.internal import builder as _builder
|
||||
from google.protobuf import descriptor as _descriptor
|
||||
from google.protobuf import descriptor_pool as _descriptor_pool
|
||||
from google.protobuf import symbol_database as _symbol_database
|
||||
|
||||
# @@protoc_insertion_point(imports)
|
||||
|
||||
_sym_db = _symbol_database.Default()
|
||||
|
||||
|
||||
from opentelemetry.exporter.prometheus_remote_write.gen import (
|
||||
types_pb2 as opentelemetry_dot_exporter_dot_prometheus__remote__write_dot_gen_dot_types__pb2,
|
||||
)
|
||||
from opentelemetry.exporter.prometheus_remote_write.gen.gogoproto import (
|
||||
gogo_pb2 as opentelemetry_dot_exporter_dot_prometheus__remote__write_dot_gen_dot_gogoproto_dot_gogo__pb2,
|
||||
)
|
||||
|
||||
|
||||
DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(
|
||||
b'\n?opentelemetry/exporter/prometheus_remote_write/gen/remote.proto\x12\nprometheus\x1a>opentelemetry/exporter/prometheus_remote_write/gen/types.proto\x1aGopentelemetry/exporter/prometheus_remote_write/gen/gogoproto/gogo.proto"z\n\x0cWriteRequest\x12\x30\n\ntimeseries\x18\x01 \x03(\x0b\x32\x16.prometheus.TimeSeriesB\x04\xc8\xde\x1f\x00\x12\x32\n\x08metadata\x18\x03 \x03(\x0b\x32\x1a.prometheus.MetricMetadataB\x04\xc8\xde\x1f\x00J\x04\x08\x02\x10\x03"\xae\x01\n\x0bReadRequest\x12"\n\x07queries\x18\x01 \x03(\x0b\x32\x11.prometheus.Query\x12\x45\n\x17\x61\x63\x63\x65pted_response_types\x18\x02 \x03(\x0e\x32$.prometheus.ReadRequest.ResponseType"4\n\x0cResponseType\x12\x0b\n\x07SAMPLES\x10\x00\x12\x17\n\x13STREAMED_XOR_CHUNKS\x10\x01"8\n\x0cReadResponse\x12(\n\x07results\x18\x01 \x03(\x0b\x32\x17.prometheus.QueryResult"\x8f\x01\n\x05Query\x12\x1a\n\x12start_timestamp_ms\x18\x01 \x01(\x03\x12\x18\n\x10\x65nd_timestamp_ms\x18\x02 \x01(\x03\x12*\n\x08matchers\x18\x03 \x03(\x0b\x32\x18.prometheus.LabelMatcher\x12$\n\x05hints\x18\x04 \x01(\x0b\x32\x15.prometheus.ReadHints"9\n\x0bQueryResult\x12*\n\ntimeseries\x18\x01 \x03(\x0b\x32\x16.prometheus.TimeSeries"]\n\x13\x43hunkedReadResponse\x12\x31\n\x0e\x63hunked_series\x18\x01 \x03(\x0b\x32\x19.prometheus.ChunkedSeries\x12\x13\n\x0bquery_index\x18\x02 \x01(\x03\x42\x08Z\x06prompbb\x06proto3'
|
||||
)
|
||||
|
||||
_builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, globals())
|
||||
_builder.BuildTopDescriptorsAndMessages(
|
||||
DESCRIPTOR,
|
||||
"opentelemetry.exporter.prometheus_remote_write.gen.remote_pb2",
|
||||
globals(),
|
||||
)
|
||||
if _descriptor._USE_C_DESCRIPTORS == False:
|
||||
|
||||
DESCRIPTOR._options = None
|
||||
DESCRIPTOR._serialized_options = b"Z\006prompb"
|
||||
_WRITEREQUEST.fields_by_name["timeseries"]._options = None
|
||||
_WRITEREQUEST.fields_by_name[
|
||||
"timeseries"
|
||||
]._serialized_options = b"\310\336\037\000"
|
||||
_WRITEREQUEST.fields_by_name["metadata"]._options = None
|
||||
_WRITEREQUEST.fields_by_name[
|
||||
"metadata"
|
||||
]._serialized_options = b"\310\336\037\000"
|
||||
_WRITEREQUEST._serialized_start = 216
|
||||
_WRITEREQUEST._serialized_end = 338
|
||||
_READREQUEST._serialized_start = 341
|
||||
_READREQUEST._serialized_end = 515
|
||||
_READREQUEST_RESPONSETYPE._serialized_start = 463
|
||||
_READREQUEST_RESPONSETYPE._serialized_end = 515
|
||||
_READRESPONSE._serialized_start = 517
|
||||
_READRESPONSE._serialized_end = 573
|
||||
_QUERY._serialized_start = 576
|
||||
_QUERY._serialized_end = 719
|
||||
_QUERYRESULT._serialized_start = 721
|
||||
_QUERYRESULT._serialized_end = 778
|
||||
_CHUNKEDREADRESPONSE._serialized_start = 780
|
||||
_CHUNKEDREADRESPONSE._serialized_end = 873
|
||||
# @@protoc_insertion_point(module_scope)
|
||||
|
|
@ -0,0 +1,86 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
# Generated by the protocol buffer compiler. DO NOT EDIT!
|
||||
# source: opentelemetry/exporter/prometheus_remote_write/gen/types.proto
|
||||
"""Generated protocol buffer code."""
|
||||
from google.protobuf.internal import builder as _builder
|
||||
from google.protobuf import descriptor as _descriptor
|
||||
from google.protobuf import descriptor_pool as _descriptor_pool
|
||||
from google.protobuf import symbol_database as _symbol_database
|
||||
|
||||
# @@protoc_insertion_point(imports)
|
||||
|
||||
_sym_db = _symbol_database.Default()
|
||||
|
||||
|
||||
from opentelemetry.exporter.prometheus_remote_write.gen.gogoproto import (
|
||||
gogo_pb2 as opentelemetry_dot_exporter_dot_prometheus__remote__write_dot_gen_dot_gogoproto_dot_gogo__pb2,
|
||||
)
|
||||
|
||||
|
||||
DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(
|
||||
b'\n>opentelemetry/exporter/prometheus_remote_write/gen/types.proto\x12\nprometheus\x1aGopentelemetry/exporter/prometheus_remote_write/gen/gogoproto/gogo.proto"\xf8\x01\n\x0eMetricMetadata\x12\x33\n\x04type\x18\x01 \x01(\x0e\x32%.prometheus.MetricMetadata.MetricType\x12\x1a\n\x12metric_family_name\x18\x02 \x01(\t\x12\x0c\n\x04help\x18\x04 \x01(\t\x12\x0c\n\x04unit\x18\x05 \x01(\t"y\n\nMetricType\x12\x0b\n\x07UNKNOWN\x10\x00\x12\x0b\n\x07\x43OUNTER\x10\x01\x12\t\n\x05GAUGE\x10\x02\x12\r\n\tHISTOGRAM\x10\x03\x12\x12\n\x0eGAUGEHISTOGRAM\x10\x04\x12\x0b\n\x07SUMMARY\x10\x05\x12\x08\n\x04INFO\x10\x06\x12\x0c\n\x08STATESET\x10\x07"*\n\x06Sample\x12\r\n\x05value\x18\x01 \x01(\x01\x12\x11\n\ttimestamp\x18\x02 \x01(\x03"U\n\x08\x45xemplar\x12\'\n\x06labels\x18\x01 \x03(\x0b\x32\x11.prometheus.LabelB\x04\xc8\xde\x1f\x00\x12\r\n\x05value\x18\x02 \x01(\x01\x12\x11\n\ttimestamp\x18\x03 \x01(\x03"\x8f\x01\n\nTimeSeries\x12\'\n\x06labels\x18\x01 \x03(\x0b\x32\x11.prometheus.LabelB\x04\xc8\xde\x1f\x00\x12)\n\x07samples\x18\x02 \x03(\x0b\x32\x12.prometheus.SampleB\x04\xc8\xde\x1f\x00\x12-\n\texemplars\x18\x03 \x03(\x0b\x32\x14.prometheus.ExemplarB\x04\xc8\xde\x1f\x00"$\n\x05Label\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t"1\n\x06Labels\x12\'\n\x06labels\x18\x01 \x03(\x0b\x32\x11.prometheus.LabelB\x04\xc8\xde\x1f\x00"\x82\x01\n\x0cLabelMatcher\x12+\n\x04type\x18\x01 \x01(\x0e\x32\x1d.prometheus.LabelMatcher.Type\x12\x0c\n\x04name\x18\x02 \x01(\t\x12\r\n\x05value\x18\x03 \x01(\t"(\n\x04Type\x12\x06\n\x02\x45Q\x10\x00\x12\x07\n\x03NEQ\x10\x01\x12\x06\n\x02RE\x10\x02\x12\x07\n\x03NRE\x10\x03"|\n\tReadHints\x12\x0f\n\x07step_ms\x18\x01 \x01(\x03\x12\x0c\n\x04\x66unc\x18\x02 \x01(\t\x12\x10\n\x08start_ms\x18\x03 \x01(\x03\x12\x0e\n\x06\x65nd_ms\x18\x04 \x01(\x03\x12\x10\n\x08grouping\x18\x05 \x03(\t\x12\n\n\x02\x62y\x18\x06 \x01(\x08\x12\x10\n\x08range_ms\x18\x07 \x01(\x03"\x8b\x01\n\x05\x43hunk\x12\x13\n\x0bmin_time_ms\x18\x01 \x01(\x03\x12\x13\n\x0bmax_time_ms\x18\x02 \x01(\x03\x12(\n\x04type\x18\x03 \x01(\x0e\x32\x1a.prometheus.Chunk.Encoding\x12\x0c\n\x04\x64\x61ta\x18\x04 \x01(\x0c" \n\x08\x45ncoding\x12\x0b\n\x07UNKNOWN\x10\x00\x12\x07\n\x03XOR\x10\x01"a\n\rChunkedSeries\x12\'\n\x06labels\x18\x01 \x03(\x0b\x32\x11.prometheus.LabelB\x04\xc8\xde\x1f\x00\x12\'\n\x06\x63hunks\x18\x02 \x03(\x0b\x32\x11.prometheus.ChunkB\x04\xc8\xde\x1f\x00\x42\x08Z\x06prompbb\x06proto3'
|
||||
)
|
||||
|
||||
_builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, globals())
|
||||
_builder.BuildTopDescriptorsAndMessages(
|
||||
DESCRIPTOR,
|
||||
"opentelemetry.exporter.prometheus_remote_write.gen.types_pb2",
|
||||
globals(),
|
||||
)
|
||||
if _descriptor._USE_C_DESCRIPTORS == False:
|
||||
|
||||
DESCRIPTOR._options = None
|
||||
DESCRIPTOR._serialized_options = b"Z\006prompb"
|
||||
_EXEMPLAR.fields_by_name["labels"]._options = None
|
||||
_EXEMPLAR.fields_by_name[
|
||||
"labels"
|
||||
]._serialized_options = b"\310\336\037\000"
|
||||
_TIMESERIES.fields_by_name["labels"]._options = None
|
||||
_TIMESERIES.fields_by_name[
|
||||
"labels"
|
||||
]._serialized_options = b"\310\336\037\000"
|
||||
_TIMESERIES.fields_by_name["samples"]._options = None
|
||||
_TIMESERIES.fields_by_name[
|
||||
"samples"
|
||||
]._serialized_options = b"\310\336\037\000"
|
||||
_TIMESERIES.fields_by_name["exemplars"]._options = None
|
||||
_TIMESERIES.fields_by_name[
|
||||
"exemplars"
|
||||
]._serialized_options = b"\310\336\037\000"
|
||||
_LABELS.fields_by_name["labels"]._options = None
|
||||
_LABELS.fields_by_name["labels"]._serialized_options = b"\310\336\037\000"
|
||||
_CHUNKEDSERIES.fields_by_name["labels"]._options = None
|
||||
_CHUNKEDSERIES.fields_by_name[
|
||||
"labels"
|
||||
]._serialized_options = b"\310\336\037\000"
|
||||
_CHUNKEDSERIES.fields_by_name["chunks"]._options = None
|
||||
_CHUNKEDSERIES.fields_by_name[
|
||||
"chunks"
|
||||
]._serialized_options = b"\310\336\037\000"
|
||||
_METRICMETADATA._serialized_start = 152
|
||||
_METRICMETADATA._serialized_end = 400
|
||||
_METRICMETADATA_METRICTYPE._serialized_start = 279
|
||||
_METRICMETADATA_METRICTYPE._serialized_end = 400
|
||||
_SAMPLE._serialized_start = 402
|
||||
_SAMPLE._serialized_end = 444
|
||||
_EXEMPLAR._serialized_start = 446
|
||||
_EXEMPLAR._serialized_end = 531
|
||||
_TIMESERIES._serialized_start = 534
|
||||
_TIMESERIES._serialized_end = 677
|
||||
_LABEL._serialized_start = 679
|
||||
_LABEL._serialized_end = 715
|
||||
_LABELS._serialized_start = 717
|
||||
_LABELS._serialized_end = 766
|
||||
_LABELMATCHER._serialized_start = 769
|
||||
_LABELMATCHER._serialized_end = 899
|
||||
_LABELMATCHER_TYPE._serialized_start = 859
|
||||
_LABELMATCHER_TYPE._serialized_end = 899
|
||||
_READHINTS._serialized_start = 901
|
||||
_READHINTS._serialized_end = 1025
|
||||
_CHUNK._serialized_start = 1028
|
||||
_CHUNK._serialized_end = 1167
|
||||
_CHUNK_ENCODING._serialized_start = 1135
|
||||
_CHUNK_ENCODING._serialized_end = 1167
|
||||
_CHUNKEDSERIES._serialized_start = 1169
|
||||
_CHUNKEDSERIES._serialized_end = 1266
|
||||
# @@protoc_insertion_point(module_scope)
|
||||
|
|
@ -0,0 +1,15 @@
|
|||
# Copyright The OpenTelemetry Authors
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
__version__ = "0.34b0"
|
||||
|
|
@ -0,0 +1,13 @@
|
|||
# Copyright The OpenTelemetry Authors
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
|
@ -0,0 +1,66 @@
|
|||
import random
|
||||
|
||||
import pytest
|
||||
|
||||
import opentelemetry.test.metrictestutil as metric_util
|
||||
from opentelemetry.exporter.prometheus_remote_write import (
|
||||
PrometheusRemoteWriteMetricsExporter,
|
||||
)
|
||||
from opentelemetry.sdk.metrics.export import (
|
||||
AggregationTemporality,
|
||||
Histogram,
|
||||
HistogramDataPoint,
|
||||
Metric,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def prom_rw():
|
||||
return PrometheusRemoteWriteMetricsExporter(
|
||||
"http://victoria:8428/api/v1/write"
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def metric(request):
|
||||
if hasattr(request, "param"):
|
||||
type_ = request.param
|
||||
else:
|
||||
type_ = random.choice(["gauge", "sum"])
|
||||
|
||||
if type_ == "gauge":
|
||||
return metric_util._generate_gauge(
|
||||
"test.gauge", random.randint(0, 100)
|
||||
)
|
||||
if type_ == "sum":
|
||||
return metric_util._generate_sum(
|
||||
"test.sum", random.randint(0, 9_999_999_999)
|
||||
)
|
||||
if type_ == "histogram":
|
||||
return _generate_histogram("test_histogram")
|
||||
|
||||
raise ValueError(f"Unsupported metric type '{type_}'.")
|
||||
|
||||
|
||||
def _generate_histogram(name):
|
||||
dp = HistogramDataPoint(
|
||||
attributes={"foo": "bar", "baz": 42},
|
||||
start_time_unix_nano=1641946016139533244,
|
||||
time_unix_nano=1641946016139533244,
|
||||
count=5,
|
||||
sum=420,
|
||||
bucket_counts=[1, 4],
|
||||
explicit_bounds=[10.0],
|
||||
min=8,
|
||||
max=80,
|
||||
)
|
||||
data = Histogram(
|
||||
[dp],
|
||||
AggregationTemporality.CUMULATIVE,
|
||||
)
|
||||
return Metric(
|
||||
name,
|
||||
"foo",
|
||||
"tu",
|
||||
data=data,
|
||||
)
|
||||
|
|
@ -0,0 +1,309 @@
|
|||
# Copyright The OpenTelemetry Authors
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
import pytest
|
||||
|
||||
from opentelemetry.exporter.prometheus_remote_write import (
|
||||
PrometheusRemoteWriteMetricsExporter,
|
||||
)
|
||||
from opentelemetry.exporter.prometheus_remote_write.gen.types_pb2 import ( # pylint: disable=E0611
|
||||
TimeSeries,
|
||||
)
|
||||
from opentelemetry.sdk.metrics.export import (
|
||||
Histogram,
|
||||
HistogramDataPoint,
|
||||
MetricExportResult,
|
||||
MetricsData,
|
||||
NumberDataPoint,
|
||||
ResourceMetrics,
|
||||
ScopeMetrics,
|
||||
)
|
||||
from opentelemetry.sdk.resources import Resource
|
||||
from opentelemetry.sdk.util.instrumentation import InstrumentationScope
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"name,result",
|
||||
[
|
||||
("abc.124", "abc_124"),
|
||||
(":abc", ":abc"),
|
||||
("abc.name.hi", "abc_name_hi"),
|
||||
("service.name...", "service_name___"),
|
||||
("4hellowor:ld5∂©∑", "_hellowor:ld5___"),
|
||||
],
|
||||
)
|
||||
def test_regex(name, result, prom_rw):
|
||||
assert prom_rw._sanitize_string(name, "name") == result
|
||||
|
||||
|
||||
def test_regex_invalid(prom_rw):
|
||||
with pytest.raises(TypeError):
|
||||
prom_rw("foo_bar", "A random type")
|
||||
|
||||
|
||||
def test_parse_data_point(prom_rw):
|
||||
|
||||
attrs = {"Foo": "Bar", "Baz": 42}
|
||||
timestamp = 1641946016139533244
|
||||
value = 242.42
|
||||
dp = NumberDataPoint(attrs, 0, timestamp, value)
|
||||
name = "abc.123_42"
|
||||
labels, sample = prom_rw._parse_data_point(dp, name)
|
||||
|
||||
name = "abc_123_42"
|
||||
assert labels == (("Foo", "Bar"), ("Baz", 42), ("__name__", name))
|
||||
assert sample == (value, timestamp // 1_000_000)
|
||||
|
||||
|
||||
def test_parse_histogram_dp(prom_rw):
|
||||
attrs = {"foo": "bar", "baz": 42}
|
||||
timestamp = 1641946016139533244
|
||||
bounds = [10.0, 20.0]
|
||||
dp = HistogramDataPoint(
|
||||
attributes=attrs,
|
||||
start_time_unix_nano=1641946016139533244,
|
||||
time_unix_nano=timestamp,
|
||||
count=9,
|
||||
sum=180,
|
||||
bucket_counts=[1, 4, 4],
|
||||
explicit_bounds=bounds,
|
||||
min=8,
|
||||
max=80,
|
||||
)
|
||||
name = "foo_histogram"
|
||||
label_sample_pairs = prom_rw._parse_histogram_data_point(dp, name)
|
||||
timestamp = timestamp // 1_000_000
|
||||
bounds.append("+Inf")
|
||||
for pos, bound in enumerate(bounds):
|
||||
# We have to attributes, we kinda assume the bucket label is last...
|
||||
assert ("le", str(bound)) == label_sample_pairs[pos][0][-1]
|
||||
# Check and make sure we are putting the bucket counts in there
|
||||
assert (dp.bucket_counts[pos], timestamp) == label_sample_pairs[pos][1]
|
||||
|
||||
# Last two are the sum & total count
|
||||
assert ("__name__", f"{name}_sum") in label_sample_pairs[-2][0]
|
||||
assert (dp.sum, timestamp) == label_sample_pairs[-2][1]
|
||||
|
||||
assert ("__name__", f"{name}_count") in label_sample_pairs[-1][0]
|
||||
assert (dp.count, timestamp) == label_sample_pairs[-1][1]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"metric",
|
||||
[
|
||||
"gauge",
|
||||
"sum",
|
||||
"histogram",
|
||||
],
|
||||
indirect=["metric"],
|
||||
)
|
||||
def test_parse_metric(metric, prom_rw):
|
||||
"""
|
||||
Ensures output from parse_metrics are TimeSeries with expected data/size
|
||||
"""
|
||||
attributes = {
|
||||
"service_name": "foo",
|
||||
"bool_value": True,
|
||||
}
|
||||
|
||||
assert (
|
||||
len(metric.data.data_points) == 1
|
||||
), "We can only support a single datapoint in tests"
|
||||
series = prom_rw._parse_metric(metric, tuple(attributes.items()))
|
||||
timestamp = metric.data.data_points[0].time_unix_nano // 1_000_000
|
||||
for single_series in series:
|
||||
labels = str(single_series.labels)
|
||||
# Its a bit easier to validate these stringified where we dont have to
|
||||
# worry about ordering and protobuf TimeSeries object structure
|
||||
# This doesn't guarantee the labels aren't mixed up, but our other
|
||||
# test cases already do.
|
||||
assert "__name__" in labels
|
||||
assert prom_rw._sanitize_string(metric.name, "name") in labels
|
||||
combined_attrs = list(attributes.items()) + list(
|
||||
metric.data.data_points[0].attributes.items()
|
||||
)
|
||||
for name, value in combined_attrs:
|
||||
assert prom_rw._sanitize_string(name, "label") in labels
|
||||
assert str(value) in labels
|
||||
if isinstance(metric.data, Histogram):
|
||||
values = [
|
||||
metric.data.data_points[0].count,
|
||||
metric.data.data_points[0].sum,
|
||||
metric.data.data_points[0].bucket_counts[0],
|
||||
metric.data.data_points[0].bucket_counts[1],
|
||||
]
|
||||
else:
|
||||
values = [
|
||||
metric.data.data_points[0].value,
|
||||
]
|
||||
for sample in single_series.samples:
|
||||
assert sample.timestamp == timestamp
|
||||
assert sample.value in values
|
||||
|
||||
|
||||
class TestValidation(unittest.TestCase):
|
||||
# Test cases to ensure exporter parameter validation works as intended
|
||||
def test_valid_standard_param(self):
|
||||
exporter = PrometheusRemoteWriteMetricsExporter(
|
||||
endpoint="/prom/test_endpoint",
|
||||
)
|
||||
self.assertEqual(exporter.endpoint, "/prom/test_endpoint")
|
||||
|
||||
def test_valid_basic_auth_param(self):
|
||||
exporter = PrometheusRemoteWriteMetricsExporter(
|
||||
endpoint="/prom/test_endpoint",
|
||||
basic_auth={
|
||||
"username": "test_username",
|
||||
"password": "test_password",
|
||||
},
|
||||
)
|
||||
self.assertEqual(exporter.basic_auth["username"], "test_username")
|
||||
self.assertEqual(exporter.basic_auth["password"], "test_password")
|
||||
|
||||
def test_invalid_no_endpoint_param(self):
|
||||
with self.assertRaises(ValueError):
|
||||
PrometheusRemoteWriteMetricsExporter("")
|
||||
|
||||
def test_invalid_no_username_param(self):
|
||||
with self.assertRaises(ValueError):
|
||||
PrometheusRemoteWriteMetricsExporter(
|
||||
endpoint="/prom/test_endpoint",
|
||||
basic_auth={"password": "test_password"},
|
||||
)
|
||||
|
||||
def test_invalid_no_password_param(self):
|
||||
with self.assertRaises(ValueError):
|
||||
PrometheusRemoteWriteMetricsExporter(
|
||||
endpoint="/prom/test_endpoint",
|
||||
basic_auth={"username": "test_username"},
|
||||
)
|
||||
|
||||
def test_invalid_conflicting_passwords_param(self):
|
||||
with self.assertRaises(ValueError):
|
||||
PrometheusRemoteWriteMetricsExporter(
|
||||
endpoint="/prom/test_endpoint",
|
||||
basic_auth={
|
||||
"username": "test_username",
|
||||
"password": "test_password",
|
||||
"password_file": "test_file",
|
||||
},
|
||||
)
|
||||
|
||||
def test_invalid_timeout_param(self):
|
||||
with self.assertRaises(ValueError):
|
||||
PrometheusRemoteWriteMetricsExporter(
|
||||
endpoint="/prom/test_endpoint", timeout=0
|
||||
)
|
||||
|
||||
def test_valid_tls_config_param(self):
|
||||
tls_config = {
|
||||
"ca_file": "test_ca_file",
|
||||
"cert_file": "test_cert_file",
|
||||
"key_file": "test_key_file",
|
||||
"insecure_skip_verify": True,
|
||||
}
|
||||
exporter = PrometheusRemoteWriteMetricsExporter(
|
||||
endpoint="/prom/test_endpoint", tls_config=tls_config
|
||||
)
|
||||
self.assertEqual(exporter.tls_config["ca_file"], tls_config["ca_file"])
|
||||
self.assertEqual(
|
||||
exporter.tls_config["cert_file"], tls_config["cert_file"]
|
||||
)
|
||||
self.assertEqual(
|
||||
exporter.tls_config["key_file"], tls_config["key_file"]
|
||||
)
|
||||
self.assertEqual(
|
||||
exporter.tls_config["insecure_skip_verify"],
|
||||
tls_config["insecure_skip_verify"],
|
||||
)
|
||||
|
||||
# if cert_file is provided, then key_file must also be provided
|
||||
def test_invalid_tls_config_cert_only_param(self):
|
||||
tls_config = {"cert_file": "value"}
|
||||
with self.assertRaises(ValueError):
|
||||
PrometheusRemoteWriteMetricsExporter(
|
||||
endpoint="/prom/test_endpoint", tls_config=tls_config
|
||||
)
|
||||
|
||||
# if cert_file is provided, then key_file must also be provided
|
||||
def test_invalid_tls_config_key_only_param(self):
|
||||
tls_config = {"cert_file": "value"}
|
||||
with self.assertRaises(ValueError):
|
||||
PrometheusRemoteWriteMetricsExporter(
|
||||
endpoint="/prom/test_endpoint", tls_config=tls_config
|
||||
)
|
||||
|
||||
|
||||
# Ensures export is successful with valid export_records and config
|
||||
@patch("requests.post")
|
||||
def test_valid_export(mock_post, prom_rw, metric):
|
||||
mock_post.return_value.configure_mock(**{"status_code": 200})
|
||||
|
||||
# Assumed a "None" for Scope or Resource aren't valid, so build them here
|
||||
scope = ScopeMetrics(
|
||||
InstrumentationScope(name="prom-rw-test"), [metric], None
|
||||
)
|
||||
resource = ResourceMetrics(
|
||||
Resource({"service.name": "foo"}), [scope], None
|
||||
)
|
||||
record = MetricsData([resource])
|
||||
|
||||
result = prom_rw.export(record)
|
||||
assert result == MetricExportResult.SUCCESS
|
||||
assert mock_post.call_count == 1
|
||||
|
||||
result = prom_rw.export([])
|
||||
assert result == MetricExportResult.SUCCESS
|
||||
|
||||
|
||||
def test_invalid_export(prom_rw):
|
||||
record = MetricsData([])
|
||||
|
||||
result = prom_rw.export(record)
|
||||
assert result == MetricExportResult.FAILURE
|
||||
|
||||
|
||||
@patch("requests.post")
|
||||
def test_valid_send_message(mock_post, prom_rw):
|
||||
mock_post.return_value.configure_mock(**{"ok": True})
|
||||
result = prom_rw._send_message(bytes(), {})
|
||||
assert mock_post.call_count == 1
|
||||
assert result == MetricExportResult.SUCCESS
|
||||
|
||||
|
||||
def test_invalid_send_message(prom_rw):
|
||||
result = prom_rw._send_message(bytes(), {})
|
||||
assert result == MetricExportResult.FAILURE
|
||||
|
||||
|
||||
# Verifies that build_message calls snappy.compress and returns SerializedString
|
||||
@patch("snappy.compress", return_value=bytes())
|
||||
def test_build_message(mock_compress, prom_rw):
|
||||
message = prom_rw._build_message([TimeSeries()])
|
||||
assert mock_compress.call_count == 1
|
||||
assert isinstance(message, bytes)
|
||||
|
||||
|
||||
# Ensure correct headers are added when valid config is provided
|
||||
def test_build_headers(prom_rw):
|
||||
prom_rw.headers = {"Custom Header": "test_header"}
|
||||
|
||||
headers = prom_rw._build_headers()
|
||||
assert headers["Content-Encoding"] == "snappy"
|
||||
assert headers["Content-Type"] == "application/x-protobuf"
|
||||
assert headers["X-Prometheus-Remote-Write-Version"] == "0.1.0"
|
||||
assert headers["Custom Header"] == "test_header"
|
||||
7
tox.ini
7
tox.ini
|
|
@ -104,6 +104,9 @@ envlist =
|
|||
; opentelemetry-exporter-richconsole
|
||||
py3{7,8,9,10}-test-exporter-richconsole
|
||||
|
||||
; opentelemetry-exporter-prometheus-remote-write
|
||||
py3{6,7,8,9,10}-test-exporter-prometheus-remote-write
|
||||
|
||||
; opentelemetry-instrumentation-mysql
|
||||
py3{7,8,9,10}-test-instrumentation-mysql
|
||||
pypy3-test-instrumentation-mysql
|
||||
|
|
@ -300,6 +303,7 @@ changedir =
|
|||
test-propagator-aws: propagator/opentelemetry-propagator-aws-xray/tests
|
||||
test-propagator-ot-trace: propagator/opentelemetry-propagator-ot-trace/tests
|
||||
test-exporter-richconsole: exporter/opentelemetry-exporter-richconsole/tests
|
||||
test-exporter-prometheus-remote-write: exporter/opentelemetry-exporter-prometheus-remote-write/tests
|
||||
|
||||
commands_pre =
|
||||
; Install without -e to test the actual installation
|
||||
|
|
@ -387,6 +391,8 @@ commands_pre =
|
|||
|
||||
richconsole: pip install flaky {toxinidir}/exporter/opentelemetry-exporter-richconsole[test]
|
||||
|
||||
prometheus: pip install {toxinidir}/exporter/opentelemetry-exporter-prometheus-remote-write[test]
|
||||
|
||||
sklearn: pip install {toxinidir}/instrumentation/opentelemetry-instrumentation-sklearn[test]
|
||||
|
||||
sqlalchemy{11,14}: pip install {toxinidir}/instrumentation/opentelemetry-instrumentation-sqlalchemy[test]
|
||||
|
|
@ -498,6 +504,7 @@ commands_pre =
|
|||
python -m pip install -e {toxinidir}/instrumentation/opentelemetry-instrumentation-aws-lambda[test]
|
||||
python -m pip install -e {toxinidir}/instrumentation/opentelemetry-instrumentation-system-metrics[test]
|
||||
python -m pip install -e {toxinidir}/exporter/opentelemetry-exporter-richconsole[test]
|
||||
python -m pip install -e {toxinidir}/exporter/opentelemetry-exporter-prometheus-remote-write[test]
|
||||
python -m pip install -e {toxinidir}/sdk-extension/opentelemetry-sdk-extension-aws[test]
|
||||
python -m pip install -e {toxinidir}/propagator/opentelemetry-propagator-aws-xray[test]
|
||||
python -m pip install -e {toxinidir}/propagator/opentelemetry-propagator-ot-trace[test]
|
||||
|
|
|
|||
Loading…
Reference in New Issue