Compare commits

..

No commits in common. "main" and "release-1.30.0" have entirely different histories.

4363 changed files with 76487 additions and 661228 deletions

View File

@ -1,31 +0,0 @@
# For a description of configuration options, see
# https://www.streetsidesoftware.com/vscode-spell-checker/docs/configuration/
# cSpell:ignore textlint textlintrc
version: '0.2'
caseSensitive: true
ignorePaths:
- '*.svg'
- node_modules
- public
- tmp
patterns:
- name: CodeBlock
pattern: |
/
^(\s*[~`]{3,}) # code-block start
.* # all languages and options, e.g. shell {hl_lines=[12]}
[\s\S]*? # content
\1 # code-block end - cSpell:disable-next-line
/igmx
languageSettings:
- languageId: markdown
ignoreRegExpList:
- CodeBlock
dictionaryDefinitions:
- name: project-words
path: .cspell/project-words.txt
- name: project-names
path: .cspell/project-names.g.txt
dictionaries:
- project-words
- project-names

View File

@ -1,60 +0,0 @@
# Maintainers (current and former)
Yuri Shkuro yurishkuro
Pavol Loffay pavolloffay
Joe Farro
Juraci Paixão Kröhling
# Other names (sometimes from github links)
robbert
## LFX Mentorship June-Aug 2025 (Term 2)
Chahat Sagar
Vishvamsinh Vaghela
## GSOC 2025
Minh Nguyen
## LFX Mentorship March-May 2025 (Term 1)
Manik Mehta
Hariom Gupta
## LFX Mentorship Sep-Nov 2024 (Term 3)
Ankit Kurmi
Mehul Gautam
## LFX Mentorship June-August 2024 (Term 2)
Harshith Mente
Raghuram Kannan
Saransh Shankar
## LFX Mentorship March-May 2024 (Term 1)
Harshvir Potpose
James Ryans
Pushkar Mishra
## LFX Mentorship Sep-Nov 2023
Ansh Goyal
Prathamesh Mutkure
Yashwanth Reddy
## LFX Mentorship Jun-Aug 2023
Afzal Ansari
GLVS Kiriti
## Google Summer of Code 2023
Ha Anh Vu
## Outreachy Dec'20 - Mar'21
Ashmita Bohara
## Outreachy Dec'18 - Mar'19
Gaby Soria
## Outreachy May'18 - Aug'18
Prakriti Bansal
## Outreachy Dec'17 - Mar'18
Kara de la Marck
# Prometheus
Julius Volz

View File

@ -1,138 +0,0 @@
alertmanager
autoscale
autoscaler
bitnami
bootcamp
cassandra
certfile
clickhouse
cncf
configtls
containerd
cqlsh
cqlshrc
daemonset
darwin
datacenter
datasource
datastax
developercertificate
Docsy
downsampling
dropwizard
dsoie
eddsa
emsgsize
errorf
expvar
fargate
findRESubmatch
flink
fluentd
glvs
gobin
gopath
goroutines
Hashicorp
hasparent
healthcheck
healthcheckv2
healthz
hostnames
hostport
htmltest
htpasswd
hugo
ingester
ingester's
ingesters
istio
istio's
jaegertracing
jaxrs
jdoe
jemalloc
keycloak
keyn
keyregistry
keyserver
keyspace
kube
kubernetes
logtostderr
logz
lookback
mailgroup
mentee
mentees
mentorships
metricsquery
metricsstore
metricstore
microsim
myapp
myappnamespace
mynamespace
myproject
myservice
myversion
navtitle
nginx
nssm
nvmrc
olap
openpgp
opensearch
openshift
opentelemetry
opentracing
openzipkin
operatorhub
otel
otelcol
otlp
outreachy
parentbased
podman
pprof
procs
prometheus
prometheusexporter
promscale
proto
protobuf
quickstart
ratelimited
ratelimiting
rawhtml
readyz
rebalance
reindex
reindexing
rolebinding
shortcode
spanmetrics
stackoverflow
statefulset
strimzi
strimzi's
struct
subpages
tchannel
thriftrw
tolerations
tracegen
uber
uberctx
unmarshal
unmarshaling
upsample
usercert
userkey
valuen
vlog
warnf
weaveworks
workqueue
xvzf
zipkin

View File

@ -1,11 +0,0 @@
# To get started with Dependabot version updates, you'll need to specify which
# package ecosystems to update and where the package manifests are located.
# Please see the documentation for all configuration options:
# https://docs.github.com/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file
version: 2
updates:
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "daily"

View File

@ -1,33 +1,19 @@
name: Release
on:
workflow_dispatch:
inputs:
version_v1:
required: true
type: string
description: "Version number for v1 docs, in #.#.# format"
version_v2:
required: true
type: string
description: "Version number for v2 docs, in #.#.# format"
dry_run:
required: true
type: boolean
description: Do a dry run.
create:
branches: [ main ]
tags: ["release-*.*.*"]
jobs:
prepare-release:
if: ${{ github.repository == 'jaegertracing/documentation' }}
release:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v2
with:
submodules: true
- uses: actions/setup-python@v5
- uses: actions/setup-python@v2
with:
python-version: 3.8
@ -38,12 +24,12 @@ jobs:
git config user.email "jaegertracingbot+jaeger-tracing@googlegroups.com"
# git config credential.helper "store --file=.git/credentials"
# echo "https://${{ secrets.JAEGERTRACINGBOT_GITHUB_TOKEN }}" > .git/credentials
export DRY_RUN=${{ inputs.dry_run }}
./scripts/release.sh ${{ inputs.version_v1 }} ${{ inputs.version_v2 }}
export RELEASE_TAG=${GITHUB_REF##*/}
./scripts/release.sh
- name: GH CLI create PR
run: |
export TAG="${{ inputs.version_v2 }}/${{ inputs.version_v1 }}"
export TAG=${GITHUB_REF##*/}
git checkout -b gen-release-${TAG} # branch is need for GH CLI
git push origin gen-release-${TAG} # branch has to be on remote before PR is opened
gh pr create --base main --title "Release ${TAG}" --body "Release ${TAG}. This PR is created from CI and is part of the release process."

View File

@ -1,36 +0,0 @@
name: Test
# cSpell:ignore github wjdp
on:
workflow_dispatch:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
build-and-check-links:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with: { node-version-file: .nvmrc }
- run: npm install
- name: Build and check links
run: npm run check:links:all # command also builds the site
spellcheck:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with: { node-version-file: .nvmrc }
- run: npm install
- run: npm run check:spelling
block-pr-from-main-branch:
runs-on: ubuntu-latest
steps:
- name: Ensure PR is not on main branch
uses: jaegertracing/jaeger/.github/actions/block-pr-from-main-branch@main

19
.gitignore vendored
View File

@ -1,20 +1,7 @@
# Generated files
.cspell/*.g.txt
# npm assets
node_modules/
package-lock.json
# Hugo-generated assets
.hugo_build.lock
/public
/resources
/.netlify
public/
resources/
.DS_Store
.idea
bin/
tmp/
.idea
.vscode/

3
.htmltest.external.yml Normal file
View File

@ -0,0 +1,3 @@
DirectoryPath: public
IgnoreDirectoryMissingTrailingSlash: true
IgnoreAltMissing: true

View File

@ -1,9 +0,0 @@
DirectoryPath: public
CheckExternal: false
IgnoreAltMissing: true
IgnoreURLs:
- /livereload.js # cSpell:disable-line
- ^/docs/latest/ # ignore redirect of latest
# Valid URL, whether canonical (before reorg) or not (and handled via a redirect)
- ^/docs/1.[6-9]\d/
- ^\.\./\.\./sampling/

View File

@ -1,21 +1,4 @@
CacheExpires: 13300h # ~ 18 months
DirectoryPath: public
IgnoreDirectoryMissingTrailingSlash: true
CheckExternal: false
IgnoreAltMissing: true
IgnoreDirs:
- docs/1.*
IgnoreURLs:
- /livereload.js # cSpell:disable-line
- ^https://www.jaegertracing.io/js/lunr
# Valid URL, whether canonical (before reorg) or not (and handled via a redirect)
- ^/docs/1.[6-9]\d/sampling/
- ^/docs/latest/architecture/
# Valid URLs, but servers yield 403 or similar errors
- ^https://(twitter|x).com/[Jj]aeger[Tt]racing
- ^https://calendar.google.com/
- ^https://eng.uber.com/(distributed-tracing/)?
- ^https://war.ukraine.ua/support-ukraine/
- ^https://www.oracle.com/cloud
- ^https://stackoverflow.com/questions/tagged/jaeger
# Temporary: invalid URLs for which replacements need to be found
# Temporary: as we work on https://github.com/jaegertracing/documentation/issues/889
- ^https://github.com/jaegertracing/jaeger/(blob|tree)/(v2|main/(internal|model|pkg))

1
.nvmrc
View File

@ -1 +0,0 @@
lts/*

View File

@ -1 +0,0 @@
**

View File

@ -118,14 +118,9 @@ then you just add a line to every git commit message:
using your real name (sorry, no pseudonyms or anonymous contributions.)
You can add the sign off when creating the git commit via `git commit -s`. Make sure that your name and email are correctly configured in your local git and match those on your GitHub account:
You can add the sign off when creating the git commit via `git commit -s`.
```shell
git config --global user.name "FIRST_NAME LAST_NAME"
git config --global user.email "MY_NAME@example.com"
```
If you want signing to be automatic you can set up some aliases:
If you want this to be automatic you can set up some aliases:
```
git config --add alias.amend "commit -s --amend"

114
Makefile
View File

@ -1,84 +1,56 @@
# Copyright (c) The Jaeger Authors.
# SPDX-License-Identifier: Apache-2.0
#
# cSpell:ignore refcache
HTMLPROOFER = bundle exec htmlproofer
HUGO_THEME = jaeger-docs
THEME_DIR := themes/$(HUGO_THEME)
HTMLTEST ?= htmltest
HTMLTEST_DIR ?= tmp/.htmltest
client-libs-docs:
# invoke one group of commands in bash to allow pushd/popd
@for d in $(shell ls -d content/docs/*); do \
/bin/bash -c "pushd $$d/; \
rm -f client-libraries.md client-features.md; \
ln -s ../../_client_libs/client-libraries.md; \
ln -s ../../_client_libs/client-features.md; \
popd"; \
echo "sym-linked content/_client_libs/*.md -> $$d/"; \
done
# Use $(HTMLTEST) in PATH, if available; otherwise, we'll get a copy
ifeq (, $(shell which $(HTMLTEST)))
override HTMLTEST=$(HTMLTEST_DIR)/bin/htmltest
ifeq (, $(shell which $(HTMLTEST)))
GET_LINK_CHECKER_IF_NEEDED=get-link-checker
endif
endif
generate: client-libs-docs
# generate currently doesn't do anything, but can be useful in the future.
generate:
develop: generate
npm run serve
develop: generate
HUGO_PREVIEW=true hugo server \
--buildDrafts \
--buildFuture \
--ignoreCache
clean:
rm -rf $(HTMLTEST_DIR) public/* resources
rm -rf public
get-link-checker:
rm -Rf $(HTMLTEST_DIR)/bin
curl https://raw.githubusercontent.com/wjdp/htmltest/master/godownloader.sh \
| bash -s -- -b $(HTMLTEST_DIR)/bin
netlify-production-build: generate
hugo --minify
# Use --keep-going to ensure that the refcache gets saved even if there are
# link-checking errors.
check-links: $(GET_LINK_CHECKER_IF_NEEDED)
$(MAKE) --keep-going GET_LINK_CHECKER_IF_NEEDED= \
_restore-refcache _check-links _save-refcache
netlify-deploy-preview: generate
HUGO_PREVIEW=true hugo \
--buildDrafts \
--buildFuture \
--baseURL $(DEPLOY_PRIME_URL) \
--minify
_restore-refcache:
mkdir -p $(HTMLTEST_DIR)
cp data/refcache.json $(HTMLTEST_DIR)/refcache.json
netlify-branch-deploy: generate
hugo \
--buildDrafts \
--buildFuture \
--baseURL $(DEPLOY_PRIME_URL) \
--minify
_check-links:
$(HTMLTEST) --log-level 1 --conf .htmltest.yml
build: clean generate
hugo -v
_save-refcache:
@echo "Saving refcache.json to data/refcache.json"
jq . $(HTMLTEST_DIR)/refcache.json > data/refcache.json
link-checker-setup:
curl https://raw.githubusercontent.com/wjdp/htmltest/master/godownloader.sh | bash
check-links-all: check-links check-links-older
run-link-checker:
bin/htmltest
check-links-older: $(GET_LINK_CHECKER_IF_NEEDED)
$(HTMLTEST) --log-level 1 --conf .htmltest.old-versions.yml
check-internal-links: clean build link-checker-setup run-link-checker
check-links-internal:
$(HTMLTEST) --log-level 1 --skip-external --conf .htmltest.yml
.cspell/project-names.g.txt: .cspell/project-names-src.txt
cat .cspell/project-names-src.txt | grep -v '^#' | grep -v '^\s*$$' | tr ' ' '\n' > .cspell/project-names.g.txt
_spellcheck: .cspell/project-names.g.txt
./scripts/spellcheck.sh
fetch-blog-feed:
curl -s -o assets/data/medium.xml https://medium.com/feed/jaegertracing
fetch-roadmap:
python3 scripts/generate_roadmap.py
# only x.y.0 semver values are valid for kicking off a new release.
SEMVER_REGEX := ^([0-9]+\.){2}0$$
VALID_VERSION := $(shell echo "$(VERSION)" | grep -E "$(SEMVER_REGEX)")
ifneq "$(VALID_VERSION)" "$(VERSION)"
validate-version:
@echo "ERROR: Invalid VERSION=$(VERSION). Must be in format x.y.0."
@false # Exit with an error code
else
validate-version:
@echo "VERSION=$(VERSION) is valid."
endif
# `make start-release VERSION=x.y.0
start-release: validate-version
git tag release-$(VERSION)
git push upstream release-$(VERSION)
check-all-links: clean build link-checker-setup
bin/htmltest --conf .htmltest.external.yml

View File

@ -2,32 +2,28 @@
# Jaeger website
This repo houses all the assets used to build the Jaeger website, available at <https://jaegertracing.io>.
This repo houses all the assets used to build the Jaeger website, available at https://jaegertracing.io.
The site is built with [Hugo](https://gohugo.io/) and hosted by [Netlify](https://www.netlify.com/).
## Setup
Install the active LTS version of Node.js, then run the following command from
the directory of this repo's clone:
Install the "extended" Hugo binary from [hugo/releases](https://github.com/gohugoio/hugo/releases) (use one of the `hugo_extended_*` binaries) or
use a package manager if it is available for your operating system.
```bash
npm install
```
> The "extended" version of Hugo supports [Sass](https://sass-lang.org), which is necessary to build the site locally.
This will also install the required version of Hugo.
The currently used version of Hugo is defined in the [`netlify.toml`](./netlify.toml) configuration file.
## Running the site locally
If you want to develop the site locally, you can run a single command (assuming that you've run the [setup](#setup)):
```bash
make develop
$ make develop
```
Alternatively, you can run `npm run serve`.
These commands will start up a local server on [localhost:1313](http://localhost:1313). When you make changes to either the content of the website (in [`content`](content)) *or* to the Sass and JavaScript assets of the page (in [`themes/jaeger-docs/assets`](themes/jaeger-docs/assets)), the browser will automatically update to reflect those changes (usually in under one second).
This will start up a local server on localhost port 1313. When you make changes to either the content of the website (in [`content`](content)) *or* to the Sass and JavaScript assets of the page (in [`themes/jaeger-docs/assets`](themes/jaeger-docs/assets)), the browser will automatically update to reflect those changes (usually in under one second).
## Publishing the site
@ -81,50 +77,6 @@ You can check internal links by running `make check-internal-links` and all link
When new pages are added to the documentation, please add a corresponding entry to [themes/jaeger-docs/layouts/index.redirects](./themes/jaeger-docs/layouts/index.redirects).
## Generating Roadmap page
To generate the `content/roadmap.md` document, run `make fetch-roadmap`.
This script fetches issues from the [GitHub project board](https://github.com/orgs/jaegertracing/projects/4/views/1?layout=table), extracts the required information, and generates the roadmap document. Make sure to set the `GITHUB_TOKEN` environment variable with your GitHub API token before running the script, or save the token in `~/.github_token` file (protect the file so only you can read it: `chmod 0600 <file>`). Personal tokens can be created at https://github.com/settings/tokens/.
## Updating Medium Blog Feed
The homepage displays the latest blog posts from the [Jaeger Medium blog](https://medium.com/jaegertracing).
To avoid network calls during builds and to ensure fast, reliable local development, the Medium RSS feed is downloaded and stored as a static XML file.
### Prerequisites
Ensure you have [`curl`](https://curl.se/) installed on your system to download the RSS feed.
Most Linux and macOS systems already have `curl` pre-installed.
You can verify installation by running:
```bash
curl --version
```
### To update the Medium blog feed
Run the following command to fetch and save the feed locally as XML:
```bash
make fetch-blog-feed
```
This will download the RSS feed from Medium and save it to:
```bash
assets/data/medium.xml
```
After updating, commit the changes:
```bash
git add assets/data/medium.xml
git commit -m "chore: update Medium blog feed"
```
### How it's used
The Hugo site reads and parses data/medium.xml using resources.Get and transform.Unmarshal.
This converts the XML into structured data at build time, allowing full control over the content without relying on live network requests.
## License
[Apache 2.0 License](./LICENSE).

View File

@ -1,23 +1,32 @@
# Release instructions
Each Jaeger version is documented in a separate directory e.g. [content/docs/1.8/](./content/docs/1.8/). A special directory [content/docs/next-release/](./content/docs/next-release/) is reserved for the changes to be published as the next version. If you are adding documentation for features that are not yet released in the main Jaeger repository, add your changes to the `next-release` directory. If you're adding documentation for already released features, you may need to make the same change twice, i.e. in the most recent release (e.g. `1.8`) and in the `next-release` directories.
Jaeger v2 next-release documentation is in the `next-release-v2` directory.
<!-- BEGIN_CHECKLIST -->
Before creating a new release:
- Make sure all outstanding PRs for that version are merged to `next-release` directory.
- Make sure the actual Jaeger release is done and Docker images for the new version are published.
- If there are new Jaeger binaries or new storage options added to the release, make sure the CLI docs config file `data/cli/next-release/config.json` is updated accordingly (see below).
- Make sure you have git remote `upstream` pointing to the official repository, e.g.
`git remote add upstream git@github.com:jaegertracing/documentation.git`
- Make sure you are on your `main` branch.
To create a new release:
- Manually trigger the [Release](https://github.com/jaegertracing/documentation/actions/workflows/ci-release.yml) workflow on GitHub. It will ask for v1 and v2 version numbers (same versions as in the main Jaeger repo), and create a [pull request](https://github.com/jaegertracing/documentation/pulls) with the documentation changes.
Then create a release by pushing a tag corresponding to the jaegertracing/jaeger version `release-X.Y.Z`, e.g.
```shell
git tag release-1.12.0
git push upstream release-1.12.0
```
- Wait for the [CI job](https://github.com/jaegertracing/documentation/actions) to create a
[pull request](https://github.com/jaegertracing/documentation/pulls) with the documentation
changes for the new version.
- Approve and merge that pull request.
- Because the site is statically generated, the release is completed after the merge.
<!-- END_CHECKLIST -->
TODO: shouldn't the tag only specify major/minor, not patch? I don't think the process will work twice for the same major.minor
### Auto-generated documentation for CLI flags
The docs for the Jaeger CLI tools are generated by the automated release process described above using the Python script [`scripts/gen-cli-data.py`](./scripts/gen-cli-data.py). It uses the configuration file `data/cli/next-release/config.json` (automatically copied to `data/cli/${NEXT_VERSION}/config.json`) that describes which Jaeger binaries to include in the documentation, and which storage options each binary supports. The script invokes the `docs` command on the respective Docker images for each binary and creates a set of YAML files under `data/cli/${NEXT_VERSION}/`, which are then used by the template engine to render the CLI docs.

File diff suppressed because one or more lines are too long

36
config.toml Normal file
View File

@ -0,0 +1,36 @@
baseURL = "https://www.jaegertracing.io"
title = "Jaeger: open source, end-to-end distributed tracing"
languageCode = "en"
pygmentsCodeFences = true
pygmentsUseClasses = true
disableKinds = ["taxonomy", "taxonomyTerm"]
canonifyURLs = true
theme = "jaeger-docs"
googleAnalytics = "UA-148208642-1"
[mediaTypes."text/netlify"]
delimiter = ""
[outputFormats.REDIRECTS]
mediaType = "text/netlify"
baseName = "_redirects"
[outputs]
home = [ "HTML", "JSON", "REDIRECTS" ]
[params]
tagline = "Monitor and troubleshoot transactions in complex distributed systems"
githubrepo = "jaegertracing/jaeger"
twitterhandle = "JaegerTracing"
mediumhandle = "jaegertracing"
opengraphimage = "img/jaeger-icon-color.png"
latest = "1.29"
binariesLatest = "1.29.0"
versions = ["1.29","1.28","1.27","1.26","1.25","1.24","1.23","1.22","1.21","1.20","1.19","1.18","1.17","1.16","1.15","1.14","1.13","1.12","1.11", "1.10", "1.9", "1.8", "1.7", "1.6"]
[navbar]
[[links]]
title = "Docs"
url = "/docs"

View File

@ -0,0 +1,18 @@
---
title: Client Library Features
widescreen: true
hasparent: true
weight: 5
---
{{< warning >}}
Jaeger clients are being retired. Please refer to this [notice](../client-libraries/#deprecating-jaeger-clients).
{{< /warning >}}
The table below provides a feature matrix for the existing client libraries. Cells marked with `?` indicate that it's not known if the given client supports the given feature and additional research & documentation update is required.
{{< info >}}
The table is auto-generated from [data/clients.yaml](https://github.com/jaegertracing/documentation/blob/main/data/clients.yaml)
{{< /info >}}
{{< featuresTable >}}

View File

@ -0,0 +1,238 @@
---
title: Client Libraries
weight: 5
children:
- title: Client Features
url: client-features
---
{{< warning >}}
Jaeger clients are being retired.
{{< /warning >}}
## Deprecating Jaeger clients
The Jaeger clients have faithfully served our community for several years. We pioneered many new features, such as remotely controlled samplers and per-operation / adaptive sampling, which were critical to the success of distributed tracing deployments at large organizations. However, now that the larger community in OpenTelemetry has caught up with the Jaeger clients in terms of feature parity and there is full support for exporting data to Jaeger, we believe it is time to **decommission Jaeger's native clients and refocus the efforts on the OpenTelemetry SDKs**.
For new applications, we recommend using the [OpenTelemetry](https://opentelemetry.io/) APIs and SDKs. For existing applications that are already instrumented with the OpenTracing API, we recommend replacing the Jaeger clients with the corresponding OpenTelemetry SDKs and the OpenTracing shim/bridge available in most languages supported by Jaeger.
### Timeline
We plan to continue accepting pull requests and making new releases of Jaeger clients **through the end of 2021**. In January 2022 we will enter a code freeze period **for 6 months**, during which time we will no longer accept pull requests with new features, with the exception of security-related fixes. After that we will archive the client library repositories and will no longer accept new changes.
### Migration to OpenTelemetry
The OpenTelemetry project is working on publishing the migration guides from OpenTracing API to OpenTelemetry SDKs via OpenTracing bridges/shims. There may be different levels of maturity and features in the SDKs. We will keep updating the information below as more of it becomes available.
#### Baggage support
OpenTelemetry implements baggage propagation differently from OpenTracing and they are not completely equivalent. In OpenTelemetry the `context` layer sits below the tracing API and relies on immutable context objects, whereas baggage in OpenTracing is stored in a `span` which is mutable (and may occasionally lead to tricky race conditions when starting children spans).
#### We need your help!
If you find inaccuracies or have information that can be added, please open an issue or a PR to the [documentation repo](https://github.com/jaegertracing/documentation). If some features are missing and you need them, please open tickets in the respective OpenTelemetry repos or contibute. For example, Jaeger's remote samplers are not yet implemented in every OpenTelemetry SDK, but porting them from the Jaeger codebase is a fairly straightforward task.
#### Copying Jaeger code
We encourage OpenTelemetry SDK authors to copy relevant pieces of the Jaeger clients instead of depending on Jaeger modules directly. This is why we use a liberal APL2 license. When copying code, the correct way to respect the license requirements is to keep the copyright notices. For example, Jaeger authors did the same with the code originally written at Uber:
```
// Copyright (c) 2019 The Jaeger Authors.
// Copyright (c) 2017 Uber Technologies, Inc.
// ... <rest of Apache notice> ...
```
#### Java
* OpenTelemetry SDK: https://github.com/open-telemetry/opentelemetry-java
* Remote sampling: [sdk-extensions/jaeger-remote-sampler](https://github.com/open-telemetry/opentelemetry-java/tree/main/sdk-extensions/jaeger-remote-sampler)
* Internal SDK metrics: n/a
* OpenTracing Bridge: [opentracing-shim](https://github.com/open-telemetry/opentelemetry-java/tree/main/opentracing-shim)
* Migration Guide: ["Migrating from Jaeger client to OpenTelemetry SDK"][blog-otel-java]
#### Python
* OpenTelemetry SDK: https://github.com/open-telemetry/opentelemetry-python
* Remote sampling: ?
* Internal SDK metrics: n/a
* OpenTracing Bridge: [opentelemetry-opentracing-shim](https://github.com/open-telemetry/opentelemetry-python/tree/main/shim/opentelemetry-opentracing-shim)
* Migration Guide: available in the shim's README
#### Node.js
* OpenTelemetry SDK: https://github.com/open-telemetry/opentelemetry-js
* Remote sampling: ?
* Internal SDK metrics: n/a
* OpenTracing Bridge: [opentelemetry-shim-opentracing](https://github.com/open-telemetry/opentelemetry-js/tree/main/packages/opentelemetry-shim-opentracing)
* Migration Guide: available in the shim's README and the corresponding readthedocs
#### Go
* OpenTelemetry SDK: https://github.com/open-telemetry/opentelemetry-go
* Remote sampling: https://github.com/open-telemetry/opentelemetry-go-contrib/pull/936
* Internal SDK metrics: n/a
* OpenTracing Bridge: [bridge/opentracing](https://github.com/open-telemetry/opentelemetry-go/tree/main/bridge/opentracing)
* Migration Guide: ?
#### C# / .NET
* OpenTelemetry SDK: https://github.com/open-telemetry/opentelemetry-dotnet
* Remote sampling: ?
* Internal SDK metrics: n/a
* OpenTracing Bridge: [OpenTelemetry.Shims.OpenTracing](https://github.com/open-telemetry/opentelemetry-dotnet/tree/main/src/OpenTelemetry.Shims.OpenTracing)
* Migration Guide: available in the shim's README, but is very terse
#### C++
* OpenTelemetry SDK: https://github.com/open-telemetry/opentelemetry-cpp
* Remote sampling: ?
* Internal SDK metrics: n/a
* OpenTracing Bridge: n/a
* Migration Guide: n/a
## Intro
All Jaeger client libraries support the [OpenTracing APIs](http://opentracing.io). The following resources provide more information about instrumenting your application with OpenTracing:
* [OpenTracing tutorials](https://github.com/yurishkuro/opentracing-tutorial) for Java, Go, Python, Node.js and C#
* A deep dive blog post [Tracing HTTP request latency in Go][http-latency-medium]
* The official OpenTracing documentation and other materials at [opentracing.io](http://opentracing.io)
* The [`opentracing-contrib` org on GitHub](https://github.com/opentracing-contrib) contains many repositories with off-the-shelf instrumentation for many popular frameworks, including JAXRS & Dropwizard (Java), Flask & Django (Python), Go std library, etc.
The rest of this page contains information about configuring and instantiating a Jaeger tracer in an application that is already instrumented with OpenTracing API.
## Terminology
We use the terms *client library*, *instrumentation library*, and *tracer* interchangeably in this document.
## Supported Libraries
The following client libraries are officially supported:
{{< clientsTable >}}
Libraries in other languages are currently under development, please see [issue #366](https://github.com/jaegertracing/jaeger/issues/366).
## Initializing Jaeger Tracer
The initialization syntax is slightly different in each languages, please refer to the README's in the respective repositories.
The general pattern is to not create the Tracer explicitly, but use a Configuration class to do that. Configuration allows
simpler parameterization of the Tracer, such as changing the default sampler or the location of Jaeger agent.
## Tracer Internals
### Sampling
See [Architecture | Sampling](../sampling#client-sampling-configuration).
### Reporters
Jaeger tracers use **reporters** to process finished {{< tip "spans" "span" >}}. Typically Jaeger libraries ship with the following reporters:
* **NullReporter** does nothing with the span. It can be useful in unit tests.
* **LoggingReporter** simply logs the fact that a span was finished, usually by printing the trace and span ID and the operation name.
* **CompositeReporter** takes a list of other reporters and invokes them one by one.
* **RemoteReporter** (default) buffers a certain number of finished spans in memory and uses a **sender** to submit a batch of spans out of process to Jaeger backend. The sender is responsible for serializing the span to the wire format (e.g. Thrift or JSON) and communicating with the backend components (e.g. over UDP or HTTP).
#### EMSGSIZE and UDP buffer limits
By default Jaeger libraries use a UDP sender to report finished {{< tip "spans" "span" >}} to the `jaeger-agent` daemon.
The default max packet size is 65,000 bytes, which can be transmitted without segmentation when
connecting to the agent via loopback interface. However, some OSs (in particular, MacOS), limit
the max buffer size for UDP packets, as raised in [this GitHub issue](https://github.com/uber/jaeger-client-node/issues/124).
If you run into issue with `EMSGSIZE` errors, consider raising the limits in your kernel (see the issue for examples).
You can also configure the client libraries to use a smaller max packet size, but that may cause
issues if you have large spans, e.g. if you log big chunks of data. Spans that exceed max packet size
are dropped by the clients (with metrics emitted to indicate that). Another alternative is
to use non-UDP transports, such as [HttpSender in Java][HttpSender] (not currently available for all languages).
### Metrics
Jaeger tracers emit various metrics about how many spans or traces they have started and finished, how many of them were sampled or not sampled, if there were any errors in decoding trace context from inbound requests or reporting spans to the backend.
TODO standardize and describe the metric names and labels (issues [#572](https://github.com/jaegertracing/jaeger/issues/572), [#611](https://github.com/jaegertracing/jaeger/issues/611)).
## Propagation Format
When `SpanContext` is encoded on the wire as part of the request to another service, Jaeger client libraries default to the Jaeger native propagation format specified below. In addition, Jaeger clients support [Zipkin B3 format](https://github.com/openzipkin/b3-propagation) and [W3C Trace-Context](https://github.com/w3c/distributed-tracing).
### Trace/Span Identity
#### Key
`uber-trace-id`
* Case-insensitive in HTTP
* Lower-case in protocols that preserve header case
#### Value
`{trace-id}:{span-id}:{parent-span-id}:{flags}`
* `{trace-id}`
* 64-bit or 128-bit random number in base16 format
* Can be variable length, shorter values are 0-padded on the left
* Receivers MUST accept hex-strings shorter than 32 characters and 0-pad them on the left
* Senders SHOULD generate hex strings of exactly 16 or 32 characters in length
* Clients in some languages support 128-bit, migration pending
* Value of 0 is not valid
* `{span-id}`
* 64-bit random number in base16 format
* Can be variable length, shorter values are 0-padded on the left
* Receivers MUST accept hex-strings shorter than 16 characters and 0-pad them on the left
* Senders SHOULD generate hex strings of exactly 16 characters in length
* Value of 0 is not valid
* `{parent-span-id}`
* 64-bit value in base16 format representing parent span id
* Deprecated, most Jaeger clients ignore on the receiving side, but still include it on the sending side
* 0 value is valid and means “root span” (when not ignored)
* `{flags}`
* One byte bitmap, as one or two hex digits (leading zero may be omitted)
* Bit 1 (right-most, least significant, bit mask `0x01`) is "sampled" flag
* 1 means the trace is sampled and all downstream services are advised to respect that
* 0 means the trace is not sampled and all downstream services are advised to respect that
* Were considering a new feature that allows downstream services to upsample if they find their tracing level is too low
* Bit 2 (bit mask `0x02` ) is "debug" flag
* Debug flag should only be set when the sampled flag is set
* Instructs the backend to try really hard not to drop this trace
* Bit 3 (bit mask `0x04` ) is not used
* Bit 4 (bit mask `0x08` ) is "firehose" flag
* Spans tagged as "firehose" are excluded from being indexed in the storage
* The traces can only be retrieved by trace ID (usually available from other sources, like logs)
* Other bits are unused
### Baggage
* Key: `uberctx-{baggage-key}`
* Value: `{baggage-value}` as a string (see Value Encoding below)
* Limitation: since HTTP headers dont preserve the case, Jaeger recommends baggage keys to be lowercase-kebab-case,
e.g. `my-baggage-key-1`.
Example: the following code sequence:
```
span.SetBaggageItem("key1", "value1")
span.SetBaggageItem("key2", "value2")
```
will result in the following HTTP headers:
```
uberctx-key1: value1
uberctx-key2: value2
```
#### Value Encoding
OpenTracing defines two formats for plain text headers: `HTTP_HEADERS` and `TEXT_MAP`. The former was introduced to deal with restrictions imposed by the HTTP protocol on the context of the headers, whereas the latter does not impose any restrictions, e.g. it can be used with Kafka Record Headers. The main difference between these two formats in the Jaeger SDKs is that the baggage values are URL-encoded when using the `HTTP_HEADERS` propagation format.
Example: when using the `HTTP_HEADERS` propagation format, the following code sequence:
```
span.SetBaggageItem("key1", "value 1 / blah")
```
will result in the following HTTP header:
```
uberctx-key1: value%201%20%2F%20blah
```
[HttpSender]: https://github.com/jaegertracing/jaeger-client-java/blob/master/jaeger-thrift/src/main/java/io/jaegertracing/thrift/internal/senders/HttpSender.java
[http-latency-medium]: https://medium.com/@YuriShkuro/tracing-http-request-latency-in-go-with-opentracing-7cc1282a100a
[blog-otel-java]: https://medium.com/jaegertracing/migrating-from-jaeger-client-to-opentelemetry-sdk-bd337d796759

View File

@ -1,3 +0,0 @@
---
title: 'Jaeger: open source, distributed tracing platform'
---

2
content/docs/.gitignore vendored Normal file
View File

@ -0,0 +1,2 @@
*/client-libraries.md
*/client-features.md

View File

@ -0,0 +1,66 @@
---
title: Introduction
weight: 1
---
Welcome to Jaeger's documentation portal! Below, you'll find information for beginners and experienced Jaeger users.
If you can't find what you are looking for, or have an issue not covered here, we'd love to [hear from you](/get-in-touch).
## About
Jaeger, inspired by [Dapper][dapper] and [OpenZipkin](http://zipkin.io),
is a distributed tracing system released as open source by [Uber Technologies][ubeross].
It is used for monitoring and troubleshooting microservices-based distributed systems, including:
* Distributed context propagation
* Distributed transaction monitoring
* Root cause analysis
* Service dependency analysis
* Performance / latency optimization
Uber published a blog post, [Evolving Distributed Tracing at Uber](https://eng.uber.com/distributed-tracing/), where they explain the history and reasons for the architectural choices made in Jaeger.
## Features
* [OpenTracing](http://opentracing.io/) compatible data model and instrumentation libraries
* in [Go](https://github.com/jaegertracing/jaeger-client-go), [Java](https://github.com/jaegertracing/jaeger-client-java), [Node](https://github.com/jaegertracing/jaeger-client-node), [Python](https://github.com/jaegertracing/jaeger-client-python)
and [C++](https://github.com/jaegertracing/cpp-client)
* Uses consistent upfront sampling with individual per service/endpoint probabilities
* Multiple storage backends: Cassandra, Elasticsearch, memory.
* Adaptive sampling (coming soon)
* Post-collection data processing pipeline (coming soon)
See [Features](./features/) page for more details.
## Technical Specs
* Backend components implemented in Go
* React/Javascript UI
* Supported storage backends:
* [Cassandra 3.4+](./deployment/#cassandra)
* [Elasticsearch 5.x, 6.x](./deployment/#elasticsearch)
* [Kafka](./deployment/#kafka)
* memory storage
## Quick Start
See [running a docker all in one image](getting-started#all-in-one).
## Screenshots
### Traces View
[![Traces View](/img/traces-ss.png)](/img/traces-ss.png)
### Trace Detail View
[![Detail View](/img/trace-detail-ss.png)](/img/trace-detail-ss.png)
## Related links
- [Evolving Distributed tracing At Uber Engineering](https://eng.uber.com/distributed-tracing/)
- [Tracing HTTP request latency in Go with OpenTracing](https://medium.com/opentracing/tracing-http-request-latency-in-go-with-opentracing-7cc1282a100a)
- [Distributed Tracing with Jaeger & Prometheus on Kubernetes](https://blog.openshift.com/openshift-commons-briefing-82-distributed-tracing-with-jaeger-prometheus-on-kubernetes/)
- [Using Jaeger with Istio](https://istio.io/docs/tasks/telemetry/distributed-tracing.html)
- [Using Jaeger with Envoy](https://www.envoyproxy.io/docs/envoy/latest/start/sandboxes/jaeger_tracing.html)
[dapper]: https://research.google.com/pubs/pub36356.html
[ubeross]: http://uber.github.io

View File

@ -0,0 +1,355 @@
---
title: Deployment
weight: 7
children:
- title: Frontend/UI
url: frontend-ui
---
The main Jaeger backend components are released as Docker images on Docker Hub:
Component | Repository
--------------------- | ---
**jaeger-agent** | [hub.docker.com/r/jaegertracing/jaeger-agent/](https://hub.docker.com/r/jaegertracing/jaeger-agent/)
**jaeger-collector** | [hub.docker.com/r/jaegertracing/jaeger-collector/](https://hub.docker.com/r/jaegertracing/jaeger-collector/)
**jaeger-query** | [hub.docker.com/r/jaegertracing/jaeger-query/](https://hub.docker.com/r/jaegertracing/jaeger-query/)
**jaeger-ingester** | [hub.docker.com/r/jaegertracing/jaeger-ingester/](https://hub.docker.com/r/jaegertracing/jaeger-ingester/)
There are orchestration templates for running Jaeger with:
* Kubernetes: [github.com/jaegertracing/jaeger-kubernetes](https://github.com/jaegertracing/jaeger-kubernetes),
* OpenShift: [github.com/jaegertracing/jaeger-openshift](https://github.com/jaegertracing/jaeger-openshift).
## Configuration Options
Jaeger binaries can be configured in a number of ways (in the order of decreasing priority):
* command line arguments,
* environment variables,
* configuration files in JSON, TOML, YAML, HCL, or Java properties formats.
To see the complete list of options, run the binary with `help` command. Options that are specific to a certain storage backend are only listed if the storage type is selected. For example, to see all available options in the Collector with Cassandra storage:
```sh
$ docker run --rm \
-e SPAN_STORAGE_TYPE=cassandra \
jaegertracing/jaeger-collector:{{< currentVersion >}} \
help
```
In order to provide configuration parameters via environment variables, find the respective command line option and convert its name to UPPER_SNAKE_CASE, for example:
Command line option | Environment variable
-----------------------------------|-------------------------------
`--cassandra.connections-per-host` | `CASSANDRA_CONNECTIONS_PER_HOST`
`--metrics-backend` | `METRICS_BACKEND`
## Agent
Jaeger client libraries expect **jaeger-agent** process to run locally on each host.
The agent exposes the following ports:
Port | Protocol | Function
---- | ------- | ---
5775 | UDP | accept zipkin.thrift over compact thrift protocol
6831 | UDP | accept jaeger.thrift over compact thrift protocol
6832 | UDP | accept jaeger.thrift over binary thrift protocol
5778 | HTTP | serve configs, sampling strategies
It can be executed directly on the host or via Docker, as follows:
```sh
## make sure to expose only the ports you use in your deployment scenario!
docker run \
--rm \
-p5775:5775/udp \
-p6831:6831/udp \
-p6832:6832/udp \
-p5778:5778/tcp \
jaegertracing/jaeger-agent:{{< currentVersion >}}
```
### Discovery System Integration
The agents can connect point to point to a single collector address, which could be
load balanced by another infrastructure component (e.g. DNS) across multiple collectors.
The agent can also be configured with a static list of collector addresses.
On Docker, a command like the following can be used:
```sh
docker run \
--rm \
-p5775:5775/udp \
-p6831:6831/udp \
-p6832:6832/udp \
-p5778:5778/tcp \
jaegertracing/jaeger-agent:{{< currentVersion >}} \
--reporter.tchannel.host-port=jaeger-collector.jaeger-infra.svc:14267
```
Or add `--reporter.type=grpc` and `--reporter.grpc.host-port=jaeger-collector.jaeger-infra.svc:14250` to use gRPC
communication with the collector. Then the `tchannel` option can be removed.
When using gRPC, you have several options for load balancing and name resolution:
* Single connection and no load balancing. This is the default if you specify a single `host:port`. (example: `--reporter.grpc.host-port=jaeger-collector.jaeger-infra.svc:14250`)
* Static list of hostnames and round-robin load balancing. This is what you get with a comma-separated list of addresses. (example: `reporter.grpc.host-port=jaeger-collector1:14250,jaeger-collector2:14250,jaeger-collector3:14250`)
* Dynamic DNS resolution and round-robin load balancing. To get this behaviour, prefix the address with `dns:///` and gRPC will attempt to resolve the hostname using SRV records (for [external load balancing](https://github.com/grpc/grpc/blob/master/doc/load-balancing.md)), TXT records (for [service configs](https://github.com/grpc/grpc/blob/master/doc/service_config.md)), and A records. Refer to the [gRPC Name Resolution docs](https://github.com/grpc/grpc/blob/master/doc/naming.md) and the [dns_resolver.go implementation](https://github.com/grpc/grpc-go/blob/master/resolver/dns/dns_resolver.go) for more info. (example: `--reporter.grpc.host-port=dns:///jaeger-collector.jaeger-infra.svc:14250`)
In the future we will support different service discovery systems to dynamically load balance
across several collectors ([issue 213](https://github.com/jaegertracing/jaeger/issues/213)).
## Collectors
The collectors are stateless and thus many instances of **jaeger-collector** can be run in parallel.
Collectors require almost no configuration, except for the location of Cassandra cluster,
via `--cassandra.keyspace` and `--cassandra.servers` options, or the location of Elasticsearch cluster, via
`--es.server-urls`, depending on which storage is specified. To see all command line options run
```sh
go run ./cmd/collector/main.go -h
```
or, if you don't have the source code
```sh
docker run -it --rm jaegertracing/jaeger-collector:{{< currentVersion >}} -h
```
At default settings the collector exposes the following ports:
Port | Protocol | Function
----- | ------- | ---
14267 | TChannel | used by **jaeger-agent** to send spans in jaeger.thrift format
14250 | gRPC | used by **jaeger-agent** to send spans in model.proto format
14268 | HTTP | can accept spans directly from clients in jaeger.thrift format over binary thrift protocol
9411 | HTTP | can accept Zipkin spans in JSON or Thrift (disabled by default)
14269 | HTTP | Health check at **/**
## Storage Backends
Collectors require a persistent storage backend. Cassandra and Elasticsearch are the primary supported storage backends. Additional backends are [discussed here](https://github.com/jaegertracing/jaeger/issues/638).
The storage type can be passed via `SPAN_STORAGE_TYPE` environment variable. Valid values are `cassandra`, `elasticsearch`, `kafka` (only as a buffer) and `memory` (only for all-in-one binary).
As of version 1.6.0, it's possible to use multiple storage types at the same time by providing a comma-separated list of valid types to the `SPAN_STORAGE_TYPE` environment variable.
It's important to note that all listed storage types are used for writing, but only the first type in the list will be used for reading and archiving.
### Memory
The in-memory storage is not intended for production workloads. It's intended as a simple solution to get started quickly and
data will be lost once the process is gone.
By default, there's no limit in the amount of traces stored in memory but a limit can be established by passing an
integer value via `--memory.max-traces`.
### Cassandra
Supported versions: 3.4+
Deploying Cassandra itself is out of scope for our documentation. One good
source of documentation is the [Apache Cassandra Docs](https://cassandra.apache.org/doc/latest/).
#### Configuration
##### Minimal
```sh
docker run \
-e SPAN_STORAGE_TYPE=cassandra \
-e CASSANDRA_SERVERS=<...> \
jaegertracing/jaeger-collector:{{< currentVersion >}}
```
##### All options
To view the full list of configuration options, you can run the following command:
```sh
docker run \
-e SPAN_STORAGE_TYPE=cassandra \
jaegertracing/jaeger-collector:{{< currentVersion >}} \
--help
```
#### Schema script
A script is provided to initialize Cassandra keyspace and schema
using Cassandra's interactive shell [`cqlsh`][cqlsh]:
```sh
MODE=test sh ./plugin/storage/cassandra/schema/create.sh | cqlsh
```
For production deployment, pass `MODE=prod DATACENTER={datacenter}` arguments to the script,
where `{datacenter}` is the name used in the Cassandra configuration / network topology.
The script also allows overriding TTL, keyspace name, replication factor, etc.
Run the script without arguments to see the full list of recognized parameters.
#### TLS support
Jaeger supports TLS client to node connections as long as you've configured
your Cassandra cluster correctly. After verifying with e.g. `cqlsh`, you can
configure the collector and query like so:
```sh
docker run \
-e CASSANDRA_SERVERS=<...> \
-e CASSANDRA_TLS=true \
-e CASSANDRA_TLS_SERVER_NAME="CN-in-certificate" \
-e CASSANDRA_TLS_KEY=<path to client key file> \
-e CASSANDRA_TLS_CERT=<path to client cert file> \
-e CASSANDRA_TLS_CA=<path to your CA cert file> \
jaegertracing/jaeger-collector:{{< currentVersion >}}
```
The schema tool also supports TLS. You need to make a custom cqlshrc file like
so:
```
# Creating schema in a cassandra cluster requiring client TLS certificates.
#
# Create a volume for the schema docker container containing four files:
# cqlshrc: this file
# ca-cert: the cert authority for your keys
# client-key: the keyfile for your client
# client-cert: the cert file matching client-key
#
# if there is any sort of DNS mismatch and you want to ignore server validation
# issues, then uncomment validate = false below.
#
# When running the container, map this volume to /root/.cassandra and set the
# environment variable CQLSH_SSL=--ssl
[ssl]
certfile = ~/.cassandra/ca-cert
userkey = ~/.cassandra/client-key
usercert = ~/.cassandra/client-cert
# validate = false
```
### Elasticsearch
Supported in Jaeger since 0.6.0
Supported versions: 5.x, 6.x
Elasticsearch does not require initialization other than
[installing and running Elasticsearch](https://www.elastic.co/downloads/elasticsearch).
Once it is running, pass the correct configuration values to the Jaeger collector and query service.
#### Configuration
##### Minimal
```sh
docker run \
-e SPAN_STORAGE_TYPE=elasticsearch \
-e ES_SERVER_URLS=<...> \
jaegertracing/jaeger-collector:{{< currentVersion >}}
```
##### All options
To view the full list of configuration options, you can run the following command:
```sh
docker run \
-e SPAN_STORAGE_TYPE=elasticsearch \
jaegertracing/jaeger-collector:{{< currentVersion >}} \
--help
```
See the [README](https://github.com/jaegertracing/jaeger/tree/master/plugin/storage/es/README.md) for an in-depth overview of how Jaeger uses Elasticsearch for storage.
#### Shards and Replicas for Elasticsearch indices
Shards and replicas are some configuration values to take special attention to, because this is decided upon
index creation. [This article](https://qbox.io/blog/optimizing-elasticsearch-how-many-shards-per-index) goes into
more information about choosing how many shards should be chosen for optimization.
### Kafka
Supported in Jaeger since 1.6.0
Supported Kafka versions: 0.9+
Starting from version 1.7.0, a new component [Ingester](#ingester) was added to support reading from Kafka and storing it in another storage backend (Elasticsearch or Cassandra).
Writing to Kafka is particularly useful for building post-processing data pipelines.
#### Configuration
##### Minimal
```sh
docker run \
-e SPAN_STORAGE_TYPE=kafka \
-e KAFKA_BROKERS=<...> \
-e KAFKA_TOPIC=<...> \
jaegertracing/jaeger-collector:{{< currentVersion >}}
```
##### All options
To view the full list of configuration options, you can run the following command:
```sh
docker run \
-e SPAN_STORAGE_TYPE=kafka \
jaegertracing/jaeger-collector:{{< currentVersion >}} \
--help
```
#### Topic & partitions
Unless your Kafka cluster is configured to automatically create topics, you will need to create it ahead of time. You can refer to [the Kafka quickstart documentation](https://kafka.apache.org/documentation/#quickstart_createtopic) to learn how.
You can find more information about topics and partitions in general in the [official documentation](https://kafka.apache.org/documentation/#intro_topics). [This article](https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/) provide more details about how to choose the number of partitions.
## Ingester
**jaeger-ingester** is a service which reads span data from Kafka topic and writes it to another storage backend (Elasticsearch or Cassandra).
Port | Protocol | Function
----- | ------- | ---
14270 | HTTP | Health check at **/**
14271 | HTTP | Metrics endpoint
To view all exposed configuration options run the following command:
```sh
docker run \
-e SPAN_STORAGE_TYPE=cassandra \
jaegertracing/jaeger-ingester:{{< currentVersion >}}
--help
```
## Query Service & UI
**jaeger-query** serves the API endpoints and a React/Javascript UI.
The service is stateless and is typically run behind a load balancer, such as **nginx**.
At default settings the query service exposes the following port(s):
Port | Protocol | Function
----- | ------- | ---
16686 | HTTP | **/api/*** endpoints and Jaeger UI at **/**
16687 | HTTP | Health check at **/**
### Minimal deployment example (Elasticsearch backend):
```sh
docker run -d --rm \
-p 16686:16686 \
-p 16687:16687 \
-e SPAN_STORAGE_TYPE=elasticsearch \
-e ES_SERVER_URLS=http://<ES_SERVER_IP>:<ES_SERVER_PORT> \
jaegertracing/jaeger-query:{{< currentVersion >}}
```
### UI Base Path
The base path for all **jaeger-query** HTTP routes can be set to a non-root value, e.g. `/jaeger` would cause all UI URLs to start with `/jaeger`. This can be useful when running **jaeger-query** behind a reverse proxy.
The base path can be configured via the `--query.base-path` command line parameter or the `QUERY_BASE_PATH` environment variable.
### UI Customization and Embedding
Please refer to the [dedicated Frontend/UI page](../frontend-ui/).
## Aggregation Jobs for Service Dependencies
Production deployments need an external process which aggregates data and creates dependency links between services.
Project [spark-dependencies](https://github.com/jaegertracing/spark-dependencies) is a Spark job which derives
dependency links and stores them directly to the storage.
## Configuration
All binaries accepts command line properties and environmental variables which are managed by
by [viper](https://github.com/spf13/viper) and [cobra](https://github.com/spf13/cobra).
The names of environmental properties are capital letters and characters `-` and `.` are replaced with `_`.
To list all configuration properties call `jaeger-binary -h`.
[cqlsh]: http://cassandra.apache.org/doc/latest/tools/cqlsh.html

View File

@ -0,0 +1,57 @@
---
title: Features
weight: 3
---
Jaeger is used for monitoring and troubleshooting microservices-based distributed systems, including:
* Distributed context propagation
* Distributed transaction monitoring
* Root cause analysis
* Service dependency analysis
* Performance / latency optimization
## High Scalability
Jaeger backend is designed to have no single points of failure and to scale with the business needs.
For example, any given Jaeger installation at Uber is typically processing several billion {{< tip "spans" "span" >}} per day.
## Native support for OpenTracing
Jaeger backend, Web UI, and instrumentation libraries have been designed from ground up to support the OpenTracing standard.
* Represent {{< tip "traces" "trace" >}} as {{< tip "directed acyclic graphs" "directed acyclic graph" >}} (not just trees) via [span references](https://github.com/opentracing/specification/blob/master/specification.md#references-between-spans)
* Support strongly typed span _tags_ and _structured logs_
* Support general distributed context propagation mechanism via _baggage_
## Multiple storage backends
Jaeger supports two popular open source NoSQL databases as trace storage backends: Cassandra 3.4+ and Elasticsearch 5.x/6.x.
There are ongoing community experiments using other databases, such as ScyllaDB, InfluxDB, Amazon DynamoDB. Jaeger also ships
with a simple in-memory storage for testing setups.
## Modern Web UI
Jaeger Web UI is implemented in Javascript using popular open source frameworks like React. Several performance
improvements have been released in v1.0 to allow the UI to efficiently deal with large volumes of data, and to display
{{< tip "traces" "trace" >}} with tens of thousands of {{< tip "spans" "span" >}} (e.g. we tried a trace with 80,000 spans).
## Cloud Native Deployment
Jaeger backend is distributed as a collection of Docker images. The binaries support various configuration methods,
including command line options, environment variables, and configuration files in multiple formats (yaml, toml, etc.)
Deployment to Kubernetes clusters is assisted by [Kubernetes templates](https://github.com/jaegertracing/jaeger-kubernetes)
and a [Helm chart](https://github.com/kubernetes/charts/tree/master/incubator/jaeger).
## Observability
All Jaeger backend components expose [Prometheus](https://prometheus.io/) metrics by default (other metrics backends are
also supported). Logs are written to standard out using the structured logging library [zap](https://github.com/uber-go/zap).
## Backwards compatibility with Zipkin
Although we recommend instrumenting applications with OpenTracing API and binding to Jaeger client libraries to benefit
from advanced features not available elsewhere, if your organization has already invested in the instrumentation
using Zipkin libraries, you do not have to rewrite all that code. Jaeger provides backwards compatibility with Zipkin
by accepting spans in Zipkin formats (Thrift or JSON v1/v2) over HTTP. Switching from Zipkin backend is just a matter
of routing the traffic from Zipkin libraries to the Jaeger backend.

View File

@ -0,0 +1,196 @@
---
title: Frontend/UI
hasparent: true
weight: 10
---
## Configuration
Several aspects of the UI can be configured:
* The Dependencies section can be enabled / configured
* Google Analytics tracking can be enabled / configured
* Additional menu options can be added to the global nav
These options can be configured by a JSON configuration file. The `--query.ui-config` command line parameter of the query service must then be set to the path to the JSON file when the query service is started.
An example configuration file:
```json
{
"dependencies": {
"dagMaxNumServices": 200,
"menuEnabled": true
},
"archiveEnabled": true,
"tracking": {
"gaID": "UA-000000-2",
"trackErrors": true
},
"menu": [
{
"label": "About Jaeger",
"items": [
{
"label": "GitHub",
"url": "https://github.com/jaegertracing/jaeger"
},
{
"label": "Docs",
"url": "http://jaeger.readthedocs.io/en/latest/"
}
]
}
]
}
```
`dependencies.dagMaxNumServices` defines the maximum number of services allowed before the DAG dependency view is disabled. Default: `200`.
`dependencies.menuEnabled` enables (`true`) or disables (`false`) the dependencies menu button. Default: `true`.
`archiveEnabled` enables (`true`) or disables (`false`) the archive traces button. Default: `false`. It requires a configuration of an archive storage in Query service. Archived traces are only accessible directly by ID, they are not searchable.
`tracking.gaID` defines the Google Analytics tracking ID. This is required for Google Analytics tracking, and setting it to a non-`null` value enables Google Analytics tracking. Default: `null`.
`tracking.trackErrors` enables (`true`) or disables (`false`) error tracking via Google Analytics. Errors can only be tracked if a valid Google Analytics ID is provided. For additional details on error tracking via Google Analytics see the [tracking README](https://github.com/jaegertracing/jaeger-ui/blob/c622330546afc1be59a42f874bcc1c2fadf7e69a/src/utils/tracking/README.md) in the UI repo. Default: `true`.
`menu` allows additional links to be added to the global nav. The additional links are right-aligned.
In the sample JSON config above, the configured menu will have a dropdown labeled "About Jaeger" with sub-options for "GitHub" and "Docs". The format for a link in the top right menu is as follows:
```json
{
"label": "Some text here",
"url": "https://example.com"
}
```
Links can either be members of the `menu` Array, directly, or they can be grouped into a dropdown menu option. The format for a group of links is:
```json
{
"label": "Dropdown button",
"items": [ ]
}
```
The `items` Array should contain one or more link configurations.
## Embedded Mode
Starting with version 1.9, Jaeger UI provides an "embedded" layout mode which is intended to support integrating Jaeger UI into other applications. Currently (as of `v0`), the approach taken is to remove various UI elements from the page to make the UI better suited for space-constrained layouts.
The embedded mode is induced and configured via URL query parameters.
To enter embedded mode, the `uiEmbed=v0` query parameter and value must be added to the URL. For example, the following URL will show the trace with ID `abc123` in embedded mode:
```
http://localhost:16686/trace/abc123?uiEmbed=v0
```
`uiEmbed=v0` is required.
Further, each page supported has an <img src="/img/frontend-ui/embed-open-icon.png" style="width: 20px; height:20px;display:inline;" alt="Embed open window"> button added that will open the non-embedded page in a new tab.
The following pages support embedded mode:
* Search Page
* Trace Page
### Search Page
To integrate the Search Trace Page to our application we have to indicate to the Jaeger UI that we want to use the embed mode with `uiEmbed=v0`.
For example:
```
http://localhost:16686/search?
service=my-service&
start=1543917759557000&
end=1543921359557000&
limit=20&
lookback=1h&
maxDuration&
minDuration&
uiEmbed=v0
```
![Embed Search Traces](/img/frontend-ui/embed-search-traces.png)
#### Configuration options
The following query parameter can be used to configure the layout of the search page :
* `uiSearchHideGraph=1` - disables the display of the scatter plot above the search results
```
http://localhost:16686/search?
service=my-service&
start=1543917759557000&
end=1543921359557000&
limit=20&
lookback=1h&
maxDuration&
minDuration&
uiEmbed=v0&
uiSearchHideGraph=1
```
![Embed Search Traces without Graph](/img/frontend-ui/embed-search-traces-hide-graph.png)
### Trace Page
To integrate the Trace Page to our application we have to indicate to the Jaeger UI that we want to use the embed mode with `uiEmbed=v0`.
For example:
```sh
http://localhost:16686/trace/{trace-id}?uiEmbed=v0
```
![Embed Trace view](/img/frontend-ui/embed-trace-view.png)
If we have navigated to this view from the search traces page we'll have a button to go back to the results page.
![Embed Trace view](/img/frontend-ui/embed-trace-view-with-back-button.png)
#### Configuration options
The following query parameters can be used to configure the layout of the trace page :
* `uiTimelineCollapseTitle=1` causes the trace header to start out collapsed, which hides the summary and the minimap.
```
http://localhost:16686/trace/{trace-id}?
uiEmbed=v0&
uiTimelineCollapseTitle=1
```
![Embed Trace view](/img/frontend-ui/embed-trace-view-with-collapse.png)
* `uiTimelineHideMinimap=1` removes the minimap, entirely, regardless of whether the trace header is expanded or not.
```
http://localhost:16686/trace/{trace-id}?
uiEmbed=v0&
uiTimelineHideMinimap=1
```
![Embed Trace view](/img/frontend-ui/embed-trace-view-with-hide-minimap.png)
* `uiTimelineHideSummary=1` - removes the trace summary information (number of services, etc.) entirely, regardless of whether the trace header is expanded or not.
```
http://localhost:16686/trace/{trace-id}?
uiEmbed=v0&
uiTimelineHideSummary=1
```
![Embed Trace view](/img/frontend-ui/embed-trace-view-with-hide-summary.png)
We can also combine the options:
```
http://localhost:16686/trace/{trace-id}?
uiEmbed=v0&
uiTimelineHideMinimap=1&
uiTimelineHideSummary=1
```
![Embed Trace view](/img/frontend-ui/embed-trace-view-with-hide-details-and-hide-minimap.png)

View File

@ -0,0 +1,129 @@
---
title: Getting started
description: Get up and running with Jaeger in your local environment
weight: 2
---
## Instrumentation
Your applications must be instrumented before they can send tracing data to Jaeger backend. Check the [Client Libraries](../client-libraries) section for information about how to use the OpenTracing API and how to initialize and configure Jaeger tracers.
## All in One
All-in-one is an executable designed for quick local testing, launches the Jaeger UI, collector, query, and agent, with an in memory storage component.
The simplest way to start the all-in-one is to use the pre-built image published to DockerHub (a single command line).
```bash
$ docker run -d --name jaeger \
-e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
-p 5775:5775/udp \
-p 6831:6831/udp \
-p 6832:6832/udp \
-p 5778:5778 \
-p 16686:16686 \
-p 14268:14268 \
-p 9411:9411 \
jaegertracing/all-in-one:{{< currentVersion >}}
```
Or run the `jaeger-all-in-one(.exe)` executable from the [binary distribution archives][download]:
```bash
$ jaeger-all-in-one --collector.zipkin.http-port=9411
```
You can then navigate to `http://localhost:16686` to access the Jaeger UI.
The container exposes the following ports:
Port | Protocol | Component | Function
----- | ------- | --------- | ---
5775 | UDP | agent | accept `zipkin.thrift` over compact thrift protocol (deprecated, used by legacy clients only)
6831 | UDP | agent | accept `jaeger.thrift` over compact thrift protocol
6832 | UDP | agent | accept `jaeger.thrift` over binary thrift protocol
5778 | HTTP | agent | serve configs
16686 | HTTP | query | serve frontend
14268 | HTTP | collector | accept `jaeger.thrift` directly from clients
14250 | HTTP | collector | accept `model.proto`
9411 | HTTP | collector | Zipkin compatible endpoint (optional)
## Kubernetes and OpenShift
* Kubernetes templates: https://github.com/jaegertracing/jaeger-kubernetes
* Kubernetes Operator: https://github.com/jaegertracing/jaeger-operator
* OpenShift templates: https://github.com/jaegertracing/jaeger-openshift
## Sample App: HotROD
HotROD (Rides on Demand) is a demo application that consists of several microservices and
illustrates the use of the [OpenTracing API](http://opentracing.io).
A tutorial / walkthrough is available in the blog post:
[Take OpenTracing for a HotROD ride][hotrod-tutorial].
It can be run standalone, but requires Jaeger backend to view the traces.
### Features
- Discover architecture of the whole system via data-driven dependency
diagram.
- View request timeline and errors; understand how the app works.
- Find sources of latency and lack of concurrency.
- Highly contextualized logging.
- Use baggage propagation to:
- Diagnose inter-request contention (queueing).
- Attribute time spent in a service.
- Use open source libraries with OpenTracing integration to get
vendor-neutral instrumentation for free.
### Prerequisites
- You need Go 1.11 or higher installed on your machine to run from source.
- Requires a [running Jaeger backend](#all-in-one) to view the traces.
### Running
#### From Source
```bash
mkdir -p $GOPATH/src/github.com/jaegertracing
cd $GOPATH/src/github.com/jaegertracing
git clone git@github.com:jaegertracing/jaeger.git jaeger
cd jaeger
make install
go run ./examples/hotrod/main.go all
```
#### From docker
```bash
$ docker run --rm -it \
--link jaeger \
-p8080-8083:8080-8083 \
-e JAEGER_AGENT_HOST="jaeger" \
jaegertracing/example-hotrod:{{< currentVersion >}} \
all
```
#### From binary distribution
Run `example-hotrod(.exe)` executable from the [binary distribution archives][download]:
```bash
$ example-hotrod all
```
Then navigate to `http://localhost:8080`.
## Migrating from Zipkin
Collector service exposes Zipkin compatible REST API `/api/v1/spans` which accepts both Thrift and JSON. Also there is `/api/v2/spans` for JSON only.
By default it's disabled. It can be enabled with `--collector.zipkin.http-port=9411`.
Zipkin Thrift IDL file can be found in [jaegertracing/jaeger-idl](https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/zipkincore.thrift).
It's compatible with [openzipkin/zipkin-api](https://github.com/openzipkin/zipkin-api/blob/master/thrift/zipkinCore.thrift)
[hotrod-tutorial]: https://medium.com/@YuriShkuro/take-opentracing-for-a-hotrod-ride-f6e3141f7941
[download]: ../../../download/

View File

@ -0,0 +1,66 @@
---
title: Introduction
weight: 1
---
Welcome to Jaeger's documentation portal! Below, you'll find information for beginners and experienced Jaeger users.
If you can't find what you are looking for, or have an issue not covered here, we'd love to [hear from you](/get-in-touch).
## About
Jaeger, inspired by [Dapper][dapper] and [OpenZipkin](http://zipkin.io),
is a distributed tracing system released as open source by [Uber Technologies][ubeross].
It is used for monitoring and troubleshooting microservices-based distributed systems, including:
* Distributed context propagation
* Distributed transaction monitoring
* Root cause analysis
* Service dependency analysis
* Performance / latency optimization
Uber published a blog post, [Evolving Distributed Tracing at Uber](https://eng.uber.com/distributed-tracing/), where they explain the history and reasons for the architectural choices made in Jaeger.
## Features
* [OpenTracing](http://opentracing.io/) compatible data model and instrumentation libraries
* in [Go](https://github.com/jaegertracing/jaeger-client-go), [Java](https://github.com/jaegertracing/jaeger-client-java), [Node](https://github.com/jaegertracing/jaeger-client-node), [Python](https://github.com/jaegertracing/jaeger-client-python)
and [C++](https://github.com/jaegertracing/cpp-client)
* Uses consistent upfront sampling with individual per service/endpoint probabilities
* Multiple storage backends: Cassandra, Elasticsearch, memory.
* Adaptive sampling (coming soon)
* Post-collection data processing pipeline (coming soon)
See [Features](./features/) page for more details.
## Technical Specs
* Backend components implemented in Go
* React/Javascript UI
* Supported storage backends:
* [Cassandra 3.4+](./deployment/#cassandra)
* [Elasticsearch 5.x, 6.x](./deployment/#elasticsearch)
* [Kafka](./deployment/#kafka)
* memory storage
## Quick Start
See [running a docker all in one image](getting-started#all-in-one).
## Screenshots
### Traces View
[![Traces View](/img/traces-ss.png)](/img/traces-ss.png)
### Trace Detail View
[![Detail View](/img/trace-detail-ss.png)](/img/trace-detail-ss.png)
## Related links
- [Evolving Distributed tracing At Uber Engineering](https://eng.uber.com/distributed-tracing/)
- [Tracing HTTP request latency in Go with OpenTracing](https://medium.com/opentracing/tracing-http-request-latency-in-go-with-opentracing-7cc1282a100a)
- [Distributed Tracing with Jaeger & Prometheus on Kubernetes](https://blog.openshift.com/openshift-commons-briefing-82-distributed-tracing-with-jaeger-prometheus-on-kubernetes/)
- [Using Jaeger with Istio](https://istio.io/docs/tasks/telemetry/distributed-tracing.html)
- [Using Jaeger with Envoy](https://www.envoyproxy.io/docs/envoy/latest/start/sandboxes/jaeger_tracing.html)
[dapper]: https://research.google.com/pubs/pub36356.html
[ubeross]: http://uber.github.io

View File

@ -0,0 +1,355 @@
---
title: Deployment
weight: 7
children:
- title: Frontend/UI
url: frontend-ui
---
The main Jaeger backend components are released as Docker images on Docker Hub:
Component | Repository
--------------------- | ---
**jaeger-agent** | [hub.docker.com/r/jaegertracing/jaeger-agent/](https://hub.docker.com/r/jaegertracing/jaeger-agent/)
**jaeger-collector** | [hub.docker.com/r/jaegertracing/jaeger-collector/](https://hub.docker.com/r/jaegertracing/jaeger-collector/)
**jaeger-query** | [hub.docker.com/r/jaegertracing/jaeger-query/](https://hub.docker.com/r/jaegertracing/jaeger-query/)
**jaeger-ingester** | [hub.docker.com/r/jaegertracing/jaeger-ingester/](https://hub.docker.com/r/jaegertracing/jaeger-ingester/)
There are orchestration templates for running Jaeger with:
* Kubernetes: [github.com/jaegertracing/jaeger-kubernetes](https://github.com/jaegertracing/jaeger-kubernetes),
* OpenShift: [github.com/jaegertracing/jaeger-openshift](https://github.com/jaegertracing/jaeger-openshift).
## Configuration Options
Jaeger binaries can be configured in a number of ways (in the order of decreasing priority):
* command line arguments,
* environment variables,
* configuration files in JSON, TOML, YAML, HCL, or Java properties formats.
To see the complete list of options, run the binary with `help` command. Options that are specific to a certain storage backend are only listed if the storage type is selected. For example, to see all available options in the Collector with Cassandra storage:
```sh
$ docker run --rm \
-e SPAN_STORAGE_TYPE=cassandra \
jaegertracing/jaeger-collector:{{< currentVersion >}} \
help
```
In order to provide configuration parameters via environment variables, find the respective command line option and convert its name to UPPER_SNAKE_CASE, for example:
Command line option | Environment variable
-----------------------------------|-------------------------------
`--cassandra.connections-per-host` | `CASSANDRA_CONNECTIONS_PER_HOST`
`--metrics-backend` | `METRICS_BACKEND`
## Agent
Jaeger client libraries expect **jaeger-agent** process to run locally on each host.
The agent exposes the following ports:
Port | Protocol | Function
---- | ------- | ---
5775 | UDP | accept zipkin.thrift over compact thrift protocol
6831 | UDP | accept jaeger.thrift over compact thrift protocol
6832 | UDP | accept jaeger.thrift over binary thrift protocol
5778 | HTTP | serve configs, sampling strategies
It can be executed directly on the host or via Docker, as follows:
```sh
## make sure to expose only the ports you use in your deployment scenario!
docker run \
--rm \
-p5775:5775/udp \
-p6831:6831/udp \
-p6832:6832/udp \
-p5778:5778/tcp \
jaegertracing/jaeger-agent:{{< currentVersion >}}
```
### Discovery System Integration
The agents can connect point to point to a single collector address, which could be
load balanced by another infrastructure component (e.g. DNS) across multiple collectors.
The agent can also be configured with a static list of collector addresses.
On Docker, a command like the following can be used:
```sh
docker run \
--rm \
-p5775:5775/udp \
-p6831:6831/udp \
-p6832:6832/udp \
-p5778:5778/tcp \
jaegertracing/jaeger-agent:{{< currentVersion >}} \
--reporter.grpc.host-port=jaeger-collector.jaeger-infra.svc:14250
```
Or use `--reporter.tchannel.host-port=jaeger-collector.jaeger-infra.svc:14267` to use
legacy tchannel reporter.
When using gRPC, you have several options for load balancing and name resolution:
* Single connection and no load balancing. This is the default if you specify a single `host:port`. (example: `--reporter.grpc.host-port=jaeger-collector.jaeger-infra.svc:14250`)
* Static list of hostnames and round-robin load balancing. This is what you get with a comma-separated list of addresses. (example: `reporter.grpc.host-port=jaeger-collector1:14250,jaeger-collector2:14250,jaeger-collector3:14250`)
* Dynamic DNS resolution and round-robin load balancing. To get this behaviour, prefix the address with `dns:///` and gRPC will attempt to resolve the hostname using SRV records (for [external load balancing](https://github.com/grpc/grpc/blob/master/doc/load-balancing.md)), TXT records (for [service configs](https://github.com/grpc/grpc/blob/master/doc/service_config.md)), and A records. Refer to the [gRPC Name Resolution docs](https://github.com/grpc/grpc/blob/master/doc/naming.md) and the [dns_resolver.go implementation](https://github.com/grpc/grpc-go/blob/master/resolver/dns/dns_resolver.go) for more info. (example: `--reporter.grpc.host-port=dns:///jaeger-collector.jaeger-infra.svc:14250`)
## Collectors
The collectors are stateless and thus many instances of **jaeger-collector** can be run in parallel.
Collectors require almost no configuration, except for the location of Cassandra cluster,
via `--cassandra.keyspace` and `--cassandra.servers` options, or the location of Elasticsearch cluster, via
`--es.server-urls`, depending on which storage is specified. To see all command line options run
```sh
go run ./cmd/collector/main.go -h
```
or, if you don't have the source code
```sh
docker run -it --rm jaegertracing/jaeger-collector:{{< currentVersion >}} -h
```
At default settings the collector exposes the following ports:
Port | Protocol | Function
----- | ------- | ---
14267 | TChannel | used by **jaeger-agent** to send spans in jaeger.thrift format
14250 | gRPC | used by **jaeger-agent** to send spans in model.proto format
14268 | HTTP | can accept spans directly from clients in jaeger.thrift format over binary thrift protocol
9411 | HTTP | can accept Zipkin spans in JSON or Thrift (disabled by default)
14269 | HTTP | Health check at **/**
## Storage Backends
Collectors require a persistent storage backend. Cassandra and Elasticsearch are the primary supported storage backends. Additional backends are [discussed here](https://github.com/jaegertracing/jaeger/issues/638).
The storage type can be passed via `SPAN_STORAGE_TYPE` environment variable. Valid values are `cassandra`, `elasticsearch`, `kafka` (only as a buffer) and `memory` (only for all-in-one binary).
As of version 1.6.0, it's possible to use multiple storage types at the same time by providing a comma-separated list of valid types to the `SPAN_STORAGE_TYPE` environment variable.
It's important to note that all listed storage types are used for writing, but only the first type in the list will be used for reading and archiving.
### Memory
The in-memory storage is not intended for production workloads. It's intended as a simple solution to get started quickly and
data will be lost once the process is gone.
By default, there's no limit in the amount of traces stored in memory but a limit can be established by passing an
integer value via `--memory.max-traces`.
### Cassandra
Supported versions: 3.4+
Deploying Cassandra itself is out of scope for our documentation. One good
source of documentation is the [Apache Cassandra Docs](https://cassandra.apache.org/doc/latest/).
#### Configuration
##### Minimal
```sh
docker run \
-e SPAN_STORAGE_TYPE=cassandra \
-e CASSANDRA_SERVERS=<...> \
jaegertracing/jaeger-collector:{{< currentVersion >}}
```
##### All options
To view the full list of configuration options, you can run the following command:
```sh
docker run \
-e SPAN_STORAGE_TYPE=cassandra \
jaegertracing/jaeger-collector:{{< currentVersion >}} \
--help
```
#### Schema script
A script is provided to initialize Cassandra keyspace and schema
using Cassandra's interactive shell [`cqlsh`][cqlsh]:
```sh
MODE=test sh ./plugin/storage/cassandra/schema/create.sh | cqlsh
```
For production deployment, pass `MODE=prod DATACENTER={datacenter}` arguments to the script,
where `{datacenter}` is the name used in the Cassandra configuration / network topology.
The script also allows overriding TTL, keyspace name, replication factor, etc.
Run the script without arguments to see the full list of recognized parameters.
#### TLS support
Jaeger supports TLS client to node connections as long as you've configured
your Cassandra cluster correctly. After verifying with e.g. `cqlsh`, you can
configure the collector and query like so:
```sh
docker run \
-e CASSANDRA_SERVERS=<...> \
-e CASSANDRA_TLS=true \
-e CASSANDRA_TLS_SERVER_NAME="CN-in-certificate" \
-e CASSANDRA_TLS_KEY=<path to client key file> \
-e CASSANDRA_TLS_CERT=<path to client cert file> \
-e CASSANDRA_TLS_CA=<path to your CA cert file> \
jaegertracing/jaeger-collector:{{< currentVersion >}}
```
The schema tool also supports TLS. You need to make a custom cqlshrc file like
so:
```
# Creating schema in a cassandra cluster requiring client TLS certificates.
#
# Create a volume for the schema docker container containing four files:
# cqlshrc: this file
# ca-cert: the cert authority for your keys
# client-key: the keyfile for your client
# client-cert: the cert file matching client-key
#
# if there is any sort of DNS mismatch and you want to ignore server validation
# issues, then uncomment validate = false below.
#
# When running the container, map this volume to /root/.cassandra and set the
# environment variable CQLSH_SSL=--ssl
[ssl]
certfile = ~/.cassandra/ca-cert
userkey = ~/.cassandra/client-key
usercert = ~/.cassandra/client-cert
# validate = false
```
### Elasticsearch
Supported in Jaeger since 0.6.0
Supported versions: 5.x, 6.x
Elasticsearch does not require initialization other than
[installing and running Elasticsearch](https://www.elastic.co/downloads/elasticsearch).
Once it is running, pass the correct configuration values to the Jaeger collector and query service.
#### Configuration
##### Minimal
```sh
docker run \
-e SPAN_STORAGE_TYPE=elasticsearch \
-e ES_SERVER_URLS=<...> \
jaegertracing/jaeger-collector:{{< currentVersion >}}
```
##### All options
To view the full list of configuration options, you can run the following command:
```sh
docker run \
-e SPAN_STORAGE_TYPE=elasticsearch \
jaegertracing/jaeger-collector:{{< currentVersion >}} \
--help
```
See the [README](https://github.com/jaegertracing/jaeger/tree/master/plugin/storage/es/README.md) for an in-depth overview of how Jaeger uses Elasticsearch for storage.
#### Shards and Replicas for Elasticsearch indices
Shards and replicas are some configuration values to take special attention to, because this is decided upon
index creation. [This article](https://qbox.io/blog/optimizing-elasticsearch-how-many-shards-per-index) goes into
more information about choosing how many shards should be chosen for optimization.
### Kafka
Supported in Jaeger since 1.6.0
Supported Kafka versions: 0.9+
Kafka can be used as an intermediary buffer between collector and an actual storage.
The collector is configured with `SPAN_STORAGE_TYPE=kafka` that makes it write all received spans
into a Kafka topic. A new component [Ingester](#ingester), added in version 1.7.0, is used to read from
Kafka and store spans in another storage backend (Elasticsearch or Cassandra).
Writing to Kafka is particularly useful for building post-processing data pipelines.
#### Configuration
##### Minimal
```sh
docker run \
-e SPAN_STORAGE_TYPE=kafka \
-e KAFKA_BROKERS=<...> \
-e KAFKA_TOPIC=<...> \
jaegertracing/jaeger-collector:{{< currentVersion >}}
```
##### All options
To view the full list of configuration options, you can run the following command:
```sh
docker run \
-e SPAN_STORAGE_TYPE=kafka \
jaegertracing/jaeger-collector:{{< currentVersion >}} \
--help
```
#### Topic & partitions
Unless your Kafka cluster is configured to automatically create topics, you will need to create it ahead of time. You can refer to [the Kafka quickstart documentation](https://kafka.apache.org/documentation/#quickstart_createtopic) to learn how.
You can find more information about topics and partitions in general in the [official documentation](https://kafka.apache.org/documentation/#intro_topics). [This article](https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/) provide more details about how to choose the number of partitions.
## Ingester
**jaeger-ingester** is a service which reads span data from Kafka topic and writes it to another storage backend (Elasticsearch or Cassandra).
Port | Protocol | Function
----- | ------- | ---
14270 | HTTP | Health check at **/**
14271 | HTTP | Metrics endpoint
To view all exposed configuration options run the following command:
```sh
docker run \
-e SPAN_STORAGE_TYPE=cassandra \
jaegertracing/jaeger-ingester:{{< currentVersion >}}
--help
```
## Query Service & UI
**jaeger-query** serves the API endpoints and a React/Javascript UI.
The service is stateless and is typically run behind a load balancer, such as **nginx**.
At default settings the query service exposes the following port(s):
Port | Protocol | Function
----- | ------- | ---
16686 | HTTP | **/api/*** endpoints and Jaeger UI at **/**
16687 | HTTP | Health check at **/**
### Minimal deployment example (Elasticsearch backend):
```sh
docker run -d --rm \
-p 16686:16686 \
-p 16687:16687 \
-e SPAN_STORAGE_TYPE=elasticsearch \
-e ES_SERVER_URLS=http://<ES_SERVER_IP>:<ES_SERVER_PORT> \
jaegertracing/jaeger-query:{{< currentVersion >}}
```
### UI Base Path
The base path for all **jaeger-query** HTTP routes can be set to a non-root value, e.g. `/jaeger` would cause all UI URLs to start with `/jaeger`. This can be useful when running **jaeger-query** behind a reverse proxy.
The base path can be configured via the `--query.base-path` command line parameter or the `QUERY_BASE_PATH` environment variable.
### UI Customization and Embedding
Please refer to the [dedicated Frontend/UI page](../frontend-ui/).
## Aggregation Jobs for Service Dependencies
Production deployments need an external process which aggregates data and creates dependency links between services.
Project [spark-dependencies](https://github.com/jaegertracing/spark-dependencies) is a Spark job which derives
dependency links and stores them directly to the storage.
## Configuration
All binaries accepts command line properties and environmental variables which are managed by
by [viper](https://github.com/spf13/viper) and [cobra](https://github.com/spf13/cobra).
The names of environmental properties are capital letters and characters `-` and `.` are replaced with `_`.
To list all configuration properties call `jaeger-binary -h`.
[cqlsh]: http://cassandra.apache.org/doc/latest/tools/cqlsh.html

16
content/docs/1.11/faq.md Normal file
View File

@ -0,0 +1,16 @@
---
title: Frequently Asked Questions
description: Answers to some frequently asked questions about Jaeger.
weight: 11
---
## Why is the Dependencies page empty?
The Dependencies page shows a graph of services traced by Jaeger and connections between them. When you are using `all-in-one` binary with in-memory storage, the graph is calculated on-demand from all the traces stored in memory. However, if you are using a real distributed storage like Cassandra or Elasticsearch, it is too expensive to scan all the data in the database to build the service graph. Instead, the Jaeger project provides "big data" jobs that can be used to extract the service graph data from traces:
* https://github.com/jaegertracing/spark-dependencies - the older Spark job that can be run periodically
* https://github.com/jaegertracing/jaeger-analytics - the new (experimental) streaming Flink jobs that run continuously and build the service graph in smaller time intervals
## Why do I not see any spans in Jaeger?
Please refer to the [Troubleshooting](../troubleshooting/) guide.

View File

@ -0,0 +1,57 @@
---
title: Features
weight: 3
---
Jaeger is used for monitoring and troubleshooting microservices-based distributed systems, including:
* Distributed context propagation
* Distributed transaction monitoring
* Root cause analysis
* Service dependency analysis
* Performance / latency optimization
## High Scalability
Jaeger backend is designed to have no single points of failure and to scale with the business needs.
For example, any given Jaeger installation at Uber is typically processing several billion {{< tip "spans" "span" >}} per day.
## Native support for OpenTracing
Jaeger backend, Web UI, and instrumentation libraries have been designed from ground up to support the OpenTracing standard.
* Represent {{< tip "traces" "trace" >}} as {{< tip "directed acyclic graphs" "directed acyclic graph" >}} (not just trees) via [span references](https://github.com/opentracing/specification/blob/master/specification.md#references-between-spans)
* Support strongly typed span _tags_ and _structured logs_
* Support general distributed context propagation mechanism via _baggage_
## Multiple storage backends
Jaeger supports two popular open source NoSQL databases as trace storage backends: Cassandra 3.4+ and Elasticsearch 5.x/6.x.
There are ongoing community experiments using other databases, such as ScyllaDB, InfluxDB, Amazon DynamoDB. Jaeger also ships
with a simple in-memory storage for testing setups.
## Modern Web UI
Jaeger Web UI is implemented in Javascript using popular open source frameworks like React. Several performance
improvements have been released in v1.0 to allow the UI to efficiently deal with large volumes of data, and to display
{{< tip "traces" "trace" >}} with tens of thousands of {{< tip "spans" "span" >}} (e.g. we tried a trace with 80,000 spans).
## Cloud Native Deployment
Jaeger backend is distributed as a collection of Docker images. The binaries support various configuration methods,
including command line options, environment variables, and configuration files in multiple formats (yaml, toml, etc.).
Deployment to Kubernetes clusters is assisted by a [Kubernetes operator](https://github.com/jaegertracing/jaeger-operator), [Kubernetes templates](https://github.com/jaegertracing/jaeger-kubernetes)
and a [Helm chart](https://github.com/kubernetes/charts/tree/master/incubator/jaeger).
## Observability
All Jaeger backend components expose [Prometheus](https://prometheus.io/) metrics by default (other metrics backends are
also supported). Logs are written to standard out using the structured logging library [zap](https://github.com/uber-go/zap).
## Backwards compatibility with Zipkin
Although we recommend instrumenting applications with OpenTracing API and binding to Jaeger client libraries to benefit
from advanced features not available elsewhere, if your organization has already invested in the instrumentation
using Zipkin libraries, you do not have to rewrite all that code. Jaeger provides backwards compatibility with Zipkin
by accepting spans in Zipkin formats (Thrift or JSON v1/v2) over HTTP. Switching from Zipkin backend is just a matter
of routing the traffic from Zipkin libraries to the Jaeger backend.

View File

@ -0,0 +1,196 @@
---
title: Frontend/UI
hasparent: true
weight: 10
---
## Configuration
Several aspects of the UI can be configured:
* The Dependencies section can be enabled / configured
* Google Analytics tracking can be enabled / configured
* Additional menu options can be added to the global nav
These options can be configured by a JSON configuration file. The `--query.ui-config` command line parameter of the query service must then be set to the path to the JSON file when the query service is started.
An example configuration file:
```json
{
"dependencies": {
"dagMaxNumServices": 200,
"menuEnabled": true
},
"archiveEnabled": true,
"tracking": {
"gaID": "UA-000000-2",
"trackErrors": true
},
"menu": [
{
"label": "About Jaeger",
"items": [
{
"label": "GitHub",
"url": "https://github.com/jaegertracing/jaeger"
},
{
"label": "Docs",
"url": "http://jaeger.readthedocs.io/en/latest/"
}
]
}
]
}
```
`dependencies.dagMaxNumServices` defines the maximum number of services allowed before the DAG dependency view is disabled. Default: `200`.
`dependencies.menuEnabled` enables (`true`) or disables (`false`) the dependencies menu button. Default: `true`.
`archiveEnabled` enables (`true`) or disables (`false`) the archive traces button. Default: `false`. It requires a configuration of an archive storage in Query service. Archived traces are only accessible directly by ID, they are not searchable.
`tracking.gaID` defines the Google Analytics tracking ID. This is required for Google Analytics tracking, and setting it to a non-`null` value enables Google Analytics tracking. Default: `null`.
`tracking.trackErrors` enables (`true`) or disables (`false`) error tracking via Google Analytics. Errors can only be tracked if a valid Google Analytics ID is provided. For additional details on error tracking via Google Analytics see the [tracking README](https://github.com/jaegertracing/jaeger-ui/blob/c622330546afc1be59a42f874bcc1c2fadf7e69a/src/utils/tracking/README.md) in the UI repo. Default: `true`.
`menu` allows additional links to be added to the global nav. The additional links are right-aligned.
In the sample JSON config above, the configured menu will have a dropdown labeled "About Jaeger" with sub-options for "GitHub" and "Docs". The format for a link in the top right menu is as follows:
```json
{
"label": "Some text here",
"url": "https://example.com"
}
```
Links can either be members of the `menu` Array, directly, or they can be grouped into a dropdown menu option. The format for a group of links is:
```json
{
"label": "Dropdown button",
"items": [ ]
}
```
The `items` Array should contain one or more link configurations.
## Embedded Mode
Starting with version 1.9, Jaeger UI provides an "embedded" layout mode which is intended to support integrating Jaeger UI into other applications. Currently (as of `v0`), the approach taken is to remove various UI elements from the page to make the UI better suited for space-constrained layouts.
The embedded mode is induced and configured via URL query parameters.
To enter embedded mode, the `uiEmbed=v0` query parameter and value must be added to the URL. For example, the following URL will show the trace with ID `abc123` in embedded mode:
```
http://localhost:16686/trace/abc123?uiEmbed=v0
```
`uiEmbed=v0` is required.
Further, each page supported has an <img src="/img/frontend-ui/embed-open-icon.png" style="width: 20px; height:20px;display:inline;" alt="Embed open window"> button added that will open the non-embedded page in a new tab.
The following pages support embedded mode:
* Search Page
* Trace Page
### Search Page
To integrate the Search Trace Page to our application we have to indicate to the Jaeger UI that we want to use the embed mode with `uiEmbed=v0`.
For example:
```
http://localhost:16686/search?
service=my-service&
start=1543917759557000&
end=1543921359557000&
limit=20&
lookback=1h&
maxDuration&
minDuration&
uiEmbed=v0
```
![Embed Search Traces](/img/frontend-ui/embed-search-traces.png)
#### Configuration options
The following query parameter can be used to configure the layout of the search page :
* `uiSearchHideGraph=1` - disables the display of the scatter plot above the search results
```
http://localhost:16686/search?
service=my-service&
start=1543917759557000&
end=1543921359557000&
limit=20&
lookback=1h&
maxDuration&
minDuration&
uiEmbed=v0&
uiSearchHideGraph=1
```
![Embed Search Traces without Graph](/img/frontend-ui/embed-search-traces-hide-graph.png)
### Trace Page
To integrate the Trace Page to our application we have to indicate to the Jaeger UI that we want to use the embed mode with `uiEmbed=v0`.
For example:
```sh
http://localhost:16686/trace/{trace-id}?uiEmbed=v0
```
![Embed Trace view](/img/frontend-ui/embed-trace-view.png)
If we have navigated to this view from the search traces page we'll have a button to go back to the results page.
![Embed Trace view](/img/frontend-ui/embed-trace-view-with-back-button.png)
#### Configuration options
The following query parameters can be used to configure the layout of the trace page :
* `uiTimelineCollapseTitle=1` causes the trace header to start out collapsed, which hides the summary and the minimap.
```
http://localhost:16686/trace/{trace-id}?
uiEmbed=v0&
uiTimelineCollapseTitle=1
```
![Embed Trace view](/img/frontend-ui/embed-trace-view-with-collapse.png)
* `uiTimelineHideMinimap=1` removes the minimap, entirely, regardless of whether the trace header is expanded or not.
```
http://localhost:16686/trace/{trace-id}?
uiEmbed=v0&
uiTimelineHideMinimap=1
```
![Embed Trace view](/img/frontend-ui/embed-trace-view-with-hide-minimap.png)
* `uiTimelineHideSummary=1` - removes the trace summary information (number of services, etc.) entirely, regardless of whether the trace header is expanded or not.
```
http://localhost:16686/trace/{trace-id}?
uiEmbed=v0&
uiTimelineHideSummary=1
```
![Embed Trace view](/img/frontend-ui/embed-trace-view-with-hide-summary.png)
We can also combine the options:
```
http://localhost:16686/trace/{trace-id}?
uiEmbed=v0&
uiTimelineHideMinimap=1&
uiTimelineHideSummary=1
```
![Embed Trace view](/img/frontend-ui/embed-trace-view-with-hide-details-and-hide-minimap.png)

View File

@ -0,0 +1,129 @@
---
title: Getting started
description: Get up and running with Jaeger in your local environment
weight: 2
---
## Instrumentation
Your applications must be instrumented before they can send tracing data to Jaeger backend. Check the [Client Libraries](../client-libraries) section for information about how to use the OpenTracing API and how to initialize and configure Jaeger tracers.
## All in One
All-in-one is an executable designed for quick local testing, launches the Jaeger UI, collector, query, and agent, with an in memory storage component.
The simplest way to start the all-in-one is to use the pre-built image published to DockerHub (a single command line).
```bash
$ docker run -d --name jaeger \
-e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
-p 5775:5775/udp \
-p 6831:6831/udp \
-p 6832:6832/udp \
-p 5778:5778 \
-p 16686:16686 \
-p 14268:14268 \
-p 9411:9411 \
jaegertracing/all-in-one:{{< currentVersion >}}
```
Or run the `jaeger-all-in-one(.exe)` executable from the [binary distribution archives][download]:
```bash
$ jaeger-all-in-one --collector.zipkin.http-port=9411
```
You can then navigate to `http://localhost:16686` to access the Jaeger UI.
The container exposes the following ports:
Port | Protocol | Component | Function
----- | ------- | --------- | ---
5775 | UDP | agent | accept `zipkin.thrift` over compact thrift protocol (deprecated, used by legacy clients only)
6831 | UDP | agent | accept `jaeger.thrift` over compact thrift protocol
6832 | UDP | agent | accept `jaeger.thrift` over binary thrift protocol
5778 | HTTP | agent | serve configs
16686 | HTTP | query | serve frontend
14268 | HTTP | collector | accept `jaeger.thrift` directly from clients
14250 | HTTP | collector | accept `model.proto`
9411 | HTTP | collector | Zipkin compatible endpoint (optional)
## Kubernetes and OpenShift
* Kubernetes templates: https://github.com/jaegertracing/jaeger-kubernetes
* Kubernetes Operator: https://github.com/jaegertracing/jaeger-operator
* OpenShift templates: https://github.com/jaegertracing/jaeger-openshift
## Sample App: HotROD
HotROD (Rides on Demand) is a demo application that consists of several microservices and
illustrates the use of the [OpenTracing API](http://opentracing.io).
A tutorial / walkthrough is available in the blog post:
[Take OpenTracing for a HotROD ride][hotrod-tutorial].
It can be run standalone, but requires Jaeger backend to view the traces.
### Features
- Discover architecture of the whole system via data-driven dependency
diagram.
- View request timeline and errors; understand how the app works.
- Find sources of latency and lack of concurrency.
- Highly contextualized logging.
- Use baggage propagation to:
- Diagnose inter-request contention (queueing).
- Attribute time spent in a service.
- Use open source libraries with OpenTracing integration to get
vendor-neutral instrumentation for free.
### Prerequisites
- You need Go 1.11 or higher installed on your machine to run from source.
- Requires a [running Jaeger backend](#all-in-one) to view the traces.
### Running
#### From Source
```bash
mkdir -p $GOPATH/src/github.com/jaegertracing
cd $GOPATH/src/github.com/jaegertracing
git clone git@github.com:jaegertracing/jaeger.git jaeger
cd jaeger
make install
go run ./examples/hotrod/main.go all
```
#### From docker
```bash
$ docker run --rm -it \
--link jaeger \
-p8080-8083:8080-8083 \
-e JAEGER_AGENT_HOST="jaeger" \
jaegertracing/example-hotrod:{{< currentVersion >}} \
all
```
#### From binary distribution
Run `example-hotrod(.exe)` executable from the [binary distribution archives][download]:
```bash
$ example-hotrod all
```
Then navigate to `http://localhost:8080`.
## Migrating from Zipkin
Collector service exposes Zipkin compatible REST API `/api/v1/spans` which accepts both Thrift and JSON. Also there is `/api/v2/spans` for JSON only.
By default it's disabled. It can be enabled with `--collector.zipkin.http-port=9411`.
Zipkin Thrift IDL file can be found in [jaegertracing/jaeger-idl](https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/zipkincore.thrift).
It's compatible with [openzipkin/zipkin-api](https://github.com/openzipkin/zipkin-api/blob/master/thrift/zipkinCore.thrift)
[hotrod-tutorial]: https://medium.com/@YuriShkuro/take-opentracing-for-a-hotrod-ride-f6e3141f7941
[download]: ../../../download/

View File

@ -0,0 +1,66 @@
---
title: Introduction
weight: 1
---
Welcome to Jaeger's documentation portal! Below, you'll find information for beginners and experienced Jaeger users.
If you can't find what you are looking for, or have an issue not covered here, we'd love to [hear from you](/get-in-touch).
## About
Jaeger, inspired by [Dapper][dapper] and [OpenZipkin](http://zipkin.io),
is a distributed tracing system released as open source by [Uber Technologies][ubeross].
It is used for monitoring and troubleshooting microservices-based distributed systems, including:
* Distributed context propagation
* Distributed transaction monitoring
* Root cause analysis
* Service dependency analysis
* Performance / latency optimization
Uber published a blog post, [Evolving Distributed Tracing at Uber](https://eng.uber.com/distributed-tracing/), where they explain the history and reasons for the architectural choices made in Jaeger.
## Features
* [OpenTracing](http://opentracing.io/) compatible data model and instrumentation libraries
* in [Go](https://github.com/jaegertracing/jaeger-client-go), [Java](https://github.com/jaegertracing/jaeger-client-java), [Node](https://github.com/jaegertracing/jaeger-client-node), [Python](https://github.com/jaegertracing/jaeger-client-python)
and [C++](https://github.com/jaegertracing/cpp-client)
* Uses consistent upfront sampling with individual per service/endpoint probabilities
* Multiple storage backends: Cassandra, Elasticsearch, memory.
* Adaptive sampling (coming soon)
* Post-collection data processing pipeline (coming soon)
See [Features](./features/) page for more details.
## Technical Specs
* Backend components implemented in Go
* React/Javascript UI
* Supported storage backends:
* [Cassandra 3.4+](./deployment/#cassandra)
* [Elasticsearch 5.x, 6.x](./deployment/#elasticsearch)
* [Kafka](./deployment/#kafka)
* memory storage
## Quick Start
See [running a docker all in one image](getting-started#all-in-one).
## Screenshots
### Traces View
[![Traces View](/img/traces-ss.png)](/img/traces-ss.png)
### Trace Detail View
[![Detail View](/img/trace-detail-ss.png)](/img/trace-detail-ss.png)
## Related links
- [Evolving Distributed tracing At Uber Engineering](https://eng.uber.com/distributed-tracing/)
- [Tracing HTTP request latency in Go with OpenTracing](https://medium.com/opentracing/tracing-http-request-latency-in-go-with-opentracing-7cc1282a100a)
- [Distributed Tracing with Jaeger & Prometheus on Kubernetes](https://blog.openshift.com/openshift-commons-briefing-82-distributed-tracing-with-jaeger-prometheus-on-kubernetes/)
- [Using Jaeger with Istio](https://istio.io/docs/tasks/telemetry/distributed-tracing.html)
- [Using Jaeger with Envoy](https://www.envoyproxy.io/docs/envoy/latest/start/sandboxes/jaeger_tracing.html)
[dapper]: https://research.google.com/pubs/pub36356.html
[ubeross]: http://uber.github.io

View File

@ -0,0 +1,396 @@
---
title: Deployment
weight: 7
children:
- title: Frontend/UI
url: frontend-ui
---
The main Jaeger backend components are released as Docker images on Docker Hub:
Component | Repository
--------------------- | ---
**jaeger-agent** | [hub.docker.com/r/jaegertracing/jaeger-agent/](https://hub.docker.com/r/jaegertracing/jaeger-agent/)
**jaeger-collector** | [hub.docker.com/r/jaegertracing/jaeger-collector/](https://hub.docker.com/r/jaegertracing/jaeger-collector/)
**jaeger-query** | [hub.docker.com/r/jaegertracing/jaeger-query/](https://hub.docker.com/r/jaegertracing/jaeger-query/)
**jaeger-ingester** | [hub.docker.com/r/jaegertracing/jaeger-ingester/](https://hub.docker.com/r/jaegertracing/jaeger-ingester/)
There are orchestration templates for running Jaeger with:
* Kubernetes: [github.com/jaegertracing/jaeger-kubernetes](https://github.com/jaegertracing/jaeger-kubernetes),
* OpenShift: [github.com/jaegertracing/jaeger-openshift](https://github.com/jaegertracing/jaeger-openshift).
## Configuration Options
Jaeger binaries can be configured in a number of ways (in the order of decreasing priority):
* command line arguments,
* environment variables,
* configuration files in JSON, TOML, YAML, HCL, or Java properties formats.
To see the complete list of options, run the binary with `help` command. Options that are specific to a certain storage backend are only listed if the storage type is selected. For example, to see all available options in the Collector with Cassandra storage:
```sh
$ docker run --rm \
-e SPAN_STORAGE_TYPE=cassandra \
jaegertracing/jaeger-collector:{{< currentVersion >}} \
help
```
In order to provide configuration parameters via environment variables, find the respective command line option and convert its name to UPPER_SNAKE_CASE, for example:
Command line option | Environment variable
-----------------------------------|-------------------------------
`--cassandra.connections-per-host` | `CASSANDRA_CONNECTIONS_PER_HOST`
`--metrics-backend` | `METRICS_BACKEND`
## Agent
Jaeger client libraries expect **jaeger-agent** process to run locally on each host.
The agent exposes the following ports:
Port | Protocol | Function
----- | ------- | ---
6831 | UDP | accept [jaeger.thrift][jaeger-thrift] in `compact` Thrift protocol used by most current Jaeger clients
6832 | UDP | accept [jaeger.thrift][jaeger-thrift] in `binary` Thrift protocol used by Node.js Jaeger client (because [thriftrw][thriftrw] npm package does not support `compact` protocol)
5778 | HTTP | serve configs, sampling strategies
5775 | UDP | accept [zipkin.thrift][zipkin-thrift] in `compact` Thrift protocol (deprecated; only used by very old Jaeger clients, circa 2016)
14271 | HTTP | Healthcheck at `/` and metrics at `/metrics`
It can be executed directly on the host or via Docker, as follows:
```sh
## make sure to expose only the ports you use in your deployment scenario!
docker run \
--rm \
-p6831:6831/udp \
-p6832:6832/udp \
-p5778:5778/tcp \
-p5775:5775/udp \
jaegertracing/jaeger-agent:{{< currentVersion >}}
```
### Discovery System Integration
The agents can connect point to point to a single collector address, which could be
load balanced by another infrastructure component (e.g. DNS) across multiple collectors.
The agent can also be configured with a static list of collector addresses.
On Docker, a command like the following can be used:
```sh
docker run \
--rm \
-p5775:5775/udp \
-p6831:6831/udp \
-p6832:6832/udp \
-p5778:5778/tcp \
jaegertracing/jaeger-agent:{{< currentVersion >}} \
--reporter.grpc.host-port=jaeger-collector.jaeger-infra.svc:14250
```
Or use `--reporter.tchannel.host-port=jaeger-collector.jaeger-infra.svc:14267` to use
legacy tchannel reporter.
When using gRPC, you have several options for load balancing and name resolution:
* Single connection and no load balancing. This is the default if you specify a single `host:port`. (example: `--reporter.grpc.host-port=jaeger-collector.jaeger-infra.svc:14250`)
* Static list of hostnames and round-robin load balancing. This is what you get with a comma-separated list of addresses. (example: `reporter.grpc.host-port=jaeger-collector1:14250,jaeger-collector2:14250,jaeger-collector3:14250`)
* Dynamic DNS resolution and round-robin load balancing. To get this behaviour, prefix the address with `dns:///` and gRPC will attempt to resolve the hostname using SRV records (for [external load balancing](https://github.com/grpc/grpc/blob/master/doc/load-balancing.md)), TXT records (for [service configs](https://github.com/grpc/grpc/blob/master/doc/service_config.md)), and A records. Refer to the [gRPC Name Resolution docs](https://github.com/grpc/grpc/blob/master/doc/naming.md) and the [dns_resolver.go implementation](https://github.com/grpc/grpc-go/blob/master/resolver/dns/dns_resolver.go) for more info. (example: `--reporter.grpc.host-port=dns:///jaeger-collector.jaeger-infra.svc:14250`)
### Agent level tags
Jaeger supports agent level tags, that can be added to the process tags of all spans passing through the agent. This is supported through the command line flag `--jaeger.tags=key1=value1,key2=value2,...,keyn=valuen`. Tags can also be set through an environment flag like so - `--jaeger.tags=key=${envFlag:defaultValue}` - The tag value will be set to the value of the `envFlag` environment key and `defaultValue` if not set. This feature is not supported for the tchannel reporter, enabled using the flags `--collector.host-port` or `--reporter.tchannel.host-port`.
## Collectors
The collectors are stateless and thus many instances of **jaeger-collector** can be run in parallel.
Collectors require almost no configuration, except for the location of Cassandra cluster,
via `--cassandra.keyspace` and `--cassandra.servers` options, or the location of Elasticsearch cluster, via
`--es.server-urls`, depending on which storage is specified. To see all command line options run
```sh
go run ./cmd/collector/main.go -h
```
or, if you don't have the source code
```sh
docker run -it --rm jaegertracing/jaeger-collector:{{< currentVersion >}} -h
```
At default settings the collector exposes the following ports:
Port | Protocol | Function
----- | ------- | ---
14267 | TChannel | used by **jaeger-agent** to send spans in jaeger.thrift format
14250 | gRPC | used by **jaeger-agent** to send spans in model.proto format
14268 | HTTP | can accept spans directly from clients in jaeger.thrift format over binary thrift protocol
9411 | HTTP | can accept Zipkin spans in JSON or Thrift (disabled by default)
14269 | HTTP | Healthcheck at `/` and metrics at `/metrics`
## Storage Backends
Collectors require a persistent storage backend. Cassandra and Elasticsearch are the primary supported storage backends. Additional backends are [discussed here](https://github.com/jaegertracing/jaeger/issues/638).
The storage type can be passed via `SPAN_STORAGE_TYPE` environment variable. Valid values are `cassandra`, `elasticsearch`, `kafka` (only as a buffer), `grpc-plugin`, `badger` (only with all-in-one) and `memory` (only with all-in-one).
As of version 1.6.0, it's possible to use multiple storage types at the same time by providing a comma-separated list of valid types to the `SPAN_STORAGE_TYPE` environment variable.
It's important to note that all listed storage types are used for writing, but only the first type in the list will be used for reading and archiving.
### Memory
The in-memory storage is not intended for production workloads. It's intended as a simple solution to get started quickly and
data will be lost once the process is gone.
By default, there's no limit in the amount of traces stored in memory but a limit can be established by passing an
integer value via `--memory.max-traces`.
### Badger - local storage
Experimental since Jaeger 1.9
[Badger](https://github.com/dgraph-io/badger) is an embedded local storage, only available
with **all-in-one** distribution. By default it acts as an ephemeral storage using a temporary filesystem.
This can be overridden by using the `--badger.ephemeral=false` option.
```sh
docker run \
-e SPAN_STORAGE_TYPE=badger \
-e BADGER_EPHEMERAL=false \
-e BADGER_DIRECTORY_VALUE=/badger/data \
-e BADGER_DIRECTORY_KEY=/badger/key \
-v <storage_dir_on_host>:/badger \
-p 16686:16686 \
jaegertracing/all-in-one:{{< currentVersion >}}
```
### Cassandra
Supported versions: 3.4+
Deploying Cassandra itself is out of scope for our documentation. One good
source of documentation is the [Apache Cassandra Docs](https://cassandra.apache.org/doc/latest/).
#### Configuration
##### Minimal
```sh
docker run \
-e SPAN_STORAGE_TYPE=cassandra \
-e CASSANDRA_SERVERS=<...> \
jaegertracing/jaeger-collector:{{< currentVersion >}}
```
##### All options
To view the full list of configuration options, you can run the following command:
```sh
docker run \
-e SPAN_STORAGE_TYPE=cassandra \
jaegertracing/jaeger-collector:{{< currentVersion >}} \
--help
```
#### Schema script
A script is provided to initialize Cassandra keyspace and schema
using Cassandra's interactive shell [`cqlsh`][cqlsh]:
```sh
MODE=test sh ./plugin/storage/cassandra/schema/create.sh | cqlsh
```
For production deployment, pass `MODE=prod DATACENTER={datacenter}` arguments to the script,
where `{datacenter}` is the name used in the Cassandra configuration / network topology.
The script also allows overriding TTL, keyspace name, replication factor, etc.
Run the script without arguments to see the full list of recognized parameters.
#### TLS support
Jaeger supports TLS client to node connections as long as you've configured
your Cassandra cluster correctly. After verifying with e.g. `cqlsh`, you can
configure the collector and query like so:
```sh
docker run \
-e CASSANDRA_SERVERS=<...> \
-e CASSANDRA_TLS=true \
-e CASSANDRA_TLS_SERVER_NAME="CN-in-certificate" \
-e CASSANDRA_TLS_KEY=<path to client key file> \
-e CASSANDRA_TLS_CERT=<path to client cert file> \
-e CASSANDRA_TLS_CA=<path to your CA cert file> \
jaegertracing/jaeger-collector:{{< currentVersion >}}
```
The schema tool also supports TLS. You need to make a custom cqlshrc file like
so:
```
# Creating schema in a cassandra cluster requiring client TLS certificates.
#
# Create a volume for the schema docker container containing four files:
# cqlshrc: this file
# ca-cert: the cert authority for your keys
# client-key: the keyfile for your client
# client-cert: the cert file matching client-key
#
# if there is any sort of DNS mismatch and you want to ignore server validation
# issues, then uncomment validate = false below.
#
# When running the container, map this volume to /root/.cassandra and set the
# environment variable CQLSH_SSL=--ssl
[ssl]
certfile = ~/.cassandra/ca-cert
userkey = ~/.cassandra/client-key
usercert = ~/.cassandra/client-cert
# validate = false
```
### Elasticsearch
Supported in Jaeger since 0.6.0
Supported versions: 5.x, 6.x
Elasticsearch does not require initialization other than
[installing and running Elasticsearch](https://www.elastic.co/downloads/elasticsearch).
Once it is running, pass the correct configuration values to the Jaeger collector and query service.
#### Configuration
##### Minimal
```sh
docker run \
-e SPAN_STORAGE_TYPE=elasticsearch \
-e ES_SERVER_URLS=<...> \
jaegertracing/jaeger-collector:{{< currentVersion >}}
```
##### All options
To view the full list of configuration options, you can run the following command:
```sh
docker run \
-e SPAN_STORAGE_TYPE=elasticsearch \
jaegertracing/jaeger-collector:{{< currentVersion >}} \
--help
```
See the [README](https://github.com/jaegertracing/jaeger/tree/master/plugin/storage/es/README.md) for an in-depth overview of how Jaeger uses Elasticsearch for storage.
#### Shards and Replicas for Elasticsearch indices
Shards and replicas are some configuration values to take special attention to, because this is decided upon
index creation. [This article](https://qbox.io/blog/optimizing-elasticsearch-how-many-shards-per-index) goes into
more information about choosing how many shards should be chosen for optimization.
### Kafka
Supported in Jaeger since 1.6.0
Supported Kafka versions: 0.9+
Kafka can be used as an intermediary buffer between collector and an actual storage.
The collector is configured with `SPAN_STORAGE_TYPE=kafka` that makes it write all received spans
into a Kafka topic. A new component [Ingester](#ingester), added in version 1.7.0, is used to read from
Kafka and store spans in another storage backend (Elasticsearch or Cassandra).
Writing to Kafka is particularly useful for building post-processing data pipelines.
#### Configuration
##### Minimal
```sh
docker run \
-e SPAN_STORAGE_TYPE=kafka \
-e KAFKA_BROKERS=<...> \
-e KAFKA_TOPIC=<...> \
jaegertracing/jaeger-collector:{{< currentVersion >}}
```
##### All options
To view the full list of configuration options, you can run the following command:
```sh
docker run \
-e SPAN_STORAGE_TYPE=kafka \
jaegertracing/jaeger-collector:{{< currentVersion >}} \
--help
```
#### Topic & partitions
Unless your Kafka cluster is configured to automatically create topics, you will need to create it ahead of time. You can refer to [the Kafka quickstart documentation](https://kafka.apache.org/documentation/#quickstart_createtopic) to learn how.
You can find more information about topics and partitions in general in the [official documentation](https://kafka.apache.org/documentation/#intro_topics). [This article](https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/) provide more details about how to choose the number of partitions.
### Storage plugin
Jaeger supports gRPC based storage plugins. For more information refer to [jaeger/plugin/storage/grpc](https://github.com/jaegertracing/jaeger/tree/master/plugin/storage/grpc)
Available plugins:
* [InfluxDB](https://github.com/influxdata/jaeger-influxdb/)
```sh
docker run \
-e SPAN_STORAGE_TYPE=grpc-plugin \
-e GRPC_STORAGE_PLUGIN_BINARY=<...> \
-e GRPC_STORAGE_PLUGIN_CONFIGURATION_FILE=<...> \
jaegertracing/all-in-one:{{< currentVersion >}}
```
## Ingester
**jaeger-ingester** is a service which reads span data from Kafka topic and writes it to another storage backend (Elasticsearch or Cassandra).
Port | Protocol | Function
----- | ------- | ---
14270 | HTTP | Healthcheck at `/` and metrics at `/metrics`
To view all exposed configuration options run the following command:
```sh
docker run \
-e SPAN_STORAGE_TYPE=cassandra \
jaegertracing/jaeger-ingester:{{< currentVersion >}}
--help
```
## Query Service & UI
**jaeger-query** serves the API endpoints and a React/Javascript UI.
The service is stateless and is typically run behind a load balancer, such as **nginx**.
At default settings the query service exposes the following port(s):
Port | Protocol | Function
----- | ------- | ---
16686 | HTTP | `/api/*` endpoints and Jaeger UI at `/`
16687 | HTTP | Healthcheck at `/` and metrics at `/metrics`
### Minimal deployment example (Elasticsearch backend):
```sh
docker run -d --rm \
-p 16686:16686 \
-p 16687:16687 \
-e SPAN_STORAGE_TYPE=elasticsearch \
-e ES_SERVER_URLS=http://<ES_SERVER_IP>:<ES_SERVER_PORT> \
jaegertracing/jaeger-query:{{< currentVersion >}}
```
### UI Base Path
The base path for all **jaeger-query** HTTP routes can be set to a non-root value, e.g. `/jaeger` would cause all UI URLs to start with `/jaeger`. This can be useful when running **jaeger-query** behind a reverse proxy.
The base path can be configured via the `--query.base-path` command line parameter or the `QUERY_BASE_PATH` environment variable.
### UI Customization and Embedding
Please refer to the [dedicated Frontend/UI page](../frontend-ui/).
## Aggregation Jobs for Service Dependencies
Production deployments need an external process which aggregates data and creates dependency links between services.
Project [spark-dependencies](https://github.com/jaegertracing/spark-dependencies) is a Spark job which derives
dependency links and stores them directly to the storage.
## Configuration
All binaries accepts command line properties and environmental variables which are managed by
by [viper](https://github.com/spf13/viper) and [cobra](https://github.com/spf13/cobra).
The names of environmental properties are capital letters and characters `-` and `.` are replaced with `_`.
To list all configuration properties call `jaeger-binary -h`.
[cqlsh]: http://cassandra.apache.org/doc/latest/tools/cqlsh.html
[zipkin-thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/zipkincore.thrift
[jaeger-thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/jaeger.thrift
[thriftrw]: https://www.npmjs.com/package/thriftrw

16
content/docs/1.12/faq.md Normal file
View File

@ -0,0 +1,16 @@
---
title: Frequently Asked Questions
description: Answers to some frequently asked questions about Jaeger.
weight: 11
---
## Why is the Dependencies page empty?
The Dependencies page shows a graph of services traced by Jaeger and connections between them. When you are using `all-in-one` binary with in-memory storage, the graph is calculated on-demand from all the traces stored in memory. However, if you are using a real distributed storage like Cassandra or Elasticsearch, it is too expensive to scan all the data in the database to build the service graph. Instead, the Jaeger project provides "big data" jobs that can be used to extract the service graph data from traces:
* https://github.com/jaegertracing/spark-dependencies - the older Spark job that can be run periodically
* https://github.com/jaegertracing/jaeger-analytics - the new (experimental) streaming Flink jobs that run continuously and build the service graph in smaller time intervals
## Why do I not see any spans in Jaeger?
Please refer to the [Troubleshooting](../troubleshooting/) guide.

View File

@ -0,0 +1,57 @@
---
title: Features
weight: 3
---
Jaeger is used for monitoring and troubleshooting microservices-based distributed systems, including:
* Distributed context propagation
* Distributed transaction monitoring
* Root cause analysis
* Service dependency analysis
* Performance / latency optimization
## High Scalability
Jaeger backend is designed to have no single points of failure and to scale with the business needs.
For example, any given Jaeger installation at Uber is typically processing several billion {{< tip "spans" "span" >}} per day.
## Native support for OpenTracing
Jaeger backend, Web UI, and instrumentation libraries have been designed from ground up to support the OpenTracing standard.
* Represent {{< tip "traces" "trace" >}} as {{< tip "directed acyclic graphs" "directed acyclic graph" >}} (not just trees) via [span references](https://github.com/opentracing/specification/blob/master/specification.md#references-between-spans)
* Support strongly typed span _tags_ and _structured logs_
* Support general distributed context propagation mechanism via _baggage_
## Multiple storage backends
Jaeger supports two popular open source NoSQL databases as trace storage backends: Cassandra 3.4+ and Elasticsearch 5.x/6.x.
There are ongoing community experiments using other databases, such as ScyllaDB, InfluxDB, Amazon DynamoDB. Jaeger also ships
with a simple in-memory storage for testing setups.
## Modern Web UI
Jaeger Web UI is implemented in Javascript using popular open source frameworks like React. Several performance
improvements have been released in v1.0 to allow the UI to efficiently deal with large volumes of data, and to display
{{< tip "traces" "trace" >}} with tens of thousands of {{< tip "spans" "span" >}} (e.g. we tried a trace with 80,000 spans).
## Cloud Native Deployment
Jaeger backend is distributed as a collection of Docker images. The binaries support various configuration methods,
including command line options, environment variables, and configuration files in multiple formats (yaml, toml, etc.).
Deployment to Kubernetes clusters is assisted by a [Kubernetes operator](https://github.com/jaegertracing/jaeger-operator), [Kubernetes templates](https://github.com/jaegertracing/jaeger-kubernetes)
and a [Helm chart](https://github.com/kubernetes/charts/tree/master/incubator/jaeger).
## Observability
All Jaeger backend components expose [Prometheus](https://prometheus.io/) metrics by default (other metrics backends are
also supported). Logs are written to standard out using the structured logging library [zap](https://github.com/uber-go/zap).
## Backwards compatibility with Zipkin
Although we recommend instrumenting applications with OpenTracing API and binding to Jaeger client libraries to benefit
from advanced features not available elsewhere, if your organization has already invested in the instrumentation
using Zipkin libraries, you do not have to rewrite all that code. Jaeger provides backwards compatibility with Zipkin
by accepting spans in Zipkin formats (Thrift or JSON v1/v2) over HTTP. Switching from Zipkin backend is just a matter
of routing the traffic from Zipkin libraries to the Jaeger backend.

View File

@ -0,0 +1,196 @@
---
title: Frontend/UI
hasparent: true
weight: 10
---
## Configuration
Several aspects of the UI can be configured:
* The Dependencies section can be enabled / configured
* Google Analytics tracking can be enabled / configured
* Additional menu options can be added to the global nav
These options can be configured by a JSON configuration file. The `--query.ui-config` command line parameter of the query service must then be set to the path to the JSON file when the query service is started.
An example configuration file:
```json
{
"dependencies": {
"dagMaxNumServices": 200,
"menuEnabled": true
},
"archiveEnabled": true,
"tracking": {
"gaID": "UA-000000-2",
"trackErrors": true
},
"menu": [
{
"label": "About Jaeger",
"items": [
{
"label": "GitHub",
"url": "https://github.com/jaegertracing/jaeger"
},
{
"label": "Docs",
"url": "http://jaeger.readthedocs.io/en/latest/"
}
]
}
]
}
```
`dependencies.dagMaxNumServices` defines the maximum number of services allowed before the DAG dependency view is disabled. Default: `200`.
`dependencies.menuEnabled` enables (`true`) or disables (`false`) the dependencies menu button. Default: `true`.
`archiveEnabled` enables (`true`) or disables (`false`) the archive traces button. Default: `false`. It requires a configuration of an archive storage in Query service. Archived traces are only accessible directly by ID, they are not searchable.
`tracking.gaID` defines the Google Analytics tracking ID. This is required for Google Analytics tracking, and setting it to a non-`null` value enables Google Analytics tracking. Default: `null`.
`tracking.trackErrors` enables (`true`) or disables (`false`) error tracking via Google Analytics. Errors can only be tracked if a valid Google Analytics ID is provided. For additional details on error tracking via Google Analytics see the [tracking README](https://github.com/jaegertracing/jaeger-ui/blob/c622330546afc1be59a42f874bcc1c2fadf7e69a/src/utils/tracking/README.md) in the UI repo. Default: `true`.
`menu` allows additional links to be added to the global nav. The additional links are right-aligned.
In the sample JSON config above, the configured menu will have a dropdown labeled "About Jaeger" with sub-options for "GitHub" and "Docs". The format for a link in the top right menu is as follows:
```json
{
"label": "Some text here",
"url": "https://example.com"
}
```
Links can either be members of the `menu` Array, directly, or they can be grouped into a dropdown menu option. The format for a group of links is:
```json
{
"label": "Dropdown button",
"items": [ ]
}
```
The `items` Array should contain one or more link configurations.
## Embedded Mode
Starting with version 1.9, Jaeger UI provides an "embedded" layout mode which is intended to support integrating Jaeger UI into other applications. Currently (as of `v0`), the approach taken is to remove various UI elements from the page to make the UI better suited for space-constrained layouts.
The embedded mode is induced and configured via URL query parameters.
To enter embedded mode, the `uiEmbed=v0` query parameter and value must be added to the URL. For example, the following URL will show the trace with ID `abc123` in embedded mode:
```
http://localhost:16686/trace/abc123?uiEmbed=v0
```
`uiEmbed=v0` is required.
Further, each page supported has an <img src="/img/frontend-ui/embed-open-icon.png" style="width: 20px; height:20px;display:inline;" alt="Embed open window"> button added that will open the non-embedded page in a new tab.
The following pages support embedded mode:
* Search Page
* Trace Page
### Search Page
To integrate the Search Trace Page to our application we have to indicate to the Jaeger UI that we want to use the embed mode with `uiEmbed=v0`.
For example:
```
http://localhost:16686/search?
service=my-service&
start=1543917759557000&
end=1543921359557000&
limit=20&
lookback=1h&
maxDuration&
minDuration&
uiEmbed=v0
```
![Embed Search Traces](/img/frontend-ui/embed-search-traces.png)
#### Configuration options
The following query parameter can be used to configure the layout of the search page :
* `uiSearchHideGraph=1` - disables the display of the scatter plot above the search results
```
http://localhost:16686/search?
service=my-service&
start=1543917759557000&
end=1543921359557000&
limit=20&
lookback=1h&
maxDuration&
minDuration&
uiEmbed=v0&
uiSearchHideGraph=1
```
![Embed Search Traces without Graph](/img/frontend-ui/embed-search-traces-hide-graph.png)
### Trace Page
To integrate the Trace Page to our application we have to indicate to the Jaeger UI that we want to use the embed mode with `uiEmbed=v0`.
For example:
```sh
http://localhost:16686/trace/{trace-id}?uiEmbed=v0
```
![Embed Trace view](/img/frontend-ui/embed-trace-view.png)
If we have navigated to this view from the search traces page we'll have a button to go back to the results page.
![Embed Trace view](/img/frontend-ui/embed-trace-view-with-back-button.png)
#### Configuration options
The following query parameters can be used to configure the layout of the trace page :
* `uiTimelineCollapseTitle=1` causes the trace header to start out collapsed, which hides the summary and the minimap.
```
http://localhost:16686/trace/{trace-id}?
uiEmbed=v0&
uiTimelineCollapseTitle=1
```
![Embed Trace view](/img/frontend-ui/embed-trace-view-with-collapse.png)
* `uiTimelineHideMinimap=1` removes the minimap, entirely, regardless of whether the trace header is expanded or not.
```
http://localhost:16686/trace/{trace-id}?
uiEmbed=v0&
uiTimelineHideMinimap=1
```
![Embed Trace view](/img/frontend-ui/embed-trace-view-with-hide-minimap.png)
* `uiTimelineHideSummary=1` - removes the trace summary information (number of services, etc.) entirely, regardless of whether the trace header is expanded or not.
```
http://localhost:16686/trace/{trace-id}?
uiEmbed=v0&
uiTimelineHideSummary=1
```
![Embed Trace view](/img/frontend-ui/embed-trace-view-with-hide-summary.png)
We can also combine the options:
```
http://localhost:16686/trace/{trace-id}?
uiEmbed=v0&
uiTimelineHideMinimap=1&
uiTimelineHideSummary=1
```
![Embed Trace view](/img/frontend-ui/embed-trace-view-with-hide-details-and-hide-minimap.png)

View File

@ -0,0 +1,129 @@
---
title: Getting started
description: Get up and running with Jaeger in your local environment
weight: 2
---
## Instrumentation
Your applications must be instrumented before they can send tracing data to Jaeger backend. Check the [Client Libraries](../client-libraries) section for information about how to use the OpenTracing API and how to initialize and configure Jaeger tracers.
## All in One
All-in-one is an executable designed for quick local testing, launches the Jaeger UI, collector, query, and agent, with an in memory storage component.
The simplest way to start the all-in-one is to use the pre-built image published to DockerHub (a single command line).
```bash
$ docker run -d --name jaeger \
-e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
-p 5775:5775/udp \
-p 6831:6831/udp \
-p 6832:6832/udp \
-p 5778:5778 \
-p 16686:16686 \
-p 14268:14268 \
-p 9411:9411 \
jaegertracing/all-in-one:{{< currentVersion >}}
```
Or run the `jaeger-all-in-one(.exe)` executable from the [binary distribution archives][download]:
```bash
$ jaeger-all-in-one --collector.zipkin.http-port=9411
```
You can then navigate to `http://localhost:16686` to access the Jaeger UI.
The container exposes the following ports:
Port | Protocol | Component | Function
----- | ------- | --------- | ---
5775 | UDP | agent | accept `zipkin.thrift` over compact thrift protocol (deprecated, used by legacy clients only)
6831 | UDP | agent | accept `jaeger.thrift` over compact thrift protocol
6832 | UDP | agent | accept `jaeger.thrift` over binary thrift protocol
5778 | HTTP | agent | serve configs
16686 | HTTP | query | serve frontend
14268 | HTTP | collector | accept `jaeger.thrift` directly from clients
14250 | HTTP | collector | accept `model.proto`
9411 | HTTP | collector | Zipkin compatible endpoint (optional)
## Kubernetes and OpenShift
* Kubernetes templates: https://github.com/jaegertracing/jaeger-kubernetes
* Kubernetes Operator: https://github.com/jaegertracing/jaeger-operator
* OpenShift templates: https://github.com/jaegertracing/jaeger-openshift
## Sample App: HotROD
HotROD (Rides on Demand) is a demo application that consists of several microservices and
illustrates the use of the [OpenTracing API](http://opentracing.io).
A tutorial / walkthrough is available in the blog post:
[Take OpenTracing for a HotROD ride][hotrod-tutorial].
It can be run standalone, but requires Jaeger backend to view the traces.
### Features
- Discover architecture of the whole system via data-driven dependency
diagram.
- View request timeline and errors; understand how the app works.
- Find sources of latency and lack of concurrency.
- Highly contextualized logging.
- Use baggage propagation to:
- Diagnose inter-request contention (queueing).
- Attribute time spent in a service.
- Use open source libraries with OpenTracing integration to get
vendor-neutral instrumentation for free.
### Prerequisites
- You need Go 1.11 or higher installed on your machine to run from source.
- Requires a [running Jaeger backend](#all-in-one) to view the traces.
### Running
#### From Source
```bash
mkdir -p $GOPATH/src/github.com/jaegertracing
cd $GOPATH/src/github.com/jaegertracing
git clone git@github.com:jaegertracing/jaeger.git jaeger
cd jaeger
make install
go run ./examples/hotrod/main.go all
```
#### From docker
```bash
$ docker run --rm -it \
--link jaeger \
-p8080-8083:8080-8083 \
-e JAEGER_AGENT_HOST="jaeger" \
jaegertracing/example-hotrod:{{< currentVersion >}} \
all
```
#### From binary distribution
Run `example-hotrod(.exe)` executable from the [binary distribution archives][download]:
```bash
$ example-hotrod all
```
Then navigate to `http://localhost:8080`.
## Migrating from Zipkin
Collector service exposes Zipkin compatible REST API `/api/v1/spans` which accepts both Thrift and JSON. Also there is `/api/v2/spans` for JSON only.
By default it's disabled. It can be enabled with `--collector.zipkin.http-port=9411`.
Zipkin Thrift IDL file can be found in [jaegertracing/jaeger-idl](https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/zipkincore.thrift).
It's compatible with [openzipkin/zipkin-api](https://github.com/openzipkin/zipkin-api/blob/master/thrift/zipkinCore.thrift)
[hotrod-tutorial]: https://medium.com/@YuriShkuro/take-opentracing-for-a-hotrod-ride-f6e3141f7941
[download]: ../../../download/

View File

@ -0,0 +1,71 @@
---
title: Introduction
weight: 1
children:
- title: Features
url: features
---
Welcome to Jaeger's documentation portal! Below, you'll find information for beginners and experienced Jaeger users.
If you can't find what you are looking for, or have an issue not covered here, we'd love to [hear from you](/get-in-touch).
## About
Jaeger, inspired by [Dapper][dapper] and [OpenZipkin](http://zipkin.io),
is a distributed tracing system released as open source by [Uber Technologies][ubeross].
It is used for monitoring and troubleshooting microservices-based distributed systems, including:
* Distributed context propagation
* Distributed transaction monitoring
* Root cause analysis
* Service dependency analysis
* Performance / latency optimization
Uber published a blog post, [Evolving Distributed Tracing at Uber](https://eng.uber.com/distributed-tracing/), where they explain the history and reasons for the architectural choices made in Jaeger. [Yuri Shkuro](https://shkuro.com), creator of Jaeger, also published a book [Mastering Distributed Tracing](https://shkuro.com/books/2019-mastering-distributed-tracing/) that covers in-depth many aspects of Jaeger design and operation, as well as distributed tracing in general.
## Features
* [OpenTracing](http://opentracing.io/) compatible data model and instrumentation libraries
* in [Go](https://github.com/jaegertracing/jaeger-client-go), [Java](https://github.com/jaegertracing/jaeger-client-java), [Node](https://github.com/jaegertracing/jaeger-client-node), [Python](https://github.com/jaegertracing/jaeger-client-python)
and [C++](https://github.com/jaegertracing/cpp-client)
* Uses consistent upfront sampling with individual per service/endpoint probabilities
* Multiple storage backends: Cassandra, Elasticsearch, memory.
* Adaptive sampling (coming soon)
* Post-collection data processing pipeline (coming soon)
See [Features](./features/) page for more details.
## Technical Specs
* Backend components implemented in Go
* React/Javascript UI
* Supported storage backends:
* [Cassandra 3.4+](./deployment/#cassandra)
* [Elasticsearch 5.x, 6.x](./deployment/#elasticsearch)
* [Kafka](./deployment/#kafka)
* memory storage
## Quick Start
See [running a docker all in one image](getting-started#all-in-one).
## Screenshots
### Traces View
[![Traces View](/img/traces-ss.png)](/img/traces-ss.png)
### Trace Detail View
[![Detail View](/img/trace-detail-ss.png)](/img/trace-detail-ss.png)
## Related links
- [Evolving Distributed tracing At Uber Engineering](https://eng.uber.com/distributed-tracing/)
- [Mastering Distributed Tracing](https://shkuro.com/books/2019-mastering-distributed-tracing/)
- [Tracing HTTP request latency in Go with OpenTracing](https://medium.com/opentracing/tracing-http-request-latency-in-go-with-opentracing-7cc1282a100a)
- [Distributed Tracing with Jaeger & Prometheus on Kubernetes](https://blog.openshift.com/openshift-commons-briefing-82-distributed-tracing-with-jaeger-prometheus-on-kubernetes/)
- [Using Jaeger with Istio](https://istio.io/docs/tasks/telemetry/distributed-tracing.html)
- [Using Jaeger with Envoy](https://www.envoyproxy.io/docs/envoy/latest/start/sandboxes/jaeger_tracing.html)
[dapper]: https://research.google.com/pubs/pub36356.html
[ubeross]: http://uber.github.io

87
content/docs/1.13/apis.md Normal file
View File

@ -0,0 +1,87 @@
---
title: APIs
hasparent: true
---
Jaeger components implement various APIs for saving or retrieving trace data.
The following labels are used to describe API compatibility guarantees.
* **stable** - the API guarantees backwards compatibility. If breaking changes are going to be made in the future, they will result in a new API version, e.g. `/api/v2` URL prefix or a different namespace in the IDL.
* **internal** - the APIs intended for internal communications between Jaeger components and not recommended for use by external components.
* **deprecated** - the APIs that are only maintained for legacy reasons and will be phased out in the future.
## Span reporting APIs
Agent and Collector are the two components of the Jaeger backend that can receive spans. At this time they support two sets of non-overlapping APIs.
### Thrift over UDP (stable)
The Agent can only receive spans over UDP in Thrift format. The primary API is a UDP packet that contains a Thrift-encoded `Batch` struct defined in [jaeger.thrift][jaeger.thrift] IDL file, located in the [jaeger-idl][jaeger-idl] repository. Most Jaeger Clients use Thrift's `compact` encoding, however some client libraries do not support it (notably, Node.js) and use Thrift's `binary` encoding (sent to a different UDP port). The Agent's API is defined by [agent.thrift][agent.thrift] IDL file.
For legacy reasons, the Agent also accepts spans in Zipkin format, however, only very old versions of Jaeger clients can send data in that format and it is officially deprecated.
### Protobuf via gRPC (stable)
In a typical Jaeger deployment, Agents receive spans from Clients and forward them to Collectors. Since Jaeger version 1.11 the official and recommended protocol between Agents and Collectors is gRPC with Protobuf.
The Protobuf IDL [collector.proto][collector.proto] is currently located in the main Jaeger repository, under [model/proto/api_v2][collector.proto]. In the future it will be moved to [jaeger-idl][jaeger-idl] repository ([jaeger-idl/issues/55](https://github.com/jaegertracing/jaeger-idl/issues/55)).
### Thrift over HTTP (stable)
In some cases it is not feasible to deploy Jaeger Agent next to the application, for example, when the application code is running as AWS Lambda function. In these scenarios the Jaeger Clients can be configured to submit spans directly to the Collectors over HTTP/HTTPS.
The same [jaeger.thrift][jaeger.thrift] payload can be submitted in HTTP POST request to `/api/traces` endpoint, for example, `https://jaeger-collector:14268/api/traces`. The `Batch` struct needs to be encoded using Thrift's `binary` encoding, and the HTTP request should specify the content type header:
```
Content-Type: application/vnd.apache.thrift.binary
```
### JSON over HTTP (n/a)
There is no official Jaeger JSON format that can be accepted by the collector. In the future the Protobuf-generated JSON may be supported.
### Thrift via TChannel (deprecated)
Agent and Collector can communicate using TChannel protocol. This protocol is generally not supported by the routing infrastructure and has been deprecated. It will be eventually removed from Jaeger.
### Zipkin Formats (stable)
Jaeger Collector can also accept spans in several Zipkin data format, namely JSON v1/v2 and Thrift. The Collector needs to be configured to enable Zipkin HTTP server, e.g. on port 9411 used by Zipkin collectors. The server enables two endpoints that expect POST requests:
* `/api/v1/spans` for submitting spans in Zipkin JSON v1 or Zipkin Thrift format.
* `/api/v2/spans` for submitting spans in Zipkin JSON v2.
## Trace retrieval APIs
Traces saved in the storage can be retrieved by calling Jaeger Query Service.
### gRPC/Protobuf (stable)
The recommended way for programmatically retrieving traces and other data is via gRPC endpoint defined in [query.proto][query.proto] IDL file (located in the main Jaeger repository, similar to [collector.proto][collector.proto]).
### HTTP JSON (internal)
Jaeger UI communicates with Jaeger Query Service via JSON API. For example, a trace can be retrieved via GET request to `https://jaeger-query:16686/api/traces/{trace-id-hex-string}`. This JSON API is intentionally undocumented and subject to change.
## Clients configuration (internal)
Client libraries not only submit finished spans to Jaeger backend, but also periodically poll the Agents for various configurations, such as sampling strategies. The schema for the payload is defined by [sampling.thrift][sampling.thrift], encoded as JSON using Thrift's built-in JSON generation capabilities.
## Service dependencies graph (internal)
Can be retrieved from Query Service at `/api/dependencies` endpoint. The GET request expects two parameters:
* `endTs` (number of milliseconds since epoch) - the end of the time interval
* `lookback` (in milliseconds) - the length the time interval (i.e. start-time + lookback = end-time).
The returned JSON is a list of edges represented as tuples `(caller, callee, count)`.
For programmatic access to service graph, the recommended API is gRPC/Protobuf described above.
[jaeger-idl]: https://github.com/jaegertracing/jaeger-idl/
[jaeger.thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/jaeger.thrift
[agent.thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/agent.thrift
[sampling.thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/sampling.thrift
[collector.proto]: https://github.com/jaegertracing/jaeger/blob/master/model/proto/api_v2/collector.proto
[query.proto]: https://github.com/jaegertracing/jaeger/blob/master/model/proto/api_v2/query.proto

13
content/docs/1.13/cli.md Normal file
View File

@ -0,0 +1,13 @@
---
title: CLI flags
widescreen: true
hasparent: true
---
This is auto-generated documentation for CLI flags supported by Jaeger binaries.
* CLI flags for some binaries change depending on the `SPAN_STORAGE_TYPE` environment variable. Relevant variations are included below.
* Some binaries support _commands_ (mostly informational), such as `env`, `docs`, and `version`. These commands are not included here.
* All parameters can be also provided via environment variables, by changing all letters to upper-case and replacing all punctuation characters with underscore `_`. For example, the value for the flag `--cassandra.connections-per-host` can be provided via `CASSANDRA_CONNECTIONS_PER_HOST` environment variable.
{{< cli/tools-list >}}

View File

@ -0,0 +1,397 @@
---
title: Deployment
weight: 4
children:
- title: Operator for Kubernetes
navtitle: Kubernetes
url: operator
- title: Frontend/UI
url: frontend-ui
- title: CLI Flags
url: cli
---
The main Jaeger backend components are released as Docker images on Docker Hub:
Component | Repository
--------------------- | ---
**jaeger-agent** | [hub.docker.com/r/jaegertracing/jaeger-agent/](https://hub.docker.com/r/jaegertracing/jaeger-agent/)
**jaeger-collector** | [hub.docker.com/r/jaegertracing/jaeger-collector/](https://hub.docker.com/r/jaegertracing/jaeger-collector/)
**jaeger-query** | [hub.docker.com/r/jaegertracing/jaeger-query/](https://hub.docker.com/r/jaegertracing/jaeger-query/)
**jaeger-ingester** | [hub.docker.com/r/jaegertracing/jaeger-ingester/](https://hub.docker.com/r/jaegertracing/jaeger-ingester/)
There are orchestration templates for running Jaeger with:
* Kubernetes: [github.com/jaegertracing/jaeger-kubernetes](https://github.com/jaegertracing/jaeger-kubernetes),
* OpenShift: [github.com/jaegertracing/jaeger-openshift](https://github.com/jaegertracing/jaeger-openshift).
## Configuration Options
Jaeger binaries can be configured in a number of ways (in the order of decreasing priority):
* command line arguments,
* environment variables,
* configuration files in JSON, TOML, YAML, HCL, or Java properties formats.
To see the complete list of options, run the binary with `help` command. Options that are specific to a certain storage backend are only listed if the storage type is selected. For example, to see all available options in the Collector with Cassandra storage:
```sh
$ docker run --rm \
-e SPAN_STORAGE_TYPE=cassandra \
jaegertracing/jaeger-collector:{{< currentVersion >}} \
help
```
In order to provide configuration parameters via environment variables, find the respective command line option and convert its name to UPPER_SNAKE_CASE, for example:
Command line option | Environment variable
-----------------------------------|-------------------------------
`--cassandra.connections-per-host` | `CASSANDRA_CONNECTIONS_PER_HOST`
`--metrics-backend` | `METRICS_BACKEND`
## Agent
Jaeger client libraries expect **jaeger-agent** process to run locally on each host.
The agent exposes the following ports:
Port | Protocol | Function
----- | ------- | ---
6831 | UDP | accept [jaeger.thrift][jaeger-thrift] in `compact` Thrift protocol used by most current Jaeger clients
6832 | UDP | accept [jaeger.thrift][jaeger-thrift] in `binary` Thrift protocol used by Node.js Jaeger client (because [thriftrw][thriftrw] npm package does not support `compact` protocol)
5778 | HTTP | serve configs, sampling strategies
5775 | UDP | accept [zipkin.thrift][zipkin-thrift] in `compact` Thrift protocol (deprecated; only used by very old Jaeger clients, circa 2016)
14271 | HTTP | Healthcheck at `/` and metrics at `/metrics`
It can be executed directly on the host or via Docker, as follows:
```sh
## make sure to expose only the ports you use in your deployment scenario!
docker run \
--rm \
-p6831:6831/udp \
-p6832:6832/udp \
-p5778:5778/tcp \
-p5775:5775/udp \
jaegertracing/jaeger-agent:{{< currentVersion >}}
```
### Discovery System Integration
The agents can connect point to point to a single collector address, which could be
load balanced by another infrastructure component (e.g. DNS) across multiple collectors.
The agent can also be configured with a static list of collector addresses.
On Docker, a command like the following can be used:
```sh
docker run \
--rm \
-p5775:5775/udp \
-p6831:6831/udp \
-p6832:6832/udp \
-p5778:5778/tcp \
jaegertracing/jaeger-agent:{{< currentVersion >}} \
--reporter.grpc.host-port=jaeger-collector.jaeger-infra.svc:14250
```
Or use `--reporter.tchannel.host-port=jaeger-collector.jaeger-infra.svc:14267` to use
legacy tchannel reporter.
When using gRPC, you have several options for load balancing and name resolution:
* Single connection and no load balancing. This is the default if you specify a single `host:port`. (example: `--reporter.grpc.host-port=jaeger-collector.jaeger-infra.svc:14250`)
* Static list of hostnames and round-robin load balancing. This is what you get with a comma-separated list of addresses. (example: `reporter.grpc.host-port=jaeger-collector1:14250,jaeger-collector2:14250,jaeger-collector3:14250`)
* Dynamic DNS resolution and round-robin load balancing. To get this behaviour, prefix the address with `dns:///` and gRPC will attempt to resolve the hostname using SRV records (for [external load balancing](https://github.com/grpc/grpc/blob/master/doc/load-balancing.md)), TXT records (for [service configs](https://github.com/grpc/grpc/blob/master/doc/service_config.md)), and A records. Refer to the [gRPC Name Resolution docs](https://github.com/grpc/grpc/blob/master/doc/naming.md) and the [dns_resolver.go implementation](https://github.com/grpc/grpc-go/blob/master/resolver/dns/dns_resolver.go) for more info. (example: `--reporter.grpc.host-port=dns:///jaeger-collector.jaeger-infra.svc:14250`)
### Agent level tags
Jaeger supports agent level tags, that can be added to the process tags of all spans passing through the agent. This is supported through the command line flag `--jaeger.tags=key1=value1,key2=value2,...,keyn=valuen`. Tags can also be set through an environment flag like so - `--jaeger.tags=key=${envFlag:defaultValue}` - The tag value will be set to the value of the `envFlag` environment key and `defaultValue` if not set.
## Collectors
The collectors are stateless and thus many instances of **jaeger-collector** can be run in parallel.
Collectors require almost no configuration, except for the location of Cassandra cluster,
via `--cassandra.keyspace` and `--cassandra.servers` options, or the location of Elasticsearch cluster, via
`--es.server-urls`, depending on which storage is specified. To see all command line options run
```sh
go run ./cmd/collector/main.go -h
```
or, if you don't have the source code
```sh
docker run -it --rm jaegertracing/jaeger-collector:{{< currentVersion >}} -h
```
At default settings the collector exposes the following ports:
Port | Protocol | Function
----- | ------- | ---
14267 | TChannel | used by **jaeger-agent** to send spans in jaeger.thrift format
14250 | gRPC | used by **jaeger-agent** to send spans in model.proto format
14268 | HTTP | can accept spans directly from clients in jaeger.thrift format over binary thrift protocol
9411 | HTTP | can accept Zipkin spans in JSON or Thrift (disabled by default)
14269 | HTTP | Healthcheck at `/` and metrics at `/metrics`
## Storage Backends
Collectors require a persistent storage backend. Cassandra and Elasticsearch are the primary supported storage backends. Additional backends are [discussed here](https://github.com/jaegertracing/jaeger/issues/638).
The storage type can be passed via `SPAN_STORAGE_TYPE` environment variable. Valid values are `cassandra`, `elasticsearch`, `kafka` (only as a buffer), `grpc-plugin`, `badger` (only with all-in-one) and `memory` (only with all-in-one).
As of version 1.6.0, it's possible to use multiple storage types at the same time by providing a comma-separated list of valid types to the `SPAN_STORAGE_TYPE` environment variable.
It's important to note that all listed storage types are used for writing, but only the first type in the list will be used for reading and archiving.
### Memory
The in-memory storage is not intended for production workloads. It's intended as a simple solution to get started quickly and
data will be lost once the process is gone.
By default, there's no limit in the amount of traces stored in memory but a limit can be established by passing an
integer value via `--memory.max-traces`.
### Badger - local storage
Experimental since Jaeger 1.9
[Badger](https://github.com/dgraph-io/badger) is an embedded local storage, only available
with **all-in-one** distribution. By default it acts as an ephemeral storage using a temporary filesystem.
This can be overridden by using the `--badger.ephemeral=false` option.
```sh
docker run \
-e SPAN_STORAGE_TYPE=badger \
-e BADGER_EPHEMERAL=false \
-e BADGER_DIRECTORY_VALUE=/badger/data \
-e BADGER_DIRECTORY_KEY=/badger/key \
-v <storage_dir_on_host>:/badger \
-p 16686:16686 \
jaegertracing/all-in-one:{{< currentVersion >}}
```
### Cassandra
Supported versions: 3.4+
Deploying Cassandra itself is out of scope for our documentation. One good
source of documentation is the [Apache Cassandra Docs](https://cassandra.apache.org/doc/latest/).
#### Configuration
##### Minimal
```sh
docker run \
-e SPAN_STORAGE_TYPE=cassandra \
-e CASSANDRA_SERVERS=<...> \
jaegertracing/jaeger-collector:{{< currentVersion >}}
```
##### All options
To view the full list of configuration options, you can run the following command:
```sh
docker run \
-e SPAN_STORAGE_TYPE=cassandra \
jaegertracing/jaeger-collector:{{< currentVersion >}} \
--help
```
#### Schema script
A script is provided to initialize Cassandra keyspace and schema
using Cassandra's interactive shell [`cqlsh`][cqlsh]:
```sh
MODE=test sh ./plugin/storage/cassandra/schema/create.sh | cqlsh
```
For production deployment, pass `MODE=prod DATACENTER={datacenter}` arguments to the script,
where `{datacenter}` is the name used in the Cassandra configuration / network topology.
The script also allows overriding TTL, keyspace name, replication factor, etc.
Run the script without arguments to see the full list of recognized parameters.
#### TLS support
Jaeger supports TLS client to node connections as long as you've configured
your Cassandra cluster correctly. After verifying with e.g. `cqlsh`, you can
configure the collector and query like so:
```sh
docker run \
-e CASSANDRA_SERVERS=<...> \
-e CASSANDRA_TLS=true \
-e CASSANDRA_TLS_SERVER_NAME="CN-in-certificate" \
-e CASSANDRA_TLS_KEY=<path to client key file> \
-e CASSANDRA_TLS_CERT=<path to client cert file> \
-e CASSANDRA_TLS_CA=<path to your CA cert file> \
jaegertracing/jaeger-collector:{{< currentVersion >}}
```
The schema tool also supports TLS. You need to make a custom cqlshrc file like
so:
```
# Creating schema in a cassandra cluster requiring client TLS certificates.
#
# Create a volume for the schema docker container containing four files:
# cqlshrc: this file
# ca-cert: the cert authority for your keys
# client-key: the keyfile for your client
# client-cert: the cert file matching client-key
#
# if there is any sort of DNS mismatch and you want to ignore server validation
# issues, then uncomment validate = false below.
#
# When running the container, map this volume to /root/.cassandra and set the
# environment variable CQLSH_SSL=--ssl
[ssl]
certfile = ~/.cassandra/ca-cert
userkey = ~/.cassandra/client-key
usercert = ~/.cassandra/client-cert
# validate = false
```
### Elasticsearch
Supported in Jaeger since 0.6.0
Supported versions: 5.x, 6.x
Elasticsearch does not require initialization other than
[installing and running Elasticsearch](https://www.elastic.co/downloads/elasticsearch).
Once it is running, pass the correct configuration values to the Jaeger collector and query service.
#### Configuration
##### Minimal
```sh
docker run \
-e SPAN_STORAGE_TYPE=elasticsearch \
-e ES_SERVER_URLS=<...> \
jaegertracing/jaeger-collector:{{< currentVersion >}}
```
##### All options
To view the full list of configuration options, you can run the following command:
```sh
docker run \
-e SPAN_STORAGE_TYPE=elasticsearch \
jaegertracing/jaeger-collector:{{< currentVersion >}} \
--help
```
See the [README](https://github.com/jaegertracing/jaeger/tree/master/plugin/storage/es/README.md) for an in-depth overview of how Jaeger uses Elasticsearch for storage.
#### Shards and Replicas for Elasticsearch indices
Shards and replicas are some configuration values to take special attention to, because this is decided upon
index creation. [This article](https://qbox.io/blog/optimizing-elasticsearch-how-many-shards-per-index) goes into
more information about choosing how many shards should be chosen for optimization.
### Kafka
Supported in Jaeger since 1.6.0
Supported Kafka versions: 0.9+
Kafka can be used as an intermediary buffer between collector and an actual storage.
The collector is configured with `SPAN_STORAGE_TYPE=kafka` that makes it write all received spans
into a Kafka topic. A new component [Ingester](#ingester), added in version 1.7.0, is used to read from
Kafka and store spans in another storage backend (Elasticsearch or Cassandra).
Writing to Kafka is particularly useful for building post-processing data pipelines.
#### Configuration
##### Minimal
```sh
docker run \
-e SPAN_STORAGE_TYPE=kafka \
-e KAFKA_BROKERS=<...> \
-e KAFKA_TOPIC=<...> \
jaegertracing/jaeger-collector:{{< currentVersion >}}
```
##### All options
To view the full list of configuration options, you can run the following command:
```sh
docker run \
-e SPAN_STORAGE_TYPE=kafka \
jaegertracing/jaeger-collector:{{< currentVersion >}} \
--help
```
#### Topic & partitions
Unless your Kafka cluster is configured to automatically create topics, you will need to create it ahead of time. You can refer to [the Kafka quickstart documentation](https://kafka.apache.org/documentation/#quickstart_createtopic) to learn how.
You can find more information about topics and partitions in general in the [official documentation](https://kafka.apache.org/documentation/#intro_topics). [This article](https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/) provide more details about how to choose the number of partitions.
### Storage plugin
Jaeger supports gRPC based storage plugins. For more information refer to [jaeger/plugin/storage/grpc](https://github.com/jaegertracing/jaeger/tree/master/plugin/storage/grpc)
Available plugins:
* [InfluxDB](https://github.com/influxdata/jaeger-influxdb/)
```sh
docker run \
-e SPAN_STORAGE_TYPE=grpc-plugin \
-e GRPC_STORAGE_PLUGIN_BINARY=<...> \
-e GRPC_STORAGE_PLUGIN_CONFIGURATION_FILE=<...> \
jaegertracing/all-in-one:{{< currentVersion >}}
```
## Ingester
**jaeger-ingester** is a service which reads span data from Kafka topic and writes it to another storage backend (Elasticsearch or Cassandra).
Port | Protocol | Function
----- | ------- | ---
14270 | HTTP | Healthcheck at `/` and metrics at `/metrics`
To view all exposed configuration options run the following command:
```sh
docker run \
-e SPAN_STORAGE_TYPE=cassandra \
jaegertracing/jaeger-ingester:{{< currentVersion >}}
--help
```
## Query Service & UI
**jaeger-query** serves the API endpoints and a React/Javascript UI.
The service is stateless and is typically run behind a load balancer, such as [**NGINX**](https://www.nginx.com/).
At default settings the query service exposes the following port(s):
Port | Protocol | Function
----- | ------- | ---
16686 | HTTP | `/api/*` endpoints and Jaeger UI at `/`
16687 | HTTP | Healthcheck at `/` and metrics at `/metrics`
### Minimal deployment example (Elasticsearch backend):
```sh
docker run -d --rm \
-p 16686:16686 \
-p 16687:16687 \
-e SPAN_STORAGE_TYPE=elasticsearch \
-e ES_SERVER_URLS=http://<ES_SERVER_IP>:<ES_SERVER_PORT> \
jaegertracing/jaeger-query:{{< currentVersion >}}
```
### UI Base Path
The base path for all **jaeger-query** HTTP routes can be set to a non-root value, e.g. `/jaeger` would cause all UI URLs to start with `/jaeger`. This can be useful when running **jaeger-query** behind a reverse proxy.
The base path can be configured via the `--query.base-path` command line parameter or the `QUERY_BASE_PATH` environment variable.
### UI Customization and Embedding
Please refer to the [dedicated Frontend/UI page](../frontend-ui/).
## Aggregation Jobs for Service Dependencies
Production deployments need an external process which aggregates data and creates dependency links between services. Project [spark-dependencies](https://github.com/jaegertracing/spark-dependencies) is a Spark job which derives dependency links and stores them directly to the storage.
## Configuration
All binaries accepts command line properties and environmental variables, power by [viper](https://github.com/spf13/viper) and [cobra](https://github.com/spf13/cobra) libraries. Please refer to the [CLI Flags](../cli/) page for more information.
[cqlsh]: http://cassandra.apache.org/doc/latest/tools/cqlsh.html
[zipkin-thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/zipkincore.thrift
[jaeger-thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/jaeger.thrift
[thriftrw]: https://www.npmjs.com/package/thriftrw

17
content/docs/1.13/faq.md Normal file
View File

@ -0,0 +1,17 @@
---
title: Frequently Asked Questions
navtitle: FAQs
description: Answers to some frequently asked questions about Jaeger.
weight: 11
---
## Why is the Dependencies page empty?
The Dependencies page shows a graph of services traced by Jaeger and connections between them. When you are using `all-in-one` binary with in-memory storage, the graph is calculated on-demand from all the traces stored in memory. However, if you are using a real distributed storage like Cassandra or Elasticsearch, it is too expensive to scan all the data in the database to build the service graph. Instead, the Jaeger project provides "big data" jobs that can be used to extract the service graph data from traces:
* https://github.com/jaegertracing/spark-dependencies - the older Spark job that can be run periodically
* https://github.com/jaegertracing/jaeger-analytics - the new (experimental) streaming Flink jobs that run continuously and builds the service graph in smaller time intervals
## Why do I not see any spans in Jaeger?
Please refer to the [Troubleshooting](../troubleshooting/) guide.

View File

@ -0,0 +1,57 @@
---
title: Features
hasparent: true
---
Jaeger is used for monitoring and troubleshooting microservices-based distributed systems, including:
* Distributed context propagation
* Distributed transaction monitoring
* Root cause analysis
* Service dependency analysis
* Performance / latency optimization
## High Scalability
Jaeger backend is designed to have no single points of failure and to scale with the business needs.
For example, any given Jaeger installation at Uber is typically processing several billion {{< tip "spans" "span" >}} per day.
## Native support for OpenTracing
Jaeger backend, Web UI, and instrumentation libraries have been designed from ground up to support the OpenTracing standard.
* Represent {{< tip "traces" "trace" >}} as {{< tip "directed acyclic graphs" "directed acyclic graph" >}} (not just trees) via [span references](https://github.com/opentracing/specification/blob/master/specification.md#references-between-spans)
* Support strongly typed span _tags_ and _structured logs_
* Support general distributed context propagation mechanism via _baggage_
## Multiple storage backends
Jaeger supports two popular open source NoSQL databases as trace storage backends: Cassandra 3.4+ and Elasticsearch 5.x/6.x.
There are ongoing community experiments using other databases, such as ScyllaDB, InfluxDB, Amazon DynamoDB. Jaeger also ships
with a simple in-memory storage for testing setups.
## Modern Web UI
Jaeger Web UI is implemented in Javascript using popular open source frameworks like React. Several performance
improvements have been released in v1.0 to allow the UI to efficiently deal with large volumes of data, and to display
{{< tip "traces" "trace" >}} with tens of thousands of {{< tip "spans" "span" >}} (e.g. we tried a trace with 80,000 spans).
## Cloud Native Deployment
Jaeger backend is distributed as a collection of Docker images. The binaries support various configuration methods,
including command line options, environment variables, and configuration files in multiple formats (yaml, toml, etc.).
Deployment to Kubernetes clusters is assisted by a [Kubernetes operator](https://github.com/jaegertracing/jaeger-operator), [Kubernetes templates](https://github.com/jaegertracing/jaeger-kubernetes)
and a [Helm chart](https://github.com/kubernetes/charts/tree/master/incubator/jaeger).
## Observability
All Jaeger backend components expose [Prometheus](https://prometheus.io/) metrics by default (other metrics backends are
also supported). Logs are written to standard out using the structured logging library [zap](https://github.com/uber-go/zap).
## Backwards compatibility with Zipkin
Although we recommend instrumenting applications with OpenTracing API and binding to Jaeger client libraries to benefit
from advanced features not available elsewhere, if your organization has already invested in the instrumentation
using Zipkin libraries, you do not have to rewrite all that code. Jaeger provides backwards compatibility with Zipkin
by accepting spans in Zipkin formats (Thrift or JSON v1/v2) over HTTP. Switching from Zipkin backend is just a matter
of routing the traffic from Zipkin libraries to the Jaeger backend.

View File

@ -0,0 +1,224 @@
---
title: Frontend/UI Configuration
navtitle: Frontend/UI
hasparent: true
weight: 7
---
## Configuration
Several aspects of the UI can be configured:
* The Dependencies section can be enabled / configured
* Google Analytics tracking can be enabled / configured
* Additional menu options can be added to the global nav
These options can be configured by a JSON configuration file. The `--query.ui-config` command line parameter of the query service must then be set to the path to the JSON file when the query service is started.
An example configuration file:
```json
{
"dependencies": {
"dagMaxNumServices": 200,
"menuEnabled": true
},
"archiveEnabled": true,
"tracking": {
"gaID": "UA-000000-2",
"trackErrors": true
},
"menu": [
{
"label": "About Jaeger",
"items": [
{
"label": "GitHub",
"url": "https://github.com/jaegertracing/jaeger"
},
{
"label": "Docs",
"url": "http://jaeger.readthedocs.io/en/latest/"
}
]
}
],
"linkPatterns": [{
"type": "process",
"key": "jaeger.version",
"url": "https://github.com/jaegertracing/jaeger-client-java/releases/tag/#{jaeger.version}",
"text": "Information about Jaeger release #{jaeger.version}"
}]
}
```
### Dependencies
`dependencies.dagMaxNumServices` defines the maximum number of services allowed before the DAG dependency view is disabled. Default: `200`.
`dependencies.menuEnabled` enables (`true`) or disables (`false`) the dependencies menu button. Default: `true`.
### Archive Support
`archiveEnabled` enables (`true`) or disables (`false`) the archive traces button. Default: `false`. It requires a configuration of an archive storage in Query service. Archived traces are only accessible directly by ID, they are not searchable.
### Google Analytics Tracking
`tracking.gaID` defines the Google Analytics tracking ID. This is required for Google Analytics tracking, and setting it to a non-`null` value enables Google Analytics tracking. Default: `null`.
`tracking.trackErrors` enables (`true`) or disables (`false`) error tracking via Google Analytics. Errors can only be tracked if a valid Google Analytics ID is provided. For additional details on error tracking via Google Analytics see the [tracking README](https://github.com/jaegertracing/jaeger-ui/blob/c622330546afc1be59a42f874bcc1c2fadf7e69a/src/utils/tracking/README.md) in the UI repo. Default: `true`.
### Custom Menu Items
`menu` allows additional links to be added to the global nav. The additional links are right-aligned.
In the sample JSON config above, the configured menu will have a dropdown labeled "About Jaeger" with sub-options for "GitHub" and "Docs". The format for a link in the top right menu is as follows:
```json
{
"label": "Some text here",
"url": "https://example.com"
}
```
Links can either be members of the `menu` Array, directly, or they can be grouped into a dropdown menu option. The format for a group of links is:
```json
{
"label": "Dropdown button",
"items": [ ]
}
```
The `items` Array should contain one or more link configurations.
### Link Patterns
The `linkPatterns` node can be used to create links from fields displayed in the Jaeger UI.
Field | Description
------|------------
type | The metadata section in which your link will be added: process, tags, logs
key | The name of tag/process/log attribute which value will be displayed as a link
url | The URL where the link should point to, it can be an external site or relative path in Jaeger UI
text | The text displayed in the tooltip for the link
Both `url` and `text` can be defined as templates (i.e. using `#{field-name}`) where Jaeger UI will dynamically substitute values based on tags/logs data.
## Embedded Mode
Starting with version 1.9, Jaeger UI provides an "embedded" layout mode which is intended to support integrating Jaeger UI into other applications. Currently (as of `v0`), the approach taken is to remove various UI elements from the page to make the UI better suited for space-constrained layouts.
The embedded mode is induced and configured via URL query parameters.
To enter embedded mode, the `uiEmbed=v0` query parameter and value must be added to the URL. For example, the following URL will show the trace with ID `abc123` in embedded mode:
```
http://localhost:16686/trace/abc123?uiEmbed=v0
```
`uiEmbed=v0` is required.
Further, each page supported has an <img src="/img/frontend-ui/embed-open-icon.png" style="width: 20px; height:20px;display:inline;" alt="Embed open window"> button added that will open the non-embedded page in a new tab.
The following pages support embedded mode:
* Search Page
* Trace Page
### Search Page
To integrate the Search Trace Page to our application we have to indicate to the Jaeger UI that we want to use the embed mode with `uiEmbed=v0`.
For example:
```
http://localhost:16686/search?
service=my-service&
start=1543917759557000&
end=1543921359557000&
limit=20&
lookback=1h&
maxDuration&
minDuration&
uiEmbed=v0
```
![Embed Search Traces](/img/frontend-ui/embed-search-traces.png)
#### Configuration options
The following query parameter can be used to configure the layout of the search page :
* `uiSearchHideGraph=1` - disables the display of the scatter plot above the search results
```
http://localhost:16686/search?
service=my-service&
start=1543917759557000&
end=1543921359557000&
limit=20&
lookback=1h&
maxDuration&
minDuration&
uiEmbed=v0&
uiSearchHideGraph=1
```
![Embed Search Traces without Graph](/img/frontend-ui/embed-search-traces-hide-graph.png)
### Trace Page
To integrate the Trace Page to our application we have to indicate to the Jaeger UI that we want to use the embed mode with `uiEmbed=v0`.
For example:
```sh
http://localhost:16686/trace/{trace-id}?uiEmbed=v0
```
![Embed Trace view](/img/frontend-ui/embed-trace-view.png)
If we have navigated to this view from the search traces page we'll have a button to go back to the results page.
![Embed Trace view](/img/frontend-ui/embed-trace-view-with-back-button.png)
#### Configuration options
The following query parameters can be used to configure the layout of the trace page :
* `uiTimelineCollapseTitle=1` causes the trace header to start out collapsed, which hides the summary and the minimap.
```
http://localhost:16686/trace/{trace-id}?
uiEmbed=v0&
uiTimelineCollapseTitle=1
```
![Embed Trace view](/img/frontend-ui/embed-trace-view-with-collapse.png)
* `uiTimelineHideMinimap=1` removes the minimap, entirely, regardless of whether the trace header is expanded or not.
```
http://localhost:16686/trace/{trace-id}?
uiEmbed=v0&
uiTimelineHideMinimap=1
```
![Embed Trace view](/img/frontend-ui/embed-trace-view-with-hide-minimap.png)
* `uiTimelineHideSummary=1` - removes the trace summary information (number of services, etc.) entirely, regardless of whether the trace header is expanded or not.
```
http://localhost:16686/trace/{trace-id}?
uiEmbed=v0&
uiTimelineHideSummary=1
```
![Embed Trace view](/img/frontend-ui/embed-trace-view-with-hide-summary.png)
We can also combine the options:
```
http://localhost:16686/trace/{trace-id}?
uiEmbed=v0&
uiTimelineHideMinimap=1&
uiTimelineHideSummary=1
```
![Embed Trace view](/img/frontend-ui/embed-trace-view-with-hide-details-and-hide-minimap.png)

View File

@ -0,0 +1,129 @@
---
title: Getting Started
description: Get up and running with Jaeger in your local environment
weight: 2
---
## Instrumentation
Your applications must be instrumented before they can send tracing data to Jaeger backend. Check the [Client Libraries](../client-libraries) section for information about how to use the OpenTracing API and how to initialize and configure Jaeger tracers.
## All in One
All-in-one is an executable designed for quick local testing, launches the Jaeger UI, collector, query, and agent, with an in memory storage component.
The simplest way to start the all-in-one is to use the pre-built image published to DockerHub (a single command line).
```bash
$ docker run -d --name jaeger \
-e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
-p 5775:5775/udp \
-p 6831:6831/udp \
-p 6832:6832/udp \
-p 5778:5778 \
-p 16686:16686 \
-p 14268:14268 \
-p 9411:9411 \
jaegertracing/all-in-one:{{< currentVersion >}}
```
Or run the `jaeger-all-in-one(.exe)` executable from the [binary distribution archives][download]:
```bash
$ jaeger-all-in-one --collector.zipkin.http-port=9411
```
You can then navigate to `http://localhost:16686` to access the Jaeger UI.
The container exposes the following ports:
Port | Protocol | Component | Function
----- | ------- | --------- | ---
5775 | UDP | agent | accept `zipkin.thrift` over compact thrift protocol (deprecated, used by legacy clients only)
6831 | UDP | agent | accept `jaeger.thrift` over compact thrift protocol
6832 | UDP | agent | accept `jaeger.thrift` over binary thrift protocol
5778 | HTTP | agent | serve configs
16686 | HTTP | query | serve frontend
14268 | HTTP | collector | accept `jaeger.thrift` directly from clients
14250 | HTTP | collector | accept `model.proto`
9411 | HTTP | collector | Zipkin compatible endpoint (optional)
## Kubernetes and OpenShift
* Kubernetes templates: https://github.com/jaegertracing/jaeger-kubernetes
* Kubernetes Operator: https://github.com/jaegertracing/jaeger-operator
* OpenShift templates: https://github.com/jaegertracing/jaeger-openshift
## Sample App: HotROD
HotROD (Rides on Demand) is a demo application that consists of several microservices and
illustrates the use of the [OpenTracing API](http://opentracing.io).
A tutorial / walkthrough is available in the blog post:
[Take OpenTracing for a HotROD ride][hotrod-tutorial].
It can be run standalone, but requires Jaeger backend to view the traces.
### Features
- Discover architecture of the whole system via data-driven dependency
diagram.
- View request timeline and errors; understand how the app works.
- Find sources of latency and lack of concurrency.
- Highly contextualized logging.
- Use baggage propagation to:
- Diagnose inter-request contention (queueing).
- Attribute time spent in a service.
- Use open source libraries with OpenTracing integration to get
vendor-neutral instrumentation for free.
### Prerequisites
- You need Go 1.11 or higher installed on your machine to run from source.
- Requires a [running Jaeger backend](#all-in-one) to view the traces.
### Running
#### From Source
```bash
mkdir -p $GOPATH/src/github.com/jaegertracing
cd $GOPATH/src/github.com/jaegertracing
git clone git@github.com:jaegertracing/jaeger.git jaeger
cd jaeger
make install
go run ./examples/hotrod/main.go all
```
#### From docker
```bash
$ docker run --rm -it \
--link jaeger \
-p8080-8083:8080-8083 \
-e JAEGER_AGENT_HOST="jaeger" \
jaegertracing/example-hotrod:{{< currentVersion >}} \
all
```
#### From binary distribution
Run `example-hotrod(.exe)` executable from the [binary distribution archives][download]:
```bash
$ example-hotrod all
```
Then navigate to `http://localhost:8080`.
## Migrating from Zipkin
Collector service exposes Zipkin compatible REST API `/api/v1/spans` which accepts both Thrift and JSON. Also there is `/api/v2/spans` for JSON only.
By default it's disabled. It can be enabled with `--collector.zipkin.http-port=9411`.
Zipkin Thrift IDL file can be found in [jaegertracing/jaeger-idl](https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/zipkincore.thrift).
It's compatible with [openzipkin/zipkin-api](https://github.com/openzipkin/zipkin-api/blob/master/thrift/zipkinCore.thrift)
[hotrod-tutorial]: https://medium.com/@YuriShkuro/take-opentracing-for-a-hotrod-ride-f6e3141f7941
[download]: ../../../download/

View File

@ -0,0 +1,755 @@
---
title: Operator for Kubernetes
hasparent: true
---
# Understanding Operators
The Jaeger Operator is an implementation of a [Kubernetes Operator](https://coreos.com/operators/). Operators are pieces of software that ease the operational complexity of running another piece of software. More technically, _Operators_ are a method of packaging, deploying, and managing a Kubernetes application.
A Kubernetes application is an application that is both deployed on Kubernetes and managed using the Kubernetes APIs and `kubectl` (kubernetes) or `oc` (OKD) tooling. To be able to make the most of Kubernetes, you need a set of cohesive APIs to extend in order to service and manage your apps that run on Kubernetes. Think of Operators as the runtime that manages this type of app on Kubernetes.
# Installing the Operator
{{< info >}}
The Jaeger Operator version tracks one version of the Jaeger components (Query, Collector, Agent). When a new version of the Jaeger components is released, a new version of the operator will be released that understands how running instances of the previous version can be upgraded to the new version.
{{< /info >}}
## Installing the Operator on Kubernetes
The following instructions will create the `observability` namespace and install the Jaeger Operator.
{{< info >}}
Make sure your `kubectl` command is properly configured to talk to a valid Kubernetes cluster. If you don't have a cluster, you can create one locally using [`minikube`](https://kubernetes.io/docs/tasks/tools/install-minikube/).
{{< /info >}}
To install the operator, run:
<!--TODO - Does Kubernetes have privileged users? Needs to be run as a system:admin on OKD/OpenShift.-->
```bash
kubectl create namespace observability # <1>
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/crds/jaegertracing.io_jaegers_crd.yaml # <2>
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/service_account.yaml
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role.yaml
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role_binding.yaml
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/operator.yaml
```
<1> This creates the namespace used by default in the deployment files. If you want to install the Jaeger operator in a different namespace, you must edit the deployment files to change `observability` to the desired namespace value.
<2> This installs the "Custom Resource Definition" for the `apiVersion: jaegertracing.io/v1`
At this point, there should be a `jaeger-operator` deployment available. You can view it by running the following command:
```bash
$ kubectl get deployment jaeger-operator -n observability
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
jaeger-operator 1 1 1 1 48s
```
The operator is now ready to create Jaeger instances.
## Installing the Operator on OKD/OpenShift
<!-- TODO: Add instructions for installing via the operatorhub? -->
The instructions from the previous section also work for installing the operator on OKD or OpenShift. Make sure you are logged in as a privileged user, when you install the role based acces control (RBAC) rules, the custom resource definition, and the operator.
```bash
oc login -u <privileged user>
oc new-project observability # <1>
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/crds/jaegertracing.io_jaegers_crd.yaml # <2>
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/service_account.yaml
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role.yaml
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role_binding.yaml
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/operator.yaml
```
<1> This creates the namespace used by default in the deployment files. If you want to install the Jaeger operator in a different namespace, you must edit the deployment files to change `observability` to the desired namespace value.
<2> This installs the "Custom Resource Definition" for the `apiVersion: jaegertracing.io/v1`
Once the operator is installed, grant the role `jaeger-operator` to users who should be able to install individual Jaeger instances. The following example creates a role binding allowing the user `developer` to create Jaeger instances:
```bash
oc create \
rolebinding developer-jaeger-operator \
--role=jaeger-operator \
--user=developer
```
After the role is granted, switch back to a non-privileged user.
Jaeger Agent can be configured to be deployed as a `DaemonSet` using a `HostPort` to allow Jaeger clients in the same node to discover the agent. In OpenShift, a `HostPort` can only be set when a special security context is set. A separate service account can be used by the Jaeger Agent with the permission to bind to `HostPort`, as follows:
```bash
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/examples/openshift/hostport-scc-daemonset.yaml # <1>
oc new-project myappnamespace
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/examples/openshift/service_account_jaeger-agent-daemonset.yaml # <2>
oc adm policy add-scc-to-user daemonset-with-hostport -z jaeger-agent-daemonset # <3>
oc apply -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/examples/openshift/agent-as-daemonset.yaml # <4>
```
<1> The `SecurityContextConstraints` with the `allowHostPorts` policy
<2> The `ServiceAccount` to be used by the Jaeger Agent
<3> Adds the security policy to the service account
<4> Creates the Jaeger Instance using the `serviceAccount` created in the steps above
{{< warning >}}
Without such a policy, errors like the following will prevent a `DaemonSet` to be created: `Warning FailedCreate 4s (x14 over 45s) daemonset-controller Error creating: pods "agent-as-daemonset-agent-daemonset-" is forbidden: unable to validate against any security context constraint: [spec.containers[0].securityContext.containers[0].hostPort: Invalid value: 5775: Host ports are not allowed to be used`
{{< /warning >}}
After a few seconds, the `DaemonSet` should be up and running:
```bash
$ oc get daemonset agent-as-daemonset-agent-daemonset
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE
agent-as-daemonset-agent-daemonset 1 1 1 1 1
```
# Quick Start - Deploying the AllInOne image
The simplest possible way to create a Jaeger instance is by creating a YAML file like the following example. This will install the default AllInOne strategy, which deploys the "all-in-one" image (agent, collector, query, ingester, Jaeger UI) in a single pod, using in-memory storage by default.
{{< info >}}
This default strategy is intended for development, testing, and demo purposes, not for production.
{{< /info >}}
```yaml
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: simplest
```
The YAML file can then be used with `kubectl`:
<!-- TODO - Add OKD commands and tabs shortcode. -->
```bash
kubectl apply -f simplest.yaml
```
In a few seconds, a new in-memory all-in-one instance of Jaeger will be available, suitable for quick demos and development purposes. To check the instances that were created, list the `jaeger` objects:
```bash
$ kubectl get jaegers
NAME CREATED AT
simplest 28s
```
To get the pod name, query for the pods belonging to the `simplest` Jaeger instance:
```bash
$ kubectl get pods -l app.kubernetes.io/instance=simplest
NAME READY STATUS RESTARTS AGE
simplest-6499bb6cdd-kqx75 1/1 Running 0 2m
```
Similarly, the logs can be queried either from the pod directly using the pod name obtained from the previous example, or from all pods belonging to our instance:
```bash
$ kubectl logs -l app.kubernetes.io/instance=simplest
...
{"level":"info","ts":1535385688.0951214,"caller":"healthcheck/handler.go:133","msg":"Health Check state change","status":"ready"}
```
{{< info >}}
On OKD/OpenShift the container name must be specified.
{{< /info >}}
```bash
$ kubectl logs -l app.kubernetes.io/instance=simplest -c jaeger
...
{"level":"info","ts":1535385688.0951214,"caller":"healthcheck/handler.go:133","msg":"Health Check state change","status":"ready"}
```
# Deployment Strategies
When you create a Jaeger instance, it is associated with a strategy. The strategy is defined in the custom resource file, and determines the architecture to be used for the Jaeger backend. The default strategy is `allInOne`. The other possible values are `production` and `streaming`.
The available strategies are described in the following sections.
## AllInOne (Default) strategy
This strategy is intended for development, testing, and demo purposes.
The main backend components, agent, collector and query service, are all packaged into a single executable which is configured (by default) to use in-memory storage.
## Production strategy
The `production` strategy is intended (as the name suggests) for production environments, where long term storage of trace data is important, as well as a more scalable and highly available architecture is required. Each of the backend components is therefore separately deployed.
The agent can be injected as a sidecar on the instrumented application or as a daemonset.
The query and collector services are configured with a supported storage type - currently Cassandra or Elasticsearch. Multiple instances of each of these components can be provisioned as required for performance and resilience purposes.
The main additional requirement is to provide the details of the storage type and options, for example:
```yaml
storage:
type: elasticsearch
options:
es:
server-urls: http://elasticsearch:9200
```
## Streaming strategy
The `streaming` strategy is designed to augment the `production` strategy by providing a streaming capability that effectively sits between the collector and the backend storage (Cassandra or Elasticsearch). This provides the benefit of reducing the pressure on the backend storage, under high load situations, and enables other trace post-processing capabilities to tap into the real time span data directly from the streaming platform (Kafka).
The only additional information required is to provide the details for accessing the Kafka platform, which is configured in the `collector` component (as producer) and `ingester` component (as consumer):
```yaml
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: simple-streaming
spec:
strategy: streaming
collector:
options:
kafka: # <1>
producer:
topic: jaeger-spans
brokers: my-cluster-kafka-brokers.kafka:9092
ingester:
options:
kafka: # <1>
consumer:
topic: jaeger-spans
brokers: my-cluster-kafka-brokers.kafka:9092
ingester:
deadlockInterval: 0 # <2>
storage:
type: elasticsearch
options:
es:
server-urls: http://elasticsearch:9200
```
<1> Identifies the Kafka configuration used by the collector, to produce the messages, and the ingester to consume the messages.
<2> The deadlock interval can be disabled to avoid the ingester being terminated when no messages arrive within the default 1 minute period
{{< info >}}
A Kafka environment can be configured using [Strimzi's Kafka operator](https://strimzi.io/).
{{< /info >}}
# Understanding Custom Resource Definitions
In the Kubernetes API, a resource is an endpoint that stores a collection of API objects of a certain kind. For example, the built-in Pods resource contains a collection of Pod objects. A _Custom Resource Definition_ (CRD) object defines a new, unique object `Kind` in the cluster and lets the Kubernetes API server handle its entire lifecycle.
To create _Custom Resource_ (CR) objects, cluster administrators must first create a Custom Resource Definition (CRD). The CRDs allow cluster users to create CRs to add the new resource types into their projects. An Operator watches for custom resource objects to be created, and when it sees a custom resource being created, it creates the application based on the parameters defined in the custom resource object.
{{< info >}}
While only cluster administrators can create CRDs, developers can create the CR from an existing CRD if they have read and write permission to it.
{{< /info >}}
<!--
## Jaeger Custom Resource Parameters
TODO Create a TABLE with all the parameters, descriptions/notes, valid values, and defaults.
Figure out if we can generate the options? Can we filter them in any way?
https://github.com/jaegertracing/jaeger/issues/1537
https://github.com/jaegertracing/documentation/issues/250-->
For reference, here's how you can create a more complex all-in-one instance:
```yaml
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: my-jaeger
spec:
strategy: allInOne # <1>
allInOne:
image: jaegertracing/all-in-one:latest # <2>
options: # <3>
log-level: debug # <4>
storage:
type: memory # <5>
options: # <6>
memory: # <7>
max-traces: 100000
ingress:
enabled: false # <8>
agent:
strategy: DaemonSet # <9>
annotations:
scheduler.alpha.kubernetes.io/critical-pod: "" # <10>
```
<1> The default strategy is `allInOne`. The other possible values are `production` and `streaming`.
<2> The image to use, in a regular Docker syntax.
<3> The (non-storage related) options to be passed verbatim to the underlying binary. Refer to the Jaeger documentation and/or to the `--help` option from the related binary for all the available options.
<4> The option is a simple `key: value` map. In this case, we want the option `--log-level=debug` to be passed to the binary.
<5> The storage type to be used. By default it will be `memory`, but can be any other supported storage type (Cassandra, Elasticsearch, Kafka).
<6> All storage related options should be placed here, rather than under the 'allInOne' or other component options.
<7> Some options are namespaced and we can alternatively break them into nested objects. We could have specified `memory.max-traces: 100000`.
<8> By default, an ingress object is created for the query service. It can be disabled by setting its `enabled` option to `false`. If deploying on OpenShift, this will be represented by a Route object.
<9> By default, the operator assumes that agents are deployed as sidecars within the target pods. Specifying the strategy as "DaemonSet" changes that and makes the operator deploy the agent as DaemonSet. Note that your tracer client will probably have to override the "JAEGER_AGENT_HOST" environment variable to use the node's IP.
<10> Define annotations to be applied to all deployments (not services). These can be overridden by annotations defined on the individual components.
You can view example custom resources for different Jaeger configurations [on GitHub](https://github.com/jaegertracing/jaeger-operator/tree/master/examples).
# Configuring the Custom Resource
<!--TODO
esIndexCleaner
Spark dependencies
-->
You can use the simplest example (shown above) and create a Jaeger instance using the defaults, or you can create your own custom resource file.
## Storage options
### Cassandra storage
When the storage type is set to Cassandra, the operator will automatically create a batch job that creates the required schema for Jaeger to run. This batch job will block the Jaeger installation, so that it starts only after the schema is successfuly created. The creation of this batch job can be disabled by setting the `enabled` property to `false`:
```yaml
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: cassandra-without-create-schema
spec:
strategy: allInOne
storage:
type: cassandra
cassandraCreateSchema:
enabled: false # <1>
```
<1> Defaults to `true`
Further aspects of the batch job can be configured as well. An example with all the possible options is shown below:
```yaml
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: cassandra-with-create-schema
spec:
strategy: allInOne # <1>
storage:
type: cassandra
options: # <2>
cassandra:
servers: cassandra
keyspace: jaeger_v1_datacenter3
cassandraCreateSchema: # <3>
datacenter: "datacenter3"
mode: "test"
```
<1> The same works for `production` and `streaming`.
<2> These options are for the regular Jaeger components, like `collector` and `query`.
<3> The options for the `create-schema` job.
{{< info >}}
The default create-schema job uses `MODE=prod`, which implies a replication factor of `2`, using `NetworkTopologyStrategy` as the class, effectively meaning that at least 3 nodes are required in the Cassandra cluster. If a `SimpleStrategy` is desired, set the mode to `test`, which then sets the replication factor of `1`. Refer to the [create-schema script](https://github.com/jaegertracing/jaeger/blob/master/plugin/storage/cassandra/schema/create.sh) for more details.
{{< /info >}}
### Elasticsearch storage
Under some circumstances, the Jaeger Operator can make use of the [Elasticsearch Operator](https://github.com/openshift/elasticsearch-operator) to provision a suitable Elasticsearch cluster.
{{< warning >}}
This feature is only tested on OpenShift clusters. Spark dependencies are not supported with this feature [Issue #294](https://github.com/jaegertracing/jaeger-operator/issues/294).
{{< /warning >}}
When there are no `es.server-urls` options as part of a Jaeger `production` instance and `elasticsearch` is set as the storage type, the Jaeger Operator creates an Elasticsearch cluster via the Elasticsearch Operator by creating a Custom Resource based on the configuration provided in storage section. The Elasticsearch cluster is meant to be dedicated for a single Jaeger instance.
The self-provision of an Elasticsearch cluster can be disabled by setting the flag `--es-provision` to `false`. The default value is `auto`, which will make the Jaeger Operator query the Kubernetes cluster for its ability to handle a `Elasticsearch` custom resource. This is usually set by the Elasticsearch Operator during its installation process, so, if the Elasticsearch Operator is expected to run *after* the Jaeger Operator, the flag can be set to `true`.
{{< danger >}}
At the moment there can be only one Jaeger with self-provisioned Elasticsearch instance per namespace.
{{< /danger >}}
#### Elasticsearch index cleaner job
When using `elasticsearch` storage by default a job is created to clean old traces from it, the options for it are listed below so you can configure it to your use case
```yaml
storage:
type: elasticsearch
esIndexCleaner:
enabled: false # turn the job deployment on and off
numberOfDays: 7 # number of days to wait before deleting a record
schedule: "55 23 * * *" # cron expression for it to run
image: jaegertracing/jaeger-es-index-cleaner # image of the job
```
## Auto-injecting Jaeger Agent Sidecars
The operator can inject Jaeger Agent sidecars in `Deployment` workloads, provided that the deployment has the annotation `sidecar.jaegertracing.io/inject` with a suitable value. The values can be either `"true"` (as string), or the Jaeger instance name, as returned by `kubectl get jaegers`. When `"true"` is used, there should be exactly *one* Jaeger instance for the same namespace as the deployment, otherwise, the operator can't figure out automatically which Jaeger instance to use.
The following snippet shows a simple application that will get a sidecar injected, with the Jaeger Agent pointing to the single Jaeger instance available in the same namespace:
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
annotations:
"sidecar.jaegertracing.io/inject": "true" # <1>
spec:
selector:
matchLabels:
app: myapp
template:
metadata:
labels:
app: myapp
spec:
containers:
- name: myapp
image: acme/myapp:myversion
```
<1> Either `"true"` (as string) or the Jaeger instance name.
A complete sample deployment is available at [`deploy/examples/business-application-injected-sidecar.yaml`](https://github.com/jaegertracing/jaeger-operator/blob/master/examples/business-application-injected-sidecar.yaml).
When the sidecar is injected, the Jaeger Agent can then be accessed at its default location on `localhost`.
## Installing the Agent as DaemonSet
By default, the Operator expects the agents to be deployed as sidecars to the target applications. This is convenient for several purposes, like in a multi-tenant scenario or to have better load balancing, but there are scenarios where you might want to install the agent as a `DaemonSet`. In that case, specify the Agent's strategy to `DaemonSet`, as follows:
```yaml
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: my-jaeger
spec:
agent:
strategy: DaemonSet
```
{{< danger >}}
If you attempt to install two Jaeger instances on the same cluster with `DaemonSet` as the strategy, only *one* will end up deploying a `DaemonSet`, as the agent is required to bind to well-known ports on the node. Because of that, the second daemon set will fail to bind to those ports.
{{< /danger >}}
Your tracer client will then most likely need to be told where the agent is located. This is usually done by setting the environment variable `JAEGER_AGENT_HOST` to the value of the Kubernetes node's IP, for example:
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
spec:
selector:
matchLabels:
app: myapp
template:
metadata:
labels:
app: myapp
spec:
containers:
- name: myapp
image: acme/myapp:myversion
env:
- name: JAEGER_AGENT_HOST
valueFrom:
fieldRef:
fieldPath: status.hostIP
```
## Secrets Support
The Operator supports passing secrets to the Collector, Query and All-In-One deployments. This can be used for example, to pass credentials (username/password) to access the underlying storage backend (for example: Elasticsearch).
The secrets are available as environment variables in the (Collector/Query/All-In-One) nodes.
```yaml
storage:
type: elasticsearch
options:
es:
server-urls: http://elasticsearch:9200
secretName: jaeger-secrets
```
The secret itself would be managed outside of the `jaeger-operator` custom resource.
## Configuring the UI
Information on various configuration options for the UI can be found [here](../frontend-ui/#configuration), defined in json format.
To apply UI configuration changes within the Custom Resource, the same information can be included in yaml format as shown below:
```yaml
ui:
options:
dependencies:
menuEnabled: false
tracking:
gaID: UA-000000-2
menu:
- label: "About Jaeger"
items:
- label: "Documentation"
url: "https://www.jaegertracing.io/docs/latest"
linkPatterns:
- type: "logs"
key: "customer_id"
url: /search?limit=20&lookback=1h&service=frontend&tags=%7B%22customer_id%22%3A%22#{customer_id}%22%7D
text: "Search for other traces for customer_id=#{customer_id}"
```
## Defining Sampling Strategies
{{< info >}}
This is not relevant if a trace was started by the Istio proxy as the sampling decision is made there. And the Jaeger sampling decisions are only relevant when you are using the Jaeger tracer (client).
{{< /info >}}
The operator can be used to define sampling strategies that will be supplied to tracers that have been configured to use a remote sampler:
```yaml
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: with-sampling
spec:
strategy: allInOne
sampling:
options:
default_strategy:
type: probabilistic
param: 0.5
```
This example defines a default sampling strategy that is probabilistic, with a 50% chance of the trace instances being sampled.
Refer to the Jaeger documentation on [Collector Sampling Configuration](https://www.jaegertracing.io/docs/latest/sampling/#collector-sampling-configuration) to see how service and endpoint sampling can be configured. The JSON representation described in that documentation can be used in the operator by converting to YAML.
## Finer grained configuration
The custom resource can be used to define finer grained Kubernetes configuration applied to all Jaeger components or at the individual component level.
When a common definition (for all Jaeger components) is required, it is defined under the `spec` node. When the definition relates to an individual component, it is placed under the `spec/<component>` node.
The types of supported configuration include:
* [affinity](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity) to determine which nodes a pod can be allocated to
* [annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/)
* [labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/)
* [resources](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container) to limit cpu and memory
* [tolerations](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/) in conjunction with `taints` to enable pods to avoid being repelled from a node
* [volumes](https://kubernetes.io/docs/concepts/storage/volumes/) and volume mounts
* [serviceAccount](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/) to run each component with separate identity
* [securityContext](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/) to define privileges of running components
```yaml
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: simple-prod
spec:
strategy: production
storage:
type: elasticsearch
options:
es:
server-urls: http://elasticsearch:9200
annotations:
key1: value1
labels:
key2: value2
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/e2e-az-name
operator: In
values:
- e2e-az1
- e2e-az2
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 1
preference:
matchExpressions:
- key: another-node-label-key
operator: In
values:
- another-node-label-value
tolerations:
- key: "key1"
operator: "Equal"
value: "value1"
effect: "NoSchedule"
- key: "key1"
operator: "Equal"
value: "value1"
effect: "NoExecute"
serviceAccount: nameOfServiceAccount
securityContext:
runAsUser: 1000
volumeMounts:
- name: config-vol
mountPath: /etc/config
volumes:
- name: config-vol
configMap:
name: log-config
items:
- key: log_level
path: log_level
```
# Accessing the Jaeger Console (UI)
<!-- TODO Add tabs shortcode -->
## Kubernetes
The operator creates a Kubernetes [`ingress`](https://kubernetes.io/docs/concepts/services-networking/ingress/) route, which is the Kubernetes' standard for exposing a service to the outside world, but by default it does not come with Ingress providers.
Check the [Kubernetes documentation](https://kubernetes.github.io/ingress-nginx/deploy/#verify-installation) for the most appropriate way to achieve an Ingress provider for your platform. The following command enables the Ingress provider on `minikube`:
```bash
minikube addons enable ingress
```
Once Ingress is enabled, the address for the Jaeger console can be found by querying the Ingress object:
```bash
$ kubectl get ingress
NAME HOSTS ADDRESS PORTS AGE
simplest-query * 192.168.122.34 80 3m
```
In this example, the Jaeger UI is available at http://192.168.122.34.
## OpenShift
When the Operator is running on OpenShift, the Operator will automatically create a `Route` object for the query services. Use the following command to check the hostname/port:
```bash
oc get routes
```
{{< info >}}
Make sure to use `https` with the hostname/port you get from the command above, otherwise you'll see a message like: "Application is not available".
{{< /info >}}
By default, the Jaeger UI is protected with OpenShift's OAuth service and any valid user is able to login. To disable this feature and leave the Jaeger UI unsecured, set the Ingress property `security` to `none` in the custom resource file:
```yaml
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: disable-oauth-proxy
spec:
ingress:
security: none
```
Custom `SAR` and `Delegate URL` values can be specified as part of the `.Spec.Ingress.OpenShift.SAR` and `.Spec.Ingress.Openshift.DelegateURLs`, as follows:
```yaml
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: custom-sar-oauth-proxy
spec:
ingress:
openshift:
sar: '{"namespace": "default", "resource": "pods", "verb": "get"}'
delegate-urls: '{"/":{"namespace": "default", "resource": "pods", "verb": "get"}}'
```
When the `delegate-urls` is set, the Jaeger Operator needs to create a new `ClusterRoleBinding` between the service account used by the UI Proxy (`{InstanceName}-ui-proxy`) and the role `system:auth-delegator`, as required by the OpenShift OAuth Proxy. Because of that, the service account used by the operator itself needs to have the same cluster role binding. To accomplish that, a `ClusterRoleBinding` such as the following has to be created:
```yaml
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: jaeger-operator-with-auth-delegator
namespace: observability
subjects:
- kind: ServiceAccount
name: jaeger-operator
namespace: observability
roleRef:
kind: ClusterRole
name: system:auth-delegator
apiGroup: rbac.authorization.k8s.io
```
Cluster administrators not comfortable in letting users deploy Jaeger instances with this cluster role are free to not add this cluster role to the operator's service account. In that case, the Operator will auto-detect that the required permissions are missing and will log a message similar to: `the requested instance specifies the delegate-urls option for the OAuth Proxy, but this operator cannot assign the proper cluster role to it (system:auth-delegator). Create a cluster role binding between the operator's service account and the cluster role 'system:auth-delegator' in order to allow instances to use 'delegate-urls'`.
# Updating a Jaeger instance (experimental)
A Jaeger instance can be updated by changing the `CustomResource`, either via `kubectl edit jaeger simplest`, where `simplest` is the Jaeger's instance name, or by applying the updated YAML file via `kubectl apply -f simplest.yaml`.
{{< danger >}}
The name of the Jaeger instance cannot be updated, as it is part of the identifying information for the resource.
{{< /danger >}}
Simpler changes such as changing the replica sizes can be applied without much concern, whereas changes to the strategy should be watched closely and might potentially cause an outage for individual components (collector/query/agent).
While changing the backing storage is supported, migration of the data is not.
# Removing a Jaeger instance
<!-- TODO Add OKD/OpenShift commands and tabs shortcode-->
To remove an instance, use the `delete` command with the custom resource file used when you created the instance:
```bash
kubectl delete -f simplest.yaml
```
Alternatively, you can remove a Jaeger instance by running:
```bash
kubectl delete jaeger simplest
```
{{< info >}}
Deleting the instance will not remove the data from any permanent storage used with this instance. Data from in-memory instances, however, will be lost.
{{< /info >}}
# Monitoring the operator
The Jaeger Operator starts a Prometheus-compatible endpoint on `0.0.0.0:8383/metrics` with internal metrics that can be used to monitor the process.
{{< info >}}
The Jaeger Operator does not yet publish its own metrics. Rather, it makes available metrics reported by the components it uses, such as the Operator SDK.
{{< /info >}}
# Uninstalling the operator
<!-- TODO Add OKD/OpenShift commands and tabs shortcode -->
To uninstall the operator, run the following commands:
```bash
kubectl delete -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/operator.yaml
kubectl delete -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role_binding.yaml
kubectl delete -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role.yaml
kubectl delete -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/service_account.yaml
kubectl delete -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/crds/jaegertracing.io_jaegers_crd.yaml
```

View File

@ -0,0 +1,95 @@
---
title: Performance Tuning Guide
description: Tweaking your Jaeger instance to achieve a better performance
weight: 10
---
Jaeger was built from day 1 to be able to ingest huge amounts of data in a resilient way. To better utilize resources that might cause delays, such as storage or network communications, Jaeger buffers and batches data. When more spans are generated than Jaeger is able to safely process, spans might get dropped. However, the defaults might not fit all scenarios: for instance, Agents running as a sidecar might have more memory constraints than Agents running as a daemon in bare metal.
## Deployment considerations
Although performance tuning the individual components is important, the way Jaeger is deployed can be decisive in obtaining optimal performance.
### Scale the Collector up and down
Use the auto-scaling capabilities of your platform: the Collector is nearly horizontally scalable so that more instances can be added and removed on-demand. A good way to scale up and down is by checking the `jaeger_collector_queue_length` metric: add instances when the length is higher than 50% of the maximum size for extended periods of time. Another metric that can be taken into consideration is `jaeger_collector_in_queue_latency_bucket`, which is a histogram indicating how long spans have been waiting in the queue before a worker picked it up. When the queue latency gets higher over time, its a good indication to increase the number of the workers, or to improve the storage performance.
Adding Collector instances is recommended when your platform provides auto-scaling capabilities, or when it's easier to start/stop Collector instances than changing existing, running instances. Scaling horizontally is also indicated when the CPU usage should be spread across nodes.
### Make sure the storage can keep up
Each span is written to the storage by the Collector using one worker, blocking it until the span has been stored. When the storage is too slow, the number of workers blocked by the storage might be too high, causing spans to be dropped. To help diagnose this situation, the histogram `jaeger_collector_save_latency_bucket` can be analyzed. Ideally, the latency should remain the same over time. When the histogram shows that most spans are taking longer and longer over time, its a good indication that your storage might need some attention.
### Place the Agents close to your applications
The Agent is meant to be placed on the same host as the instrumented application, in order to avoid UDP packet loss over the network. This is typically accomplished by having one Agent per bare metal host for traditional applications, or as a sidecar in container environments like Kubernetes, as this helps spread the load handled by Agents with the additional advantage of allowing each Agent to be tweaked individually, according to the applications needs and importance.
### Consider using Apache Kafka as intermediate buffer
Jaeger [can use Apache Kafka](../architecture/) as a buffer between the Collector and the actual backing storage (Elasticsearch, Apache Cassandra). This is ideal for cases where the traffic spikes are relatively frequent (prime time traffic) but the storage can eventually catch up once the traffic normalizes. For that, the `SPAN_STORAGE_TYPE` environment variable should be set to `kafka` in the Collector and the Jaeger Ingester component can be used, reading data from Kafka and writing it to the storage.
In addition to the performance aspects, having spans written to Kafka is useful for building real time data pipeline for aggregations and feature extraction from traces.
## Client (Tracer) settings
The Jaeger Clients are built to have minimal effect to the instrumented application. As such, it has conservative defaults that might not be suitable for all cases. Note that Jaeger Clients can be configured programmatically or via [environment variables](../client-features/).
### Adjust the sampling configuration
Together, the `JAEGER_SAMPLER_TYPE` and `JAEGER_SAMPLER_PARAM` specify how often traces should be "sampled", ie, recorded and sent to the Jaeger backend. For applications generating a large number of spans, setting the sampling type to `probabilistic` and the value to `0.001` (the default) will cause traces to be reported with a 1/1000th chance. Note that the sampling decision is made at the root span and propagated down to all child spans.
For applications with low to medium traffic, setting the sampling type to `const` and value to `1` will cause all spans to be reported. Similarly, tracing can be disabled by setting the value to `0`, while context propagation will continue to work.
Some Clients support the setting `JAEGER_DISABLED` to completely disable the Jaeger Tracer. This is recommended only if the Tracer is behaving in a way that causes problems to the instrumented application, as it will not propagate the context to the downstream services.
{{< info >}}
We recommend to set your clients to use the [`remote` sampling strategy](../sampling/#collector-sampling-configuration), so that admins can centrally set the concrete sampling strategy for each service.
{{< /info >}}
### Increase in-memory queue size
Most of the Jaeger clients, such as the Java, Go, and C# clients, buffer spans in memory before sending them to the Jaeger Agent/Collector. The maximum size of this buffer is defined by the environment variable `JAEGER_REPORTER_MAX_QUEUE_SIZE` (default value: about `100` spans): the larger the size, the higher the potential memory consumption. When the instrumented application is generating a large number of spans, its possible that the queue will be full causing the Client to discard the new spans (metric `jaeger_tracer_reporter_spans_total{result="dropped",}`).
In most common scenarios, the queue will be close to empty (metric: `jaeger_tracer_reporter_queue_length`), as spans are flushed to the Agent or Collector at regular intervals or when a certain size of the batch is reached. The detailed behavior of this queue is described in this [GitHub issue](https://github.com/jaegertracing/jaeger-client-java/issues/607).
### Modify the batched spans flush interval
The Java, Go, NodeJS, Python and C# Clients allow the customization of the flush interval (default value: `1000` milliseconds, or 1 second) used by the reporters, such as the `RemoteReporter`, to trigger a `flush` operation, sending all in-memory spans to the Agent or Collector. The lower the flush interval is set to, the more frequent the flush operations happen. As most reporters will wait until enough data is in the queue, this setting will force a flush operation at periodic intervals, so that spans are sent to the backend in a timely fashion.
When the instrumented application is generating a large number of spans and the Agent/Collector is close to the application, the networking overhead might be low, justifying a higher number of flush operations. When the `HttpSender` is being used and the Collector is not close enough to the application, the networking overhead might be too high so that a higher value for this property makes sense.
## Agent settings
Jaeger Agents receive data from Clients, sending them in batches to the Collector. When not properly configured, it might end up discarding data even if the host machine has plenty of resources.
### Adjust server queue sizes
The set of “server queue size” properties ( `processor.jaeger-binary.server-queue-size`, `processor.jaeger-compact.server-queue-size`, `processor.zipkin-compact.server-queue-size`) indicate the maximum number of span batches that the Agent can accept and store in memory. Its safe to assume that `jaeger-compact` is the most important processor in your Agent setup, as its the only one available in most Clients, such as the Java and Go Clients.
The default value for each queue is `1000` span batches. Given that each span batch has up to 64KiB worth of spans, each queue can hold up to 64MiB worth of spans.
In typical scenarios, the queue will be close to empty (metric `jaeger_agent_thrift_udp_server_queue_size`) as span batches should be quickly picked up and processed by a worker. However, sudden spikes in the number of span batches submitted by Clients might occur, causing the batches to be queued. When the queue is full, the older batches are overridden causing spans to be discarded (metric `jaeger_agent_thrift_udp_server_packets_dropped_total`).
### Adjust processor workers
The set of “processor workers” properties ( `processor.jaeger-binary.workers`, `processor.jaeger-compact.workers`, `processor.zipkin-compact.workers`) indicate the number of parallel span batch processors to start. Each worker type has a default size of `10`. In general, span batches are processed as soon as they are placed in the server queue and will block a worker until the whole packet is sent to the Collector. For Agents processing data from multiple Clients, the number of workers should be increased. Given that the cost of each worker is low, a good rule of thumb is 10 workers per Client with moderate traffic: given that each span batch might contain up to 64KiB worth of spans, it means that 10 workers are able to send about 640KiB concurrently to a Collector.
## Collector settings
The Collector receives data from Clients and Agents. When not properly configured, it might process less data than what would be possible on the same host, or it might overload the host by consuming more memory than permitted.
### Adjust queue size
Similar to the Agent, the Collector is able to receive spans and place them in an internal queue for processing. This allows the Collector to return immediately to the Client/Agent instead of waiting for the span to make its way to the storage.
The setting `collector.queue-size` (default: `2000`) dictates how many spans the queue should support. In the typical scenario, the queue will be close to empty, as enough workers should exist picking up spans from the queue and sending them to the storage. When the number of items in the queue (metric `jaeger_collector_queue_length`) is permanently high, its an indication that either the number of workers should be increased or that the storage cannot keep up with the volume of data that its receiving. When the queue is full, the older items in the queue are overridden, causing spans to be discarded (metric `jaeger_collector_spans_dropped_total`).
{{< warning >}}
The queue size for the Agent is about _span batches_, whereas the queue size for the Collector is about _individual spans_.
{{< /warning >}}
Given that the queue size should be close to empty most of the time, this setting should be as high as the available memory for the Collector, to provide maximum protection against sudden traffic spikes. However, if your storage layer is under-provisioned and cannot keep up, even a large queue will quickly fill up and start dropping data.
### Adjust processor workers
Items from the span queue in the Collector are picked up by workers. Each worker picks one span from the queue and persists it to the storage. The number of workers can be specified by the setting `collector.num-workers` (default: `50`) and should be as high as needed to keep the queue close to zero. The general rule is: the faster the backing storage, the lower the number of workers can be. Given that workers are relatively cheap, this number can be increased at will. As a general rule, one worker per 50 items in the queue should be sufficient when the storage is fast. With a `collector.queue-size` of `2000`, having about `40` workers should be sufficient. For slower storage mechanisms, this ratio should be adjusted accordingly, having more workers per queue item.

View File

@ -0,0 +1,66 @@
---
title: Sampling
hasparent: true
---
Jaeger libraries implement consistent upfront (or head-based) sampling. For example, assume we have a simple call graph where service A calls service B, and B calls service C: `A -> B -> C`. When service A receives a request that contains no tracing information, Jaeger tracer will start a new {{< tip "trace" >}}, assign it a random trace ID, and make a sampling decision based on the currently installed sampling strategy. The sampling decision will be propagated with the requests to B and to C, so those services will not be making the sampling decision again but instead will respect the decision made by the top service A. This approach guarantees that if a trace is sampled, all its {{< tip "spans" "span" >}} will be recorded in the backend. If each service was making its own sampling decision we would rarely get complete traces in the backend.
## Client Sampling Configuration
When using configuration object to instantiate the tracer, the type of sampling can be selected via `sampler.type` and `sampler.param` properties. Jaeger libraries support the following samplers:
* **Constant** (`sampler.type=const`) sampler always makes the same decision for all traces. It either samples all traces (`sampler.param=1`) or none of them (`sampler.param=0`).
* **Probabilistic** (`sampler.type=probabilistic`) sampler makes a random sampling decision with the probability of sampling equal to the value of `sampler.param` property. For example, with `sampler.param=0.1` approximately 1 in 10 traces will be sampled.
* **Rate Limiting** (`sampler.type=ratelimiting`) sampler uses a leaky bucket rate limiter to ensure that traces are sampled with a certain constant rate. For example, when `sampler.param=2.0` it will sample requests with the rate of 2 traces per second.
* **Remote** (`sampler.type=remote`, which is also the default) sampler consults Jaeger agent for the appropriate sampling strategy to use in the current service. This allows controlling the sampling strategies in the services from a [central configuration](#collector-sampling-configuration) in Jaeger backend, or even dynamically (see [Adaptive Sampling](https://github.com/jaegertracing/jaeger/issues/365)).
### Adaptive Sampler
Adaptive sampler is a composite sampler that combines two functions:
* It makes sampling decisions on a per-operation basis, i.e. based on {{< tip "span" >}} operation name. This is especially useful in the API services whose endpoints may have very different traffic volumes and using a single probabilistic sampler for the whole service might starve (never sample) some of the low QPS endpoints.
* It supports a minimum guaranteed rate of sampling, such as always allowing up to N {{< tip "traces" "trace" >}} per seconds and then sampling anything above that with a certain probability (everything is per-operation, not per-service).
Per-operation parameters can be configured statically or pulled periodically from Jaeger backend with the help of Remote sampler. Adaptive sampler is designed to work with the upcoming [Adaptive Sampling](https://github.com/jaegertracing/jaeger/issues/365) feature of the Jaeger backend.
## Collector Sampling Configuration
Collectors can be instantiated with static sampling strategies (which are propagated to the respective service if configured with Remote sampler) via the `--sampling.strategies-file` option. This option requires a path to a json file which have the sampling strategies defined.
Example `strategies.json`
```json
{
"service_strategies": [
{
"service": "foo",
"type": "probabilistic",
"param": 0.8,
"operation_strategies": [
{
"operation": "op1",
"type": "probabilistic",
"param": 0.2
},
{
"operation": "op2",
"type": "probabilistic",
"param": 0.4
}
]
},
{
"service": "bar",
"type": "ratelimiting",
"param": 5
}
],
"default_strategy": {
"type": "probabilistic",
"param": 0.5
}
}
```
`service_strategies` defines service specific sampling strategies and `operation_strategies` defines operation specific sampling strategies. There are 2 types of strategies possible: `probabilistic` and `ratelimiting` which are described [above](#client-sampling-configuration) (NOTE: `ratelimiting` is not supported for `operation_strategies`). `default_strategy` defines the catch-all sampling strategy that is propagated if the service is not included as part of `service_strategies`.
In the above example, all service `foo` operations are sampled probabilistically with a probability of 0.8 except `op1` and `op2` which are probabilistically sampled with a probability of 0.2 and 0.4 respectively. All operations for service `bar` are ratelimited at 5 traces per second. Any other service is probabilistically sampled with a probability of 0.5.

View File

@ -0,0 +1,70 @@
---
title: Introduction
weight: 1
children:
- title: Features
url: features
---
Welcome to Jaeger's documentation portal! Below, you'll find information for beginners and experienced Jaeger users.
If you can't find what you are looking for, or have an issue not covered here, we'd love to [hear from you](/get-in-touch).
## About
Jaeger, inspired by [Dapper][dapper] and [OpenZipkin](http://zipkin.io),
is a distributed tracing system released as open source by [Uber Technologies][ubeross].
It is used for monitoring and troubleshooting microservices-based distributed systems, including:
* Distributed context propagation
* Distributed transaction monitoring
* Root cause analysis
* Service dependency analysis
* Performance / latency optimization
Uber published a blog post, [Evolving Distributed Tracing at Uber](https://eng.uber.com/distributed-tracing/), where they explain the history and reasons for the architectural choices made in Jaeger. [Yuri Shkuro](https://shkuro.com), creator of Jaeger, also published a book [Mastering Distributed Tracing](https://shkuro.com/books/2019-mastering-distributed-tracing/) that covers in-depth many aspects of Jaeger design and operation, as well as distributed tracing in general.
## Features
* [OpenTracing](http://opentracing.io/) compatible data model and instrumentation libraries
* in [Go](https://github.com/jaegertracing/jaeger-client-go), [Java](https://github.com/jaegertracing/jaeger-client-java), [Node](https://github.com/jaegertracing/jaeger-client-node), [Python](https://github.com/jaegertracing/jaeger-client-python)
and [C++](https://github.com/jaegertracing/cpp-client)
* Uses consistent upfront sampling with individual per service/endpoint probabilities
* Multiple storage backends: Cassandra, Elasticsearch, memory.
* Adaptive sampling (coming soon)
* Post-collection data processing pipeline (coming soon)
See [Features](./features/) page for more details.
## Technical Specs
* Backend components implemented in Go
* React/Javascript UI
* Supported storage backends:
* [Cassandra 3.4+](./deployment/#cassandra)
* [Elasticsearch 5.x, 6.x, 7.x](./deployment/#elasticsearch)
* [Kafka](./deployment/#kafka)
* memory storage
## Quick Start
See [running a docker all in one image](getting-started#all-in-one).
## Screenshots
### Traces View
[![Traces View](/img/traces-ss.png)](/img/traces-ss.png)
### Trace Detail View
[![Detail View](/img/trace-detail-ss.png)](/img/trace-detail-ss.png)
## Related links
- [Evolving Distributed tracing At Uber Engineering](https://eng.uber.com/distributed-tracing/)
- [Mastering Distributed Tracing](https://shkuro.com/books/2019-mastering-distributed-tracing/)
- [Tracing HTTP request latency in Go with OpenTracing](https://medium.com/opentracing/tracing-http-request-latency-in-go-with-opentracing-7cc1282a100a)
- [Distributed Tracing with Jaeger & Prometheus on Kubernetes](https://blog.openshift.com/openshift-commons-briefing-82-distributed-tracing-with-jaeger-prometheus-on-kubernetes/)
- [Using Jaeger with Istio](https://istio.io/docs/tasks/telemetry/distributed-tracing.html)
- [Using Jaeger with Envoy](https://www.envoyproxy.io/docs/envoy/latest/start/sandboxes/jaeger_tracing.html)
[dapper]: https://research.google.com/pubs/pub36356.html
[ubeross]: http://uber.github.io

87
content/docs/1.14/apis.md Normal file
View File

@ -0,0 +1,87 @@
---
title: APIs
hasparent: true
---
Jaeger components implement various APIs for saving or retrieving trace data.
The following labels are used to describe API compatibility guarantees.
* **stable** - the API guarantees backwards compatibility. If breaking changes are going to be made in the future, they will result in a new API version, e.g. `/api/v2` URL prefix or a different namespace in the IDL.
* **internal** - the APIs intended for internal communications between Jaeger components and not recommended for use by external components.
* **deprecated** - the APIs that are only maintained for legacy reasons and will be phased out in the future.
## Span reporting APIs
Agent and Collector are the two components of the Jaeger backend that can receive spans. At this time they support two sets of non-overlapping APIs.
### Thrift over UDP (stable)
The Agent can only receive spans over UDP in Thrift format. The primary API is a UDP packet that contains a Thrift-encoded `Batch` struct defined in [jaeger.thrift][jaeger.thrift] IDL file, located in the [jaeger-idl][jaeger-idl] repository. Most Jaeger Clients use Thrift's `compact` encoding, however some client libraries do not support it (notably, Node.js) and use Thrift's `binary` encoding (sent to a different UDP port). The Agent's API is defined by [agent.thrift][agent.thrift] IDL file.
For legacy reasons, the Agent also accepts spans in Zipkin format, however, only very old versions of Jaeger clients can send data in that format and it is officially deprecated.
### Protobuf via gRPC (stable)
In a typical Jaeger deployment, Agents receive spans from Clients and forward them to Collectors. Since Jaeger version 1.11 the official and recommended protocol between Agents and Collectors is gRPC with Protobuf.
The Protobuf IDL [collector.proto][collector.proto] is currently located in the main Jaeger repository, under [model/proto/api_v2][collector.proto]. In the future it will be moved to [jaeger-idl][jaeger-idl] repository ([jaeger-idl/issues/55](https://github.com/jaegertracing/jaeger-idl/issues/55)).
### Thrift over HTTP (stable)
In some cases it is not feasible to deploy Jaeger Agent next to the application, for example, when the application code is running as AWS Lambda function. In these scenarios the Jaeger Clients can be configured to submit spans directly to the Collectors over HTTP/HTTPS.
The same [jaeger.thrift][jaeger.thrift] payload can be submitted in HTTP POST request to `/api/traces` endpoint, for example, `https://jaeger-collector:14268/api/traces`. The `Batch` struct needs to be encoded using Thrift's `binary` encoding, and the HTTP request should specify the content type header:
```
Content-Type: application/vnd.apache.thrift.binary
```
### JSON over HTTP (n/a)
There is no official Jaeger JSON format that can be accepted by the collector. In the future the Protobuf-generated JSON may be supported.
### Thrift via TChannel (deprecated)
Agent and Collector can communicate using TChannel protocol. This protocol is generally not supported by the routing infrastructure and has been deprecated. It will be eventually removed from Jaeger.
### Zipkin Formats (stable)
Jaeger Collector can also accept spans in several Zipkin data format, namely JSON v1/v2 and Thrift. The Collector needs to be configured to enable Zipkin HTTP server, e.g. on port 9411 used by Zipkin collectors. The server enables two endpoints that expect POST requests:
* `/api/v1/spans` for submitting spans in Zipkin JSON v1 or Zipkin Thrift format.
* `/api/v2/spans` for submitting spans in Zipkin JSON v2.
## Trace retrieval APIs
Traces saved in the storage can be retrieved by calling Jaeger Query Service.
### gRPC/Protobuf (stable)
The recommended way for programmatically retrieving traces and other data is via gRPC endpoint defined in [query.proto][query.proto] IDL file (located in the main Jaeger repository, similar to [collector.proto][collector.proto]).
### HTTP JSON (internal)
Jaeger UI communicates with Jaeger Query Service via JSON API. For example, a trace can be retrieved via GET request to `https://jaeger-query:16686/api/traces/{trace-id-hex-string}`. This JSON API is intentionally undocumented and subject to change.
## Clients configuration (internal)
Client libraries not only submit finished spans to Jaeger backend, but also periodically poll the Agents for various configurations, such as sampling strategies. The schema for the payload is defined by [sampling.thrift][sampling.thrift], encoded as JSON using Thrift's built-in JSON generation capabilities.
## Service dependencies graph (internal)
Can be retrieved from Query Service at `/api/dependencies` endpoint. The GET request expects two parameters:
* `endTs` (number of milliseconds since epoch) - the end of the time interval
* `lookback` (in milliseconds) - the length the time interval (i.e. start-time + lookback = end-time).
The returned JSON is a list of edges represented as tuples `(caller, callee, count)`.
For programmatic access to service graph, the recommended API is gRPC/Protobuf described above.
[jaeger-idl]: https://github.com/jaegertracing/jaeger-idl/
[jaeger.thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/jaeger.thrift
[agent.thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/agent.thrift
[sampling.thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/sampling.thrift
[collector.proto]: https://github.com/jaegertracing/jaeger/blob/master/model/proto/api_v2/collector.proto
[query.proto]: https://github.com/jaegertracing/jaeger/blob/master/model/proto/api_v2/query.proto

13
content/docs/1.14/cli.md Normal file
View File

@ -0,0 +1,13 @@
---
title: CLI flags
widescreen: true
hasparent: true
---
This is auto-generated documentation for CLI flags supported by Jaeger binaries.
* CLI flags for some binaries change depending on the `SPAN_STORAGE_TYPE` environment variable. Relevant variations are included below.
* Some binaries support _commands_ (mostly informational), such as `env`, `docs`, and `version`. These commands are not included here.
* All parameters can be also provided via environment variables, by changing all letters to upper-case and replacing all punctuation characters with underscore `_`. For example, the value for the flag `--cassandra.connections-per-host` can be provided via `CASSANDRA_CONNECTIONS_PER_HOST` environment variable.
{{< cli/tools-list >}}

View File

@ -0,0 +1,467 @@
---
title: Deployment
weight: 4
children:
- title: Operator for Kubernetes
navtitle: Kubernetes
url: operator
- title: Frontend/UI
url: frontend-ui
- title: CLI Flags
url: cli
---
The main Jaeger backend components are released as Docker images on Docker Hub:
Component | Repository
--------------------- | ---
**jaeger-agent** | [hub.docker.com/r/jaegertracing/jaeger-agent/](https://hub.docker.com/r/jaegertracing/jaeger-agent/)
**jaeger-collector** | [hub.docker.com/r/jaegertracing/jaeger-collector/](https://hub.docker.com/r/jaegertracing/jaeger-collector/)
**jaeger-query** | [hub.docker.com/r/jaegertracing/jaeger-query/](https://hub.docker.com/r/jaegertracing/jaeger-query/)
**jaeger-ingester** | [hub.docker.com/r/jaegertracing/jaeger-ingester/](https://hub.docker.com/r/jaegertracing/jaeger-ingester/)
There are orchestration templates for running Jaeger with:
* Kubernetes: [github.com/jaegertracing/jaeger-kubernetes](https://github.com/jaegertracing/jaeger-kubernetes),
* OpenShift: [github.com/jaegertracing/jaeger-openshift](https://github.com/jaegertracing/jaeger-openshift).
## Configuration Options
Jaeger binaries can be configured in a number of ways (in the order of decreasing priority):
* command line arguments,
* environment variables,
* configuration files in JSON, TOML, YAML, HCL, or Java properties formats.
To see the complete list of options, run the binary with `help` command. Options that are specific to a certain storage backend are only listed if the storage type is selected. For example, to see all available options in the Collector with Cassandra storage:
```sh
$ docker run --rm \
-e SPAN_STORAGE_TYPE=cassandra \
jaegertracing/jaeger-collector:{{< currentVersion >}} \
help
```
In order to provide configuration parameters via environment variables, find the respective command line option and convert its name to UPPER_SNAKE_CASE, for example:
Command line option | Environment variable
-----------------------------------|-------------------------------
`--cassandra.connections-per-host` | `CASSANDRA_CONNECTIONS_PER_HOST`
`--metrics-backend` | `METRICS_BACKEND`
## Agent
Jaeger client libraries expect **jaeger-agent** process to run locally on each host.
The agent exposes the following ports:
Port | Protocol | Function
----- | ------- | ---
6831 | UDP | accept [jaeger.thrift][jaeger-thrift] in `compact` Thrift protocol used by most current Jaeger clients
6832 | UDP | accept [jaeger.thrift][jaeger-thrift] in `binary` Thrift protocol used by Node.js Jaeger client (because [thriftrw][thriftrw] npm package does not support `compact` protocol)
5778 | HTTP | serve configs, sampling strategies
5775 | UDP | accept [zipkin.thrift][zipkin-thrift] in `compact` Thrift protocol (deprecated; only used by very old Jaeger clients, circa 2016)
14271 | HTTP | Healthcheck at `/` and metrics at `/metrics`
It can be executed directly on the host or via Docker, as follows:
```sh
## make sure to expose only the ports you use in your deployment scenario!
docker run \
--rm \
-p6831:6831/udp \
-p6832:6832/udp \
-p5778:5778/tcp \
-p5775:5775/udp \
jaegertracing/jaeger-agent:{{< currentVersion >}}
```
### Discovery System Integration
The agents can connect point to point to a single collector address, which could be
load balanced by another infrastructure component (e.g. DNS) across multiple collectors.
The agent can also be configured with a static list of collector addresses.
On Docker, a command like the following can be used:
```sh
docker run \
--rm \
-p5775:5775/udp \
-p6831:6831/udp \
-p6832:6832/udp \
-p5778:5778/tcp \
jaegertracing/jaeger-agent:{{< currentVersion >}} \
--reporter.grpc.host-port=jaeger-collector.jaeger-infra.svc:14250
```
Or use `--reporter.tchannel.host-port=jaeger-collector.jaeger-infra.svc:14267` to use
legacy tchannel reporter.
When using gRPC, you have several options for load balancing and name resolution:
* Single connection and no load balancing. This is the default if you specify a single `host:port`. (example: `--reporter.grpc.host-port=jaeger-collector.jaeger-infra.svc:14250`)
* Static list of hostnames and round-robin load balancing. This is what you get with a comma-separated list of addresses. (example: `reporter.grpc.host-port=jaeger-collector1:14250,jaeger-collector2:14250,jaeger-collector3:14250`)
* Dynamic DNS resolution and round-robin load balancing. To get this behaviour, prefix the address with `dns:///` and gRPC will attempt to resolve the hostname using SRV records (for [external load balancing](https://github.com/grpc/grpc/blob/master/doc/load-balancing.md)), TXT records (for [service configs](https://github.com/grpc/grpc/blob/master/doc/service_config.md)), and A records. Refer to the [gRPC Name Resolution docs](https://github.com/grpc/grpc/blob/master/doc/naming.md) and the [dns_resolver.go implementation](https://github.com/grpc/grpc-go/blob/master/resolver/dns/dns_resolver.go) for more info. (example: `--reporter.grpc.host-port=dns:///jaeger-collector.jaeger-infra.svc:14250`)
### Agent level tags
Jaeger supports agent level tags, that can be added to the process tags of all spans passing through the agent. This is supported through the command line flag `--jaeger.tags=key1=value1,key2=value2,...,keyn=valuen`. Tags can also be set through an environment flag like so - `--jaeger.tags=key=${envFlag:defaultValue}` - The tag value will be set to the value of the `envFlag` environment key and `defaultValue` if not set. This feature is not supported for the tchannel reporter, enabled using the flags `--collector.host-port` or `--reporter.tchannel.host-port`.
## Collectors
The collectors are stateless and thus many instances of **jaeger-collector** can be run in parallel.
Collectors require almost no configuration, except for the location of Cassandra cluster,
via `--cassandra.keyspace` and `--cassandra.servers` options, or the location of Elasticsearch cluster, via
`--es.server-urls`, depending on which storage is specified. To see all command line options run
```sh
go run ./cmd/collector/main.go -h
```
or, if you don't have the source code
```sh
docker run -it --rm jaegertracing/jaeger-collector:{{< currentVersion >}} -h
```
At default settings the collector exposes the following ports:
Port | Protocol | Function
----- | ------- | ---
14267 | TChannel | used by **jaeger-agent** to send spans in jaeger.thrift format
14250 | gRPC | used by **jaeger-agent** to send spans in model.proto format
14268 | HTTP | can accept spans directly from clients in jaeger.thrift format over binary thrift protocol
9411 | HTTP | can accept Zipkin spans in Thrift, JSON and Proto (disabled by default)
14269 | HTTP | Healthcheck at `/` and metrics at `/metrics`
## Storage Backends
Collectors require a persistent storage backend. Cassandra and Elasticsearch are the primary supported storage backends. Additional backends are [discussed here](https://github.com/jaegertracing/jaeger/issues/638).
The storage type can be passed via `SPAN_STORAGE_TYPE` environment variable. Valid values are `cassandra`, `elasticsearch`, `kafka` (only as a buffer), `grpc-plugin`, `badger` (only with all-in-one) and `memory` (only with all-in-one).
As of version 1.6.0, it's possible to use multiple storage types at the same time by providing a comma-separated list of valid types to the `SPAN_STORAGE_TYPE` environment variable.
It's important to note that all listed storage types are used for writing, but only the first type in the list will be used for reading and archiving.
### Memory
The in-memory storage is not intended for production workloads. It's intended as a simple solution to get started quickly and
data will be lost once the process is gone.
By default, there's no limit in the amount of traces stored in memory but a limit can be established by passing an
integer value via `--memory.max-traces`.
### Badger - local storage
Experimental since Jaeger 1.9
[Badger](https://github.com/dgraph-io/badger) is an embedded local storage, only available
with **all-in-one** distribution. By default it acts as an ephemeral storage using a temporary filesystem.
This can be overridden by using the `--badger.ephemeral=false` option.
```sh
docker run \
-e SPAN_STORAGE_TYPE=badger \
-e BADGER_EPHEMERAL=false \
-e BADGER_DIRECTORY_VALUE=/badger/data \
-e BADGER_DIRECTORY_KEY=/badger/key \
-v <storage_dir_on_host>:/badger \
-p 16686:16686 \
jaegertracing/all-in-one:{{< currentVersion >}}
```
### Cassandra
Supported versions: 3.4+
Deploying Cassandra itself is out of scope for our documentation. One good
source of documentation is the [Apache Cassandra Docs](https://cassandra.apache.org/doc/latest/).
#### Configuration
##### Minimal
```sh
docker run \
-e SPAN_STORAGE_TYPE=cassandra \
-e CASSANDRA_SERVERS=<...> \
jaegertracing/jaeger-collector:{{< currentVersion >}}
```
##### All options
To view the full list of configuration options, you can run the following command:
```sh
docker run \
-e SPAN_STORAGE_TYPE=cassandra \
jaegertracing/jaeger-collector:{{< currentVersion >}} \
--help
```
#### Schema script
A script is provided to initialize Cassandra keyspace and schema
using Cassandra's interactive shell [`cqlsh`][cqlsh]:
```sh
MODE=test sh ./plugin/storage/cassandra/schema/create.sh | cqlsh
```
For production deployment, pass `MODE=prod DATACENTER={datacenter}` arguments to the script,
where `{datacenter}` is the name used in the Cassandra configuration / network topology.
The script also allows overriding TTL, keyspace name, replication factor, etc.
Run the script without arguments to see the full list of recognized parameters.
#### TLS support
Jaeger supports TLS client to node connections as long as you've configured
your Cassandra cluster correctly. After verifying with e.g. `cqlsh`, you can
configure the collector and query like so:
```sh
docker run \
-e CASSANDRA_SERVERS=<...> \
-e CASSANDRA_TLS=true \
-e CASSANDRA_TLS_SERVER_NAME="CN-in-certificate" \
-e CASSANDRA_TLS_KEY=<path to client key file> \
-e CASSANDRA_TLS_CERT=<path to client cert file> \
-e CASSANDRA_TLS_CA=<path to your CA cert file> \
jaegertracing/jaeger-collector:{{< currentVersion >}}
```
The schema tool also supports TLS. You need to make a custom cqlshrc file like
so:
```
# Creating schema in a cassandra cluster requiring client TLS certificates.
#
# Create a volume for the schema docker container containing four files:
# cqlshrc: this file
# ca-cert: the cert authority for your keys
# client-key: the keyfile for your client
# client-cert: the cert file matching client-key
#
# if there is any sort of DNS mismatch and you want to ignore server validation
# issues, then uncomment validate = false below.
#
# When running the container, map this volume to /root/.cassandra and set the
# environment variable CQLSH_SSL=--ssl
[ssl]
certfile = ~/.cassandra/ca-cert
userkey = ~/.cassandra/client-key
usercert = ~/.cassandra/client-cert
# validate = false
```
### Elasticsearch
Supported in Jaeger since 0.6.0
Supported versions: 5.x, 6.x, 7.x
Elasticsearch version is automatically retrieved from root/ping endpoint.
Based on this version Jaeger uses compatible index mappings and Elasticsearch REST API.
The version can be explicitly provided via `--es.version=` flag.
Elasticsearch does not require initialization other than
[installing and running Elasticsearch](https://www.elastic.co/downloads/elasticsearch).
Once it is running, pass the correct configuration values to the Jaeger collector and query service.
#### Configuration
##### Minimal
```sh
docker run \
-e SPAN_STORAGE_TYPE=elasticsearch \
-e ES_SERVER_URLS=<...> \
jaegertracing/jaeger-collector:{{< currentVersion >}}
```
##### All options
To view the full list of configuration options, you can run the following command:
```sh
docker run \
-e SPAN_STORAGE_TYPE=elasticsearch \
jaegertracing/jaeger-collector:{{< currentVersion >}} \
--help
```
#### Shards and Replicas for Elasticsearch indices
Shards and replicas are some configuration values to take special attention to, because this is decided upon
index creation. [This article](https://qbox.io/blog/optimizing-elasticsearch-how-many-shards-per-index) goes into
more information about choosing how many shards should be chosen for optimization.
#### Upgrade Elasticsearch version
Elasticsearch defines wire and index compatibility versions. The index compatibility defines
the minimal version a node can read data from. For example Elasticsearch 7 can read indices
created by Elasticsearch 6, however it cannot read indices created by Elasticsearch 5 even
though they use the same index mappings. Therefore upgrade from Elasticsearch 6 to 7 does not require any
data migration. However, upgrade from Elasticsearch 5 to 7 has to be done through Elasticsearch 6 and wait
until indices created by ES 5.x are removed or explicitly reindexed.
Refer to the Elasticsearch [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current//setup-upgrade.html)
for wire and index compatibility versions. Generally this information can be retrieved from root/ping REST endpoint.
##### Reindex
Manual reindexing can be used when upgrading from Elasticsearch 5 to 7 (through Elasticsearch 6)
without waiting until indices created by Elasticsearch 5 are removed.
1. Reindex all span indices to new indices with suffix `-1`:
```bash
curl -ivX POST -H "Content-Type: application/json" http://localhost:9200/_reindex -d @reindex.json
{
"source": {
"index": "jaeger-span-*"
},
"dest": {
"index": "jaeger-span"
},
"script": {
"lang": "painless",
"source": "ctx._index = 'jaeger-span-' + (ctx._index.substring('jaeger-span-'.length(), ctx._index.length())) + '-1'"
}
}
```
2. Delete indices with old mapping:
```bash
curl -ivX DELETE -H "Content-Type: application/json" http://localhost:9200/jaeger-span-\*,-\*-1
```
3. Create indices without `-1` suffix:
```bash
curl -ivX POST -H "Content-Type: application/json" http://localhost:9200/_reindex -d @reindex.json
{
"source": {
"index": "jaeger-span-*"
},
"dest": {
"index": "jaeger-span"
},
"script": {
"lang": "painless",
"source": "ctx._index = 'jaeger-span-' + (ctx._index.substring('jaeger-span-'.length(), ctx._index.length() - 2))"
}
}
```
4. Remove suffixed indices:
```bash
curl -ivX DELETE -H "Content-Type: application/json" http://localhost:9200/jaeger-span-\*-1
```
Run the commands analogically for other Jaeger indices.
There might exist more effective migration procedure. Please share with the community any findings.
### Kafka
Supported in Jaeger since 1.6.0
Supported Kafka versions: 0.9+
Kafka can be used as an intermediary buffer between collector and an actual storage.
The collector is configured with `SPAN_STORAGE_TYPE=kafka` that makes it write all received spans
into a Kafka topic. A new component [Ingester](#ingester), added in version 1.7.0, is used to read from
Kafka and store spans in another storage backend (Elasticsearch or Cassandra).
Writing to Kafka is particularly useful for building post-processing data pipelines.
#### Configuration
##### Minimal
```sh
docker run \
-e SPAN_STORAGE_TYPE=kafka \
-e KAFKA_BROKERS=<...> \
-e KAFKA_TOPIC=<...> \
jaegertracing/jaeger-collector:{{< currentVersion >}}
```
##### All options
To view the full list of configuration options, you can run the following command:
```sh
docker run \
-e SPAN_STORAGE_TYPE=kafka \
jaegertracing/jaeger-collector:{{< currentVersion >}} \
--help
```
#### Topic & partitions
Unless your Kafka cluster is configured to automatically create topics, you will need to create it ahead of time. You can refer to [the Kafka quickstart documentation](https://kafka.apache.org/documentation/#quickstart_createtopic) to learn how.
You can find more information about topics and partitions in general in the [official documentation](https://kafka.apache.org/documentation/#intro_topics). [This article](https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/) provide more details about how to choose the number of partitions.
### Storage plugin
Jaeger supports gRPC based storage plugins. For more information refer to [jaeger/plugin/storage/grpc](https://github.com/jaegertracing/jaeger/tree/master/plugin/storage/grpc)
Available plugins:
* [InfluxDB](https://github.com/influxdata/jaeger-influxdb/)
```sh
docker run \
-e SPAN_STORAGE_TYPE=grpc-plugin \
-e GRPC_STORAGE_PLUGIN_BINARY=<...> \
-e GRPC_STORAGE_PLUGIN_CONFIGURATION_FILE=<...> \
jaegertracing/all-in-one:{{< currentVersion >}}
```
## Ingester
**jaeger-ingester** is a service which reads span data from Kafka topic and writes it to another storage backend (Elasticsearch or Cassandra).
Port | Protocol | Function
----- | ------- | ---
14270 | HTTP | Healthcheck at `/` and metrics at `/metrics`
To view all exposed configuration options run the following command:
```sh
docker run \
-e SPAN_STORAGE_TYPE=cassandra \
jaegertracing/jaeger-ingester:{{< currentVersion >}}
--help
```
## Query Service & UI
**jaeger-query** serves the API endpoints and a React/Javascript UI.
The service is stateless and is typically run behind a load balancer, such as [**NGINX**](https://www.nginx.com/).
At default settings the query service exposes the following port(s):
Port | Protocol | Function
----- | ------- | ---
16686 | HTTP | `/api/*` endpoints and Jaeger UI at `/`
16687 | HTTP | Healthcheck at `/` and metrics at `/metrics`
### Minimal deployment example (Elasticsearch backend):
```sh
docker run -d --rm \
-p 16686:16686 \
-p 16687:16687 \
-e SPAN_STORAGE_TYPE=elasticsearch \
-e ES_SERVER_URLS=http://<ES_SERVER_IP>:<ES_SERVER_PORT> \
jaegertracing/jaeger-query:{{< currentVersion >}}
```
### UI Base Path
The base path for all **jaeger-query** HTTP routes can be set to a non-root value, e.g. `/jaeger` would cause all UI URLs to start with `/jaeger`. This can be useful when running **jaeger-query** behind a reverse proxy.
The base path can be configured via the `--query.base-path` command line parameter or the `QUERY_BASE_PATH` environment variable.
### UI Customization and Embedding
Please refer to the [dedicated Frontend/UI page](../frontend-ui/).
## Aggregation Jobs for Service Dependencies
Production deployments need an external process which aggregates data and creates dependency links between services. Project [spark-dependencies](https://github.com/jaegertracing/spark-dependencies) is a Spark job which derives dependency links and stores them directly to the storage.
## Configuration
All binaries accepts command line properties and environmental variables, power by [viper](https://github.com/spf13/viper) and [cobra](https://github.com/spf13/cobra) libraries. Please refer to the [CLI Flags](../cli/) page for more information.
[cqlsh]: http://cassandra.apache.org/doc/latest/tools/cqlsh.html
[zipkin-thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/zipkincore.thrift
[jaeger-thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/jaeger.thrift
[thriftrw]: https://www.npmjs.com/package/thriftrw

17
content/docs/1.14/faq.md Normal file
View File

@ -0,0 +1,17 @@
---
title: Frequently Asked Questions
navtitle: FAQs
description: Answers to some frequently asked questions about Jaeger.
weight: 11
---
## Why is the Dependencies page empty?
The Dependencies page shows a graph of services traced by Jaeger and connections between them. When you are using `all-in-one` binary with in-memory storage, the graph is calculated on-demand from all the traces stored in memory. However, if you are using a real distributed storage like Cassandra or Elasticsearch, it is too expensive to scan all the data in the database to build the service graph. Instead, the Jaeger project provides "big data" jobs that can be used to extract the service graph data from traces:
* https://github.com/jaegertracing/spark-dependencies - the older Spark job that can be run periodically
* https://github.com/jaegertracing/jaeger-analytics - the new (experimental) streaming Flink jobs that run continuously and builds the service graph in smaller time intervals
## Why do I not see any spans in Jaeger?
Please refer to the [Troubleshooting](../troubleshooting/) guide.

View File

@ -0,0 +1,57 @@
---
title: Features
hasparent: true
---
Jaeger is used for monitoring and troubleshooting microservices-based distributed systems, including:
* Distributed context propagation
* Distributed transaction monitoring
* Root cause analysis
* Service dependency analysis
* Performance / latency optimization
## High Scalability
Jaeger backend is designed to have no single points of failure and to scale with the business needs.
For example, any given Jaeger installation at Uber is typically processing several billion {{< tip "spans" "span" >}} per day.
## Native support for OpenTracing
Jaeger backend, Web UI, and instrumentation libraries have been designed from ground up to support the OpenTracing standard.
* Represent {{< tip "traces" "trace" >}} as {{< tip "directed acyclic graphs" "directed acyclic graph" >}} (not just trees) via [span references](https://github.com/opentracing/specification/blob/master/specification.md#references-between-spans)
* Support strongly typed span _tags_ and _structured logs_
* Support general distributed context propagation mechanism via _baggage_
## Multiple storage backends
Jaeger supports two popular open source NoSQL databases as trace storage backends: Cassandra 3.4+ and Elasticsearch 5.x/6.x/7.x.
There are ongoing community experiments using other databases, such as ScyllaDB, InfluxDB, Amazon DynamoDB. Jaeger also ships
with a simple in-memory storage for testing setups.
## Modern Web UI
Jaeger Web UI is implemented in Javascript using popular open source frameworks like React. Several performance
improvements have been released in v1.0 to allow the UI to efficiently deal with large volumes of data, and to display
{{< tip "traces" "trace" >}} with tens of thousands of {{< tip "spans" "span" >}} (e.g. we tried a trace with 80,000 spans).
## Cloud Native Deployment
Jaeger backend is distributed as a collection of Docker images. The binaries support various configuration methods,
including command line options, environment variables, and configuration files in multiple formats (yaml, toml, etc.).
Deployment to Kubernetes clusters is assisted by a [Kubernetes operator](https://github.com/jaegertracing/jaeger-operator), [Kubernetes templates](https://github.com/jaegertracing/jaeger-kubernetes)
and a [Helm chart](https://github.com/kubernetes/charts/tree/master/incubator/jaeger).
## Observability
All Jaeger backend components expose [Prometheus](https://prometheus.io/) metrics by default (other metrics backends are
also supported). Logs are written to standard out using the structured logging library [zap](https://github.com/uber-go/zap).
## Backwards compatibility with Zipkin
Although we recommend instrumenting applications with OpenTracing API and binding to Jaeger client libraries to benefit
from advanced features not available elsewhere, if your organization has already invested in the instrumentation
using Zipkin libraries, you do not have to rewrite all that code. Jaeger provides backwards compatibility with Zipkin
by accepting spans in Zipkin formats (Thrift, JSON v1/v2 and Protobuf) over HTTP. Switching from Zipkin backend is just a matter
of routing the traffic from Zipkin libraries to the Jaeger backend.

View File

@ -0,0 +1,224 @@
---
title: Frontend/UI Configuration
navtitle: Frontend/UI
hasparent: true
weight: 7
---
## Configuration
Several aspects of the UI can be configured:
* The Dependencies section can be enabled / configured
* Google Analytics tracking can be enabled / configured
* Additional menu options can be added to the global nav
These options can be configured by a JSON configuration file. The `--query.ui-config` command line parameter of the query service must then be set to the path to the JSON file when the query service is started.
An example configuration file:
```json
{
"dependencies": {
"dagMaxNumServices": 200,
"menuEnabled": true
},
"archiveEnabled": true,
"tracking": {
"gaID": "UA-000000-2",
"trackErrors": true
},
"menu": [
{
"label": "About Jaeger",
"items": [
{
"label": "GitHub",
"url": "https://github.com/jaegertracing/jaeger"
},
{
"label": "Docs",
"url": "http://jaeger.readthedocs.io/en/latest/"
}
]
}
],
"linkPatterns": [{
"type": "process",
"key": "jaeger.version",
"url": "https://github.com/jaegertracing/jaeger-client-java/releases/tag/#{jaeger.version}",
"text": "Information about Jaeger release #{jaeger.version}"
}]
}
```
### Dependencies
`dependencies.dagMaxNumServices` defines the maximum number of services allowed before the DAG dependency view is disabled. Default: `200`.
`dependencies.menuEnabled` enables (`true`) or disables (`false`) the dependencies menu button. Default: `true`.
### Archive Support
`archiveEnabled` enables (`true`) or disables (`false`) the archive traces button. Default: `false`. It requires a configuration of an archive storage in Query service. Archived traces are only accessible directly by ID, they are not searchable.
### Google Analytics Tracking
`tracking.gaID` defines the Google Analytics tracking ID. This is required for Google Analytics tracking, and setting it to a non-`null` value enables Google Analytics tracking. Default: `null`.
`tracking.trackErrors` enables (`true`) or disables (`false`) error tracking via Google Analytics. Errors can only be tracked if a valid Google Analytics ID is provided. For additional details on error tracking via Google Analytics see the [tracking README](https://github.com/jaegertracing/jaeger-ui/blob/c622330546afc1be59a42f874bcc1c2fadf7e69a/src/utils/tracking/README.md) in the UI repo. Default: `true`.
### Custom Menu Items
`menu` allows additional links to be added to the global nav. The additional links are right-aligned.
In the sample JSON config above, the configured menu will have a dropdown labeled "About Jaeger" with sub-options for "GitHub" and "Docs". The format for a link in the top right menu is as follows:
```json
{
"label": "Some text here",
"url": "https://example.com"
}
```
Links can either be members of the `menu` Array, directly, or they can be grouped into a dropdown menu option. The format for a group of links is:
```json
{
"label": "Dropdown button",
"items": [ ]
}
```
The `items` Array should contain one or more link configurations.
### Link Patterns
The `linkPatterns` node can be used to create links from fields displayed in the Jaeger UI.
Field | Description
------|------------
type | The metadata section in which your link will be added: process, tags, logs
key | The name of tag/process/log attribute which value will be displayed as a link
url | The URL where the link should point to, it can be an external site or relative path in Jaeger UI
text | The text displayed in the tooltip for the link
Both `url` and `text` can be defined as templates (i.e. using `#{field-name}`) where Jaeger UI will dynamically substitute values based on tags/logs data.
## Embedded Mode
Starting with version 1.9, Jaeger UI provides an "embedded" layout mode which is intended to support integrating Jaeger UI into other applications. Currently (as of `v0`), the approach taken is to remove various UI elements from the page to make the UI better suited for space-constrained layouts.
The embedded mode is induced and configured via URL query parameters.
To enter embedded mode, the `uiEmbed=v0` query parameter and value must be added to the URL. For example, the following URL will show the trace with ID `abc123` in embedded mode:
```
http://localhost:16686/trace/abc123?uiEmbed=v0
```
`uiEmbed=v0` is required.
Further, each page supported has an <img src="/img/frontend-ui/embed-open-icon.png" style="width: 20px; height:20px;display:inline;" alt="Embed open window"> button added that will open the non-embedded page in a new tab.
The following pages support embedded mode:
* Search Page
* Trace Page
### Search Page
To integrate the Search Trace Page to our application we have to indicate to the Jaeger UI that we want to use the embed mode with `uiEmbed=v0`.
For example:
```
http://localhost:16686/search?
service=my-service&
start=1543917759557000&
end=1543921359557000&
limit=20&
lookback=1h&
maxDuration&
minDuration&
uiEmbed=v0
```
![Embed Search Traces](/img/frontend-ui/embed-search-traces.png)
#### Configuration options
The following query parameter can be used to configure the layout of the search page :
* `uiSearchHideGraph=1` - disables the display of the scatter plot above the search results
```
http://localhost:16686/search?
service=my-service&
start=1543917759557000&
end=1543921359557000&
limit=20&
lookback=1h&
maxDuration&
minDuration&
uiEmbed=v0&
uiSearchHideGraph=1
```
![Embed Search Traces without Graph](/img/frontend-ui/embed-search-traces-hide-graph.png)
### Trace Page
To integrate the Trace Page to our application we have to indicate to the Jaeger UI that we want to use the embed mode with `uiEmbed=v0`.
For example:
```sh
http://localhost:16686/trace/{trace-id}?uiEmbed=v0
```
![Embed Trace view](/img/frontend-ui/embed-trace-view.png)
If we have navigated to this view from the search traces page we'll have a button to go back to the results page.
![Embed Trace view](/img/frontend-ui/embed-trace-view-with-back-button.png)
#### Configuration options
The following query parameters can be used to configure the layout of the trace page :
* `uiTimelineCollapseTitle=1` causes the trace header to start out collapsed, which hides the summary and the minimap.
```
http://localhost:16686/trace/{trace-id}?
uiEmbed=v0&
uiTimelineCollapseTitle=1
```
![Embed Trace view](/img/frontend-ui/embed-trace-view-with-collapse.png)
* `uiTimelineHideMinimap=1` removes the minimap, entirely, regardless of whether the trace header is expanded or not.
```
http://localhost:16686/trace/{trace-id}?
uiEmbed=v0&
uiTimelineHideMinimap=1
```
![Embed Trace view](/img/frontend-ui/embed-trace-view-with-hide-minimap.png)
* `uiTimelineHideSummary=1` - removes the trace summary information (number of services, etc.) entirely, regardless of whether the trace header is expanded or not.
```
http://localhost:16686/trace/{trace-id}?
uiEmbed=v0&
uiTimelineHideSummary=1
```
![Embed Trace view](/img/frontend-ui/embed-trace-view-with-hide-summary.png)
We can also combine the options:
```
http://localhost:16686/trace/{trace-id}?
uiEmbed=v0&
uiTimelineHideMinimap=1&
uiTimelineHideSummary=1
```
![Embed Trace view](/img/frontend-ui/embed-trace-view-with-hide-details-and-hide-minimap.png)

View File

@ -0,0 +1,129 @@
---
title: Getting Started
description: Get up and running with Jaeger in your local environment
weight: 2
---
## Instrumentation
Your applications must be instrumented before they can send tracing data to Jaeger backend. Check the [Client Libraries](../client-libraries) section for information about how to use the OpenTracing API and how to initialize and configure Jaeger tracers.
## All in One
All-in-one is an executable designed for quick local testing, launches the Jaeger UI, collector, query, and agent, with an in memory storage component.
The simplest way to start the all-in-one is to use the pre-built image published to DockerHub (a single command line).
```bash
$ docker run -d --name jaeger \
-e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
-p 5775:5775/udp \
-p 6831:6831/udp \
-p 6832:6832/udp \
-p 5778:5778 \
-p 16686:16686 \
-p 14268:14268 \
-p 9411:9411 \
jaegertracing/all-in-one:{{< currentVersion >}}
```
Or run the `jaeger-all-in-one(.exe)` executable from the [binary distribution archives][download]:
```bash
$ jaeger-all-in-one --collector.zipkin.http-port=9411
```
You can then navigate to `http://localhost:16686` to access the Jaeger UI.
The container exposes the following ports:
Port | Protocol | Component | Function
----- | ------- | --------- | ---
5775 | UDP | agent | accept `zipkin.thrift` over compact thrift protocol (deprecated, used by legacy clients only)
6831 | UDP | agent | accept `jaeger.thrift` over compact thrift protocol
6832 | UDP | agent | accept `jaeger.thrift` over binary thrift protocol
5778 | HTTP | agent | serve configs
16686 | HTTP | query | serve frontend
14268 | HTTP | collector | accept `jaeger.thrift` directly from clients
14250 | HTTP | collector | accept `model.proto`
9411 | HTTP | collector | Zipkin compatible endpoint (optional)
## Kubernetes and OpenShift
* Kubernetes templates: https://github.com/jaegertracing/jaeger-kubernetes
* Kubernetes Operator: https://github.com/jaegertracing/jaeger-operator
* OpenShift templates: https://github.com/jaegertracing/jaeger-openshift
## Sample App: HotROD
HotROD (Rides on Demand) is a demo application that consists of several microservices and
illustrates the use of the [OpenTracing API](http://opentracing.io).
A tutorial / walkthrough is available in the blog post:
[Take OpenTracing for a HotROD ride][hotrod-tutorial].
It can be run standalone, but requires Jaeger backend to view the traces.
### Features
- Discover architecture of the whole system via data-driven dependency
diagram.
- View request timeline and errors; understand how the app works.
- Find sources of latency and lack of concurrency.
- Highly contextualized logging.
- Use baggage propagation to:
- Diagnose inter-request contention (queueing).
- Attribute time spent in a service.
- Use open source libraries with OpenTracing integration to get
vendor-neutral instrumentation for free.
### Prerequisites
- You need Go 1.11 or higher installed on your machine to run from source.
- Requires a [running Jaeger backend](#all-in-one) to view the traces.
### Running
#### From Source
```bash
mkdir -p $GOPATH/src/github.com/jaegertracing
cd $GOPATH/src/github.com/jaegertracing
git clone git@github.com:jaegertracing/jaeger.git jaeger
cd jaeger
make install
go run ./examples/hotrod/main.go all
```
#### From docker
```bash
$ docker run --rm -it \
--link jaeger \
-p8080-8083:8080-8083 \
-e JAEGER_AGENT_HOST="jaeger" \
jaegertracing/example-hotrod:{{< currentVersion >}} \
all
```
#### From binary distribution
Run `example-hotrod(.exe)` executable from the [binary distribution archives][download]:
```bash
$ example-hotrod all
```
Then navigate to `http://localhost:8080`.
## Migrating from Zipkin
Collector service exposes Zipkin compatible REST API `/api/v1/spans` which accepts both Thrift and JSON. Also there is `/api/v2/spans` for JSON and Proto.
By default it's disabled. It can be enabled with `--collector.zipkin.http-port=9411`.
Zipkin [Thrift](https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/zipkincore.thrift) IDL and Zipkin [Proto](https://github.com/jaegertracing/jaeger-idl/blob/master/proto/zipkin.proto) IDL files can be found in [jaegertracing/jaeger-idl](https://github.com/jaegertracing/jaeger-idl) repository.
They're compatible with [openzipkin/zipkin-api](https://github.com/openzipkin/zipkin-api) [Thrift](https://github.com/openzipkin/zipkin-api/blob/master/thrift/zipkinCore.thrift) and [Proto](https://github.com/openzipkin/zipkin-api/blob/master/zipkin.proto).
[hotrod-tutorial]: https://medium.com/@YuriShkuro/take-opentracing-for-a-hotrod-ride-f6e3141f7941
[download]: ../../../download/

View File

@ -0,0 +1,816 @@
---
title: Operator for Kubernetes
hasparent: true
---
# Understanding Operators
The Jaeger Operator is an implementation of a [Kubernetes Operator](https://coreos.com/operators/). Operators are pieces of software that ease the operational complexity of running another piece of software. More technically, _Operators_ are a method of packaging, deploying, and managing a Kubernetes application.
A Kubernetes application is an application that is both deployed on Kubernetes and managed using the Kubernetes APIs and `kubectl` (kubernetes) or `oc` (OKD) tooling. To be able to make the most of Kubernetes, you need a set of cohesive APIs to extend in order to service and manage your apps that run on Kubernetes. Think of Operators as the runtime that manages this type of app on Kubernetes.
# Installing the Operator
{{< info >}}
The Jaeger Operator version tracks one version of the Jaeger components (Query, Collector, Agent). When a new version of the Jaeger components is released, a new version of the operator will be released that understands how running instances of the previous version can be upgraded to the new version.
{{< /info >}}
## Installing the Operator on Kubernetes
The following instructions will create the `observability` namespace and install the Jaeger Operator.
{{< info >}}
Make sure your `kubectl` command is properly configured to talk to a valid Kubernetes cluster. If you don't have a cluster, you can create one locally using [`minikube`](https://kubernetes.io/docs/tasks/tools/install-minikube/).
{{< /info >}}
To install the operator, run:
<!--TODO - Does Kubernetes have privileged users? Needs to be run as a system:admin on OKD/OpenShift.-->
```bash
kubectl create namespace observability # <1>
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/crds/jaegertracing.io_jaegers_crd.yaml # <2>
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/service_account.yaml
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role.yaml
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role_binding.yaml
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/operator.yaml
```
<1> This creates the namespace used by default in the deployment files. If you want to install the Jaeger operator in a different namespace, you must edit the deployment files to change `observability` to the desired namespace value.
<2> This installs the "Custom Resource Definition" for the `apiVersion: jaegertracing.io/v1`
At this point, there should be a `jaeger-operator` deployment available. You can view it by running the following command:
```bash
$ kubectl get deployment jaeger-operator -n observability
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
jaeger-operator 1 1 1 1 48s
```
The operator is now ready to create Jaeger instances.
## Installing the Operator on OKD/OpenShift
<!-- TODO: Add instructions for installing via the operatorhub? -->
The instructions from the previous section also work for installing the operator on OKD or OpenShift. Make sure you are logged in as a privileged user, when you install the role based acces control (RBAC) rules, the custom resource definition, and the operator.
```bash
oc login -u <privileged user>
oc new-project observability # <1>
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/crds/jaegertracing.io_jaegers_crd.yaml # <2>
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/service_account.yaml
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role.yaml
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role_binding.yaml
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/operator.yaml
```
<1> This creates the namespace used by default in the deployment files. If you want to install the Jaeger operator in a different namespace, you must edit the deployment files to change `observability` to the desired namespace value.
<2> This installs the "Custom Resource Definition" for the `apiVersion: jaegertracing.io/v1`
Once the operator is installed, grant the role `jaeger-operator` to users who should be able to install individual Jaeger instances. The following example creates a role binding allowing the user `developer` to create Jaeger instances:
```bash
oc create \
rolebinding developer-jaeger-operator \
--role=jaeger-operator \
--user=developer
```
After the role is granted, switch back to a non-privileged user.
# Quick Start - Deploying the AllInOne image
The simplest possible way to create a Jaeger instance is by creating a YAML file like the following example. This will install the default AllInOne strategy, which deploys the "all-in-one" image (agent, collector, query, ingester, Jaeger UI) in a single pod, using in-memory storage by default.
{{< info >}}
This default strategy is intended for development, testing, and demo purposes, not for production.
{{< /info >}}
```yaml
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: simplest
```
The YAML file can then be used with `kubectl`:
<!-- TODO - Add OKD commands and tabs shortcode. -->
```bash
kubectl apply -f simplest.yaml
```
In a few seconds, a new in-memory all-in-one instance of Jaeger will be available, suitable for quick demos and development purposes. To check the instances that were created, list the `jaeger` objects:
```bash
$ kubectl get jaegers
NAME CREATED AT
simplest 28s
```
To get the pod name, query for the pods belonging to the `simplest` Jaeger instance:
```bash
$ kubectl get pods -l app.kubernetes.io/instance=simplest
NAME READY STATUS RESTARTS AGE
simplest-6499bb6cdd-kqx75 1/1 Running 0 2m
```
Similarly, the logs can be queried either from the pod directly using the pod name obtained from the previous example, or from all pods belonging to our instance:
```bash
$ kubectl logs -l app.kubernetes.io/instance=simplest
...
{"level":"info","ts":1535385688.0951214,"caller":"healthcheck/handler.go:133","msg":"Health Check state change","status":"ready"}
```
{{< info >}}
On OKD/OpenShift the container name must be specified.
{{< /info >}}
```bash
$ kubectl logs -l app.kubernetes.io/instance=simplest -c jaeger
...
{"level":"info","ts":1535385688.0951214,"caller":"healthcheck/handler.go:133","msg":"Health Check state change","status":"ready"}
```
# Deployment Strategies
When you create a Jaeger instance, it is associated with a strategy. The strategy is defined in the custom resource file, and determines the architecture to be used for the Jaeger backend. The default strategy is `allInOne`. The other possible values are `production` and `streaming`.
The available strategies are described in the following sections.
## AllInOne (Default) strategy
This strategy is intended for development, testing, and demo purposes.
The main backend components, agent, collector and query service, are all packaged into a single executable which is configured (by default) to use in-memory storage.
## Production strategy
The `production` strategy is intended (as the name suggests) for production environments, where long term storage of trace data is important, as well as a more scalable and highly available architecture is required. Each of the backend components is therefore separately deployed.
The agent can be injected as a sidecar on the instrumented application or as a daemonset.
The query and collector services are configured with a supported storage type - currently Cassandra or Elasticsearch. Multiple instances of each of these components can be provisioned as required for performance and resilience purposes.
The main additional requirement is to provide the details of the storage type and options, for example:
```yaml
storage:
type: elasticsearch
options:
es:
server-urls: http://elasticsearch:9200
```
## Streaming strategy
The `streaming` strategy is designed to augment the `production` strategy by providing a streaming capability that effectively sits between the collector and the backend storage (Cassandra or Elasticsearch). This provides the benefit of reducing the pressure on the backend storage, under high load situations, and enables other trace post-processing capabilities to tap into the real time span data directly from the streaming platform (Kafka).
The only additional information required is to provide the details for accessing the Kafka platform, which is configured in the `collector` component (as producer) and `ingester` component (as consumer):
```yaml
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: simple-streaming
spec:
strategy: streaming
collector:
options:
kafka: # <1>
producer:
topic: jaeger-spans
brokers: my-cluster-kafka-brokers.kafka:9092
ingester:
options:
kafka: # <1>
consumer:
topic: jaeger-spans
brokers: my-cluster-kafka-brokers.kafka:9092
ingester:
deadlockInterval: 0 # <2>
storage:
type: elasticsearch
options:
es:
server-urls: http://elasticsearch:9200
```
<1> Identifies the Kafka configuration used by the collector, to produce the messages, and the ingester to consume the messages.
<2> The deadlock interval can be disabled to avoid the ingester being terminated when no messages arrive within the default 1 minute period
{{< info >}}
A Kafka environment can be configured using [Strimzi's Kafka operator](https://strimzi.io/).
{{< /info >}}
# Understanding Custom Resource Definitions
In the Kubernetes API, a resource is an endpoint that stores a collection of API objects of a certain kind. For example, the built-in Pods resource contains a collection of Pod objects. A _Custom Resource Definition_ (CRD) object defines a new, unique object `Kind` in the cluster and lets the Kubernetes API server handle its entire lifecycle.
To create _Custom Resource_ (CR) objects, cluster administrators must first create a Custom Resource Definition (CRD). The CRDs allow cluster users to create CRs to add the new resource types into their projects. An Operator watches for custom resource objects to be created, and when it sees a custom resource being created, it creates the application based on the parameters defined in the custom resource object.
{{< info >}}
While only cluster administrators can create CRDs, developers can create the CR from an existing CRD if they have read and write permission to it.
{{< /info >}}
<!--
## Jaeger Custom Resource Parameters
TODO Create a TABLE with all the parameters, descriptions/notes, valid values, and defaults.
Figure out if we can generate the options? Can we filter them in any way?
https://github.com/jaegertracing/jaeger/issues/1537
https://github.com/jaegertracing/documentation/issues/250-->
For reference, here's how you can create a more complex all-in-one instance:
```yaml
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: my-jaeger
spec:
strategy: allInOne # <1>
allInOne:
image: jaegertracing/all-in-one:latest # <2>
options: # <3>
log-level: debug # <4>
storage:
type: memory # <5>
options: # <6>
memory: # <7>
max-traces: 100000
ingress:
enabled: false # <8>
agent:
strategy: DaemonSet # <9>
annotations:
scheduler.alpha.kubernetes.io/critical-pod: "" # <10>
```
<1> The default strategy is `allInOne`. The other possible values are `production` and `streaming`.
<2> The image to use, in a regular Docker syntax.
<3> The (non-storage related) options to be passed verbatim to the underlying binary. Refer to the Jaeger documentation and/or to the `--help` option from the related binary for all the available options.
<4> The option is a simple `key: value` map. In this case, we want the option `--log-level=debug` to be passed to the binary.
<5> The storage type to be used. By default it will be `memory`, but can be any other supported storage type (Cassandra, Elasticsearch, Kafka).
<6> All storage related options should be placed here, rather than under the 'allInOne' or other component options.
<7> Some options are namespaced and we can alternatively break them into nested objects. We could have specified `memory.max-traces: 100000`.
<8> By default, an ingress object is created for the query service. It can be disabled by setting its `enabled` option to `false`. If deploying on OpenShift, this will be represented by a Route object.
<9> By default, the operator assumes that agents are deployed as sidecars within the target pods. Specifying the strategy as "DaemonSet" changes that and makes the operator deploy the agent as DaemonSet. Note that your tracer client will probably have to override the "JAEGER_AGENT_HOST" environment variable to use the node's IP.
<10> Define annotations to be applied to all deployments (not services). These can be overridden by annotations defined on the individual components.
You can view example custom resources for different Jaeger configurations [on GitHub](https://github.com/jaegertracing/jaeger-operator/tree/master/examples).
# Configuring the Custom Resource
<!--TODO
esIndexCleaner
Spark dependencies
-->
You can use the simplest example (shown above) and create a Jaeger instance using the defaults, or you can create your own custom resource file.
## Storage options
### Cassandra storage
When the storage type is set to Cassandra, the operator will automatically create a batch job that creates the required schema for Jaeger to run. This batch job will block the Jaeger installation, so that it starts only after the schema is successfuly created. The creation of this batch job can be disabled by setting the `enabled` property to `false`:
```yaml
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: cassandra-without-create-schema
spec:
strategy: allInOne
storage:
type: cassandra
cassandraCreateSchema:
enabled: false # <1>
```
<1> Defaults to `true`
Further aspects of the batch job can be configured as well. An example with all the possible options is shown below:
```yaml
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: cassandra-with-create-schema
spec:
strategy: allInOne # <1>
storage:
type: cassandra
options: # <2>
cassandra:
servers: cassandra
keyspace: jaeger_v1_datacenter3
cassandraCreateSchema: # <3>
datacenter: "datacenter3"
mode: "test"
```
<1> The same works for `production` and `streaming`.
<2> These options are for the regular Jaeger components, like `collector` and `query`.
<3> The options for the `create-schema` job.
{{< info >}}
The default create-schema job uses `MODE=prod`, which implies a replication factor of `2`, using `NetworkTopologyStrategy` as the class, effectively meaning that at least 3 nodes are required in the Cassandra cluster. If a `SimpleStrategy` is desired, set the mode to `test`, which then sets the replication factor of `1`. Refer to the [create-schema script](https://github.com/jaegertracing/jaeger/blob/master/plugin/storage/cassandra/schema/create.sh) for more details.
{{< /info >}}
### Elasticsearch storage
Under some circumstances, the Jaeger Operator can make use of the [Elasticsearch Operator](https://github.com/openshift/elasticsearch-operator) to provision a suitable Elasticsearch cluster.
{{< warning >}}
This feature is only tested on OpenShift clusters. Spark dependencies are not supported with this feature [Issue #294](https://github.com/jaegertracing/jaeger-operator/issues/294).
{{< /warning >}}
When there are no `es.server-urls` options as part of a Jaeger `production` instance and `elasticsearch` is set as the storage type, the Jaeger Operator creates an Elasticsearch cluster via the Elasticsearch Operator by creating a Custom Resource based on the configuration provided in storage section. The Elasticsearch cluster is meant to be dedicated for a single Jaeger instance.
The self-provision of an Elasticsearch cluster can be disabled by setting the flag `--es-provision` to `false`. The default value is `auto`, which will make the Jaeger Operator query the Kubernetes cluster for its ability to handle a `Elasticsearch` custom resource. This is usually set by the Elasticsearch Operator during its installation process, so, if the Elasticsearch Operator is expected to run *after* the Jaeger Operator, the flag can be set to `true`.
{{< danger >}}
At the moment there can be only one Jaeger with self-provisioned Elasticsearch instance per namespace.
{{< /danger >}}
#### Elasticsearch index cleaner job
When using `elasticsearch` storage by default a job is created to clean old traces from it, the options for it are listed below so you can configure it to your use case
```yaml
storage:
type: elasticsearch
esIndexCleaner:
enabled: false # turn the job deployment on and off
numberOfDays: 7 # number of days to wait before deleting a record
schedule: "55 23 * * *" # cron expression for it to run
image: jaegertracing/jaeger-es-index-cleaner # image of the job
```
## Deriving dependencies
The processing to derive dependencies will collect spans from storage, analyzes links between services and store them for later presentation in the UI.
This job can only be used with the `production` strategy and storage type `cassandra` or `elasticsearch`.
```yaml
storage:
type: elasticsearch
dependencies:
enabled: true # turn the job deployment on and off
schedule: "55 23 * * *" # cron expression for it to run
sparkMaster: # spark master connection string, when empty spark runs in embedded local mode
```
The connection configuration to storage is derived from storage options.
## Auto-injecting Jaeger Agent Sidecars
The operator can inject Jaeger Agent sidecars in `Deployment` workloads, provided that the deployment has the annotation `sidecar.jaegertracing.io/inject` with a suitable value. The values can be either `"true"` (as string), or the Jaeger instance name, as returned by `kubectl get jaegers`. When `"true"` is used, there should be exactly *one* Jaeger instance for the same namespace as the deployment, otherwise, the operator can't figure out automatically which Jaeger instance to use.
The following snippet shows a simple application that will get a sidecar injected, with the Jaeger Agent pointing to the single Jaeger instance available in the same namespace:
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
annotations:
"sidecar.jaegertracing.io/inject": "true" # <1>
spec:
selector:
matchLabels:
app: myapp
template:
metadata:
labels:
app: myapp
spec:
containers:
- name: myapp
image: acme/myapp:myversion
```
<1> Either `"true"` (as string) or the Jaeger instance name.
A complete sample deployment is available at [`deploy/examples/business-application-injected-sidecar.yaml`](https://github.com/jaegertracing/jaeger-operator/blob/master/examples/business-application-injected-sidecar.yaml).
When the sidecar is injected, the Jaeger Agent can be accessed at its default location on `localhost`.
## Installing the Agent as DaemonSet
By default, the Operator expects the agents to be deployed as sidecars to the target applications. This is convenient for several purposes, like in a multi-tenant scenario or to have better load balancing, but there are scenarios where you might want to install the agent as a `DaemonSet`. In that case, specify the Agent's strategy to `DaemonSet`, as follows:
```yaml
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: my-jaeger
spec:
agent:
strategy: DaemonSet
```
{{< danger >}}
If you attempt to install two Jaeger instances on the same cluster with `DaemonSet` as the strategy, only *one* will end up deploying a `DaemonSet`, as the agent is required to bind to well-known ports on the node. Because of that, the second daemon set will fail to bind to those ports.
{{< /danger >}}
Your tracer client will then most likely need to be told where the agent is located. This is usually done by setting the environment variable `JAEGER_AGENT_HOST` to the value of the Kubernetes node's IP, for example:
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
spec:
selector:
matchLabels:
app: myapp
template:
metadata:
labels:
app: myapp
spec:
containers:
- name: myapp
image: acme/myapp:myversion
env:
- name: JAEGER_AGENT_HOST
valueFrom:
fieldRef:
fieldPath: status.hostIP
```
### OpenShift
In OpenShift, a `HostPort` can only be set when a special security context is set. A separate service account can be used by the Jaeger Agent with the permission to bind to `HostPort`, as follows:
```bash
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/examples/openshift/hostport-scc-daemonset.yaml # <1>
oc new-project myappnamespace
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/examples/openshift/service_account_jaeger-agent-daemonset.yaml # <2>
oc adm policy add-scc-to-user daemonset-with-hostport -z jaeger-agent-daemonset # <3>
oc apply -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/examples/openshift/agent-as-daemonset.yaml # <4>
```
<1> The `SecurityContextConstraints` with the `allowHostPorts` policy
<2> The `ServiceAccount` to be used by the Jaeger Agent
<3> Adds the security policy to the service account
<4> Creates the Jaeger Instance using the `serviceAccount` created in the steps above
{{< warning >}}
Without such a policy, errors like the following will prevent a `DaemonSet` to be created: `Warning FailedCreate 4s (x14 over 45s) daemonset-controller Error creating: pods "agent-as-daemonset-agent-daemonset-" is forbidden: unable to validate against any security context constraint: [spec.containers[0].securityContext.containers[0].hostPort: Invalid value: 5775: Host ports are not allowed to be used`
{{< /warning >}}
After a few seconds, the `DaemonSet` should be up and running:
```bash
$ oc get daemonset agent-as-daemonset-agent-daemonset
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE
agent-as-daemonset-agent-daemonset 1 1 1 1 1
```
## Secrets Support
The Operator supports passing secrets to the Collector, Query and All-In-One deployments. This can be used for example, to pass credentials (username/password) to access the underlying storage backend (for example: Elasticsearch).
The secrets are available as environment variables in the (Collector/Query/All-In-One) nodes.
```yaml
storage:
type: elasticsearch
options:
es:
server-urls: http://elasticsearch:9200
secretName: jaeger-secrets
```
The secret itself would be managed outside of the `jaeger-operator` custom resource.
## Configuring the UI
Information on various configuration options for the UI can be found [here](../frontend-ui/#configuration), defined in json format.
To apply UI configuration changes within the Custom Resource, the same information can be included in yaml format as shown below:
```yaml
ui:
options:
dependencies:
menuEnabled: false
tracking:
gaID: UA-000000-2
menu:
- label: "About Jaeger"
items:
- label: "Documentation"
url: "https://www.jaegertracing.io/docs/latest"
linkPatterns:
- type: "logs"
key: "customer_id"
url: /search?limit=20&lookback=1h&service=frontend&tags=%7B%22customer_id%22%3A%22#{customer_id}%22%7D
text: "Search for other traces for customer_id=#{customer_id}"
```
## Defining Sampling Strategies
{{< info >}}
This is not relevant if a trace was started by the Istio proxy as the sampling decision is made there. And the Jaeger sampling decisions are only relevant when you are using the Jaeger tracer (client).
{{< /info >}}
The operator can be used to define sampling strategies that will be supplied to tracers that have been configured to use a remote sampler:
```yaml
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: with-sampling
spec:
strategy: allInOne
sampling:
options:
default_strategy:
type: probabilistic
param: 0.5
```
This example defines a default sampling strategy that is probabilistic, with a 50% chance of the trace instances being sampled.
Refer to the Jaeger documentation on [Collector Sampling Configuration](https://www.jaegertracing.io/docs/latest/sampling/#collector-sampling-configuration) to see how service and endpoint sampling can be configured. The JSON representation described in that documentation can be used in the operator by converting to YAML.
## Finer grained configuration
The custom resource can be used to define finer grained Kubernetes configuration applied to all Jaeger components or at the individual component level.
When a common definition (for all Jaeger components) is required, it is defined under the `spec` node. When the definition relates to an individual component, it is placed under the `spec/<component>` node.
The types of supported configuration include:
* [affinity](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity) to determine which nodes a pod can be allocated to
* [annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/)
* [labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/)
* [resources](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container) to limit cpu and memory
* [tolerations](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/) in conjunction with `taints` to enable pods to avoid being repelled from a node
* [volumes](https://kubernetes.io/docs/concepts/storage/volumes/) and volume mounts
* [serviceAccount](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/) to run each component with separate identity
* [securityContext](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/) to define privileges of running components
```yaml
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: simple-prod
spec:
strategy: production
storage:
type: elasticsearch
options:
es:
server-urls: http://elasticsearch:9200
annotations:
key1: value1
labels:
key2: value2
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/e2e-az-name
operator: In
values:
- e2e-az1
- e2e-az2
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 1
preference:
matchExpressions:
- key: another-node-label-key
operator: In
values:
- another-node-label-value
tolerations:
- key: "key1"
operator: "Equal"
value: "value1"
effect: "NoSchedule"
- key: "key1"
operator: "Equal"
value: "value1"
effect: "NoExecute"
serviceAccount: nameOfServiceAccount
securityContext:
runAsUser: 1000
volumeMounts:
- name: config-vol
mountPath: /etc/config
volumes:
- name: config-vol
configMap:
name: log-config
items:
- key: log_level
path: log_level
```
# Accessing the Jaeger Console (UI)
<!-- TODO Add tabs shortcode -->
## Kubernetes
The operator creates a Kubernetes [`ingress`](https://kubernetes.io/docs/concepts/services-networking/ingress/) route, which is the Kubernetes' standard for exposing a service to the outside world, but by default it does not come with Ingress providers.
Check the [Kubernetes documentation](https://kubernetes.github.io/ingress-nginx/deploy/#verify-installation) for the most appropriate way to achieve an Ingress provider for your platform. The following command enables the Ingress provider on `minikube`:
```bash
minikube addons enable ingress
```
Once Ingress is enabled, the address for the Jaeger console can be found by querying the Ingress object:
```bash
$ kubectl get ingress
NAME HOSTS ADDRESS PORTS AGE
simplest-query * 192.168.122.34 80 3m
```
In this example, the Jaeger UI is available at http://192.168.122.34.
## OpenShift
When the Operator is running on OpenShift, the Operator will automatically create a `Route` object for the query services. Use the following command to check the hostname/port:
```bash
oc get routes
```
{{< info >}}
Make sure to use `https` with the hostname/port you get from the command above, otherwise you'll see a message like: "Application is not available".
{{< /info >}}
By default, the Jaeger UI is protected with OpenShift's OAuth service and any valid user is able to login. To disable this feature and leave the Jaeger UI unsecured, set the Ingress property `security` to `none` in the custom resource file:
```yaml
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: disable-oauth-proxy
spec:
ingress:
security: none
```
Custom `SAR` and `Delegate URL` values can be specified as part of the `.Spec.Ingress.OpenShift.SAR` and `.Spec.Ingress.Openshift.DelegateURLs`, as follows:
```yaml
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: custom-sar-oauth-proxy
spec:
ingress:
openshift:
sar: '{"namespace": "default", "resource": "pods", "verb": "get"}'
delegateUrls: '{"/":{"namespace": "default", "resource": "pods", "verb": "get"}}'
```
When the `delegateUrls` is set, the Jaeger Operator needs to create a new `ClusterRoleBinding` between the service account used by the UI Proxy (`{InstanceName}-ui-proxy`) and the role `system:auth-delegator`, as required by the OpenShift OAuth Proxy. Because of that, the service account used by the operator itself needs to have the same cluster role binding. To accomplish that, a `ClusterRoleBinding` such as the following has to be created:
```yaml
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: jaeger-operator-with-auth-delegator
namespace: observability
subjects:
- kind: ServiceAccount
name: jaeger-operator
namespace: observability
roleRef:
kind: ClusterRole
name: system:auth-delegator
apiGroup: rbac.authorization.k8s.io
```
Cluster administrators not comfortable in letting users deploy Jaeger instances with this cluster role are free to not add this cluster role to the operator's service account. In that case, the Operator will auto-detect that the required permissions are missing and will log a message similar to: `the requested instance specifies the delegateUrls option for the OAuth Proxy, but this operator cannot assign the proper cluster role to it (system:auth-delegator). Create a cluster role binding between the operator's service account and the cluster role 'system:auth-delegator' in order to allow instances to use 'delegateUrls'`.
The Jaeger Operator also supports authentication using `htpasswd` files via the OpenShift OAuth Proxy. To make use of that, specify the `htpasswdFile` option within the OpenShift-specific entries, pointing to the file `htpasswd` file location in the local disk. The `htpasswd` file can be created using the `htpasswd` utility:
```console
$ htpasswd -cs /tmp/htpasswd jdoe
New password:
Re-type new password:
Adding password for user jdoe
```
This file can then be used as the input for the `kubectl create secret` command:
```console
$ kubectl create secret generic htpasswd --from-file=htpasswd=/tmp/htpasswd
secret/htpasswd created
```
Once the secret is created, it can be specified in the Jaeger CR as a volume/volume mount:
```yaml
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: with-htpasswd
spec:
ingress:
openshift:
sar: '{"namespace": "default", "resource": "pods", "verb": "get"}'
htpasswdFile: /usr/local/data/htpasswd
volumeMounts:
- name: htpasswd-volume
mountPath: /usr/local/data
volumes:
- name: htpasswd-volume
secret:
secretName: htpasswd
```
# Upgrading the Operator and its managed instances
Each version of the Jaeger Operator follows one Jaeger version. Whenever a new version of the Jaeger Operator is installed, all the Jaeger instances managed by the operator will be upgraded to the Operator's supported version. For example, an instance named `simplest` that was created with Jaeger Operator 1.12.0 will be running Jaeger 1.12.0. Once the Jaeger Operator is upgraded to 1.13.0, the instance `simplest` will be upgraded to the version 1.13.0, following the official upgrade instructions from the Jaeger project.
The Jaeger Operator can be upgraded manually by changing the deployment (`kubectl edit deployment jaeger-operator`), or via specialized tools such as the [Operator Lifecycle Manager (OLM)](https://github.com/operator-framework/operator-lifecycle-manager).
# Updating a Jaeger instance (experimental)
A Jaeger instance can be updated by changing the `CustomResource`, either via `kubectl edit jaeger simplest`, where `simplest` is the Jaeger's instance name, or by applying the updated YAML file via `kubectl apply -f simplest.yaml`.
{{< danger >}}
The name of the Jaeger instance cannot be updated, as it is part of the identifying information for the resource.
{{< /danger >}}
Simpler changes such as changing the replica sizes can be applied without much concern, whereas changes to the strategy should be watched closely and might potentially cause an outage for individual components (collector/query/agent).
While changing the backing storage is supported, migration of the data is not.
# Removing a Jaeger instance
<!-- TODO Add OKD/OpenShift commands and tabs shortcode-->
To remove an instance, use the `delete` command with the custom resource file used when you created the instance:
```bash
kubectl delete -f simplest.yaml
```
Alternatively, you can remove a Jaeger instance by running:
```bash
kubectl delete jaeger simplest
```
{{< info >}}
Deleting the instance will not remove the data from any permanent storage used with this instance. Data from in-memory instances, however, will be lost.
{{< /info >}}
# Monitoring the operator
The Jaeger Operator starts a Prometheus-compatible endpoint on `0.0.0.0:8383/metrics` with internal metrics that can be used to monitor the process.
{{< info >}}
The Jaeger Operator does not yet publish its own metrics. Rather, it makes available metrics reported by the components it uses, such as the Operator SDK.
{{< /info >}}
# Uninstalling the operator
<!-- TODO Add OKD/OpenShift commands and tabs shortcode -->
To uninstall the operator, run the following commands:
```bash
kubectl delete -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/operator.yaml
kubectl delete -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role_binding.yaml
kubectl delete -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role.yaml
kubectl delete -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/service_account.yaml
kubectl delete -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/crds/jaegertracing.io_jaegers_crd.yaml
```

View File

@ -0,0 +1,95 @@
---
title: Performance Tuning Guide
description: Tweaking your Jaeger instance to achieve a better performance
weight: 10
---
Jaeger was built from day 1 to be able to ingest huge amounts of data in a resilient way. To better utilize resources that might cause delays, such as storage or network communications, Jaeger buffers and batches data. When more spans are generated than Jaeger is able to safely process, spans might get dropped. However, the defaults might not fit all scenarios: for instance, Agents running as a sidecar might have more memory constraints than Agents running as a daemon in bare metal.
## Deployment considerations
Although performance tuning the individual components is important, the way Jaeger is deployed can be decisive in obtaining optimal performance.
### Scale the Collector up and down
Use the auto-scaling capabilities of your platform: the Collector is nearly horizontally scalable so that more instances can be added and removed on-demand. A good way to scale up and down is by checking the `jaeger_collector_queue_length` metric: add instances when the length is higher than 50% of the maximum size for extended periods of time. Another metric that can be taken into consideration is `jaeger_collector_in_queue_latency_bucket`, which is a histogram indicating how long spans have been waiting in the queue before a worker picked it up. When the queue latency gets higher over time, its a good indication to increase the number of the workers, or to improve the storage performance.
Adding Collector instances is recommended when your platform provides auto-scaling capabilities, or when it's easier to start/stop Collector instances than changing existing, running instances. Scaling horizontally is also indicated when the CPU usage should be spread across nodes.
### Make sure the storage can keep up
Each span is written to the storage by the Collector using one worker, blocking it until the span has been stored. When the storage is too slow, the number of workers blocked by the storage might be too high, causing spans to be dropped. To help diagnose this situation, the histogram `jaeger_collector_save_latency_bucket` can be analyzed. Ideally, the latency should remain the same over time. When the histogram shows that most spans are taking longer and longer over time, its a good indication that your storage might need some attention.
### Place the Agents close to your applications
The Agent is meant to be placed on the same host as the instrumented application, in order to avoid UDP packet loss over the network. This is typically accomplished by having one Agent per bare metal host for traditional applications, or as a sidecar in container environments like Kubernetes, as this helps spread the load handled by Agents with the additional advantage of allowing each Agent to be tweaked individually, according to the applications needs and importance.
### Consider using Apache Kafka as intermediate buffer
Jaeger [can use Apache Kafka](../architecture/) as a buffer between the Collector and the actual backing storage (Elasticsearch, Apache Cassandra). This is ideal for cases where the traffic spikes are relatively frequent (prime time traffic) but the storage can eventually catch up once the traffic normalizes. For that, the `SPAN_STORAGE_TYPE` environment variable should be set to `kafka` in the Collector and the Jaeger Ingester component can be used, reading data from Kafka and writing it to the storage.
In addition to the performance aspects, having spans written to Kafka is useful for building real time data pipeline for aggregations and feature extraction from traces.
## Client (Tracer) settings
The Jaeger Clients are built to have minimal effect to the instrumented application. As such, it has conservative defaults that might not be suitable for all cases. Note that Jaeger Clients can be configured programmatically or via [environment variables](../client-features/).
### Adjust the sampling configuration
Together, the `JAEGER_SAMPLER_TYPE` and `JAEGER_SAMPLER_PARAM` specify how often traces should be "sampled", ie, recorded and sent to the Jaeger backend. For applications generating a large number of spans, setting the sampling type to `probabilistic` and the value to `0.001` (the default) will cause traces to be reported with a 1/1000th chance. Note that the sampling decision is made at the root span and propagated down to all child spans.
For applications with low to medium traffic, setting the sampling type to `const` and value to `1` will cause all spans to be reported. Similarly, tracing can be disabled by setting the value to `0`, while context propagation will continue to work.
Some Clients support the setting `JAEGER_DISABLED` to completely disable the Jaeger Tracer. This is recommended only if the Tracer is behaving in a way that causes problems to the instrumented application, as it will not propagate the context to the downstream services.
{{< info >}}
We recommend to set your clients to use the [`remote` sampling strategy](../sampling/#collector-sampling-configuration), so that admins can centrally set the concrete sampling strategy for each service.
{{< /info >}}
### Increase in-memory queue size
Most of the Jaeger clients, such as the Java, Go, and C# clients, buffer spans in memory before sending them to the Jaeger Agent/Collector. The maximum size of this buffer is defined by the environment variable `JAEGER_REPORTER_MAX_QUEUE_SIZE` (default value: about `100` spans): the larger the size, the higher the potential memory consumption. When the instrumented application is generating a large number of spans, its possible that the queue will be full causing the Client to discard the new spans (metric `jaeger_tracer_reporter_spans_total{result="dropped",}`).
In most common scenarios, the queue will be close to empty (metric: `jaeger_tracer_reporter_queue_length`), as spans are flushed to the Agent or Collector at regular intervals or when a certain size of the batch is reached. The detailed behavior of this queue is described in this [GitHub issue](https://github.com/jaegertracing/jaeger-client-java/issues/607).
### Modify the batched spans flush interval
The Java, Go, NodeJS, Python and C# Clients allow the customization of the flush interval (default value: `1000` milliseconds, or 1 second) used by the reporters, such as the `RemoteReporter`, to trigger a `flush` operation, sending all in-memory spans to the Agent or Collector. The lower the flush interval is set to, the more frequent the flush operations happen. As most reporters will wait until enough data is in the queue, this setting will force a flush operation at periodic intervals, so that spans are sent to the backend in a timely fashion.
When the instrumented application is generating a large number of spans and the Agent/Collector is close to the application, the networking overhead might be low, justifying a higher number of flush operations. When the `HttpSender` is being used and the Collector is not close enough to the application, the networking overhead might be too high so that a higher value for this property makes sense.
## Agent settings
Jaeger Agents receive data from Clients, sending them in batches to the Collector. When not properly configured, it might end up discarding data even if the host machine has plenty of resources.
### Adjust server queue sizes
The set of “server queue size” properties ( `processor.jaeger-binary.server-queue-size`, `processor.jaeger-compact.server-queue-size`, `processor.zipkin-compact.server-queue-size`) indicate the maximum number of span batches that the Agent can accept and store in memory. Its safe to assume that `jaeger-compact` is the most important processor in your Agent setup, as its the only one available in most Clients, such as the Java and Go Clients.
The default value for each queue is `1000` span batches. Given that each span batch has up to 64KiB worth of spans, each queue can hold up to 64MiB worth of spans.
In typical scenarios, the queue will be close to empty (metric `jaeger_agent_thrift_udp_server_queue_size`) as span batches should be quickly picked up and processed by a worker. However, sudden spikes in the number of span batches submitted by Clients might occur, causing the batches to be queued. When the queue is full, the older batches are overridden causing spans to be discarded (metric `jaeger_agent_thrift_udp_server_packets_dropped_total`).
### Adjust processor workers
The set of “processor workers” properties ( `processor.jaeger-binary.workers`, `processor.jaeger-compact.workers`, `processor.zipkin-compact.workers`) indicate the number of parallel span batch processors to start. Each worker type has a default size of `10`. In general, span batches are processed as soon as they are placed in the server queue and will block a worker until the whole packet is sent to the Collector. For Agents processing data from multiple Clients, the number of workers should be increased. Given that the cost of each worker is low, a good rule of thumb is 10 workers per Client with moderate traffic: given that each span batch might contain up to 64KiB worth of spans, it means that 10 workers are able to send about 640KiB concurrently to a Collector.
## Collector settings
The Collector receives data from Clients and Agents. When not properly configured, it might process less data than what would be possible on the same host, or it might overload the host by consuming more memory than permitted.
### Adjust queue size
Similar to the Agent, the Collector is able to receive spans and place them in an internal queue for processing. This allows the Collector to return immediately to the Client/Agent instead of waiting for the span to make its way to the storage.
The setting `collector.queue-size` (default: `2000`) dictates how many spans the queue should support. In the typical scenario, the queue will be close to empty, as enough workers should exist picking up spans from the queue and sending them to the storage. When the number of items in the queue (metric `jaeger_collector_queue_length`) is permanently high, its an indication that either the number of workers should be increased or that the storage cannot keep up with the volume of data that its receiving. When the queue is full, the older items in the queue are overridden, causing spans to be discarded (metric `jaeger_collector_spans_dropped_total`).
{{< warning >}}
The queue size for the Agent is about _span batches_, whereas the queue size for the Collector is about _individual spans_.
{{< /warning >}}
Given that the queue size should be close to empty most of the time, this setting should be as high as the available memory for the Collector, to provide maximum protection against sudden traffic spikes. However, if your storage layer is under-provisioned and cannot keep up, even a large queue will quickly fill up and start dropping data.
### Adjust processor workers
Items from the span queue in the Collector are picked up by workers. Each worker picks one span from the queue and persists it to the storage. The number of workers can be specified by the setting `collector.num-workers` (default: `50`) and should be as high as needed to keep the queue close to zero. The general rule is: the faster the backing storage, the lower the number of workers can be. Given that workers are relatively cheap, this number can be increased at will. As a general rule, one worker per 50 items in the queue should be sufficient when the storage is fast. With a `collector.queue-size` of `2000`, having about `40` workers should be sufficient. For slower storage mechanisms, this ratio should be adjusted accordingly, having more workers per queue item.

View File

@ -0,0 +1,66 @@
---
title: Sampling
hasparent: true
---
Jaeger libraries implement consistent upfront (or head-based) sampling. For example, assume we have a simple call graph where service A calls service B, and B calls service C: `A -> B -> C`. When service A receives a request that contains no tracing information, Jaeger tracer will start a new {{< tip "trace" >}}, assign it a random trace ID, and make a sampling decision based on the currently installed sampling strategy. The sampling decision will be propagated with the requests to B and to C, so those services will not be making the sampling decision again but instead will respect the decision made by the top service A. This approach guarantees that if a trace is sampled, all its {{< tip "spans" "span" >}} will be recorded in the backend. If each service was making its own sampling decision we would rarely get complete traces in the backend.
## Client Sampling Configuration
When using configuration object to instantiate the tracer, the type of sampling can be selected via `sampler.type` and `sampler.param` properties. Jaeger libraries support the following samplers:
* **Constant** (`sampler.type=const`) sampler always makes the same decision for all traces. It either samples all traces (`sampler.param=1`) or none of them (`sampler.param=0`).
* **Probabilistic** (`sampler.type=probabilistic`) sampler makes a random sampling decision with the probability of sampling equal to the value of `sampler.param` property. For example, with `sampler.param=0.1` approximately 1 in 10 traces will be sampled.
* **Rate Limiting** (`sampler.type=ratelimiting`) sampler uses a leaky bucket rate limiter to ensure that traces are sampled with a certain constant rate. For example, when `sampler.param=2.0` it will sample requests with the rate of 2 traces per second.
* **Remote** (`sampler.type=remote`, which is also the default) sampler consults Jaeger agent for the appropriate sampling strategy to use in the current service. This allows controlling the sampling strategies in the services from a [central configuration](#collector-sampling-configuration) in Jaeger backend, or even dynamically (see [Adaptive Sampling](https://github.com/jaegertracing/jaeger/issues/365)).
### Adaptive Sampler
Adaptive sampler is a composite sampler that combines two functions:
* It makes sampling decisions on a per-operation basis, i.e. based on {{< tip "span" >}} operation name. This is especially useful in the API services whose endpoints may have very different traffic volumes and using a single probabilistic sampler for the whole service might starve (never sample) some of the low QPS endpoints.
* It supports a minimum guaranteed rate of sampling, such as always allowing up to N {{< tip "traces" "trace" >}} per seconds and then sampling anything above that with a certain probability (everything is per-operation, not per-service).
Per-operation parameters can be configured statically or pulled periodically from Jaeger backend with the help of Remote sampler. Adaptive sampler is designed to work with the upcoming [Adaptive Sampling](https://github.com/jaegertracing/jaeger/issues/365) feature of the Jaeger backend.
## Collector Sampling Configuration
Collectors can be instantiated with static sampling strategies (which are propagated to the respective service if configured with Remote sampler) via the `--sampling.strategies-file` option. This option requires a path to a json file which have the sampling strategies defined.
Example `strategies.json`
```json
{
"service_strategies": [
{
"service": "foo",
"type": "probabilistic",
"param": 0.8,
"operation_strategies": [
{
"operation": "op1",
"type": "probabilistic",
"param": 0.2
},
{
"operation": "op2",
"type": "probabilistic",
"param": 0.4
}
]
},
{
"service": "bar",
"type": "ratelimiting",
"param": 5
}
],
"default_strategy": {
"type": "probabilistic",
"param": 0.5
}
}
```
`service_strategies` defines service specific sampling strategies and `operation_strategies` defines operation specific sampling strategies. There are 2 types of strategies possible: `probabilistic` and `ratelimiting` which are described [above](#client-sampling-configuration) (NOTE: `ratelimiting` is not supported for `operation_strategies`). `default_strategy` defines the catch-all sampling strategy that is propagated if the service is not included as part of `service_strategies`.
In the above example, all service `foo` operations are sampled probabilistically with a probability of 0.8 except `op1` and `op2` which are probabilistically sampled with a probability of 0.2 and 0.4 respectively. All operations for service `bar` are ratelimited at 5 traces per second. Any other service is probabilistically sampled with a probability of 0.5.

View File

@ -0,0 +1,70 @@
---
title: Introduction
weight: 1
children:
- title: Features
url: features
---
Welcome to Jaeger's documentation portal! Below, you'll find information for beginners and experienced Jaeger users.
If you can't find what you are looking for, or have an issue not covered here, we'd love to [hear from you](/get-in-touch).
## About
Jaeger, inspired by [Dapper][dapper] and [OpenZipkin](http://zipkin.io),
is a distributed tracing system released as open source by [Uber Technologies][ubeross].
It is used for monitoring and troubleshooting microservices-based distributed systems, including:
* Distributed context propagation
* Distributed transaction monitoring
* Root cause analysis
* Service dependency analysis
* Performance / latency optimization
Uber published a blog post, [Evolving Distributed Tracing at Uber](https://eng.uber.com/distributed-tracing/), where they explain the history and reasons for the architectural choices made in Jaeger. [Yuri Shkuro](https://shkuro.com), creator of Jaeger, also published a book [Mastering Distributed Tracing](https://shkuro.com/books/2019-mastering-distributed-tracing/) that covers in-depth many aspects of Jaeger design and operation, as well as distributed tracing in general.
## Features
* [OpenTracing](http://opentracing.io/) compatible data model and instrumentation libraries
* in [Go](https://github.com/jaegertracing/jaeger-client-go), [Java](https://github.com/jaegertracing/jaeger-client-java), [Node](https://github.com/jaegertracing/jaeger-client-node), [Python](https://github.com/jaegertracing/jaeger-client-python)
and [C++](https://github.com/jaegertracing/cpp-client)
* Uses consistent upfront sampling with individual per service/endpoint probabilities
* Multiple storage backends: Cassandra, Elasticsearch, memory.
* Adaptive sampling (coming soon)
* Post-collection data processing pipeline (coming soon)
See [Features](./features/) page for more details.
## Technical Specs
* Backend components implemented in Go
* React/Javascript UI
* Supported storage backends:
* [Cassandra 3.4+](./deployment/#cassandra)
* [Elasticsearch 5.x, 6.x, 7.x](./deployment/#elasticsearch)
* [Kafka](./deployment/#kafka)
* memory storage
## Quick Start
See [running a docker all in one image](getting-started#all-in-one).
## Screenshots
### Traces View
[![Traces View](/img/traces-ss.png)](/img/traces-ss.png)
### Trace Detail View
[![Detail View](/img/trace-detail-ss.png)](/img/trace-detail-ss.png)
## Related links
- [Evolving Distributed tracing At Uber Engineering](https://eng.uber.com/distributed-tracing/)
- [Mastering Distributed Tracing](https://shkuro.com/books/2019-mastering-distributed-tracing/)
- [Tracing HTTP request latency in Go with OpenTracing](https://medium.com/opentracing/tracing-http-request-latency-in-go-with-opentracing-7cc1282a100a)
- [Distributed Tracing with Jaeger & Prometheus on Kubernetes](https://blog.openshift.com/openshift-commons-briefing-82-distributed-tracing-with-jaeger-prometheus-on-kubernetes/)
- [Using Jaeger with Istio](https://istio.io/docs/tasks/telemetry/distributed-tracing.html)
- [Using Jaeger with Envoy](https://www.envoyproxy.io/docs/envoy/latest/start/sandboxes/jaeger_tracing.html)
[dapper]: https://research.google.com/pubs/pub36356.html
[ubeross]: http://uber.github.io

87
content/docs/1.15/apis.md Normal file
View File

@ -0,0 +1,87 @@
---
title: APIs
hasparent: true
---
Jaeger components implement various APIs for saving or retrieving trace data.
The following labels are used to describe API compatibility guarantees.
* **stable** - the API guarantees backwards compatibility. If breaking changes are going to be made in the future, they will result in a new API version, e.g. `/api/v2` URL prefix or a different namespace in the IDL.
* **internal** - the APIs intended for internal communications between Jaeger components and not recommended for use by external components.
* **deprecated** - the APIs that are only maintained for legacy reasons and will be phased out in the future.
## Span reporting APIs
Agent and Collector are the two components of the Jaeger backend that can receive spans. At this time they support two sets of non-overlapping APIs.
### Thrift over UDP (stable)
The Agent can only receive spans over UDP in Thrift format. The primary API is a UDP packet that contains a Thrift-encoded `Batch` struct defined in [jaeger.thrift][jaeger.thrift] IDL file, located in the [jaeger-idl][jaeger-idl] repository. Most Jaeger Clients use Thrift's `compact` encoding, however some client libraries do not support it (notably, Node.js) and use Thrift's `binary` encoding (sent to a different UDP port). The Agent's API is defined by [agent.thrift][agent.thrift] IDL file.
For legacy reasons, the Agent also accepts spans in Zipkin format, however, only very old versions of Jaeger clients can send data in that format and it is officially deprecated.
### Protobuf via gRPC (stable)
In a typical Jaeger deployment, Agents receive spans from Clients and forward them to Collectors. Since Jaeger version 1.11 the official and recommended protocol between Agents and Collectors is gRPC with Protobuf.
The Protobuf IDL [collector.proto][collector.proto] is currently located in the main Jaeger repository, under [model/proto/api_v2][collector.proto]. In the future it will be moved to [jaeger-idl][jaeger-idl] repository ([jaeger-idl/issues/55](https://github.com/jaegertracing/jaeger-idl/issues/55)).
### Thrift over HTTP (stable)
In some cases it is not feasible to deploy Jaeger Agent next to the application, for example, when the application code is running as AWS Lambda function. In these scenarios the Jaeger Clients can be configured to submit spans directly to the Collectors over HTTP/HTTPS.
The same [jaeger.thrift][jaeger.thrift] payload can be submitted in HTTP POST request to `/api/traces` endpoint, for example, `https://jaeger-collector:14268/api/traces`. The `Batch` struct needs to be encoded using Thrift's `binary` encoding, and the HTTP request should specify the content type header:
```
Content-Type: application/vnd.apache.thrift.binary
```
### JSON over HTTP (n/a)
There is no official Jaeger JSON format that can be accepted by the collector. In the future the Protobuf-generated JSON may be supported.
### Thrift via TChannel (deprecated)
Agent and Collector can communicate using TChannel protocol. This protocol is generally not supported by the routing infrastructure and has been deprecated. It will be eventually removed from Jaeger.
### Zipkin Formats (stable)
Jaeger Collector can also accept spans in several Zipkin data format, namely JSON v1/v2 and Thrift. The Collector needs to be configured to enable Zipkin HTTP server, e.g. on port 9411 used by Zipkin collectors. The server enables two endpoints that expect POST requests:
* `/api/v1/spans` for submitting spans in Zipkin JSON v1 or Zipkin Thrift format.
* `/api/v2/spans` for submitting spans in Zipkin JSON v2.
## Trace retrieval APIs
Traces saved in the storage can be retrieved by calling Jaeger Query Service.
### gRPC/Protobuf (stable)
The recommended way for programmatically retrieving traces and other data is via gRPC endpoint defined in [query.proto][query.proto] IDL file (located in the main Jaeger repository, similar to [collector.proto][collector.proto]).
### HTTP JSON (internal)
Jaeger UI communicates with Jaeger Query Service via JSON API. For example, a trace can be retrieved via GET request to `https://jaeger-query:16686/api/traces/{trace-id-hex-string}`. This JSON API is intentionally undocumented and subject to change.
## Clients configuration (internal)
Client libraries not only submit finished spans to Jaeger backend, but also periodically poll the Agents for various configurations, such as sampling strategies. The schema for the payload is defined by [sampling.thrift][sampling.thrift], encoded as JSON using Thrift's built-in JSON generation capabilities.
## Service dependencies graph (internal)
Can be retrieved from Query Service at `/api/dependencies` endpoint. The GET request expects two parameters:
* `endTs` (number of milliseconds since epoch) - the end of the time interval
* `lookback` (in milliseconds) - the length the time interval (i.e. start-time + lookback = end-time).
The returned JSON is a list of edges represented as tuples `(caller, callee, count)`.
For programmatic access to service graph, the recommended API is gRPC/Protobuf described above.
[jaeger-idl]: https://github.com/jaegertracing/jaeger-idl/
[jaeger.thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/jaeger.thrift
[agent.thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/agent.thrift
[sampling.thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/sampling.thrift
[collector.proto]: https://github.com/jaegertracing/jaeger/blob/master/model/proto/api_v2/collector.proto
[query.proto]: https://github.com/jaegertracing/jaeger/blob/master/model/proto/api_v2/query.proto

13
content/docs/1.15/cli.md Normal file
View File

@ -0,0 +1,13 @@
---
title: CLI flags
widescreen: true
hasparent: true
---
This is auto-generated documentation for CLI flags supported by Jaeger binaries.
* CLI flags for some binaries change depending on the `SPAN_STORAGE_TYPE` environment variable. Relevant variations are included below.
* Some binaries support _commands_ (mostly informational), such as `env`, `docs`, and `version`. These commands are not included here.
* All parameters can be also provided via environment variables, by changing all letters to upper-case and replacing all punctuation characters with underscore `_`. For example, the value for the flag `--cassandra.connections-per-host` can be provided via `CASSANDRA_CONNECTIONS_PER_HOST` environment variable.
{{< cli/tools-list >}}

View File

@ -0,0 +1,469 @@
---
title: Deployment
weight: 4
children:
- title: Operator for Kubernetes
navtitle: Kubernetes
url: operator
- title: Frontend/UI
url: frontend-ui
- title: Windows
url: windows
- title: CLI Flags
url: cli
---
The main Jaeger backend components are released as Docker images on Docker Hub:
Component | Repository
--------------------- | ---
**jaeger-agent** | [hub.docker.com/r/jaegertracing/jaeger-agent/](https://hub.docker.com/r/jaegertracing/jaeger-agent/)
**jaeger-collector** | [hub.docker.com/r/jaegertracing/jaeger-collector/](https://hub.docker.com/r/jaegertracing/jaeger-collector/)
**jaeger-query** | [hub.docker.com/r/jaegertracing/jaeger-query/](https://hub.docker.com/r/jaegertracing/jaeger-query/)
**jaeger-ingester** | [hub.docker.com/r/jaegertracing/jaeger-ingester/](https://hub.docker.com/r/jaegertracing/jaeger-ingester/)
There are orchestration templates for running Jaeger with:
* Kubernetes: [github.com/jaegertracing/jaeger-kubernetes](https://github.com/jaegertracing/jaeger-kubernetes),
* OpenShift: [github.com/jaegertracing/jaeger-openshift](https://github.com/jaegertracing/jaeger-openshift).
## Configuration Options
Jaeger binaries can be configured in a number of ways (in the order of decreasing priority):
* command line arguments,
* environment variables,
* configuration files in JSON, TOML, YAML, HCL, or Java properties formats.
To see the complete list of options, run the binary with `help` command. Options that are specific to a certain storage backend are only listed if the storage type is selected. For example, to see all available options in the Collector with Cassandra storage:
```sh
$ docker run --rm \
-e SPAN_STORAGE_TYPE=cassandra \
jaegertracing/jaeger-collector:{{< currentVersion >}} \
help
```
In order to provide configuration parameters via environment variables, find the respective command line option and convert its name to UPPER_SNAKE_CASE, for example:
Command line option | Environment variable
-----------------------------------|-------------------------------
`--cassandra.connections-per-host` | `CASSANDRA_CONNECTIONS_PER_HOST`
`--metrics-backend` | `METRICS_BACKEND`
## Agent
Jaeger client libraries expect **jaeger-agent** process to run locally on each host.
The agent exposes the following ports:
Port | Protocol | Function
----- | ------- | ---
6831 | UDP | accept [jaeger.thrift][jaeger-thrift] in `compact` Thrift protocol used by most current Jaeger clients
6832 | UDP | accept [jaeger.thrift][jaeger-thrift] in `binary` Thrift protocol used by Node.js Jaeger client (because [thriftrw][thriftrw] npm package does not support `compact` protocol)
5778 | HTTP | serve configs, sampling strategies
5775 | UDP | accept [zipkin.thrift][zipkin-thrift] in `compact` Thrift protocol (deprecated; only used by very old Jaeger clients, circa 2016)
14271 | HTTP | Healthcheck at `/` and metrics at `/metrics`
It can be executed directly on the host or via Docker, as follows:
```sh
## make sure to expose only the ports you use in your deployment scenario!
docker run \
--rm \
-p6831:6831/udp \
-p6832:6832/udp \
-p5778:5778/tcp \
-p5775:5775/udp \
jaegertracing/jaeger-agent:{{< currentVersion >}}
```
### Discovery System Integration
The agents can connect point to point to a single collector address, which could be
load balanced by another infrastructure component (e.g. DNS) across multiple collectors.
The agent can also be configured with a static list of collector addresses.
On Docker, a command like the following can be used:
```sh
docker run \
--rm \
-p5775:5775/udp \
-p6831:6831/udp \
-p6832:6832/udp \
-p5778:5778/tcp \
jaegertracing/jaeger-agent:{{< currentVersion >}} \
--reporter.grpc.host-port=jaeger-collector.jaeger-infra.svc:14250
```
Or use `--reporter.tchannel.host-port=jaeger-collector.jaeger-infra.svc:14267` to use
legacy tchannel reporter.
When using gRPC, you have several options for load balancing and name resolution:
* Single connection and no load balancing. This is the default if you specify a single `host:port`. (example: `--reporter.grpc.host-port=jaeger-collector.jaeger-infra.svc:14250`)
* Static list of hostnames and round-robin load balancing. This is what you get with a comma-separated list of addresses. (example: `reporter.grpc.host-port=jaeger-collector1:14250,jaeger-collector2:14250,jaeger-collector3:14250`)
* Dynamic DNS resolution and round-robin load balancing. To get this behaviour, prefix the address with `dns:///` and gRPC will attempt to resolve the hostname using SRV records (for [external load balancing](https://github.com/grpc/grpc/blob/master/doc/load-balancing.md)), TXT records (for [service configs](https://github.com/grpc/grpc/blob/master/doc/service_config.md)), and A records. Refer to the [gRPC Name Resolution docs](https://github.com/grpc/grpc/blob/master/doc/naming.md) and the [dns_resolver.go implementation](https://github.com/grpc/grpc-go/blob/master/resolver/dns/dns_resolver.go) for more info. (example: `--reporter.grpc.host-port=dns:///jaeger-collector.jaeger-infra.svc:14250`)
### Agent level tags
Jaeger supports agent level tags, that can be added to the process tags of all spans passing through the agent. This is supported through the command line flag `--jaeger.tags=key1=value1,key2=value2,...,keyn=valuen`. Tags can also be set through an environment flag like so - `--jaeger.tags=key=${envFlag:defaultValue}` - The tag value will be set to the value of the `envFlag` environment key and `defaultValue` if not set. This feature is not supported for the tchannel reporter, enabled using the flags `--collector.host-port` or `--reporter.tchannel.host-port`.
## Collectors
The collectors are stateless and thus many instances of **jaeger-collector** can be run in parallel.
Collectors require almost no configuration, except for the location of Cassandra cluster,
via `--cassandra.keyspace` and `--cassandra.servers` options, or the location of Elasticsearch cluster, via
`--es.server-urls`, depending on which storage is specified. To see all command line options run
```sh
go run ./cmd/collector/main.go -h
```
or, if you don't have the source code
```sh
docker run -it --rm jaegertracing/jaeger-collector:{{< currentVersion >}} -h
```
At default settings the collector exposes the following ports:
Port | Protocol | Function
----- | ------- | ---
14267 | TChannel | used by **jaeger-agent** to send spans in jaeger.thrift format
14250 | gRPC | used by **jaeger-agent** to send spans in model.proto format
14268 | HTTP | can accept spans directly from clients in jaeger.thrift format over binary thrift protocol
9411 | HTTP | can accept Zipkin spans in Thrift, JSON and Proto (disabled by default)
14269 | HTTP | Healthcheck at `/` and metrics at `/metrics`
## Storage Backends
Collectors require a persistent storage backend. Cassandra and Elasticsearch are the primary supported storage backends. Additional backends are [discussed here](https://github.com/jaegertracing/jaeger/issues/638).
The storage type can be passed via `SPAN_STORAGE_TYPE` environment variable. Valid values are `cassandra`, `elasticsearch`, `kafka` (only as a buffer), `grpc-plugin`, `badger` (only with all-in-one) and `memory` (only with all-in-one).
As of version 1.6.0, it's possible to use multiple storage types at the same time by providing a comma-separated list of valid types to the `SPAN_STORAGE_TYPE` environment variable.
It's important to note that all listed storage types are used for writing, but only the first type in the list will be used for reading and archiving.
### Memory
The in-memory storage is not intended for production workloads. It's intended as a simple solution to get started quickly and
data will be lost once the process is gone.
By default, there's no limit in the amount of traces stored in memory but a limit can be established by passing an
integer value via `--memory.max-traces`.
### Badger - local storage
Experimental since Jaeger 1.9
[Badger](https://github.com/dgraph-io/badger) is an embedded local storage, only available
with **all-in-one** distribution. By default it acts as an ephemeral storage using a temporary filesystem.
This can be overridden by using the `--badger.ephemeral=false` option.
```sh
docker run \
-e SPAN_STORAGE_TYPE=badger \
-e BADGER_EPHEMERAL=false \
-e BADGER_DIRECTORY_VALUE=/badger/data \
-e BADGER_DIRECTORY_KEY=/badger/key \
-v <storage_dir_on_host>:/badger \
-p 16686:16686 \
jaegertracing/all-in-one:{{< currentVersion >}}
```
### Cassandra
Supported versions: 3.4+
Deploying Cassandra itself is out of scope for our documentation. One good
source of documentation is the [Apache Cassandra Docs](https://cassandra.apache.org/doc/latest/).
#### Configuration
##### Minimal
```sh
docker run \
-e SPAN_STORAGE_TYPE=cassandra \
-e CASSANDRA_SERVERS=<...> \
jaegertracing/jaeger-collector:{{< currentVersion >}}
```
##### All options
To view the full list of configuration options, you can run the following command:
```sh
docker run \
-e SPAN_STORAGE_TYPE=cassandra \
jaegertracing/jaeger-collector:{{< currentVersion >}} \
--help
```
#### Schema script
A script is provided to initialize Cassandra keyspace and schema
using Cassandra's interactive shell [`cqlsh`][cqlsh]:
```sh
MODE=test sh ./plugin/storage/cassandra/schema/create.sh | cqlsh
```
For production deployment, pass `MODE=prod DATACENTER={datacenter}` arguments to the script,
where `{datacenter}` is the name used in the Cassandra configuration / network topology.
The script also allows overriding TTL, keyspace name, replication factor, etc.
Run the script without arguments to see the full list of recognized parameters.
#### TLS support
Jaeger supports TLS client to node connections as long as you've configured
your Cassandra cluster correctly. After verifying with e.g. `cqlsh`, you can
configure the collector and query like so:
```sh
docker run \
-e CASSANDRA_SERVERS=<...> \
-e CASSANDRA_TLS=true \
-e CASSANDRA_TLS_SERVER_NAME="CN-in-certificate" \
-e CASSANDRA_TLS_KEY=<path to client key file> \
-e CASSANDRA_TLS_CERT=<path to client cert file> \
-e CASSANDRA_TLS_CA=<path to your CA cert file> \
jaegertracing/jaeger-collector:{{< currentVersion >}}
```
The schema tool also supports TLS. You need to make a custom cqlshrc file like
so:
```
# Creating schema in a cassandra cluster requiring client TLS certificates.
#
# Create a volume for the schema docker container containing four files:
# cqlshrc: this file
# ca-cert: the cert authority for your keys
# client-key: the keyfile for your client
# client-cert: the cert file matching client-key
#
# if there is any sort of DNS mismatch and you want to ignore server validation
# issues, then uncomment validate = false below.
#
# When running the container, map this volume to /root/.cassandra and set the
# environment variable CQLSH_SSL=--ssl
[ssl]
certfile = ~/.cassandra/ca-cert
userkey = ~/.cassandra/client-key
usercert = ~/.cassandra/client-cert
# validate = false
```
### Elasticsearch
Supported in Jaeger since 0.6.0
Supported versions: 5.x, 6.x, 7.x
Elasticsearch version is automatically retrieved from root/ping endpoint.
Based on this version Jaeger uses compatible index mappings and Elasticsearch REST API.
The version can be explicitly provided via `--es.version=` flag.
Elasticsearch does not require initialization other than
[installing and running Elasticsearch](https://www.elastic.co/downloads/elasticsearch).
Once it is running, pass the correct configuration values to the Jaeger collector and query service.
#### Configuration
##### Minimal
```sh
docker run \
-e SPAN_STORAGE_TYPE=elasticsearch \
-e ES_SERVER_URLS=<...> \
jaegertracing/jaeger-collector:{{< currentVersion >}}
```
##### All options
To view the full list of configuration options, you can run the following command:
```sh
docker run \
-e SPAN_STORAGE_TYPE=elasticsearch \
jaegertracing/jaeger-collector:{{< currentVersion >}} \
--help
```
#### Shards and Replicas for Elasticsearch indices
Shards and replicas are some configuration values to take special attention to, because this is decided upon
index creation. [This article](https://qbox.io/blog/optimizing-elasticsearch-how-many-shards-per-index) goes into
more information about choosing how many shards should be chosen for optimization.
#### Upgrade Elasticsearch version
Elasticsearch defines wire and index compatibility versions. The index compatibility defines
the minimal version a node can read data from. For example Elasticsearch 7 can read indices
created by Elasticsearch 6, however it cannot read indices created by Elasticsearch 5 even
though they use the same index mappings. Therefore upgrade from Elasticsearch 6 to 7 does not require any
data migration. However, upgrade from Elasticsearch 5 to 7 has to be done through Elasticsearch 6 and wait
until indices created by ES 5.x are removed or explicitly reindexed.
Refer to the Elasticsearch [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current//setup-upgrade.html)
for wire and index compatibility versions. Generally this information can be retrieved from root/ping REST endpoint.
##### Reindex
Manual reindexing can be used when upgrading from Elasticsearch 5 to 7 (through Elasticsearch 6)
without waiting until indices created by Elasticsearch 5 are removed.
1. Reindex all span indices to new indices with suffix `-1`:
```bash
curl -ivX POST -H "Content-Type: application/json" http://localhost:9200/_reindex -d @reindex.json
{
"source": {
"index": "jaeger-span-*"
},
"dest": {
"index": "jaeger-span"
},
"script": {
"lang": "painless",
"source": "ctx._index = 'jaeger-span-' + (ctx._index.substring('jaeger-span-'.length(), ctx._index.length())) + '-1'"
}
}
```
2. Delete indices with old mapping:
```bash
curl -ivX DELETE -H "Content-Type: application/json" http://localhost:9200/jaeger-span-\*,-\*-1
```
3. Create indices without `-1` suffix:
```bash
curl -ivX POST -H "Content-Type: application/json" http://localhost:9200/_reindex -d @reindex.json
{
"source": {
"index": "jaeger-span-*"
},
"dest": {
"index": "jaeger-span"
},
"script": {
"lang": "painless",
"source": "ctx._index = 'jaeger-span-' + (ctx._index.substring('jaeger-span-'.length(), ctx._index.length() - 2))"
}
}
```
4. Remove suffixed indices:
```bash
curl -ivX DELETE -H "Content-Type: application/json" http://localhost:9200/jaeger-span-\*-1
```
Run the commands analogically for other Jaeger indices.
There might exist more effective migration procedure. Please share with the community any findings.
### Kafka
Supported in Jaeger since 1.6.0
Supported Kafka versions: 0.9+
Kafka can be used as an intermediary buffer between collector and an actual storage.
The collector is configured with `SPAN_STORAGE_TYPE=kafka` that makes it write all received spans
into a Kafka topic. A new component [Ingester](#ingester), added in version 1.7.0, is used to read from
Kafka and store spans in another storage backend (Elasticsearch or Cassandra).
Writing to Kafka is particularly useful for building post-processing data pipelines.
#### Configuration
##### Minimal
```sh
docker run \
-e SPAN_STORAGE_TYPE=kafka \
-e KAFKA_BROKERS=<...> \
-e KAFKA_TOPIC=<...> \
jaegertracing/jaeger-collector:{{< currentVersion >}}
```
##### All options
To view the full list of configuration options, you can run the following command:
```sh
docker run \
-e SPAN_STORAGE_TYPE=kafka \
jaegertracing/jaeger-collector:{{< currentVersion >}} \
--help
```
#### Topic & partitions
Unless your Kafka cluster is configured to automatically create topics, you will need to create it ahead of time. You can refer to [the Kafka quickstart documentation](https://kafka.apache.org/documentation/#quickstart_createtopic) to learn how.
You can find more information about topics and partitions in general in the [official documentation](https://kafka.apache.org/documentation/#intro_topics). [This article](https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/) provide more details about how to choose the number of partitions.
### Storage plugin
Jaeger supports gRPC based storage plugins. For more information refer to [jaeger/plugin/storage/grpc](https://github.com/jaegertracing/jaeger/tree/master/plugin/storage/grpc)
Available plugins:
* [InfluxDB](https://github.com/influxdata/jaeger-influxdb/)
```sh
docker run \
-e SPAN_STORAGE_TYPE=grpc-plugin \
-e GRPC_STORAGE_PLUGIN_BINARY=<...> \
-e GRPC_STORAGE_PLUGIN_CONFIGURATION_FILE=<...> \
jaegertracing/all-in-one:{{< currentVersion >}}
```
## Ingester
**jaeger-ingester** is a service which reads span data from Kafka topic and writes it to another storage backend (Elasticsearch or Cassandra).
Port | Protocol | Function
----- | ------- | ---
14270 | HTTP | Healthcheck at `/` and metrics at `/metrics`
To view all exposed configuration options run the following command:
```sh
docker run \
-e SPAN_STORAGE_TYPE=cassandra \
jaegertracing/jaeger-ingester:{{< currentVersion >}}
--help
```
## Query Service & UI
**jaeger-query** serves the API endpoints and a React/Javascript UI.
The service is stateless and is typically run behind a load balancer, such as [**NGINX**](https://www.nginx.com/).
At default settings the query service exposes the following port(s):
Port | Protocol | Function
----- | ------- | ---
16686 | HTTP | `/api/*` endpoints and Jaeger UI at `/`
16687 | HTTP | Healthcheck at `/` and metrics at `/metrics`
### Minimal deployment example (Elasticsearch backend):
```sh
docker run -d --rm \
-p 16686:16686 \
-p 16687:16687 \
-e SPAN_STORAGE_TYPE=elasticsearch \
-e ES_SERVER_URLS=http://<ES_SERVER_IP>:<ES_SERVER_PORT> \
jaegertracing/jaeger-query:{{< currentVersion >}}
```
### UI Base Path
The base path for all **jaeger-query** HTTP routes can be set to a non-root value, e.g. `/jaeger` would cause all UI URLs to start with `/jaeger`. This can be useful when running **jaeger-query** behind a reverse proxy.
The base path can be configured via the `--query.base-path` command line parameter or the `QUERY_BASE_PATH` environment variable.
### UI Customization and Embedding
Please refer to the [dedicated Frontend/UI page](../frontend-ui/).
## Aggregation Jobs for Service Dependencies
Production deployments need an external process which aggregates data and creates dependency links between services. Project [spark-dependencies](https://github.com/jaegertracing/spark-dependencies) is a Spark job which derives dependency links and stores them directly to the storage.
## Configuration
All binaries accepts command line properties and environmental variables, power by [viper](https://github.com/spf13/viper) and [cobra](https://github.com/spf13/cobra) libraries. Please refer to the [CLI Flags](../cli/) page for more information.
[cqlsh]: http://cassandra.apache.org/doc/latest/tools/cqlsh.html
[zipkin-thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/zipkincore.thrift
[jaeger-thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/jaeger.thrift
[thriftrw]: https://www.npmjs.com/package/thriftrw

26
content/docs/1.15/faq.md Normal file
View File

@ -0,0 +1,26 @@
---
title: Frequently Asked Questions
navtitle: FAQs
description: Answers to some frequently asked questions about Jaeger.
weight: 11
---
## Why is the Dependencies page empty?
The Dependencies page shows a graph of services traced by Jaeger and connections between them. When you are using `all-in-one` binary with in-memory storage, the graph is calculated on-demand from all the traces stored in memory. However, if you are using a real distributed storage like Cassandra or Elasticsearch, it is too expensive to scan all the data in the database to build the service graph. Instead, the Jaeger project provides "big data" jobs that can be used to extract the service graph data from traces:
* https://github.com/jaegertracing/spark-dependencies - the older Spark job that can be run periodically
* https://github.com/jaegertracing/jaeger-analytics - the new (experimental) streaming Flink jobs that run continuously and builds the service graph in smaller time intervals
## Why do I not see any spans in Jaeger?
Please refer to the [Troubleshooting](../troubleshooting/) guide.
## Do I need to run jaeger-agent?
`jaeger-agent` is not always necessary. Jaeger client libraries can be configured to export trace data directly to `jaeger-collector`. However, the following are the reasons why running `jaeger-agent` is recommended:
* If we want Jaeger client libraries to send trace data directly to collectors, we must provide them with a URL of the HTTP endpoint. It means that our applications require additional configuration containing this parameter, especially if we are running multiple Jaeger installations (e.g. in different availability zones or regions) and want the data sent to a nearby installation. In contrast, when using the agent, the libraries require no additional configuration because the agent is always accessible via `localhost`. It acts as a sidecar and proxies the requests to the appropriate collectors.
* The agent can be configured to enrich the tracing data with infrastructure-specific metadata by adding extra tags to the spans, such as the current zone, region, etc. If the agent is running as a host daemon, it will be shared by all applications running on the same host. If the agent is running as a true sidecar, i.e. one per application, it can provide additional functionality such as strong authentication, multi-tenancy (see [this blog post](https://medium.com/jaegertracing/jaeger-and-multitenancy-99dfa1d49dc0)), pod name, etc.
* If we want Jaeger client libraries to use sampling strategies that are centrally configured in the collectors, this is only possible by using the `/sampling` HTTP endpoint on the agent. There is no technical reason why this endpoint cannot be implemented directly in the collectors, it's just [not done yet](https://github.com/jaegertracing/jaeger/issues/1971).
* Agents allow implementing traffic control to the collectors. If we have thousands of hosts in the data center, each running many applications, and each application sending data directly to the collectors, there may be too many open connections for each collector to handle. The agents can load balance this traffic with fewer connections.

View File

@ -0,0 +1,57 @@
---
title: Features
hasparent: true
---
Jaeger is used for monitoring and troubleshooting microservices-based distributed systems, including:
* Distributed context propagation
* Distributed transaction monitoring
* Root cause analysis
* Service dependency analysis
* Performance / latency optimization
## High Scalability
Jaeger backend is designed to have no single points of failure and to scale with the business needs.
For example, any given Jaeger installation at Uber is typically processing several billion {{< tip "spans" "span" >}} per day.
## Native support for OpenTracing
Jaeger backend, Web UI, and instrumentation libraries have been designed from ground up to support the OpenTracing standard.
* Represent {{< tip "traces" "trace" >}} as {{< tip "directed acyclic graphs" "directed acyclic graph" >}} (not just trees) via [span references](https://github.com/opentracing/specification/blob/master/specification.md#references-between-spans)
* Support strongly typed span _tags_ and _structured logs_
* Support general distributed context propagation mechanism via _baggage_
## Multiple storage backends
Jaeger supports two popular open source NoSQL databases as trace storage backends: Cassandra 3.4+ and Elasticsearch 5.x/6.x/7.x.
There are ongoing community experiments using other databases, such as ScyllaDB, InfluxDB, Amazon DynamoDB. Jaeger also ships
with a simple in-memory storage for testing setups.
## Modern Web UI
Jaeger Web UI is implemented in Javascript using popular open source frameworks like React. Several performance
improvements have been released in v1.0 to allow the UI to efficiently deal with large volumes of data, and to display
{{< tip "traces" "trace" >}} with tens of thousands of {{< tip "spans" "span" >}} (e.g. we tried a trace with 80,000 spans).
## Cloud Native Deployment
Jaeger backend is distributed as a collection of Docker images. The binaries support various configuration methods,
including command line options, environment variables, and configuration files in multiple formats (yaml, toml, etc.).
Deployment to Kubernetes clusters is assisted by a [Kubernetes operator](https://github.com/jaegertracing/jaeger-operator), [Kubernetes templates](https://github.com/jaegertracing/jaeger-kubernetes)
and a [Helm chart](https://github.com/kubernetes/charts/tree/master/incubator/jaeger).
## Observability
All Jaeger backend components expose [Prometheus](https://prometheus.io/) metrics by default (other metrics backends are
also supported). Logs are written to standard out using the structured logging library [zap](https://github.com/uber-go/zap).
## Backwards compatibility with Zipkin
Although we recommend instrumenting applications with OpenTracing API and binding to Jaeger client libraries to benefit
from advanced features not available elsewhere, if your organization has already invested in the instrumentation
using Zipkin libraries, you do not have to rewrite all that code. Jaeger provides backwards compatibility with Zipkin
by accepting spans in Zipkin formats (Thrift, JSON v1/v2 and Protobuf) over HTTP. Switching from Zipkin backend is just a matter
of routing the traffic from Zipkin libraries to the Jaeger backend.

View File

@ -0,0 +1,224 @@
---
title: Frontend/UI Configuration
navtitle: Frontend/UI
hasparent: true
weight: 7
---
## Configuration
Several aspects of the UI can be configured:
* The Dependencies section can be enabled / configured
* Google Analytics tracking can be enabled / configured
* Additional menu options can be added to the global nav
These options can be configured by a JSON configuration file. The `--query.ui-config` command line parameter of the query service must then be set to the path to the JSON file when the query service is started.
An example configuration file:
```json
{
"dependencies": {
"dagMaxNumServices": 200,
"menuEnabled": true
},
"archiveEnabled": true,
"tracking": {
"gaID": "UA-000000-2",
"trackErrors": true
},
"menu": [
{
"label": "About Jaeger",
"items": [
{
"label": "GitHub",
"url": "https://github.com/jaegertracing/jaeger"
},
{
"label": "Docs",
"url": "http://jaeger.readthedocs.io/en/latest/"
}
]
}
],
"linkPatterns": [{
"type": "process",
"key": "jaeger.version",
"url": "https://github.com/jaegertracing/jaeger-client-java/releases/tag/#{jaeger.version}",
"text": "Information about Jaeger release #{jaeger.version}"
}]
}
```
### Dependencies
`dependencies.dagMaxNumServices` defines the maximum number of services allowed before the DAG dependency view is disabled. Default: `200`.
`dependencies.menuEnabled` enables (`true`) or disables (`false`) the dependencies menu button. Default: `true`.
### Archive Support
`archiveEnabled` enables (`true`) or disables (`false`) the archive traces button. Default: `false`. It requires a configuration of an archive storage in Query service. Archived traces are only accessible directly by ID, they are not searchable.
### Google Analytics Tracking
`tracking.gaID` defines the Google Analytics tracking ID. This is required for Google Analytics tracking, and setting it to a non-`null` value enables Google Analytics tracking. Default: `null`.
`tracking.trackErrors` enables (`true`) or disables (`false`) error tracking via Google Analytics. Errors can only be tracked if a valid Google Analytics ID is provided. For additional details on error tracking via Google Analytics see the [tracking README](https://github.com/jaegertracing/jaeger-ui/blob/c622330546afc1be59a42f874bcc1c2fadf7e69a/src/utils/tracking/README.md) in the UI repo. Default: `true`.
### Custom Menu Items
`menu` allows additional links to be added to the global nav. The additional links are right-aligned.
In the sample JSON config above, the configured menu will have a dropdown labeled "About Jaeger" with sub-options for "GitHub" and "Docs". The format for a link in the top right menu is as follows:
```json
{
"label": "Some text here",
"url": "https://example.com"
}
```
Links can either be members of the `menu` Array, directly, or they can be grouped into a dropdown menu option. The format for a group of links is:
```json
{
"label": "Dropdown button",
"items": [ ]
}
```
The `items` Array should contain one or more link configurations.
### Link Patterns
The `linkPatterns` node can be used to create links from fields displayed in the Jaeger UI.
Field | Description
------|------------
type | The metadata section in which your link will be added: process, tags, logs
key | The name of tag/process/log attribute which value will be displayed as a link
url | The URL where the link should point to, it can be an external site or relative path in Jaeger UI
text | The text displayed in the tooltip for the link
Both `url` and `text` can be defined as templates (i.e. using `#{field-name}`) where Jaeger UI will dynamically substitute values based on tags/logs data.
## Embedded Mode
Starting with version 1.9, Jaeger UI provides an "embedded" layout mode which is intended to support integrating Jaeger UI into other applications. Currently (as of `v0`), the approach taken is to remove various UI elements from the page to make the UI better suited for space-constrained layouts.
The embedded mode is induced and configured via URL query parameters.
To enter embedded mode, the `uiEmbed=v0` query parameter and value must be added to the URL. For example, the following URL will show the trace with ID `abc123` in embedded mode:
```
http://localhost:16686/trace/abc123?uiEmbed=v0
```
`uiEmbed=v0` is required.
Further, each page supported has an <img src="/img/frontend-ui/embed-open-icon.png" style="width: 20px; height:20px;display:inline;" alt="Embed open window"> button added that will open the non-embedded page in a new tab.
The following pages support embedded mode:
* Search Page
* Trace Page
### Search Page
To integrate the Search Trace Page to our application we have to indicate to the Jaeger UI that we want to use the embed mode with `uiEmbed=v0`.
For example:
```
http://localhost:16686/search?
service=my-service&
start=1543917759557000&
end=1543921359557000&
limit=20&
lookback=1h&
maxDuration&
minDuration&
uiEmbed=v0
```
![Embed Search Traces](/img/frontend-ui/embed-search-traces.png)
#### Configuration options
The following query parameter can be used to configure the layout of the search page :
* `uiSearchHideGraph=1` - disables the display of the scatter plot above the search results
```
http://localhost:16686/search?
service=my-service&
start=1543917759557000&
end=1543921359557000&
limit=20&
lookback=1h&
maxDuration&
minDuration&
uiEmbed=v0&
uiSearchHideGraph=1
```
![Embed Search Traces without Graph](/img/frontend-ui/embed-search-traces-hide-graph.png)
### Trace Page
To integrate the Trace Page to our application we have to indicate to the Jaeger UI that we want to use the embed mode with `uiEmbed=v0`.
For example:
```sh
http://localhost:16686/trace/{trace-id}?uiEmbed=v0
```
![Embed Trace view](/img/frontend-ui/embed-trace-view.png)
If we have navigated to this view from the search traces page we'll have a button to go back to the results page.
![Embed Trace view](/img/frontend-ui/embed-trace-view-with-back-button.png)
#### Configuration options
The following query parameters can be used to configure the layout of the trace page :
* `uiTimelineCollapseTitle=1` causes the trace header to start out collapsed, which hides the summary and the minimap.
```
http://localhost:16686/trace/{trace-id}?
uiEmbed=v0&
uiTimelineCollapseTitle=1
```
![Embed Trace view](/img/frontend-ui/embed-trace-view-with-collapse.png)
* `uiTimelineHideMinimap=1` removes the minimap, entirely, regardless of whether the trace header is expanded or not.
```
http://localhost:16686/trace/{trace-id}?
uiEmbed=v0&
uiTimelineHideMinimap=1
```
![Embed Trace view](/img/frontend-ui/embed-trace-view-with-hide-minimap.png)
* `uiTimelineHideSummary=1` - removes the trace summary information (number of services, etc.) entirely, regardless of whether the trace header is expanded or not.
```
http://localhost:16686/trace/{trace-id}?
uiEmbed=v0&
uiTimelineHideSummary=1
```
![Embed Trace view](/img/frontend-ui/embed-trace-view-with-hide-summary.png)
We can also combine the options:
```
http://localhost:16686/trace/{trace-id}?
uiEmbed=v0&
uiTimelineHideMinimap=1&
uiTimelineHideSummary=1
```
![Embed Trace view](/img/frontend-ui/embed-trace-view-with-hide-details-and-hide-minimap.png)

View File

@ -0,0 +1,129 @@
---
title: Getting Started
description: Get up and running with Jaeger in your local environment
weight: 2
---
## Instrumentation
Your applications must be instrumented before they can send tracing data to Jaeger backend. Check the [Client Libraries](../client-libraries) section for information about how to use the OpenTracing API and how to initialize and configure Jaeger tracers.
## All in One
All-in-one is an executable designed for quick local testing, launches the Jaeger UI, collector, query, and agent, with an in memory storage component.
The simplest way to start the all-in-one is to use the pre-built image published to DockerHub (a single command line).
```bash
$ docker run -d --name jaeger \
-e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
-p 5775:5775/udp \
-p 6831:6831/udp \
-p 6832:6832/udp \
-p 5778:5778 \
-p 16686:16686 \
-p 14268:14268 \
-p 9411:9411 \
jaegertracing/all-in-one:{{< currentVersion >}}
```
Or run the `jaeger-all-in-one(.exe)` executable from the [binary distribution archives][download]:
```bash
$ jaeger-all-in-one --collector.zipkin.http-port=9411
```
You can then navigate to `http://localhost:16686` to access the Jaeger UI.
The container exposes the following ports:
Port | Protocol | Component | Function
----- | ------- | --------- | ---
5775 | UDP | agent | accept `zipkin.thrift` over compact thrift protocol (deprecated, used by legacy clients only)
6831 | UDP | agent | accept `jaeger.thrift` over compact thrift protocol
6832 | UDP | agent | accept `jaeger.thrift` over binary thrift protocol
5778 | HTTP | agent | serve configs
16686 | HTTP | query | serve frontend
14268 | HTTP | collector | accept `jaeger.thrift` directly from clients
14250 | HTTP | collector | accept `model.proto`
9411 | HTTP | collector | Zipkin compatible endpoint (optional)
## Kubernetes and OpenShift
* Kubernetes templates: https://github.com/jaegertracing/jaeger-kubernetes
* Kubernetes Operator: https://github.com/jaegertracing/jaeger-operator
* OpenShift templates: https://github.com/jaegertracing/jaeger-openshift
## Sample App: HotROD
HotROD (Rides on Demand) is a demo application that consists of several microservices and
illustrates the use of the [OpenTracing API](http://opentracing.io).
A tutorial / walkthrough is available in the blog post:
[Take OpenTracing for a HotROD ride][hotrod-tutorial].
It can be run standalone, but requires Jaeger backend to view the traces.
### Features
- Discover architecture of the whole system via data-driven dependency
diagram.
- View request timeline and errors; understand how the app works.
- Find sources of latency and lack of concurrency.
- Highly contextualized logging.
- Use baggage propagation to:
- Diagnose inter-request contention (queueing).
- Attribute time spent in a service.
- Use open source libraries with OpenTracing integration to get
vendor-neutral instrumentation for free.
### Prerequisites
- You need Go 1.11 or higher installed on your machine to run from source.
- Requires a [running Jaeger backend](#all-in-one) to view the traces.
### Running
#### From Source
```bash
mkdir -p $GOPATH/src/github.com/jaegertracing
cd $GOPATH/src/github.com/jaegertracing
git clone git@github.com:jaegertracing/jaeger.git jaeger
cd jaeger
make install
go run ./examples/hotrod/main.go all
```
#### From docker
```bash
$ docker run --rm -it \
--link jaeger \
-p8080-8083:8080-8083 \
-e JAEGER_AGENT_HOST="jaeger" \
jaegertracing/example-hotrod:{{< currentVersion >}} \
all
```
#### From binary distribution
Run `example-hotrod(.exe)` executable from the [binary distribution archives][download]:
```bash
$ example-hotrod all
```
Then navigate to `http://localhost:8080`.
## Migrating from Zipkin
Collector service exposes Zipkin compatible REST API `/api/v1/spans` which accepts both Thrift and JSON. Also there is `/api/v2/spans` for JSON and Proto.
By default it's disabled. It can be enabled with `--collector.zipkin.http-port=9411`.
Zipkin [Thrift](https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/zipkincore.thrift) IDL and Zipkin [Proto](https://github.com/jaegertracing/jaeger-idl/blob/master/proto/zipkin.proto) IDL files can be found in [jaegertracing/jaeger-idl](https://github.com/jaegertracing/jaeger-idl) repository.
They're compatible with [openzipkin/zipkin-api](https://github.com/openzipkin/zipkin-api) [Thrift](https://github.com/openzipkin/zipkin-api/blob/master/thrift/zipkinCore.thrift) and [Proto](https://github.com/openzipkin/zipkin-api/blob/master/zipkin.proto).
[hotrod-tutorial]: https://medium.com/@YuriShkuro/take-opentracing-for-a-hotrod-ride-f6e3141f7941
[download]: ../../../download/

View File

@ -0,0 +1,903 @@
---
title: Operator for Kubernetes
hasparent: true
---
# Understanding Operators
The Jaeger Operator is an implementation of a [Kubernetes Operator](https://coreos.com/operators/). Operators are pieces of software that ease the operational complexity of running another piece of software. More technically, _Operators_ are a method of packaging, deploying, and managing a Kubernetes application.
A Kubernetes application is an application that is both deployed on Kubernetes and managed using the Kubernetes APIs and `kubectl` (kubernetes) or `oc` (OKD) tooling. To be able to make the most of Kubernetes, you need a set of cohesive APIs to extend in order to service and manage your apps that run on Kubernetes. Think of Operators as the runtime that manages this type of app on Kubernetes.
# Installing the Operator
{{< info >}}
The Jaeger Operator version tracks one version of the Jaeger components (Query, Collector, Agent). When a new version of the Jaeger components is released, a new version of the operator will be released that understands how running instances of the previous version can be upgraded to the new version.
{{< /info >}}
## Installing the Operator on Kubernetes
The following instructions will create the `observability` namespace and install the Jaeger Operator.
{{< info >}}
Make sure your `kubectl` command is properly configured to talk to a valid Kubernetes cluster. If you don't have a cluster, you can create one locally using [`minikube`](https://kubernetes.io/docs/tasks/tools/install-minikube/).
{{< /info >}}
To install the operator, run:
<!--TODO - Does Kubernetes have privileged users? Needs to be run as a system:admin on OKD/OpenShift.-->
```bash
kubectl create namespace observability # <1>
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/crds/jaegertracing.io_jaegers_crd.yaml # <2>
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/service_account.yaml
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role.yaml
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role_binding.yaml
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/operator.yaml
```
<1> This creates the namespace used by default in the deployment files. If you want to install the Jaeger operator in a different namespace, you must edit the deployment files to change `observability` to the desired namespace value.
<2> This installs the "Custom Resource Definition" for the `apiVersion: jaegertracing.io/v1`
At this point, there should be a `jaeger-operator` deployment available. You can view it by running the following command:
```bash
$ kubectl get deployment jaeger-operator -n observability
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
jaeger-operator 1 1 1 1 48s
```
The operator is now ready to create Jaeger instances.
## Installing the Operator on OKD/OpenShift
<!-- TODO: Add instructions for installing via the operatorhub? -->
The instructions from the previous section also work for installing the operator on OKD or OpenShift. Make sure you are logged in as a privileged user, when you install the role based acces control (RBAC) rules, the custom resource definition, and the operator.
```bash
oc login -u <privileged user>
oc new-project observability # <1>
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/crds/jaegertracing.io_jaegers_crd.yaml # <2>
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/service_account.yaml
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role.yaml
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role_binding.yaml
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/operator.yaml
```
<1> This creates the namespace used by default in the deployment files. If you want to install the Jaeger operator in a different namespace, you must edit the deployment files to change `observability` to the desired namespace value.
<2> This installs the "Custom Resource Definition" for the `apiVersion: jaegertracing.io/v1`
Once the operator is installed, grant the role `jaeger-operator` to users who should be able to install individual Jaeger instances. The following example creates a role binding allowing the user `developer` to create Jaeger instances:
```bash
oc create \
rolebinding developer-jaeger-operator \
--role=jaeger-operator \
--user=developer
```
After the role is granted, switch back to a non-privileged user.
# Quick Start - Deploying the AllInOne image
The simplest possible way to create a Jaeger instance is by creating a YAML file like the following example. This will install the default AllInOne strategy, which deploys the "all-in-one" image (agent, collector, query, ingester, Jaeger UI) in a single pod, using in-memory storage by default.
{{< info >}}
This default strategy is intended for development, testing, and demo purposes, not for production.
{{< /info >}}
```yaml
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: simplest
```
The YAML file can then be used with `kubectl`:
<!-- TODO - Add OKD commands and tabs shortcode. -->
```bash
kubectl apply -f simplest.yaml
```
In a few seconds, a new in-memory all-in-one instance of Jaeger will be available, suitable for quick demos and development purposes. To check the instances that were created, list the `jaeger` objects:
```bash
$ kubectl get jaegers
NAME CREATED AT
simplest 28s
```
To get the pod name, query for the pods belonging to the `simplest` Jaeger instance:
```bash
$ kubectl get pods -l app.kubernetes.io/instance=simplest
NAME READY STATUS RESTARTS AGE
simplest-6499bb6cdd-kqx75 1/1 Running 0 2m
```
Similarly, the logs can be queried either from the pod directly using the pod name obtained from the previous example, or from all pods belonging to our instance:
```bash
$ kubectl logs -l app.kubernetes.io/instance=simplest
...
{"level":"info","ts":1535385688.0951214,"caller":"healthcheck/handler.go:133","msg":"Health Check state change","status":"ready"}
```
{{< info >}}
On OKD/OpenShift the container name must be specified.
{{< /info >}}
```bash
$ kubectl logs -l app.kubernetes.io/instance=simplest -c jaeger
...
{"level":"info","ts":1535385688.0951214,"caller":"healthcheck/handler.go:133","msg":"Health Check state change","status":"ready"}
```
# Deployment Strategies
When you create a Jaeger instance, it is associated with a strategy. The strategy is defined in the custom resource file, and determines the architecture to be used for the Jaeger backend. The default strategy is `allInOne`. The other possible values are `production` and `streaming`.
The available strategies are described in the following sections.
## AllInOne (Default) strategy
This strategy is intended for development, testing, and demo purposes.
The main backend components, agent, collector and query service, are all packaged into a single executable which is configured (by default) to use in-memory storage.
## Production strategy
The `production` strategy is intended (as the name suggests) for production environments, where long term storage of trace data is important, as well as a more scalable and highly available architecture is required. Each of the backend components is therefore separately deployed.
The agent can be injected as a sidecar on the instrumented application or as a daemonset.
The query and collector services are configured with a supported storage type - currently Cassandra or Elasticsearch. Multiple instances of each of these components can be provisioned as required for performance and resilience purposes.
The main additional requirement is to provide the details of the storage type and options, for example:
```yaml
storage:
type: elasticsearch
options:
es:
server-urls: http://elasticsearch:9200
```
## Streaming strategy
The `streaming` strategy is designed to augment the `production` strategy by providing a streaming capability that effectively sits between the collector and the backend storage (Cassandra or Elasticsearch). This provides the benefit of reducing the pressure on the backend storage, under high load situations, and enables other trace post-processing capabilities to tap into the real time span data directly from the streaming platform (Kafka).
The only additional information required is to provide the details for accessing the Kafka platform, which is configured in the `collector` component (as producer) and `ingester` component (as consumer):
```yaml
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: simple-streaming
spec:
strategy: streaming
collector:
options:
kafka: # <1>
producer:
topic: jaeger-spans
brokers: my-cluster-kafka-brokers.kafka:9092
ingester:
options:
kafka: # <1>
consumer:
topic: jaeger-spans
brokers: my-cluster-kafka-brokers.kafka:9092
ingester:
deadlockInterval: 0 # <2>
storage:
type: elasticsearch
options:
es:
server-urls: http://elasticsearch:9200
```
<1> Identifies the Kafka configuration used by the collector, to produce the messages, and the ingester to consume the messages.
<2> The deadlock interval can be disabled to avoid the ingester being terminated when no messages arrive within the default 1 minute period
{{< info >}}
A Kafka environment can be configured using [Strimzi's Kafka operator](https://strimzi.io/).
{{< /info >}}
# Understanding Custom Resource Definitions
In the Kubernetes API, a resource is an endpoint that stores a collection of API objects of a certain kind. For example, the built-in Pods resource contains a collection of Pod objects. A _Custom Resource Definition_ (CRD) object defines a new, unique object `Kind` in the cluster and lets the Kubernetes API server handle its entire lifecycle.
To create _Custom Resource_ (CR) objects, cluster administrators must first create a Custom Resource Definition (CRD). The CRDs allow cluster users to create CRs to add the new resource types into their projects. An Operator watches for custom resource objects to be created, and when it sees a custom resource being created, it creates the application based on the parameters defined in the custom resource object.
{{< info >}}
While only cluster administrators can create CRDs, developers can create the CR from an existing CRD if they have read and write permission to it.
{{< /info >}}
<!--
## Jaeger Custom Resource Parameters
TODO Create a TABLE with all the parameters, descriptions/notes, valid values, and defaults.
Figure out if we can generate the options? Can we filter them in any way?
https://github.com/jaegertracing/jaeger/issues/1537
https://github.com/jaegertracing/documentation/issues/250-->
For reference, here's how you can create a more complex all-in-one instance:
```yaml
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: my-jaeger
spec:
strategy: allInOne # <1>
allInOne:
image: jaegertracing/all-in-one:latest # <2>
options: # <3>
log-level: debug # <4>
storage:
type: memory # <5>
options: # <6>
memory: # <7>
max-traces: 100000
ingress:
enabled: false # <8>
agent:
strategy: DaemonSet # <9>
annotations:
scheduler.alpha.kubernetes.io/critical-pod: "" # <10>
```
<1> The default strategy is `allInOne`. The other possible values are `production` and `streaming`.
<2> The image to use, in a regular Docker syntax.
<3> The (non-storage related) options to be passed verbatim to the underlying binary. Refer to the Jaeger documentation and/or to the `--help` option from the related binary for all the available options.
<4> The option is a simple `key: value` map. In this case, we want the option `--log-level=debug` to be passed to the binary.
<5> The storage type to be used. By default it will be `memory`, but can be any other supported storage type (Cassandra, Elasticsearch, Kafka).
<6> All storage related options should be placed here, rather than under the 'allInOne' or other component options.
<7> Some options are namespaced and we can alternatively break them into nested objects. We could have specified `memory.max-traces: 100000`.
<8> By default, an ingress object is created for the query service. It can be disabled by setting its `enabled` option to `false`. If deploying on OpenShift, this will be represented by a Route object.
<9> By default, the operator assumes that agents are deployed as sidecars within the target pods. Specifying the strategy as "DaemonSet" changes that and makes the operator deploy the agent as DaemonSet. Note that your tracer client will probably have to override the "JAEGER_AGENT_HOST" environment variable to use the node's IP.
<10> Define annotations to be applied to all deployments (not services). These can be overridden by annotations defined on the individual components.
You can view example custom resources for different Jaeger configurations [on GitHub](https://github.com/jaegertracing/jaeger-operator/tree/master/examples).
# Configuring the Custom Resource
<!--TODO
esIndexCleaner
Spark dependencies
-->
You can use the simplest example (shown above) and create a Jaeger instance using the defaults, or you can create your own custom resource file.
## Storage options
### Cassandra storage
When the storage type is set to Cassandra, the operator will automatically create a batch job that creates the required schema for Jaeger to run. This batch job will block the Jaeger installation, so that it starts only after the schema is successfuly created. The creation of this batch job can be disabled by setting the `enabled` property to `false`:
```yaml
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: cassandra-without-create-schema
spec:
strategy: allInOne
storage:
type: cassandra
cassandraCreateSchema:
enabled: false # <1>
```
<1> Defaults to `true`
Further aspects of the batch job can be configured as well. An example with all the possible options is shown below:
```yaml
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: cassandra-with-create-schema
spec:
strategy: allInOne # <1>
storage:
type: cassandra
options: # <2>
cassandra:
servers: cassandra
keyspace: jaeger_v1_datacenter3
cassandraCreateSchema: # <3>
datacenter: "datacenter3"
mode: "test"
```
<1> The same works for `production` and `streaming`.
<2> These options are for the regular Jaeger components, like `collector` and `query`.
<3> The options for the `create-schema` job.
{{< info >}}
The default create-schema job uses `MODE=prod`, which implies a replication factor of `2`, using `NetworkTopologyStrategy` as the class, effectively meaning that at least 3 nodes are required in the Cassandra cluster. If a `SimpleStrategy` is desired, set the mode to `test`, which then sets the replication factor of `1`. Refer to the [create-schema script](https://github.com/jaegertracing/jaeger/blob/master/plugin/storage/cassandra/schema/create.sh) for more details.
{{< /info >}}
### Elasticsearch storage
By default Elasticsearch storage does not require any initialization job to be run. However Elasticsearch
storage requires a cron job to be run to clean old data from the storage.
When rollover (`es.use-aliases`) is enabled, Jaeger operator also deploys a job to initialize Elasticsearch storage
and another two cron jobs to perform required index management actions.
#### External Elasticsearch
Jaeger can be used with an external Elasticsearch cluster.
The following example shows a Jaeger CR using an external Elasticsearch cluster
with TLS CA certificate mounted from a volume and user/password stored in a secret.
```yaml
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: simple-prod
spec:
strategy: production
storage:
type: elasticsearch # <1>
options:
es:
server-urls: https://elasticsearch.default.svc:9200 # <2>
tls: # <3>
ca: /es/certificates/root-ca.pem
secretName: jaeger-secret # <4>
volumeMounts: # <5>
- name: certificates
mountPath: /es/certificates/
readOnly: true
volumes:
- name: certificates
secret:
secretName: quickstart-es-http-certs-public
```
<1> Storage type Elasticsearch.
<2> Url to Elasticsearch service running in default namespace.
<3> TLS configuration. In this case only CA certificate, but it can also contain `es.tls.key` and `es.tls.cert` when using mutual TLS.
<4> Secret which defines environment variables `ES_PASSWORD` and `ES_USERNAME`. Created by `kubectl create secret generic jaeger-secret --from-literal=ES_PASSWORD=changeme --from-literal=ES_USERNAME=elastic`
<5> Volume mounts and volumes which are mounted into all storage components.
#### Self provisioned
Under some circumstances, the Jaeger Operator can make use of the [Elasticsearch Operator](https://github.com/openshift/elasticsearch-operator) to provision a suitable Elasticsearch cluster.
{{< warning >}}
This feature is supported only on OKD/OpenShift clusters. Spark dependencies are not supported with this feature [Issue #294](https://github.com/jaegertracing/jaeger-operator/issues/294).
{{< /warning >}}
When there is no `es.server-urls` option as part of a Jaeger `production` instance and `elasticsearch` is set as the storage type, the Jaeger Operator creates an Elasticsearch cluster via the Elasticsearch Operator by creating a Custom Resource based on the configuration provided in storage section. The Elasticsearch cluster is meant to be dedicated for a single Jaeger instance.
The self-provision of an Elasticsearch cluster can be disabled by setting the flag `--es-provision` to `false`. The default value is `auto`, which will make the Jaeger Operator query the Kubernetes cluster for its ability to handle a `Elasticsearch` custom resource. This is usually set by the Elasticsearch Operator during its installation process, so, if the Elasticsearch Operator is expected to run *after* the Jaeger Operator, the flag can be set to `true`.
{{< danger >}}
At the moment there can be only one Jaeger with self-provisioned Elasticsearch instance per namespace.
{{< /danger >}}
#### Elasticsearch index cleaner job
When using `elasticsearch` storage by default a cron job is created to clean old traces from it, the options for it are listed below so you can configure it to your use case.
The connection configuration is derived from the storage options.
```yaml
storage:
type: elasticsearch
esIndexCleaner:
enabled: true // turn the cron job deployment on and off
numberOfDays: 7 // number of days to wait before deleting a record
schedule: "55 23 * * *" // cron expression for it to run
```
The connection configuration to storage is derived from storage options.
#### Elasticsearch rollover
This index management strategy is more complicated than using the default daily indices and
it requires an initialisation job to prepare the storage and two cron jobs to manage indices.
The first cron job is used for rolling-over to a new index and the second for removing
indices from read alias. The rollover feature is used when storage option `es.use-aliases` is enabled.
To learn more about rollover index management in Jaeger refer to this
[article](https://medium.com/jaegertracing/using-elasticsearch-rollover-to-manage-indices-8b3d0c77915d).
```yaml
storage:
type: elasticsearch
options:
es:
use-aliases: true
esRollover:
enabled: true // turn the cron job deployment on and off
conditions: "{\"max_age\": \"2d\"}" // conditions when to rollover to a new index
readTTL: 7d // how long should be old data available for reading
schedule: "55 23 * * *" // cron expression for it to run
```
The connection configuration to storage is derived from storage options.
## Deriving dependencies
The processing to derive dependencies will collect spans from storage, analyzes links between services and store them for later presentation in the UI.
This job can only be used with the `production` strategy and storage type `cassandra` or `elasticsearch`.
```yaml
storage:
type: elasticsearch
dependencies:
enabled: true # turn the job deployment on and off
schedule: "55 23 * * *" # cron expression for it to run
sparkMaster: # spark master connection string, when empty spark runs in embedded local mode
```
The connection configuration to storage is derived from storage options.
## Auto-injecting Jaeger Agent Sidecars
The operator can inject Jaeger Agent sidecars in `Deployment` workloads, provided that the deployment has the annotation `sidecar.jaegertracing.io/inject` with a suitable value. The values can be either `"true"` (as string), or the Jaeger instance name, as returned by `kubectl get jaegers`. When `"true"` is used, there should be exactly *one* Jaeger instance for the same namespace as the deployment, otherwise, the operator can't figure out automatically which Jaeger instance to use.
The following snippet shows a simple application that will get a sidecar injected, with the Jaeger Agent pointing to the single Jaeger instance available in the same namespace:
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
annotations:
"sidecar.jaegertracing.io/inject": "true" # <1>
spec:
selector:
matchLabels:
app: myapp
template:
metadata:
labels:
app: myapp
spec:
containers:
- name: myapp
image: acme/myapp:myversion
```
<1> Either `"true"` (as string) or the Jaeger instance name.
A complete sample deployment is available at [`deploy/examples/business-application-injected-sidecar.yaml`](https://github.com/jaegertracing/jaeger-operator/blob/master/examples/business-application-injected-sidecar.yaml).
When the sidecar is injected, the Jaeger Agent can then be accessed at its default location on `localhost`.
## Installing the Agent as DaemonSet
By default, the Operator expects the agents to be deployed as sidecars to the target applications. This is convenient for several purposes, like in a multi-tenant scenario or to have better load balancing, but there are scenarios where you might want to install the agent as a `DaemonSet`. In that case, specify the Agent's strategy to `DaemonSet`, as follows:
```yaml
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: my-jaeger
spec:
agent:
strategy: DaemonSet
```
{{< danger >}}
If you attempt to install two Jaeger instances on the same cluster with `DaemonSet` as the strategy, only *one* will end up deploying a `DaemonSet`, as the agent is required to bind to well-known ports on the node. Because of that, the second daemon set will fail to bind to those ports.
{{< /danger >}}
Your tracer client will then most likely need to be told where the agent is located. This is usually done by setting the environment variable `JAEGER_AGENT_HOST` to the value of the Kubernetes node's IP, for example:
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
spec:
selector:
matchLabels:
app: myapp
template:
metadata:
labels:
app: myapp
spec:
containers:
- name: myapp
image: acme/myapp:myversion
env:
- name: JAEGER_AGENT_HOST
valueFrom:
fieldRef:
fieldPath: status.hostIP
```
### OpenShift
In OpenShift, a `HostPort` can only be set when a special security context is set. A separate service account can be used by the Jaeger Agent with the permission to bind to `HostPort`, as follows:
```bash
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/examples/openshift/hostport-scc-daemonset.yaml # <1>
oc new-project myappnamespace
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/examples/openshift/service_account_jaeger-agent-daemonset.yaml # <2>
oc adm policy add-scc-to-user daemonset-with-hostport -z jaeger-agent-daemonset # <3>
oc apply -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/examples/openshift/agent-as-daemonset.yaml # <4>
```
<1> The `SecurityContextConstraints` with the `allowHostPorts` policy
<2> The `ServiceAccount` to be used by the Jaeger Agent
<3> Adds the security policy to the service account
<4> Creates the Jaeger Instance using the `serviceAccount` created in the steps above
{{< warning >}}
Without such a policy, errors like the following will prevent a `DaemonSet` to be created: `Warning FailedCreate 4s (x14 over 45s) daemonset-controller Error creating: pods "agent-as-daemonset-agent-daemonset-" is forbidden: unable to validate against any security context constraint: [spec.containers[0].securityContext.containers[0].hostPort: Invalid value: 5775: Host ports are not allowed to be used`
{{< /warning >}}
After a few seconds, the `DaemonSet` should be up and running:
```bash
$ oc get daemonset agent-as-daemonset-agent-daemonset
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE
agent-as-daemonset-agent-daemonset 1 1 1 1 1
```
## Secrets Support
The Operator supports passing secrets to the Collector, Query and All-In-One deployments. This can be used for example, to pass credentials (username/password) to access the underlying storage backend (for example: Elasticsearch).
The secrets are available as environment variables in the (Collector/Query/All-In-One) nodes.
```yaml
storage:
type: elasticsearch
options:
es:
server-urls: http://elasticsearch:9200
secretName: jaeger-secrets
```
The secret itself would be managed outside of the `jaeger-operator` custom resource.
## Configuring the UI
Information on various configuration options for the UI can be found [here](../frontend-ui/#configuration), defined in json format.
To apply UI configuration changes within the Custom Resource, the same information can be included in yaml format as shown below:
```yaml
ui:
options:
dependencies:
menuEnabled: false
tracking:
gaID: UA-000000-2
menu:
- label: "About Jaeger"
items:
- label: "Documentation"
url: "https://www.jaegertracing.io/docs/latest"
linkPatterns:
- type: "logs"
key: "customer_id"
url: /search?limit=20&lookback=1h&service=frontend&tags=%7B%22customer_id%22%3A%22#{customer_id}%22%7D
text: "Search for other traces for customer_id=#{customer_id}"
```
## Defining Sampling Strategies
{{< info >}}
This is not relevant if a trace was started by the Istio proxy as the sampling decision is made there. And the Jaeger sampling decisions are only relevant when you are using the Jaeger tracer (client).
{{< /info >}}
The operator can be used to define sampling strategies that will be supplied to tracers that have been configured to use a remote sampler:
```yaml
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: with-sampling
spec:
strategy: allInOne
sampling:
options:
default_strategy:
type: probabilistic
param: 0.5
```
This example defines a default sampling strategy that is probabilistic, with a 50% chance of the trace instances being sampled.
Refer to the Jaeger documentation on [Collector Sampling Configuration](https://www.jaegertracing.io/docs/latest/sampling/#collector-sampling-configuration) to see how service and endpoint sampling can be configured. The JSON representation described in that documentation can be used in the operator by converting to YAML.
## Finer grained configuration
The custom resource can be used to define finer grained Kubernetes configuration applied to all Jaeger components or at the individual component level.
When a common definition (for all Jaeger components) is required, it is defined under the `spec` node. When the definition relates to an individual component, it is placed under the `spec/<component>` node.
The types of supported configuration include:
* [affinity](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity) to determine which nodes a pod can be allocated to
* [annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/)
* [labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/)
* [resources](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container) to limit cpu and memory
* [tolerations](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/) in conjunction with `taints` to enable pods to avoid being repelled from a node
* [volumes](https://kubernetes.io/docs/concepts/storage/volumes/) and volume mounts
* [serviceAccount](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/) to run each component with separate identity
* [securityContext](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/) to define privileges of running components
```yaml
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: simple-prod
spec:
strategy: production
storage:
type: elasticsearch
options:
es:
server-urls: http://elasticsearch:9200
annotations:
key1: value1
labels:
key2: value2
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/e2e-az-name
operator: In
values:
- e2e-az1
- e2e-az2
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 1
preference:
matchExpressions:
- key: another-node-label-key
operator: In
values:
- another-node-label-value
tolerations:
- key: "key1"
operator: "Equal"
value: "value1"
effect: "NoSchedule"
- key: "key1"
operator: "Equal"
value: "value1"
effect: "NoExecute"
serviceAccount: nameOfServiceAccount
securityContext:
runAsUser: 1000
volumeMounts:
- name: config-vol
mountPath: /etc/config
volumes:
- name: config-vol
configMap:
name: log-config
items:
- key: log_level
path: log_level
```
# Accessing the Jaeger Console (UI)
<!-- TODO Add tabs shortcode -->
## Kubernetes
The operator creates a Kubernetes [`ingress`](https://kubernetes.io/docs/concepts/services-networking/ingress/) route, which is the Kubernetes' standard for exposing a service to the outside world, but by default it does not come with Ingress providers.
Check the [Kubernetes documentation](https://kubernetes.github.io/ingress-nginx/deploy/#verify-installation) for the most appropriate way to achieve an Ingress provider for your platform. The following command enables the Ingress provider on `minikube`:
```bash
minikube addons enable ingress
```
Once Ingress is enabled, the address for the Jaeger console can be found by querying the Ingress object:
```bash
$ kubectl get ingress
NAME HOSTS ADDRESS PORTS AGE
simplest-query * 192.168.122.34 80 3m
```
In this example, the Jaeger UI is available at http://192.168.122.34.
To enable TLS in the Ingress, pass a `secretName` with the name of a [Secret](https://kubernetes.io/docs/concepts/configuration/secret/) containing the TLS certificate:
```yaml
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: ingress-with-tls
spec:
ingress:
secretName: my-tls-secret
```
## OpenShift
When the Operator is running on OpenShift, the Operator will automatically create a `Route` object for the query services. Use the following command to check the hostname/port:
```bash
oc get routes
```
{{< info >}}
Make sure to use `https` with the hostname/port you get from the command above, otherwise you'll see a message like: "Application is not available".
{{< /info >}}
By default, the Jaeger UI is protected with OpenShift's OAuth service and any valid user is able to login. To disable this feature and leave the Jaeger UI unsecured, set the Ingress property `security` to `none` in the custom resource file:
```yaml
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: disable-oauth-proxy
spec:
ingress:
security: none
```
Custom `SAR` and `Delegate URL` values can be specified as part of the `.Spec.Ingress.OpenShift.SAR` and `.Spec.Ingress.Openshift.DelegateURLs`, as follows:
```yaml
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: custom-sar-oauth-proxy
spec:
ingress:
openshift:
sar: '{"namespace": "default", "resource": "pods", "verb": "get"}'
delegateUrls: '{"/":{"namespace": "default", "resource": "pods", "verb": "get"}}'
```
When the `delegateUrls` is set, the Jaeger Operator needs to create a new `ClusterRoleBinding` between the service account used by the UI Proxy (`{InstanceName}-ui-proxy`) and the role `system:auth-delegator`, as required by the OpenShift OAuth Proxy. Because of that, the service account used by the operator itself needs to have the same cluster role binding. To accomplish that, a `ClusterRoleBinding` such as the following has to be created:
```yaml
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: jaeger-operator-with-auth-delegator
namespace: observability
subjects:
- kind: ServiceAccount
name: jaeger-operator
namespace: observability
roleRef:
kind: ClusterRole
name: system:auth-delegator
apiGroup: rbac.authorization.k8s.io
```
Cluster administrators not comfortable in letting users deploy Jaeger instances with this cluster role are free to not add this cluster role to the operator's service account. In that case, the Operator will auto-detect that the required permissions are missing and will log a message similar to: `the requested instance specifies the delegateUrls option for the OAuth Proxy, but this operator cannot assign the proper cluster role to it (system:auth-delegator). Create a cluster role binding between the operator's service account and the cluster role 'system:auth-delegator' in order to allow instances to use 'delegateUrls'`.
The Jaeger Operator also supports authentication using `htpasswd` files via the OpenShift OAuth Proxy. To make use of that, specify the `htpasswdFile` option within the OpenShift-specific entries, pointing to the file `htpasswd` file location in the local disk. The `htpasswd` file can be created using the `htpasswd` utility:
```console
$ htpasswd -cs /tmp/htpasswd jdoe
New password:
Re-type new password:
Adding password for user jdoe
```
This file can then be used as the input for the `kubectl create secret` command:
```console
$ kubectl create secret generic htpasswd --from-file=htpasswd=/tmp/htpasswd
secret/htpasswd created
```
Once the secret is created, it can be specified in the Jaeger CR as a volume/volume mount:
```yaml
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: with-htpasswd
spec:
ingress:
openshift:
sar: '{"namespace": "default", "resource": "pods", "verb": "get"}'
htpasswdFile: /usr/local/data/htpasswd
volumeMounts:
- name: htpasswd-volume
mountPath: /usr/local/data
volumes:
- name: htpasswd-volume
secret:
secretName: htpasswd
```
# Upgrading the Operator and its managed instances
Each version of the Jaeger Operator follows one Jaeger version. Whenever a new version of the Jaeger Operator is installed, all the Jaeger instances managed by the operator will be upgraded to the Operator's supported version. For example, an instance named `simplest` that was created with Jaeger Operator 1.12.0 will be running Jaeger 1.12.0. Once the Jaeger Operator is upgraded to 1.13.0, the instance `simplest` will be upgraded to the version 1.13.0, following the official upgrade instructions from the Jaeger project.
The Jaeger Operator can be upgraded manually by changing the deployment (`kubectl edit deployment jaeger-operator`), or via specialized tools such as the [Operator Lifecycle Manager (OLM)](https://github.com/operator-framework/operator-lifecycle-manager).
# Updating a Jaeger instance (experimental)
A Jaeger instance can be updated by changing the `CustomResource`, either via `kubectl edit jaeger simplest`, where `simplest` is the Jaeger's instance name, or by applying the updated YAML file via `kubectl apply -f simplest.yaml`.
{{< danger >}}
The name of the Jaeger instance cannot be updated, as it is part of the identifying information for the resource.
{{< /danger >}}
Simpler changes such as changing the replica sizes can be applied without much concern, whereas changes to the strategy should be watched closely and might potentially cause an outage for individual components (collector/query/agent).
While changing the backing storage is supported, migration of the data is not.
# Removing a Jaeger instance
<!-- TODO Add OKD/OpenShift commands and tabs shortcode-->
To remove an instance, use the `delete` command with the custom resource file used when you created the instance:
```bash
kubectl delete -f simplest.yaml
```
Alternatively, you can remove a Jaeger instance by running:
```bash
kubectl delete jaeger simplest
```
{{< info >}}
Deleting the instance will not remove the data from any permanent storage used with this instance. Data from in-memory instances, however, will be lost.
{{< /info >}}
# Monitoring the operator
The Jaeger Operator starts a Prometheus-compatible endpoint on `0.0.0.0:8383/metrics` with internal metrics that can be used to monitor the process.
{{< info >}}
The Jaeger Operator does not yet publish its own metrics. Rather, it makes available metrics reported by the components it uses, such as the Operator SDK.
{{< /info >}}
# Uninstalling the operator
<!-- TODO Add OKD/OpenShift commands and tabs shortcode -->
To uninstall the operator, run the following commands:
```bash
kubectl delete -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/operator.yaml
kubectl delete -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role_binding.yaml
kubectl delete -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role.yaml
kubectl delete -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/service_account.yaml
kubectl delete -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/crds/jaegertracing.io_jaegers_crd.yaml
```

View File

@ -0,0 +1,95 @@
---
title: Performance Tuning Guide
description: Tweaking your Jaeger instance to achieve a better performance
weight: 10
---
Jaeger was built from day 1 to be able to ingest huge amounts of data in a resilient way. To better utilize resources that might cause delays, such as storage or network communications, Jaeger buffers and batches data. When more spans are generated than Jaeger is able to safely process, spans might get dropped. However, the defaults might not fit all scenarios: for instance, Agents running as a sidecar might have more memory constraints than Agents running as a daemon in bare metal.
## Deployment considerations
Although performance tuning the individual components is important, the way Jaeger is deployed can be decisive in obtaining optimal performance.
### Scale the Collector up and down
Use the auto-scaling capabilities of your platform: the Collector is nearly horizontally scalable so that more instances can be added and removed on-demand. A good way to scale up and down is by checking the `jaeger_collector_queue_length` metric: add instances when the length is higher than 50% of the maximum size for extended periods of time. Another metric that can be taken into consideration is `jaeger_collector_in_queue_latency_bucket`, which is a histogram indicating how long spans have been waiting in the queue before a worker picked it up. When the queue latency gets higher over time, its a good indication to increase the number of the workers, or to improve the storage performance.
Adding Collector instances is recommended when your platform provides auto-scaling capabilities, or when it's easier to start/stop Collector instances than changing existing, running instances. Scaling horizontally is also indicated when the CPU usage should be spread across nodes.
### Make sure the storage can keep up
Each span is written to the storage by the Collector using one worker, blocking it until the span has been stored. When the storage is too slow, the number of workers blocked by the storage might be too high, causing spans to be dropped. To help diagnose this situation, the histogram `jaeger_collector_save_latency_bucket` can be analyzed. Ideally, the latency should remain the same over time. When the histogram shows that most spans are taking longer and longer over time, its a good indication that your storage might need some attention.
### Place the Agents close to your applications
The Agent is meant to be placed on the same host as the instrumented application, in order to avoid UDP packet loss over the network. This is typically accomplished by having one Agent per bare metal host for traditional applications, or as a sidecar in container environments like Kubernetes, as this helps spread the load handled by Agents with the additional advantage of allowing each Agent to be tweaked individually, according to the applications needs and importance.
### Consider using Apache Kafka as intermediate buffer
Jaeger [can use Apache Kafka](../architecture/) as a buffer between the Collector and the actual backing storage (Elasticsearch, Apache Cassandra). This is ideal for cases where the traffic spikes are relatively frequent (prime time traffic) but the storage can eventually catch up once the traffic normalizes. For that, the `SPAN_STORAGE_TYPE` environment variable should be set to `kafka` in the Collector and the Jaeger Ingester component can be used, reading data from Kafka and writing it to the storage.
In addition to the performance aspects, having spans written to Kafka is useful for building real time data pipeline for aggregations and feature extraction from traces.
## Client (Tracer) settings
The Jaeger Clients are built to have minimal effect to the instrumented application. As such, it has conservative defaults that might not be suitable for all cases. Note that Jaeger Clients can be configured programmatically or via [environment variables](../client-features/).
### Adjust the sampling configuration
Together, the `JAEGER_SAMPLER_TYPE` and `JAEGER_SAMPLER_PARAM` specify how often traces should be "sampled", ie, recorded and sent to the Jaeger backend. For applications generating a large number of spans, setting the sampling type to `probabilistic` and the value to `0.001` (the default) will cause traces to be reported with a 1/1000th chance. Note that the sampling decision is made at the root span and propagated down to all child spans.
For applications with low to medium traffic, setting the sampling type to `const` and value to `1` will cause all spans to be reported. Similarly, tracing can be disabled by setting the value to `0`, while context propagation will continue to work.
Some Clients support the setting `JAEGER_DISABLED` to completely disable the Jaeger Tracer. This is recommended only if the Tracer is behaving in a way that causes problems to the instrumented application, as it will not propagate the context to the downstream services.
{{< info >}}
We recommend to set your clients to use the [`remote` sampling strategy](../sampling/#collector-sampling-configuration), so that admins can centrally set the concrete sampling strategy for each service.
{{< /info >}}
### Increase in-memory queue size
Most of the Jaeger clients, such as the Java, Go, and C# clients, buffer spans in memory before sending them to the Jaeger Agent/Collector. The maximum size of this buffer is defined by the environment variable `JAEGER_REPORTER_MAX_QUEUE_SIZE` (default value: about `100` spans): the larger the size, the higher the potential memory consumption. When the instrumented application is generating a large number of spans, its possible that the queue will be full causing the Client to discard the new spans (metric `jaeger_tracer_reporter_spans_total{result="dropped",}`).
In most common scenarios, the queue will be close to empty (metric: `jaeger_tracer_reporter_queue_length`), as spans are flushed to the Agent or Collector at regular intervals or when a certain size of the batch is reached. The detailed behavior of this queue is described in this [GitHub issue](https://github.com/jaegertracing/jaeger-client-java/issues/607).
### Modify the batched spans flush interval
The Java, Go, NodeJS, Python and C# Clients allow the customization of the flush interval (default value: `1000` milliseconds, or 1 second) used by the reporters, such as the `RemoteReporter`, to trigger a `flush` operation, sending all in-memory spans to the Agent or Collector. The lower the flush interval is set to, the more frequent the flush operations happen. As most reporters will wait until enough data is in the queue, this setting will force a flush operation at periodic intervals, so that spans are sent to the backend in a timely fashion.
When the instrumented application is generating a large number of spans and the Agent/Collector is close to the application, the networking overhead might be low, justifying a higher number of flush operations. When the `HttpSender` is being used and the Collector is not close enough to the application, the networking overhead might be too high so that a higher value for this property makes sense.
## Agent settings
Jaeger Agents receive data from Clients, sending them in batches to the Collector. When not properly configured, it might end up discarding data even if the host machine has plenty of resources.
### Adjust server queue sizes
The set of “server queue size” properties ( `processor.jaeger-binary.server-queue-size`, `processor.jaeger-compact.server-queue-size`, `processor.zipkin-compact.server-queue-size`) indicate the maximum number of span batches that the Agent can accept and store in memory. Its safe to assume that `jaeger-compact` is the most important processor in your Agent setup, as its the only one available in most Clients, such as the Java and Go Clients.
The default value for each queue is `1000` span batches. Given that each span batch has up to 64KiB worth of spans, each queue can hold up to 64MiB worth of spans.
In typical scenarios, the queue will be close to empty (metric `jaeger_agent_thrift_udp_server_queue_size`) as span batches should be quickly picked up and processed by a worker. However, sudden spikes in the number of span batches submitted by Clients might occur, causing the batches to be queued. When the queue is full, the older batches are overridden causing spans to be discarded (metric `jaeger_agent_thrift_udp_server_packets_dropped_total`).
### Adjust processor workers
The set of “processor workers” properties ( `processor.jaeger-binary.workers`, `processor.jaeger-compact.workers`, `processor.zipkin-compact.workers`) indicate the number of parallel span batch processors to start. Each worker type has a default size of `10`. In general, span batches are processed as soon as they are placed in the server queue and will block a worker until the whole packet is sent to the Collector. For Agents processing data from multiple Clients, the number of workers should be increased. Given that the cost of each worker is low, a good rule of thumb is 10 workers per Client with moderate traffic: given that each span batch might contain up to 64KiB worth of spans, it means that 10 workers are able to send about 640KiB concurrently to a Collector.
## Collector settings
The Collector receives data from Clients and Agents. When not properly configured, it might process less data than what would be possible on the same host, or it might overload the host by consuming more memory than permitted.
### Adjust queue size
Similar to the Agent, the Collector is able to receive spans and place them in an internal queue for processing. This allows the Collector to return immediately to the Client/Agent instead of waiting for the span to make its way to the storage.
The setting `collector.queue-size` (default: `2000`) dictates how many spans the queue should support. In the typical scenario, the queue will be close to empty, as enough workers should exist picking up spans from the queue and sending them to the storage. When the number of items in the queue (metric `jaeger_collector_queue_length`) is permanently high, its an indication that either the number of workers should be increased or that the storage cannot keep up with the volume of data that its receiving. When the queue is full, the older items in the queue are overridden, causing spans to be discarded (metric `jaeger_collector_spans_dropped_total`).
{{< warning >}}
The queue size for the Agent is about _span batches_, whereas the queue size for the Collector is about _individual spans_.
{{< /warning >}}
Given that the queue size should be close to empty most of the time, this setting should be as high as the available memory for the Collector, to provide maximum protection against sudden traffic spikes. However, if your storage layer is under-provisioned and cannot keep up, even a large queue will quickly fill up and start dropping data.
### Adjust processor workers
Items from the span queue in the Collector are picked up by workers. Each worker picks one span from the queue and persists it to the storage. The number of workers can be specified by the setting `collector.num-workers` (default: `50`) and should be as high as needed to keep the queue close to zero. The general rule is: the faster the backing storage, the lower the number of workers can be. Given that workers are relatively cheap, this number can be increased at will. As a general rule, one worker per 50 items in the queue should be sufficient when the storage is fast. With a `collector.queue-size` of `2000`, having about `40` workers should be sufficient. For slower storage mechanisms, this ratio should be adjusted accordingly, having more workers per queue item.

View File

@ -0,0 +1,66 @@
---
title: Sampling
hasparent: true
---
Jaeger libraries implement consistent upfront (or head-based) sampling. For example, assume we have a simple call graph where service A calls service B, and B calls service C: `A -> B -> C`. When service A receives a request that contains no tracing information, Jaeger tracer will start a new {{< tip "trace" >}}, assign it a random trace ID, and make a sampling decision based on the currently installed sampling strategy. The sampling decision will be propagated with the requests to B and to C, so those services will not be making the sampling decision again but instead will respect the decision made by the top service A. This approach guarantees that if a trace is sampled, all its {{< tip "spans" "span" >}} will be recorded in the backend. If each service was making its own sampling decision we would rarely get complete traces in the backend.
## Client Sampling Configuration
When using configuration object to instantiate the tracer, the type of sampling can be selected via `sampler.type` and `sampler.param` properties. Jaeger libraries support the following samplers:
* **Constant** (`sampler.type=const`) sampler always makes the same decision for all traces. It either samples all traces (`sampler.param=1`) or none of them (`sampler.param=0`).
* **Probabilistic** (`sampler.type=probabilistic`) sampler makes a random sampling decision with the probability of sampling equal to the value of `sampler.param` property. For example, with `sampler.param=0.1` approximately 1 in 10 traces will be sampled.
* **Rate Limiting** (`sampler.type=ratelimiting`) sampler uses a leaky bucket rate limiter to ensure that traces are sampled with a certain constant rate. For example, when `sampler.param=2.0` it will sample requests with the rate of 2 traces per second.
* **Remote** (`sampler.type=remote`, which is also the default) sampler consults Jaeger agent for the appropriate sampling strategy to use in the current service. This allows controlling the sampling strategies in the services from a [central configuration](#collector-sampling-configuration) in Jaeger backend, or even dynamically (see [Adaptive Sampling](https://github.com/jaegertracing/jaeger/issues/365)).
### Adaptive Sampler
Adaptive sampler is a composite sampler that combines two functions:
* It makes sampling decisions on a per-operation basis, i.e. based on {{< tip "span" >}} operation name. This is especially useful in the API services whose endpoints may have very different traffic volumes and using a single probabilistic sampler for the whole service might starve (never sample) some of the low QPS endpoints.
* It supports a minimum guaranteed rate of sampling, such as always allowing up to N {{< tip "traces" "trace" >}} per seconds and then sampling anything above that with a certain probability (everything is per-operation, not per-service).
Per-operation parameters can be configured statically or pulled periodically from Jaeger backend with the help of Remote sampler. Adaptive sampler is designed to work with the upcoming [Adaptive Sampling](https://github.com/jaegertracing/jaeger/issues/365) feature of the Jaeger backend.
## Collector Sampling Configuration
Collectors can be instantiated with static sampling strategies (which are propagated to the respective service if configured with Remote sampler) via the `--sampling.strategies-file` option. This option requires a path to a json file which have the sampling strategies defined.
Example `strategies.json`
```json
{
"service_strategies": [
{
"service": "foo",
"type": "probabilistic",
"param": 0.8,
"operation_strategies": [
{
"operation": "op1",
"type": "probabilistic",
"param": 0.2
},
{
"operation": "op2",
"type": "probabilistic",
"param": 0.4
}
]
},
{
"service": "bar",
"type": "ratelimiting",
"param": 5
}
],
"default_strategy": {
"type": "probabilistic",
"param": 0.5
}
}
```
`service_strategies` defines service specific sampling strategies and `operation_strategies` defines operation specific sampling strategies. There are 2 types of strategies possible: `probabilistic` and `ratelimiting` which are described [above](#client-sampling-configuration) (NOTE: `ratelimiting` is not supported for `operation_strategies`). `default_strategy` defines the catch-all sampling strategy that is propagated if the service is not included as part of `service_strategies`.
In the above example, all service `foo` operations are sampled probabilistically with a probability of 0.8 except `op1` and `op2` which are probabilistically sampled with a probability of 0.2 and 0.4 respectively. All operations for service `bar` are ratelimited at 5 traces per second. Any other service is probabilistically sampled with a probability of 0.5.

View File

@ -0,0 +1,43 @@
---
title: Windows Service Deployment
hasparent: true
---
In Windows environments, Jaeger processes can be hosted and managed as Windows services controlled via the `sc` utility. To configure such services on Windows, download [nssm.exe](https://nssm.cc/download) for the appropriate architecture, and issue commands similar to how Jaeger is typically run. The example below showcases a basic Elasticsearch setup, configured using both environment variables and process arguments.
## Agent
```bat
nssm install JaegerAgent C:\Jaeger\jaeger-agent.exe --reporter.grpc.host-port=localhost:14250
nssm set JaegerAgent AppStdout C:\Jaeger\jaeger-agent.out.log
nssm set JaegerAgent AppStderr C:\Jaeger\jaeger-agent.err.log
nssm set JaegerAgent Description Jaeger Agent service
nssm start JaegerAgent
```
## Collector
```bat
nssm install JaegerCollector C:\Jaeger\jaeger-collector.exe --es.server-urls=http://localhost:9200 --es.username=jaeger --es.password=PASSWORD
nssm set JaegerCollector AppStdout C:\Jaeger\jaeger-collector.out.log
nssm set JaegerCollector AppStderr C:\Jaeger\jaeger-collector.err.log
nssm set JaegerCollector Description Jaeger Collector service
nssm set JaegerCollector AppEnvironmentExtra SPAN_STORAGE_TYPE=elasticsearch
nssm start JaegerCollector
```
## Query UI
```bat
nssm install JaegerUI C:\Jaeger\jaeger-query.exe --es.server-urls=http://localhost:9200 --es.username=jaeger --es.password=PASSWORD
nssm set JaegerUI AppStdout C:\Jaeger\jaeger-ui.out.log
nssm set JaegerUI AppStderr C:\Jaeger\jaeger-ui.err.log
nssm set JaegerUI Description Jaeger Query service
nssm set JaegerUI AppEnvironmentExtra SPAN_STORAGE_TYPE=elasticsearch
nssm start JaegerUI
```
For additional information & docs, please see [the NSSM usage guide.](https://nssm.cc/usage)

View File

@ -0,0 +1,70 @@
---
title: Introduction
weight: 1
children:
- title: Features
url: features
---
Welcome to Jaeger's documentation portal! Below, you'll find information for beginners and experienced Jaeger users.
If you can't find what you are looking for, or have an issue not covered here, we'd love to [hear from you](/get-in-touch).
## About
Jaeger, inspired by [Dapper][dapper] and [OpenZipkin](http://zipkin.io),
is a distributed tracing system released as open source by [Uber Technologies][ubeross].
It is used for monitoring and troubleshooting microservices-based distributed systems, including:
* Distributed context propagation
* Distributed transaction monitoring
* Root cause analysis
* Service dependency analysis
* Performance / latency optimization
Uber published a blog post, [Evolving Distributed Tracing at Uber](https://eng.uber.com/distributed-tracing/), where they explain the history and reasons for the architectural choices made in Jaeger. [Yuri Shkuro](https://shkuro.com), creator of Jaeger, also published a book [Mastering Distributed Tracing](https://shkuro.com/books/2019-mastering-distributed-tracing/) that covers in-depth many aspects of Jaeger design and operation, as well as distributed tracing in general.
## Features
* [OpenTracing](http://opentracing.io/) compatible data model and instrumentation libraries
* in [Go](https://github.com/jaegertracing/jaeger-client-go), [Java](https://github.com/jaegertracing/jaeger-client-java), [Node](https://github.com/jaegertracing/jaeger-client-node), [Python](https://github.com/jaegertracing/jaeger-client-python)
and [C++](https://github.com/jaegertracing/cpp-client)
* Uses consistent upfront sampling with individual per service/endpoint probabilities
* Multiple storage backends: Cassandra, Elasticsearch, memory.
* Adaptive sampling (coming soon)
* Post-collection data processing pipeline (coming soon)
See [Features](./features/) page for more details.
## Technical Specs
* Backend components implemented in Go
* React/Javascript UI
* Supported storage backends:
* [Cassandra 3.4+](./deployment/#cassandra)
* [Elasticsearch 5.x, 6.x, 7.x](./deployment/#elasticsearch)
* [Kafka](./deployment/#kafka)
* memory storage
## Quick Start
See [running a docker all in one image](getting-started#all-in-one).
## Screenshots
### Traces View
[![Traces View](/img/traces-ss.png)](/img/traces-ss.png)
### Trace Detail View
[![Detail View](/img/trace-detail-ss.png)](/img/trace-detail-ss.png)
## Related links
- [Evolving Distributed tracing At Uber Engineering](https://eng.uber.com/distributed-tracing/)
- [Mastering Distributed Tracing](https://shkuro.com/books/2019-mastering-distributed-tracing/)
- [Tracing HTTP request latency in Go with OpenTracing](https://medium.com/opentracing/tracing-http-request-latency-in-go-with-opentracing-7cc1282a100a)
- [Distributed Tracing with Jaeger & Prometheus on Kubernetes](https://blog.openshift.com/openshift-commons-briefing-82-distributed-tracing-with-jaeger-prometheus-on-kubernetes/)
- [Using Jaeger with Istio](https://istio.io/docs/tasks/telemetry/distributed-tracing.html)
- [Using Jaeger with Envoy](https://www.envoyproxy.io/docs/envoy/latest/start/sandboxes/jaeger_tracing.html)
[dapper]: https://research.google.com/pubs/pub36356.html
[ubeross]: http://uber.github.io

87
content/docs/1.16/apis.md Normal file
View File

@ -0,0 +1,87 @@
---
title: APIs
hasparent: true
---
Jaeger components implement various APIs for saving or retrieving trace data.
The following labels are used to describe API compatibility guarantees.
* **stable** - the API guarantees backwards compatibility. If breaking changes are going to be made in the future, they will result in a new API version, e.g. `/api/v2` URL prefix or a different namespace in the IDL.
* **internal** - the APIs intended for internal communications between Jaeger components and not recommended for use by external components.
* **deprecated** - the APIs that are only maintained for legacy reasons and will be phased out in the future.
## Span reporting APIs
Agent and Collector are the two components of the Jaeger backend that can receive spans. At this time they support two sets of non-overlapping APIs.
### Thrift over UDP (stable)
The Agent can only receive spans over UDP in Thrift format. The primary API is a UDP packet that contains a Thrift-encoded `Batch` struct defined in [jaeger.thrift][jaeger.thrift] IDL file, located in the [jaeger-idl][jaeger-idl] repository. Most Jaeger Clients use Thrift's `compact` encoding, however some client libraries do not support it (notably, Node.js) and use Thrift's `binary` encoding (sent to a different UDP port). The Agent's API is defined by [agent.thrift][agent.thrift] IDL file.
For legacy reasons, the Agent also accepts spans in Zipkin format, however, only very old versions of Jaeger clients can send data in that format and it is officially deprecated.
### Protobuf via gRPC (stable)
In a typical Jaeger deployment, Agents receive spans from Clients and forward them to Collectors. Since Jaeger version 1.11 the official and recommended protocol between Agents and Collectors is gRPC with Protobuf.
The Protobuf IDL [collector.proto][collector.proto] is currently located in the main Jaeger repository, under [model/proto/api_v2][collector.proto]. In the future it will be moved to [jaeger-idl][jaeger-idl] repository ([jaeger-idl/issues/55](https://github.com/jaegertracing/jaeger-idl/issues/55)).
### Thrift over HTTP (stable)
In some cases it is not feasible to deploy Jaeger Agent next to the application, for example, when the application code is running as AWS Lambda function. In these scenarios the Jaeger Clients can be configured to submit spans directly to the Collectors over HTTP/HTTPS.
The same [jaeger.thrift][jaeger.thrift] payload can be submitted in HTTP POST request to `/api/traces` endpoint, for example, `https://jaeger-collector:14268/api/traces`. The `Batch` struct needs to be encoded using Thrift's `binary` encoding, and the HTTP request should specify the content type header:
```
Content-Type: application/vnd.apache.thrift.binary
```
### JSON over HTTP (n/a)
There is no official Jaeger JSON format that can be accepted by the collector. In the future the Protobuf-generated JSON may be supported.
### Thrift via TChannel (deprecated)
Agent and Collector can communicate using TChannel protocol. This protocol is generally not supported by the routing infrastructure and has been deprecated. It will be eventually removed from Jaeger.
### Zipkin Formats (stable)
Jaeger Collector can also accept spans in several Zipkin data format, namely JSON v1/v2 and Thrift. The Collector needs to be configured to enable Zipkin HTTP server, e.g. on port 9411 used by Zipkin collectors. The server enables two endpoints that expect POST requests:
* `/api/v1/spans` for submitting spans in Zipkin JSON v1 or Zipkin Thrift format.
* `/api/v2/spans` for submitting spans in Zipkin JSON v2.
## Trace retrieval APIs
Traces saved in the storage can be retrieved by calling Jaeger Query Service.
### gRPC/Protobuf (stable)
The recommended way for programmatically retrieving traces and other data is via gRPC endpoint defined in [query.proto][query.proto] IDL file (located in the main Jaeger repository, similar to [collector.proto][collector.proto]).
### HTTP JSON (internal)
Jaeger UI communicates with Jaeger Query Service via JSON API. For example, a trace can be retrieved via GET request to `https://jaeger-query:16686/api/traces/{trace-id-hex-string}`. This JSON API is intentionally undocumented and subject to change.
## Clients configuration (internal)
Client libraries not only submit finished spans to Jaeger backend, but also periodically poll the Agents for various configurations, such as sampling strategies. The schema for the payload is defined by [sampling.thrift][sampling.thrift], encoded as JSON using Thrift's built-in JSON generation capabilities.
## Service dependencies graph (internal)
Can be retrieved from Query Service at `/api/dependencies` endpoint. The GET request expects two parameters:
* `endTs` (number of milliseconds since epoch) - the end of the time interval
* `lookback` (in milliseconds) - the length the time interval (i.e. start-time + lookback = end-time).
The returned JSON is a list of edges represented as tuples `(caller, callee, count)`.
For programmatic access to service graph, the recommended API is gRPC/Protobuf described above.
[jaeger-idl]: https://github.com/jaegertracing/jaeger-idl/
[jaeger.thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/jaeger.thrift
[agent.thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/agent.thrift
[sampling.thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/sampling.thrift
[collector.proto]: https://github.com/jaegertracing/jaeger/blob/master/model/proto/api_v2/collector.proto
[query.proto]: https://github.com/jaegertracing/jaeger/blob/master/model/proto/api_v2/query.proto

13
content/docs/1.16/cli.md Normal file
View File

@ -0,0 +1,13 @@
---
title: CLI flags
widescreen: true
hasparent: true
---
This is auto-generated documentation for CLI flags supported by Jaeger binaries.
* CLI flags for some binaries change depending on the `SPAN_STORAGE_TYPE` environment variable. Relevant variations are included below.
* Some binaries support _commands_ (mostly informational), such as `env`, `docs`, and `version`. These commands are not included here.
* All parameters can be also provided via environment variables, by changing all letters to upper-case and replacing all punctuation characters with underscore `_`. For example, the value for the flag `--cassandra.connections-per-host` can be provided via `CASSANDRA_CONNECTIONS_PER_HOST` environment variable.
{{< cli/tools-list >}}

View File

@ -0,0 +1,469 @@
---
title: Deployment
weight: 4
children:
- title: Operator for Kubernetes
navtitle: Kubernetes
url: operator
- title: Frontend/UI
url: frontend-ui
- title: Windows
url: windows
- title: CLI Flags
url: cli
---
The main Jaeger backend components are released as Docker images on Docker Hub:
Component | Repository
--------------------- | ---
**jaeger-agent** | [hub.docker.com/r/jaegertracing/jaeger-agent/](https://hub.docker.com/r/jaegertracing/jaeger-agent/)
**jaeger-collector** | [hub.docker.com/r/jaegertracing/jaeger-collector/](https://hub.docker.com/r/jaegertracing/jaeger-collector/)
**jaeger-query** | [hub.docker.com/r/jaegertracing/jaeger-query/](https://hub.docker.com/r/jaegertracing/jaeger-query/)
**jaeger-ingester** | [hub.docker.com/r/jaegertracing/jaeger-ingester/](https://hub.docker.com/r/jaegertracing/jaeger-ingester/)
There are orchestration templates for running Jaeger with:
* Kubernetes: [github.com/jaegertracing/jaeger-kubernetes](https://github.com/jaegertracing/jaeger-kubernetes),
* OpenShift: [github.com/jaegertracing/jaeger-openshift](https://github.com/jaegertracing/jaeger-openshift).
## Configuration Options
Jaeger binaries can be configured in a number of ways (in the order of decreasing priority):
* command line arguments,
* environment variables,
* configuration files in JSON, TOML, YAML, HCL, or Java properties formats.
To see the complete list of options, run the binary with `help` command. Options that are specific to a certain storage backend are only listed if the storage type is selected. For example, to see all available options in the Collector with Cassandra storage:
```sh
$ docker run --rm \
-e SPAN_STORAGE_TYPE=cassandra \
jaegertracing/jaeger-collector:{{< currentVersion >}} \
help
```
In order to provide configuration parameters via environment variables, find the respective command line option and convert its name to UPPER_SNAKE_CASE, for example:
Command line option | Environment variable
-----------------------------------|-------------------------------
`--cassandra.connections-per-host` | `CASSANDRA_CONNECTIONS_PER_HOST`
`--metrics-backend` | `METRICS_BACKEND`
## Agent
Jaeger client libraries expect **jaeger-agent** process to run locally on each host.
The agent exposes the following ports:
Port | Protocol | Function
----- | ------- | ---
6831 | UDP | accept [jaeger.thrift][jaeger-thrift] in `compact` Thrift protocol used by most current Jaeger clients
6832 | UDP | accept [jaeger.thrift][jaeger-thrift] in `binary` Thrift protocol used by Node.js Jaeger client (because [thriftrw][thriftrw] npm package does not support `compact` protocol)
5778 | HTTP | serve configs, sampling strategies
5775 | UDP | accept [zipkin.thrift][zipkin-thrift] in `compact` Thrift protocol (deprecated; only used by very old Jaeger clients, circa 2016)
14271 | HTTP | Healthcheck at `/` and metrics at `/metrics`
It can be executed directly on the host or via Docker, as follows:
```sh
## make sure to expose only the ports you use in your deployment scenario!
docker run \
--rm \
-p6831:6831/udp \
-p6832:6832/udp \
-p5778:5778/tcp \
-p5775:5775/udp \
jaegertracing/jaeger-agent:{{< currentVersion >}}
```
### Discovery System Integration
The agents can connect point to point to a single collector address, which could be
load balanced by another infrastructure component (e.g. DNS) across multiple collectors.
The agent can also be configured with a static list of collector addresses.
On Docker, a command like the following can be used:
```sh
docker run \
--rm \
-p5775:5775/udp \
-p6831:6831/udp \
-p6832:6832/udp \
-p5778:5778/tcp \
jaegertracing/jaeger-agent:{{< currentVersion >}} \
--reporter.grpc.host-port=jaeger-collector.jaeger-infra.svc:14250
```
Or use `--reporter.tchannel.host-port=jaeger-collector.jaeger-infra.svc:14267` to use
legacy tchannel reporter.
When using gRPC, you have several options for load balancing and name resolution:
* Single connection and no load balancing. This is the default if you specify a single `host:port`. (example: `--reporter.grpc.host-port=jaeger-collector.jaeger-infra.svc:14250`)
* Static list of hostnames and round-robin load balancing. This is what you get with a comma-separated list of addresses. (example: `reporter.grpc.host-port=jaeger-collector1:14250,jaeger-collector2:14250,jaeger-collector3:14250`)
* Dynamic DNS resolution and round-robin load balancing. To get this behaviour, prefix the address with `dns:///` and gRPC will attempt to resolve the hostname using SRV records (for [external load balancing](https://github.com/grpc/grpc/blob/master/doc/load-balancing.md)), TXT records (for [service configs](https://github.com/grpc/grpc/blob/master/doc/service_config.md)), and A records. Refer to the [gRPC Name Resolution docs](https://github.com/grpc/grpc/blob/master/doc/naming.md) and the [dns_resolver.go implementation](https://github.com/grpc/grpc-go/blob/master/resolver/dns/dns_resolver.go) for more info. (example: `--reporter.grpc.host-port=dns:///jaeger-collector.jaeger-infra.svc:14250`)
### Agent level tags
Jaeger supports agent level tags, that can be added to the process tags of all spans passing through the agent. This is supported through the command line flag `--agent.tags=key1=value1,key2=value2,...,keyn=valuen`. Tags can also be set through an environment flag like so - `--agent.tags=key=${envFlag:defaultValue}` - The tag value will be set to the value of the `envFlag` environment key and `defaultValue` if not set. This feature is not supported for the tchannel reporter, enabled using the flags `--collector.host-port` or `--reporter.tchannel.host-port`.
## Collectors
The collectors are stateless and thus many instances of **jaeger-collector** can be run in parallel.
Collectors require almost no configuration, except for the location of Cassandra cluster,
via `--cassandra.keyspace` and `--cassandra.servers` options, or the location of Elasticsearch cluster, via
`--es.server-urls`, depending on which storage is specified. To see all command line options run
```sh
go run ./cmd/collector/main.go -h
```
or, if you don't have the source code
```sh
docker run -it --rm jaegertracing/jaeger-collector:{{< currentVersion >}} -h
```
At default settings the collector exposes the following ports:
Port | Protocol | Function
----- | ------- | ---
14267 | TChannel | used by **jaeger-agent** to send spans in jaeger.thrift format
14250 | gRPC | used by **jaeger-agent** to send spans in model.proto format
14268 | HTTP | can accept spans directly from clients in jaeger.thrift format over binary thrift protocol
9411 | HTTP | can accept Zipkin spans in Thrift, JSON and Proto (disabled by default)
14269 | HTTP | Healthcheck at `/` and metrics at `/metrics`
## Storage Backends
Collectors require a persistent storage backend. Cassandra and Elasticsearch are the primary supported storage backends. Additional backends are [discussed here](https://github.com/jaegertracing/jaeger/issues/638).
The storage type can be passed via `SPAN_STORAGE_TYPE` environment variable. Valid values are `cassandra`, `elasticsearch`, `kafka` (only as a buffer), `grpc-plugin`, `badger` (only with all-in-one) and `memory` (only with all-in-one).
As of version 1.6.0, it's possible to use multiple storage types at the same time by providing a comma-separated list of valid types to the `SPAN_STORAGE_TYPE` environment variable.
It's important to note that all listed storage types are used for writing, but only the first type in the list will be used for reading and archiving.
### Memory
The in-memory storage is not intended for production workloads. It's intended as a simple solution to get started quickly and
data will be lost once the process is gone.
By default, there's no limit in the amount of traces stored in memory but a limit can be established by passing an
integer value via `--memory.max-traces`.
### Badger - local storage
Experimental since Jaeger 1.9
[Badger](https://github.com/dgraph-io/badger) is an embedded local storage, only available
with **all-in-one** distribution. By default it acts as an ephemeral storage using a temporary filesystem.
This can be overridden by using the `--badger.ephemeral=false` option.
```sh
docker run \
-e SPAN_STORAGE_TYPE=badger \
-e BADGER_EPHEMERAL=false \
-e BADGER_DIRECTORY_VALUE=/badger/data \
-e BADGER_DIRECTORY_KEY=/badger/key \
-v <storage_dir_on_host>:/badger \
-p 16686:16686 \
jaegertracing/all-in-one:{{< currentVersion >}}
```
### Cassandra
Supported versions: 3.4+
Deploying Cassandra itself is out of scope for our documentation. One good
source of documentation is the [Apache Cassandra Docs](https://cassandra.apache.org/doc/latest/).
#### Configuration
##### Minimal
```sh
docker run \
-e SPAN_STORAGE_TYPE=cassandra \
-e CASSANDRA_SERVERS=<...> \
jaegertracing/jaeger-collector:{{< currentVersion >}}
```
##### All options
To view the full list of configuration options, you can run the following command:
```sh
docker run \
-e SPAN_STORAGE_TYPE=cassandra \
jaegertracing/jaeger-collector:{{< currentVersion >}} \
--help
```
#### Schema script
A script is provided to initialize Cassandra keyspace and schema
using Cassandra's interactive shell [`cqlsh`][cqlsh]:
```sh
MODE=test sh ./plugin/storage/cassandra/schema/create.sh | cqlsh
```
For production deployment, pass `MODE=prod DATACENTER={datacenter}` arguments to the script,
where `{datacenter}` is the name used in the Cassandra configuration / network topology.
The script also allows overriding TTL, keyspace name, replication factor, etc.
Run the script without arguments to see the full list of recognized parameters.
#### TLS support
Jaeger supports TLS client to node connections as long as you've configured
your Cassandra cluster correctly. After verifying with e.g. `cqlsh`, you can
configure the collector and query like so:
```sh
docker run \
-e CASSANDRA_SERVERS=<...> \
-e CASSANDRA_TLS=true \
-e CASSANDRA_TLS_SERVER_NAME="CN-in-certificate" \
-e CASSANDRA_TLS_KEY=<path to client key file> \
-e CASSANDRA_TLS_CERT=<path to client cert file> \
-e CASSANDRA_TLS_CA=<path to your CA cert file> \
jaegertracing/jaeger-collector:{{< currentVersion >}}
```
The schema tool also supports TLS. You need to make a custom cqlshrc file like
so:
```
# Creating schema in a cassandra cluster requiring client TLS certificates.
#
# Create a volume for the schema docker container containing four files:
# cqlshrc: this file
# ca-cert: the cert authority for your keys
# client-key: the keyfile for your client
# client-cert: the cert file matching client-key
#
# if there is any sort of DNS mismatch and you want to ignore server validation
# issues, then uncomment validate = false below.
#
# When running the container, map this volume to /root/.cassandra and set the
# environment variable CQLSH_SSL=--ssl
[ssl]
certfile = ~/.cassandra/ca-cert
userkey = ~/.cassandra/client-key
usercert = ~/.cassandra/client-cert
# validate = false
```
### Elasticsearch
Supported in Jaeger since 0.6.0
Supported versions: 5.x, 6.x, 7.x
Elasticsearch version is automatically retrieved from root/ping endpoint.
Based on this version Jaeger uses compatible index mappings and Elasticsearch REST API.
The version can be explicitly provided via `--es.version=` flag.
Elasticsearch does not require initialization other than
[installing and running Elasticsearch](https://www.elastic.co/downloads/elasticsearch).
Once it is running, pass the correct configuration values to the Jaeger collector and query service.
#### Configuration
##### Minimal
```sh
docker run \
-e SPAN_STORAGE_TYPE=elasticsearch \
-e ES_SERVER_URLS=<...> \
jaegertracing/jaeger-collector:{{< currentVersion >}}
```
##### All options
To view the full list of configuration options, you can run the following command:
```sh
docker run \
-e SPAN_STORAGE_TYPE=elasticsearch \
jaegertracing/jaeger-collector:{{< currentVersion >}} \
--help
```
#### Shards and Replicas for Elasticsearch indices
Shards and replicas are some configuration values to take special attention to, because this is decided upon
index creation. [This article](https://qbox.io/blog/optimizing-elasticsearch-how-many-shards-per-index) goes into
more information about choosing how many shards should be chosen for optimization.
#### Upgrade Elasticsearch version
Elasticsearch defines wire and index compatibility versions. The index compatibility defines
the minimal version a node can read data from. For example Elasticsearch 7 can read indices
created by Elasticsearch 6, however it cannot read indices created by Elasticsearch 5 even
though they use the same index mappings. Therefore upgrade from Elasticsearch 6 to 7 does not require any
data migration. However, upgrade from Elasticsearch 5 to 7 has to be done through Elasticsearch 6 and wait
until indices created by ES 5.x are removed or explicitly reindexed.
Refer to the Elasticsearch [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current//setup-upgrade.html)
for wire and index compatibility versions. Generally this information can be retrieved from root/ping REST endpoint.
##### Reindex
Manual reindexing can be used when upgrading from Elasticsearch 5 to 7 (through Elasticsearch 6)
without waiting until indices created by Elasticsearch 5 are removed.
1. Reindex all span indices to new indices with suffix `-1`:
```bash
curl -ivX POST -H "Content-Type: application/json" http://localhost:9200/_reindex -d @reindex.json
{
"source": {
"index": "jaeger-span-*"
},
"dest": {
"index": "jaeger-span"
},
"script": {
"lang": "painless",
"source": "ctx._index = 'jaeger-span-' + (ctx._index.substring('jaeger-span-'.length(), ctx._index.length())) + '-1'"
}
}
```
2. Delete indices with old mapping:
```bash
curl -ivX DELETE -H "Content-Type: application/json" http://localhost:9200/jaeger-span-\*,-\*-1
```
3. Create indices without `-1` suffix:
```bash
curl -ivX POST -H "Content-Type: application/json" http://localhost:9200/_reindex -d @reindex.json
{
"source": {
"index": "jaeger-span-*"
},
"dest": {
"index": "jaeger-span"
},
"script": {
"lang": "painless",
"source": "ctx._index = 'jaeger-span-' + (ctx._index.substring('jaeger-span-'.length(), ctx._index.length() - 2))"
}
}
```
4. Remove suffixed indices:
```bash
curl -ivX DELETE -H "Content-Type: application/json" http://localhost:9200/jaeger-span-\*-1
```
Run the commands analogically for other Jaeger indices.
There might exist more effective migration procedure. Please share with the community any findings.
### Kafka
Supported in Jaeger since 1.6.0
Supported Kafka versions: 0.9+
Kafka can be used as an intermediary buffer between collector and an actual storage.
The collector is configured with `SPAN_STORAGE_TYPE=kafka` that makes it write all received spans
into a Kafka topic. A new component [Ingester](#ingester), added in version 1.7.0, is used to read from
Kafka and store spans in another storage backend (Elasticsearch or Cassandra).
Writing to Kafka is particularly useful for building post-processing data pipelines.
#### Configuration
##### Minimal
```sh
docker run \
-e SPAN_STORAGE_TYPE=kafka \
-e KAFKA_PRODUCER_BROKERS=<...> \
-e KAFKA_TOPIC=<...> \
jaegertracing/jaeger-collector:{{< currentVersion >}}
```
##### All options
To view the full list of configuration options, you can run the following command:
```sh
docker run \
-e SPAN_STORAGE_TYPE=kafka \
jaegertracing/jaeger-collector:{{< currentVersion >}} \
--help
```
#### Topic & partitions
Unless your Kafka cluster is configured to automatically create topics, you will need to create it ahead of time. You can refer to [the Kafka quickstart documentation](https://kafka.apache.org/documentation/#quickstart_createtopic) to learn how.
You can find more information about topics and partitions in general in the [official documentation](https://kafka.apache.org/documentation/#intro_topics). [This article](https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/) provide more details about how to choose the number of partitions.
### Storage plugin
Jaeger supports gRPC based storage plugins. For more information refer to [jaeger/plugin/storage/grpc](https://github.com/jaegertracing/jaeger/tree/master/plugin/storage/grpc)
Available plugins:
* [InfluxDB](https://github.com/influxdata/jaeger-influxdb/)
```sh
docker run \
-e SPAN_STORAGE_TYPE=grpc-plugin \
-e GRPC_STORAGE_PLUGIN_BINARY=<...> \
-e GRPC_STORAGE_PLUGIN_CONFIGURATION_FILE=<...> \
jaegertracing/all-in-one:{{< currentVersion >}}
```
## Ingester
**jaeger-ingester** is a service which reads span data from Kafka topic and writes it to another storage backend (Elasticsearch or Cassandra).
Port | Protocol | Function
----- | ------- | ---
14270 | HTTP | Healthcheck at `/` and metrics at `/metrics`
To view all exposed configuration options run the following command:
```sh
docker run \
-e SPAN_STORAGE_TYPE=cassandra \
jaegertracing/jaeger-ingester:{{< currentVersion >}}
--help
```
## Query Service & UI
**jaeger-query** serves the API endpoints and a React/Javascript UI.
The service is stateless and is typically run behind a load balancer, such as [**NGINX**](https://www.nginx.com/).
At default settings the query service exposes the following port(s):
Port | Protocol | Function
----- | ------- | ---
16686 | HTTP | `/api/*` endpoints and Jaeger UI at `/`
16687 | HTTP | Healthcheck at `/` and metrics at `/metrics`
### Minimal deployment example (Elasticsearch backend):
```sh
docker run -d --rm \
-p 16686:16686 \
-p 16687:16687 \
-e SPAN_STORAGE_TYPE=elasticsearch \
-e ES_SERVER_URLS=http://<ES_SERVER_IP>:<ES_SERVER_PORT> \
jaegertracing/jaeger-query:{{< currentVersion >}}
```
### UI Base Path
The base path for all **jaeger-query** HTTP routes can be set to a non-root value, e.g. `/jaeger` would cause all UI URLs to start with `/jaeger`. This can be useful when running **jaeger-query** behind a reverse proxy.
The base path can be configured via the `--query.base-path` command line parameter or the `QUERY_BASE_PATH` environment variable.
### UI Customization and Embedding
Please refer to the [dedicated Frontend/UI page](../frontend-ui/).
## Aggregation Jobs for Service Dependencies
Production deployments need an external process which aggregates data and creates dependency links between services. Project [spark-dependencies](https://github.com/jaegertracing/spark-dependencies) is a Spark job which derives dependency links and stores them directly to the storage.
## Configuration
All binaries accepts command line properties and environmental variables, power by [viper](https://github.com/spf13/viper) and [cobra](https://github.com/spf13/cobra) libraries. Please refer to the [CLI Flags](../cli/) page for more information.
[cqlsh]: http://cassandra.apache.org/doc/latest/tools/cqlsh.html
[zipkin-thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/zipkincore.thrift
[jaeger-thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/jaeger.thrift
[thriftrw]: https://www.npmjs.com/package/thriftrw

26
content/docs/1.16/faq.md Normal file
View File

@ -0,0 +1,26 @@
---
title: Frequently Asked Questions
navtitle: FAQs
description: Answers to some frequently asked questions about Jaeger.
weight: 11
---
## Why is the Dependencies page empty?
The Dependencies page shows a graph of services traced by Jaeger and connections between them. When you are using `all-in-one` binary with in-memory storage, the graph is calculated on-demand from all the traces stored in memory. However, if you are using a real distributed storage like Cassandra or Elasticsearch, it is too expensive to scan all the data in the database to build the service graph. Instead, the Jaeger project provides "big data" jobs that can be used to extract the service graph data from traces:
* https://github.com/jaegertracing/spark-dependencies - the older Spark job that can be run periodically
* https://github.com/jaegertracing/jaeger-analytics - the new (experimental) streaming Flink jobs that run continuously and builds the service graph in smaller time intervals
## Why do I not see any spans in Jaeger?
Please refer to the [Troubleshooting](../troubleshooting/) guide.
## Do I need to run jaeger-agent?
`jaeger-agent` is not always necessary. Jaeger client libraries can be configured to export trace data directly to `jaeger-collector`. However, the following are the reasons why running `jaeger-agent` is recommended:
* If we want Jaeger client libraries to send trace data directly to collectors, we must provide them with a URL of the HTTP endpoint. It means that our applications require additional configuration containing this parameter, especially if we are running multiple Jaeger installations (e.g. in different availability zones or regions) and want the data sent to a nearby installation. In contrast, when using the agent, the libraries require no additional configuration because the agent is always accessible via `localhost`. It acts as a sidecar and proxies the requests to the appropriate collectors.
* The agent can be configured to enrich the tracing data with infrastructure-specific metadata by adding extra tags to the spans, such as the current zone, region, etc. If the agent is running as a host daemon, it will be shared by all applications running on the same host. If the agent is running as a true sidecar, i.e. one per application, it can provide additional functionality such as strong authentication, multi-tenancy (see [this blog post](https://medium.com/jaegertracing/jaeger-and-multitenancy-99dfa1d49dc0)), pod name, etc.
* If we want Jaeger client libraries to use sampling strategies that are centrally configured in the collectors, this is only possible by using the `/sampling` HTTP endpoint on the agent. There is no technical reason why this endpoint cannot be implemented directly in the collectors, it's just [not done yet](https://github.com/jaegertracing/jaeger/issues/1971).
* Agents allow implementing traffic control to the collectors. If we have thousands of hosts in the data center, each running many applications, and each application sending data directly to the collectors, there may be too many open connections for each collector to handle. The agents can load balance this traffic with fewer connections.

Some files were not shown because too many files have changed in this diff Show More