Compare commits
394 Commits
release-1.
...
main
Author | SHA1 | Date |
---|---|---|
|
e47275c7c6 | |
|
27107420dd | |
|
2f7339694e | |
|
abe2243476 | |
|
cd51407f41 | |
|
4b04d5038d | |
|
96b4436972 | |
|
11f8965ab9 | |
|
6379f90336 | |
|
55d6c1ff15 | |
|
ff40c080dc | |
|
25cb878e48 | |
|
2f4be0d7c7 | |
|
0f7df55b8b | |
|
991207d457 | |
|
3abc5a2fe3 | |
|
1b39d03173 | |
|
236bb7eee9 | |
|
5f2bf8e078 | |
|
260627a8f9 | |
|
839f8e3948 | |
|
5a1b33f4d0 | |
|
d8d3c6d346 | |
|
cf240acb9d | |
|
34eb385ac3 | |
|
ae17f88568 | |
|
aca909392b | |
|
4b358b05c0 | |
|
1b8318ace0 | |
|
56a1806301 | |
|
2f3b661aa9 | |
|
02a5cc267f | |
|
13635684c6 | |
|
744c28c4e0 | |
|
3fa5ff6921 | |
|
08412ebd50 | |
|
42e8127cb7 | |
|
b43f6bb6b2 | |
|
ead537303d | |
|
c15a5245c7 | |
|
6427455fee | |
|
952ea6d7ea | |
|
20148ef3f1 | |
|
fe15d425fe | |
|
2967775a44 | |
|
941b99a445 | |
|
9059411246 | |
|
48486c4f82 | |
|
726bfdf1ad | |
|
f5e43aa925 | |
|
e08677ca40 | |
|
6702472974 | |
|
1f904bf0e2 | |
|
875a2975f4 | |
|
5aad41992b | |
|
9cc4fd42ee | |
|
4b83aff91b | |
|
0d98bc772d | |
|
e362787075 | |
|
4cafa35b1e | |
|
a3020c4704 | |
|
501c53c075 | |
|
e8c7d0e02b | |
|
030fafba74 | |
|
9a4bb74023 | |
|
a8321e9272 | |
|
0f4dc8c4d8 | |
|
0a07d62f81 | |
|
496a030b8a | |
|
fe1ef1b0f5 | |
|
a1ce17bfb0 | |
|
1f0998b3eb | |
|
a845e680db | |
|
ff06500df2 | |
|
e1cd72660e | |
|
ddd01751cc | |
|
8fe5a47de7 | |
|
483dc1018f | |
|
506afcec77 | |
|
a1df2985cf | |
|
57ed087590 | |
|
96f77eb7b7 | |
|
b67c3286c2 | |
|
047356abea | |
|
c58ff3c304 | |
|
e96dea1983 | |
|
57b1ed769f | |
|
dda5852ddb | |
|
fb064fac2c | |
|
49d8f04ffe | |
|
b70af0a1ce | |
|
efde64553f | |
|
32c49824fb | |
|
af008c9934 | |
|
a989b70c27 | |
|
321e5619dc | |
|
c8b78434a2 | |
|
7e307a9f32 | |
|
33b0df9642 | |
|
d4ca50de94 | |
|
e789e16c71 | |
|
aa0cccb391 | |
|
2d5e8299e4 | |
|
3be8ee875c | |
|
b7efb67f8e | |
|
5481e1b85f | |
|
d8c40b81ff | |
|
229b8b837e | |
|
9378fff9e4 | |
|
ca9923a5b3 | |
|
d514c7d417 | |
|
0ad71e51d2 | |
|
db5829a009 | |
|
a275428202 | |
|
d863c8ed72 | |
|
9830d6939d | |
|
e4dd0afb19 | |
|
3dbd3efc1c | |
|
70dca5b115 | |
|
67dc2a57ef | |
|
448561c17f | |
|
cba706f8f5 | |
|
ab2d1efddc | |
|
808ebb6b94 | |
|
d1b8f1de42 | |
|
4f219e15ab | |
|
830752a1df | |
|
3088d1050a | |
|
1209812a79 | |
|
45a35762f3 | |
|
aee39d229a | |
|
8372518994 | |
|
f51d66b55c | |
|
4f0b5f0c83 | |
|
597e983b3b | |
|
82057aa29e | |
|
8faa5b764a | |
|
5eeb00655b | |
|
9ee5914f8b | |
|
b141dad999 | |
|
3df7655155 | |
|
9445d0b752 | |
|
13d8d0a353 | |
|
bc08f1c4b9 | |
|
b49cea93ec | |
|
303a67bc0f | |
|
cb1baba973 | |
|
fd727f5c52 | |
|
edb1c0950d | |
|
1467a50fa2 | |
|
ab06932153 | |
|
83707105ae | |
|
595c299937 | |
|
6b1ded7de1 | |
|
57be545422 | |
|
41ab6d8722 | |
|
1c9fbb949b | |
|
0fc07b038f | |
|
a77d378944 | |
|
5d1dddfbb8 | |
|
462508693a | |
|
e272409636 | |
|
a027bffc27 | |
|
59a33b9ec9 | |
|
ab458816a2 | |
|
5669398ef1 | |
|
7945007639 | |
|
5cddbf2f1f | |
|
a299a21c3c | |
|
0c0d5746ec | |
|
c8d46c99b6 | |
|
bb60377017 | |
|
7e8ef6cc63 | |
|
83ee5f00ae | |
|
36b8685831 | |
|
320d60b8fa | |
|
21676c63fb | |
|
e942c23502 | |
|
6995017ece | |
|
54f6d84ba7 | |
|
a44e51bd97 | |
|
f334558799 | |
|
869423ee30 | |
|
bc8ceee6d6 | |
|
a1b93ce7dd | |
|
4187d2bc93 | |
|
0e63cb29cc | |
|
c93d97eaeb | |
|
f8ea5c9d94 | |
|
f846d48a62 | |
|
b1af07adc1 | |
|
e879666e52 | |
|
cee94fe4d2 | |
|
9b9d0440e9 | |
|
a36d03487d | |
|
832a3ecad6 | |
|
6c61b39065 | |
|
398a0d48b3 | |
|
f302d1460f | |
|
d182994698 | |
|
f69a708b61 | |
|
204bbef2c1 | |
|
c339401bf1 | |
|
3c326283d4 | |
|
585d59d75c | |
|
f479f78f84 | |
|
f8975a5984 | |
|
2e84c710f2 | |
|
8f40818edb | |
|
4db93b42cb | |
|
a0f35ea802 | |
|
ceba9daf49 | |
|
ce627d40dd | |
|
ed20bcbeca | |
|
101c39db85 | |
|
543329a1fd | |
|
097291d6dc | |
|
0936851fef | |
|
94f929e323 | |
|
d7cd6a1e65 | |
|
663b65db49 | |
|
16a738b8a6 | |
|
ee52c3dd1f | |
|
9da1045f9f | |
|
70a5b15409 | |
|
5135a1ae50 | |
|
5ef00ac9be | |
|
89637b6f45 | |
|
916ed3a751 | |
|
c2ac9fa1ce | |
|
9a22d41ec1 | |
|
f15ab41b1c | |
|
2095173da6 | |
|
4d627e8bba | |
|
562fba4c95 | |
|
d9825f3b50 | |
|
5aeb97b026 | |
|
cb9ff33ac8 | |
|
58a41dab44 | |
|
5ac5a79cff | |
|
f701acb2d4 | |
|
82d6bdffd9 | |
|
08b96a308b | |
|
2c829b7e30 | |
|
8e299be461 | |
|
c1d6d05ec0 | |
|
89aeb810e4 | |
|
b8d1fff949 | |
|
4bb360ff4d | |
|
855ca70bf6 | |
|
aacddd0efb | |
|
b25eda44fc | |
|
7dbb843ad4 | |
|
b4fe14a344 | |
|
97524d35c2 | |
|
cefeab1b9b | |
|
be8c268411 | |
|
152e4b309c | |
|
a25acad72a | |
|
502ba1e4cd | |
|
f1f6a5e8f6 | |
|
bf25cda794 | |
|
39d36bd6b4 | |
|
f080597012 | |
|
e3af450e1c | |
|
e431e64e5f | |
|
11cbedc154 | |
|
c53f99674d | |
|
b92f905142 | |
|
cbf4222c56 | |
|
b8ac666f1d | |
|
6d6d2c549a | |
|
bd4e37352c | |
|
1b655f7081 | |
|
8456ad0cc1 | |
|
b8e14b1cea | |
|
edda500ce0 | |
|
3fea5595d9 | |
|
3183e363c3 | |
|
93d33899eb | |
|
ef5dd20e3c | |
|
48388e88eb | |
|
bcce1811c2 | |
|
5c15edbf94 | |
|
f4772434a9 | |
|
c859bfe536 | |
|
d408b4a741 | |
|
9fd06a2950 | |
|
16f0a66fbb | |
|
ff6acc2148 | |
|
6732d5698e | |
|
9809c093e3 | |
|
13e258137e | |
|
0cee2fa95e | |
|
b87ff3a5d9 | |
|
d886371e7a | |
|
64ad8e3f3e | |
|
e3c96010fc | |
|
150dd4dba3 | |
|
4213b9df45 | |
|
3c2076505f | |
|
162794986b | |
|
4fc0dcefbe | |
|
531466d6cf | |
|
3f47b0d3b8 | |
|
a37d2834bd | |
|
941353d0fe | |
|
f6224cb825 | |
|
523db89af5 | |
|
ddc8df0a95 | |
|
53c3d39616 | |
|
fd2eeae56e | |
|
5d6eb845dd | |
|
e1fd40296c | |
|
69a72d6547 | |
|
5004730332 | |
|
4f2787a52b | |
|
62fafef710 | |
|
60c00ad10b | |
|
29b4851aca | |
|
0d006f026b | |
|
0bd2f40c20 | |
|
a1cde56a05 | |
|
038c2c5946 | |
|
bf3fdec035 | |
|
4fccb87520 | |
|
6d06cc2da6 | |
|
416776a7d8 | |
|
871c43a939 | |
|
4e3a846665 | |
|
5202c2f2ba | |
|
ac7b08bdf2 | |
|
c48d249fc1 | |
|
1990d788a6 | |
|
b6d45d0309 | |
|
974f2efab3 | |
|
bb39918085 | |
|
836dbff903 | |
|
47232d5804 | |
|
d24e55f27d | |
|
446151acc3 | |
|
b3006922e6 | |
|
9c973cfaa6 | |
|
28561f4333 | |
|
0c5c560738 | |
|
5901a450fa | |
|
a916209d69 | |
|
1c7ebb28d3 | |
|
fd2957980c | |
|
903a5fee57 | |
|
f38e55512f | |
|
860aa8713d | |
|
e376091bf1 | |
|
4e7da9918b | |
|
28d8697589 | |
|
51ea050dab | |
|
793fbc4036 | |
|
ae75f207e6 | |
|
1f2ac05c95 | |
|
0217a54dae | |
|
3ddf63a3ee | |
|
356d6b7aa2 | |
|
ba3b53f04a | |
|
eacb52f332 | |
|
62c430d17f | |
|
dc4960b824 | |
|
83703bdc5b | |
|
6e78ed840e | |
|
ffa59ac9f8 | |
|
74ddbb0f58 | |
|
7f1f07f182 | |
|
90a0349d58 | |
|
9aaf8219c7 | |
|
e72f4c0a1d | |
|
fce284c411 | |
|
7de81cb9e9 | |
|
85ee7847e4 | |
|
e9de995ccc | |
|
9831b9bd73 | |
|
a7ce3da00f | |
|
2e184bed99 | |
|
485564942b | |
|
b9d8e231f7 | |
|
793c558d1e | |
|
7b3fb9f0ed | |
|
62ed0f8618 | |
|
545fb93e1c | |
|
524e640202 | |
|
4a4b714945 | |
|
c2facf7aca | |
|
7769941187 | |
|
060e2cb4db | |
|
4324fe1955 | |
|
4c2649b78c |
|
@ -0,0 +1,31 @@
|
|||
# For a description of configuration options, see
|
||||
# https://www.streetsidesoftware.com/vscode-spell-checker/docs/configuration/
|
||||
# cSpell:ignore textlint textlintrc
|
||||
version: '0.2'
|
||||
caseSensitive: true
|
||||
ignorePaths:
|
||||
- '*.svg'
|
||||
- node_modules
|
||||
- public
|
||||
- tmp
|
||||
patterns:
|
||||
- name: CodeBlock
|
||||
pattern: |
|
||||
/
|
||||
^(\s*[~`]{3,}) # code-block start
|
||||
.* # all languages and options, e.g. shell {hl_lines=[12]}
|
||||
[\s\S]*? # content
|
||||
\1 # code-block end - cSpell:disable-next-line
|
||||
/igmx
|
||||
languageSettings:
|
||||
- languageId: markdown
|
||||
ignoreRegExpList:
|
||||
- CodeBlock
|
||||
dictionaryDefinitions:
|
||||
- name: project-words
|
||||
path: .cspell/project-words.txt
|
||||
- name: project-names
|
||||
path: .cspell/project-names.g.txt
|
||||
dictionaries:
|
||||
- project-words
|
||||
- project-names
|
|
@ -0,0 +1,56 @@
|
|||
# Maintainers (current and former)
|
||||
Yuri Shkuro yurishkuro
|
||||
Pavol Loffay pavolloffay
|
||||
Joe Farro
|
||||
Juraci Paixão Kröhling
|
||||
|
||||
# Prometheus
|
||||
Julius Volz
|
||||
|
||||
# Other names (sometimes from github links)
|
||||
robbert
|
||||
|
||||
## GSOC 2025
|
||||
Minh Nguyen
|
||||
|
||||
## LFX Mentorship March-May 2025 (Term 1)
|
||||
Manik Mehta
|
||||
Hariom Gupta
|
||||
|
||||
## LFX Mentorship Sep-Nov 2024 (Term 3)
|
||||
Ankit Kurmi
|
||||
Mehul Gautam
|
||||
|
||||
## LFX Mentorship June-August 2024 (Term 2)
|
||||
Harshith Mente
|
||||
Raghuram Kannan
|
||||
Saransh Shankar
|
||||
|
||||
## LFX Mentorship March-May 2024 (Term 1)
|
||||
Harshvir Potpose
|
||||
James Ryans
|
||||
Pushkar Mishra
|
||||
|
||||
## LFX Mentorship Sep-Nov 2023
|
||||
Ansh Goyal
|
||||
Prathamesh Mutkure
|
||||
Yashwanth Reddy
|
||||
|
||||
## LFX Mentorship Jun-Aug 2023
|
||||
Afzal Ansari
|
||||
GLVS Kiriti
|
||||
|
||||
## Google Summer of Code 2023
|
||||
Ha Anh Vu
|
||||
|
||||
## Outreachy Dec'20 - Mar'21
|
||||
Ashmita Bohara
|
||||
|
||||
## Outreachy Dec'18 - Mar'19
|
||||
Gaby Soria
|
||||
|
||||
## Outreachy May'18 - Aug'18
|
||||
Prakriti Bansal
|
||||
|
||||
## Outreachy Dec'17 - Mar'18
|
||||
Kara de la Marck
|
|
@ -0,0 +1,137 @@
|
|||
alertmanager
|
||||
autoscale
|
||||
autoscaler
|
||||
bitnami
|
||||
bootcamp
|
||||
cassandra
|
||||
certfile
|
||||
clickhouse
|
||||
cncf
|
||||
configtls
|
||||
containerd
|
||||
cqlsh
|
||||
cqlshrc
|
||||
daemonset
|
||||
darwin
|
||||
datacenter
|
||||
datasource
|
||||
datastax
|
||||
developercertificate
|
||||
Docsy
|
||||
downsampling
|
||||
dropwizard
|
||||
dsoie
|
||||
eddsa
|
||||
emsgsize
|
||||
errorf
|
||||
expvar
|
||||
fargate
|
||||
findRESubmatch
|
||||
flink
|
||||
fluentd
|
||||
glvs
|
||||
gobin
|
||||
gopath
|
||||
Hashicorp
|
||||
hasparent
|
||||
healthcheck
|
||||
healthcheckv2
|
||||
healthz
|
||||
hostnames
|
||||
hostport
|
||||
htmltest
|
||||
htpasswd
|
||||
hugo
|
||||
ingester
|
||||
ingester's
|
||||
ingesters
|
||||
istio
|
||||
istio's
|
||||
jaegertracing
|
||||
jaxrs
|
||||
jdoe
|
||||
jemalloc
|
||||
keycloak
|
||||
keyn
|
||||
keyregistry
|
||||
keyserver
|
||||
keyspace
|
||||
kube
|
||||
kubernetes
|
||||
logtostderr
|
||||
logz
|
||||
lookback
|
||||
mailgroup
|
||||
mentee
|
||||
mentees
|
||||
mentorships
|
||||
metricsquery
|
||||
metricsstore
|
||||
metricstore
|
||||
microsim
|
||||
myapp
|
||||
myappnamespace
|
||||
mynamespace
|
||||
myproject
|
||||
myservice
|
||||
myversion
|
||||
navtitle
|
||||
nginx
|
||||
nssm
|
||||
nvmrc
|
||||
olap
|
||||
openpgp
|
||||
opensearch
|
||||
openshift
|
||||
opentelemetry
|
||||
opentracing
|
||||
openzipkin
|
||||
operatorhub
|
||||
otel
|
||||
otelcol
|
||||
otlp
|
||||
outreachy
|
||||
parentbased
|
||||
podman
|
||||
pprof
|
||||
procs
|
||||
prometheus
|
||||
prometheusexporter
|
||||
promscale
|
||||
proto
|
||||
protobuf
|
||||
quickstart
|
||||
ratelimited
|
||||
ratelimiting
|
||||
rawhtml
|
||||
readyz
|
||||
rebalance
|
||||
reindex
|
||||
reindexing
|
||||
rolebinding
|
||||
shortcode
|
||||
spanmetrics
|
||||
stackoverflow
|
||||
statefulset
|
||||
strimzi
|
||||
strimzi's
|
||||
struct
|
||||
subpages
|
||||
tchannel
|
||||
thriftrw
|
||||
tolerations
|
||||
tracegen
|
||||
uber
|
||||
uberctx
|
||||
unmarshal
|
||||
unmarshaling
|
||||
upsample
|
||||
usercert
|
||||
userkey
|
||||
valuen
|
||||
vlog
|
||||
warnf
|
||||
weaveworks
|
||||
workqueue
|
||||
xvzf
|
||||
zipkin
|
|
@ -0,0 +1,11 @@
|
|||
# To get started with Dependabot version updates, you'll need to specify which
|
||||
# package ecosystems to update and where the package manifests are located.
|
||||
# Please see the documentation for all configuration options:
|
||||
# https://docs.github.com/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file
|
||||
|
||||
version: 2
|
||||
updates:
|
||||
- package-ecosystem: "github-actions"
|
||||
directory: "/"
|
||||
schedule:
|
||||
interval: "daily"
|
|
@ -1,19 +1,33 @@
|
|||
name: Release
|
||||
|
||||
on:
|
||||
create:
|
||||
branches: [ main ]
|
||||
tags: ["release-*.*.*"]
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
version_v1:
|
||||
required: true
|
||||
type: string
|
||||
description: "Version number for v1 docs, in #.#.# format"
|
||||
|
||||
version_v2:
|
||||
required: true
|
||||
type: string
|
||||
description: "Version number for v2 docs, in #.#.# format"
|
||||
|
||||
dry_run:
|
||||
required: true
|
||||
type: boolean
|
||||
description: Do a dry run.
|
||||
|
||||
jobs:
|
||||
release:
|
||||
prepare-release:
|
||||
if: ${{ github.repository == 'jaegertracing/documentation' }}
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v2
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: true
|
||||
|
||||
- uses: actions/setup-python@v2
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: 3.8
|
||||
|
||||
|
@ -24,12 +38,12 @@ jobs:
|
|||
git config user.email "jaegertracingbot+jaeger-tracing@googlegroups.com"
|
||||
# git config credential.helper "store --file=.git/credentials"
|
||||
# echo "https://${{ secrets.JAEGERTRACINGBOT_GITHUB_TOKEN }}" > .git/credentials
|
||||
export RELEASE_TAG=${GITHUB_REF##*/}
|
||||
./scripts/release.sh
|
||||
export DRY_RUN=${{ inputs.dry_run }}
|
||||
./scripts/release.sh ${{ inputs.version_v1 }} ${{ inputs.version_v2 }}
|
||||
|
||||
- name: GH CLI create PR
|
||||
run: |
|
||||
export TAG=${GITHUB_REF##*/}
|
||||
export TAG="${{ inputs.version_v2 }}/${{ inputs.version_v1 }}"
|
||||
git checkout -b gen-release-${TAG} # branch is need for GH CLI
|
||||
git push origin gen-release-${TAG} # branch has to be on remote before PR is opened
|
||||
gh pr create --base main --title "Release ${TAG}" --body "Release ${TAG}. This PR is created from CI and is part of the release process."
|
||||
|
|
|
@ -0,0 +1,36 @@
|
|||
name: Test
|
||||
# cSpell:ignore github wjdp
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
push:
|
||||
branches: [main]
|
||||
pull_request:
|
||||
branches: [main]
|
||||
|
||||
jobs:
|
||||
build-and-check-links:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- uses: actions/setup-node@v4
|
||||
with: { node-version-file: .nvmrc }
|
||||
- run: npm install
|
||||
- name: Build and check links
|
||||
run: npm run check:links:all # command also builds the site
|
||||
|
||||
spellcheck:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-node@v4
|
||||
with: { node-version-file: .nvmrc }
|
||||
- run: npm install
|
||||
- run: npm run check:spelling
|
||||
|
||||
block-pr-from-main-branch:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Ensure PR is not on main branch
|
||||
uses: jaegertracing/jaeger/.github/actions/block-pr-from-main-branch@main
|
|
@ -1,7 +1,20 @@
|
|||
public/
|
||||
resources/
|
||||
# Generated files
|
||||
.cspell/*.g.txt
|
||||
|
||||
# npm assets
|
||||
node_modules/
|
||||
package-lock.json
|
||||
|
||||
# Hugo-generated assets
|
||||
.hugo_build.lock
|
||||
/public
|
||||
/resources
|
||||
|
||||
/.netlify
|
||||
|
||||
.DS_Store
|
||||
.idea
|
||||
|
||||
bin/
|
||||
tmp/
|
||||
.idea
|
||||
.vscode/
|
||||
|
|
|
@ -1,3 +0,0 @@
|
|||
DirectoryPath: public
|
||||
IgnoreDirectoryMissingTrailingSlash: true
|
||||
IgnoreAltMissing: true
|
|
@ -0,0 +1,9 @@
|
|||
DirectoryPath: public
|
||||
CheckExternal: false
|
||||
IgnoreAltMissing: true
|
||||
IgnoreURLs:
|
||||
- /livereload.js # cSpell:disable-line
|
||||
- ^/docs/latest/ # ignore redirect of latest
|
||||
# Valid URL, whether canonical (before reorg) or not (and handled via a redirect)
|
||||
- ^/docs/1.[6-9]\d/
|
||||
- ^\.\./\.\./sampling/
|
|
@ -1,4 +1,21 @@
|
|||
CacheExpires: 13300h # ~ 18 months
|
||||
DirectoryPath: public
|
||||
IgnoreDirectoryMissingTrailingSlash: true
|
||||
CheckExternal: false
|
||||
IgnoreAltMissing: true
|
||||
IgnoreDirs:
|
||||
- docs/1.*
|
||||
IgnoreURLs:
|
||||
- /livereload.js # cSpell:disable-line
|
||||
- ^https://www.jaegertracing.io/js/lunr
|
||||
# Valid URL, whether canonical (before reorg) or not (and handled via a redirect)
|
||||
- ^/docs/1.[6-9]\d/sampling/
|
||||
- ^/docs/latest/architecture/
|
||||
# Valid URLs, but servers yield 403 or similar errors
|
||||
- ^https://(twitter|x).com/[Jj]aeger[Tt]racing
|
||||
- ^https://calendar.google.com/
|
||||
- ^https://eng.uber.com/(distributed-tracing/)?
|
||||
- ^https://war.ukraine.ua/support-ukraine/
|
||||
- ^https://www.oracle.com/cloud
|
||||
- ^https://stackoverflow.com/questions/tagged/jaeger
|
||||
# Temporary: invalid URLs for which replacements need to be found
|
||||
# Temporary: as we work on https://github.com/jaegertracing/documentation/issues/889
|
||||
- ^https://github.com/jaegertracing/jaeger/(blob|tree)/(v2|main/(internal|model|pkg))
|
||||
|
|
|
@ -0,0 +1 @@
|
|||
**
|
|
@ -118,9 +118,14 @@ then you just add a line to every git commit message:
|
|||
|
||||
using your real name (sorry, no pseudonyms or anonymous contributions.)
|
||||
|
||||
You can add the sign off when creating the git commit via `git commit -s`.
|
||||
You can add the sign off when creating the git commit via `git commit -s`. Make sure that your name and email are correctly configured in your local git and match those on your GitHub account:
|
||||
|
||||
If you want this to be automatic you can set up some aliases:
|
||||
```shell
|
||||
git config --global user.name "FIRST_NAME LAST_NAME"
|
||||
git config --global user.email "MY_NAME@example.com"
|
||||
```
|
||||
|
||||
If you want signing to be automatic you can set up some aliases:
|
||||
|
||||
```
|
||||
git config --add alias.amend "commit -s --amend"
|
||||
|
|
114
Makefile
114
Makefile
|
@ -1,56 +1,84 @@
|
|||
HTMLPROOFER = bundle exec htmlproofer
|
||||
HUGO_THEME = jaeger-docs
|
||||
THEME_DIR := themes/$(HUGO_THEME)
|
||||
# Copyright (c) The Jaeger Authors.
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
#
|
||||
# cSpell:ignore refcache
|
||||
|
||||
client-libs-docs:
|
||||
# invoke one group of commands in bash to allow pushd/popd
|
||||
@for d in $(shell ls -d content/docs/*); do \
|
||||
/bin/bash -c "pushd $$d/; \
|
||||
rm -f client-libraries.md client-features.md; \
|
||||
ln -s ../../_client_libs/client-libraries.md; \
|
||||
ln -s ../../_client_libs/client-features.md; \
|
||||
popd"; \
|
||||
echo "sym-linked content/_client_libs/*.md -> $$d/"; \
|
||||
done
|
||||
HTMLTEST ?= htmltest
|
||||
HTMLTEST_DIR ?= tmp/.htmltest
|
||||
|
||||
generate: client-libs-docs
|
||||
# Use $(HTMLTEST) in PATH, if available; otherwise, we'll get a copy
|
||||
ifeq (, $(shell which $(HTMLTEST)))
|
||||
override HTMLTEST=$(HTMLTEST_DIR)/bin/htmltest
|
||||
ifeq (, $(shell which $(HTMLTEST)))
|
||||
GET_LINK_CHECKER_IF_NEEDED=get-link-checker
|
||||
endif
|
||||
endif
|
||||
|
||||
develop: generate
|
||||
HUGO_PREVIEW=true hugo server \
|
||||
--buildDrafts \
|
||||
--buildFuture \
|
||||
--ignoreCache
|
||||
# generate currently doesn't do anything, but can be useful in the future.
|
||||
generate:
|
||||
|
||||
develop: generate
|
||||
npm run serve
|
||||
|
||||
clean:
|
||||
rm -rf public
|
||||
rm -rf $(HTMLTEST_DIR) public/* resources
|
||||
|
||||
netlify-production-build: generate
|
||||
hugo --minify
|
||||
get-link-checker:
|
||||
rm -Rf $(HTMLTEST_DIR)/bin
|
||||
curl https://raw.githubusercontent.com/wjdp/htmltest/master/godownloader.sh \
|
||||
| bash -s -- -b $(HTMLTEST_DIR)/bin
|
||||
|
||||
netlify-deploy-preview: generate
|
||||
HUGO_PREVIEW=true hugo \
|
||||
--buildDrafts \
|
||||
--buildFuture \
|
||||
--baseURL $(DEPLOY_PRIME_URL) \
|
||||
--minify
|
||||
# Use --keep-going to ensure that the refcache gets saved even if there are
|
||||
# link-checking errors.
|
||||
check-links: $(GET_LINK_CHECKER_IF_NEEDED)
|
||||
$(MAKE) --keep-going GET_LINK_CHECKER_IF_NEEDED= \
|
||||
_restore-refcache _check-links _save-refcache
|
||||
|
||||
netlify-branch-deploy: generate
|
||||
hugo \
|
||||
--buildDrafts \
|
||||
--buildFuture \
|
||||
--baseURL $(DEPLOY_PRIME_URL) \
|
||||
--minify
|
||||
_restore-refcache:
|
||||
mkdir -p $(HTMLTEST_DIR)
|
||||
cp data/refcache.json $(HTMLTEST_DIR)/refcache.json
|
||||
|
||||
build: clean generate
|
||||
hugo -v
|
||||
_check-links:
|
||||
$(HTMLTEST) --log-level 1 --conf .htmltest.yml
|
||||
|
||||
link-checker-setup:
|
||||
curl https://raw.githubusercontent.com/wjdp/htmltest/master/godownloader.sh | bash
|
||||
_save-refcache:
|
||||
@echo "Saving refcache.json to data/refcache.json"
|
||||
jq . $(HTMLTEST_DIR)/refcache.json > data/refcache.json
|
||||
|
||||
run-link-checker:
|
||||
bin/htmltest
|
||||
check-links-all: check-links check-links-older
|
||||
|
||||
check-internal-links: clean build link-checker-setup run-link-checker
|
||||
check-links-older: $(GET_LINK_CHECKER_IF_NEEDED)
|
||||
$(HTMLTEST) --log-level 1 --conf .htmltest.old-versions.yml
|
||||
|
||||
check-all-links: clean build link-checker-setup
|
||||
bin/htmltest --conf .htmltest.external.yml
|
||||
check-links-internal:
|
||||
$(HTMLTEST) --log-level 1 --skip-external --conf .htmltest.yml
|
||||
|
||||
.cspell/project-names.g.txt: .cspell/project-names-src.txt
|
||||
cat .cspell/project-names-src.txt | grep -v '^#' | grep -v '^\s*$$' | tr ' ' '\n' > .cspell/project-names.g.txt
|
||||
|
||||
_spellcheck: .cspell/project-names.g.txt
|
||||
./scripts/spellcheck.sh
|
||||
|
||||
fetch-blog-feed:
|
||||
curl -s -o assets/data/medium.xml https://medium.com/feed/jaegertracing
|
||||
|
||||
fetch-roadmap:
|
||||
python3 scripts/generate_roadmap.py
|
||||
|
||||
# only x.y.0 semver values are valid for kicking off a new release.
|
||||
SEMVER_REGEX := ^([0-9]+\.){2}0$$
|
||||
VALID_VERSION := $(shell echo "$(VERSION)" | grep -E "$(SEMVER_REGEX)")
|
||||
|
||||
ifneq "$(VALID_VERSION)" "$(VERSION)"
|
||||
validate-version:
|
||||
@echo "ERROR: Invalid VERSION=$(VERSION). Must be in format x.y.0."
|
||||
@false # Exit with an error code
|
||||
else
|
||||
validate-version:
|
||||
@echo "VERSION=$(VERSION) is valid."
|
||||
endif
|
||||
|
||||
# `make start-release VERSION=x.y.0
|
||||
start-release: validate-version
|
||||
git tag release-$(VERSION)
|
||||
git push upstream release-$(VERSION)
|
||||
|
|
62
README.md
62
README.md
|
@ -2,28 +2,32 @@
|
|||
|
||||
# Jaeger website
|
||||
|
||||
This repo houses all the assets used to build the Jaeger website, available at https://jaegertracing.io.
|
||||
This repo houses all the assets used to build the Jaeger website, available at <https://jaegertracing.io>.
|
||||
|
||||
The site is built with [Hugo](https://gohugo.io/) and hosted by [Netlify](https://www.netlify.com/).
|
||||
|
||||
## Setup
|
||||
|
||||
Install the "extended" Hugo binary from [hugo/releases](https://github.com/gohugoio/hugo/releases) (use one of the `hugo_extended_*` binaries) or
|
||||
use a package manager if it is available for your operating system.
|
||||
Install the active LTS version of Node.js, then run the following command from
|
||||
the directory of this repo's clone:
|
||||
|
||||
> The "extended" version of Hugo supports [Sass](https://sass-lang.org), which is necessary to build the site locally.
|
||||
```bash
|
||||
npm install
|
||||
```
|
||||
|
||||
The currently used version of Hugo is defined in the [`netlify.toml`](./netlify.toml) configuration file.
|
||||
This will also install the required version of Hugo.
|
||||
|
||||
## Running the site locally
|
||||
|
||||
If you want to develop the site locally, you can run a single command (assuming that you've run the [setup](#setup)):
|
||||
|
||||
```bash
|
||||
$ make develop
|
||||
make develop
|
||||
```
|
||||
|
||||
This will start up a local server on localhost port 1313. When you make changes to either the content of the website (in [`content`](content)) *or* to the Sass and JavaScript assets of the page (in [`themes/jaeger-docs/assets`](themes/jaeger-docs/assets)), the browser will automatically update to reflect those changes (usually in under one second).
|
||||
Alternatively, you can run `npm run serve`.
|
||||
|
||||
These commands will start up a local server on [localhost:1313](http://localhost:1313). When you make changes to either the content of the website (in [`content`](content)) *or* to the Sass and JavaScript assets of the page (in [`themes/jaeger-docs/assets`](themes/jaeger-docs/assets)), the browser will automatically update to reflect those changes (usually in under one second).
|
||||
|
||||
## Publishing the site
|
||||
|
||||
|
@ -77,6 +81,50 @@ You can check internal links by running `make check-internal-links` and all link
|
|||
|
||||
When new pages are added to the documentation, please add a corresponding entry to [themes/jaeger-docs/layouts/index.redirects](./themes/jaeger-docs/layouts/index.redirects).
|
||||
|
||||
## Generating Roadmap page
|
||||
|
||||
To generate the `content/roadmap.md` document, run `make fetch-roadmap`.
|
||||
|
||||
This script fetches issues from the [GitHub project board](https://github.com/orgs/jaegertracing/projects/4/views/1?layout=table), extracts the required information, and generates the roadmap document. Make sure to set the `GITHUB_TOKEN` environment variable with your GitHub API token before running the script, or save the token in `~/.github_token` file (protect the file so only you can read it: `chmod 0600 <file>`). Personal tokens can be created at https://github.com/settings/tokens/.
|
||||
|
||||
## Updating Medium Blog Feed
|
||||
|
||||
The homepage displays the latest blog posts from the [Jaeger Medium blog](https://medium.com/jaegertracing).
|
||||
To avoid network calls during builds and to ensure fast, reliable local development, the Medium RSS feed is downloaded and stored as a static XML file.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
Ensure you have [`curl`](https://curl.se/) installed on your system to download the RSS feed.
|
||||
|
||||
Most Linux and macOS systems already have `curl` pre-installed.
|
||||
You can verify installation by running:
|
||||
|
||||
```bash
|
||||
curl --version
|
||||
```
|
||||
|
||||
### To update the Medium blog feed
|
||||
Run the following command to fetch and save the feed locally as XML:
|
||||
|
||||
```bash
|
||||
make fetch-blog-feed
|
||||
```
|
||||
|
||||
This will download the RSS feed from Medium and save it to:
|
||||
```bash
|
||||
assets/data/medium.xml
|
||||
```
|
||||
|
||||
After updating, commit the changes:
|
||||
```bash
|
||||
git add assets/data/medium.xml
|
||||
git commit -m "chore: update Medium blog feed"
|
||||
```
|
||||
|
||||
### How it's used
|
||||
The Hugo site reads and parses data/medium.xml using resources.Get and transform.Unmarshal.
|
||||
This converts the XML into structured data at build time, allowing full control over the content without relying on live network requests.
|
||||
|
||||
## License
|
||||
|
||||
[Apache 2.0 License](./LICENSE).
|
||||
|
|
23
RELEASE.md
23
RELEASE.md
|
@ -1,32 +1,23 @@
|
|||
# Release instructions
|
||||
|
||||
|
||||
Each Jaeger version is documented in a separate directory e.g. [content/docs/1.8/](./content/docs/1.8/). A special directory [content/docs/next-release/](./content/docs/next-release/) is reserved for the changes to be published as the next version. If you are adding documentation for features that are not yet released in the main Jaeger repository, add your changes to the `next-release` directory. If you're adding documentation for already released features, you may need to make the same change twice, i.e. in the most recent release (e.g. `1.8`) and in the `next-release` directories.
|
||||
|
||||
Jaeger v2 next-release documentation is in the `next-release-v2` directory.
|
||||
|
||||
<!-- BEGIN_CHECKLIST -->
|
||||
Before creating a new release:
|
||||
|
||||
- Make sure all outstanding PRs for that version are merged to `next-release` directory.
|
||||
- Make sure the actual Jaeger release is done and Docker images for the new version are published.
|
||||
- If there are new Jaeger binaries or new storage options added to the release, make sure the CLI docs config file `data/cli/next-release/config.json` is updated accordingly (see below).
|
||||
- Make sure you have git remote `upstream` pointing to the official repository, e.g.
|
||||
`git remote add upstream git@github.com:jaegertracing/documentation.git`
|
||||
- Make sure you are on your `main` branch.
|
||||
|
||||
Then create a release by pushing a tag corresponding to the jaegertracing/jaeger version `release-X.Y.Z`, e.g.
|
||||
|
||||
```shell
|
||||
git tag release-1.12.0
|
||||
git push upstream release-1.12.0
|
||||
```
|
||||
|
||||
- Wait for the [CI job](https://github.com/jaegertracing/documentation/actions) to create a
|
||||
[pull request](https://github.com/jaegertracing/documentation/pulls) with the documentation
|
||||
changes for the new version.
|
||||
To create a new release:
|
||||
- Manually trigger the [Release](https://github.com/jaegertracing/documentation/actions/workflows/ci-release.yml) workflow on GitHub. It will ask for v1 and v2 version numbers (same versions as in the main Jaeger repo), and create a [pull request](https://github.com/jaegertracing/documentation/pulls) with the documentation changes.
|
||||
- Approve and merge that pull request.
|
||||
- Because the site is statically generated, the release is completed after the merge.
|
||||
|
||||
TODO: shouldn't the tag only specify major/minor, not patch? I don't think the process will work twice for the same major.minor
|
||||
<!-- END_CHECKLIST -->
|
||||
|
||||
### Auto-generated documentation for CLI flags
|
||||
|
||||
The docs for the Jaeger CLI tools are generated by the automated release process described above using the Python script [`scripts/gen-cli-data.py`](./scripts/gen-cli-data.py). It uses the configuration file `data/cli/next-release/config.json` (automatically copied to `data/cli/${NEXT_VERSION}/config.json`) that describes which Jaeger binaries to include in the documentation, and which storage options each binary supports. The script invokes the `docs` command on the respective Docker images for each binary and creates a set of YAML files under `data/cli/${NEXT_VERSION}/`, which are then used by the template engine to render the CLI docs.
|
||||
|
||||
|
|
File diff suppressed because one or more lines are too long
36
config.toml
36
config.toml
|
@ -1,36 +0,0 @@
|
|||
baseURL = "https://www.jaegertracing.io"
|
||||
title = "Jaeger: open source, end-to-end distributed tracing"
|
||||
languageCode = "en"
|
||||
pygmentsCodeFences = true
|
||||
pygmentsUseClasses = true
|
||||
disableKinds = ["taxonomy", "taxonomyTerm"]
|
||||
canonifyURLs = true
|
||||
theme = "jaeger-docs"
|
||||
googleAnalytics = "UA-148208642-1"
|
||||
|
||||
[mediaTypes."text/netlify"]
|
||||
delimiter = ""
|
||||
|
||||
[outputFormats.REDIRECTS]
|
||||
mediaType = "text/netlify"
|
||||
baseName = "_redirects"
|
||||
|
||||
[outputs]
|
||||
home = [ "HTML", "JSON", "REDIRECTS" ]
|
||||
|
||||
[params]
|
||||
tagline = "Monitor and troubleshoot transactions in complex distributed systems"
|
||||
githubrepo = "jaegertracing/jaeger"
|
||||
twitterhandle = "JaegerTracing"
|
||||
mediumhandle = "jaegertracing"
|
||||
opengraphimage = "img/jaeger-icon-color.png"
|
||||
|
||||
latest = "1.29"
|
||||
binariesLatest = "1.29.0"
|
||||
versions = ["1.29","1.28","1.27","1.26","1.25","1.24","1.23","1.22","1.21","1.20","1.19","1.18","1.17","1.16","1.15","1.14","1.13","1.12","1.11", "1.10", "1.9", "1.8", "1.7", "1.6"]
|
||||
|
||||
[navbar]
|
||||
[[links]]
|
||||
title = "Docs"
|
||||
url = "/docs"
|
||||
|
|
@ -1,18 +0,0 @@
|
|||
---
|
||||
title: Client Library Features
|
||||
widescreen: true
|
||||
hasparent: true
|
||||
weight: 5
|
||||
---
|
||||
|
||||
{{< warning >}}
|
||||
Jaeger clients are being retired. Please refer to this [notice](../client-libraries/#deprecating-jaeger-clients).
|
||||
{{< /warning >}}
|
||||
|
||||
The table below provides a feature matrix for the existing client libraries. Cells marked with `?` indicate that it's not known if the given client supports the given feature and additional research & documentation update is required.
|
||||
|
||||
{{< info >}}
|
||||
The table is auto-generated from [data/clients.yaml](https://github.com/jaegertracing/documentation/blob/main/data/clients.yaml)
|
||||
{{< /info >}}
|
||||
|
||||
{{< featuresTable >}}
|
|
@ -1,238 +0,0 @@
|
|||
---
|
||||
title: Client Libraries
|
||||
weight: 5
|
||||
children:
|
||||
- title: Client Features
|
||||
url: client-features
|
||||
---
|
||||
|
||||
{{< warning >}}
|
||||
Jaeger clients are being retired.
|
||||
{{< /warning >}}
|
||||
|
||||
## Deprecating Jaeger clients
|
||||
|
||||
The Jaeger clients have faithfully served our community for several years. We pioneered many new features, such as remotely controlled samplers and per-operation / adaptive sampling, which were critical to the success of distributed tracing deployments at large organizations. However, now that the larger community in OpenTelemetry has caught up with the Jaeger clients in terms of feature parity and there is full support for exporting data to Jaeger, we believe it is time to **decommission Jaeger's native clients and refocus the efforts on the OpenTelemetry SDKs**.
|
||||
|
||||
For new applications, we recommend using the [OpenTelemetry](https://opentelemetry.io/) APIs and SDKs. For existing applications that are already instrumented with the OpenTracing API, we recommend replacing the Jaeger clients with the corresponding OpenTelemetry SDKs and the OpenTracing shim/bridge available in most languages supported by Jaeger.
|
||||
|
||||
### Timeline
|
||||
|
||||
We plan to continue accepting pull requests and making new releases of Jaeger clients **through the end of 2021**. In January 2022 we will enter a code freeze period **for 6 months**, during which time we will no longer accept pull requests with new features, with the exception of security-related fixes. After that we will archive the client library repositories and will no longer accept new changes.
|
||||
|
||||
### Migration to OpenTelemetry
|
||||
|
||||
The OpenTelemetry project is working on publishing the migration guides from OpenTracing API to OpenTelemetry SDKs via OpenTracing bridges/shims. There may be different levels of maturity and features in the SDKs. We will keep updating the information below as more of it becomes available.
|
||||
|
||||
#### Baggage support
|
||||
|
||||
OpenTelemetry implements baggage propagation differently from OpenTracing and they are not completely equivalent. In OpenTelemetry the `context` layer sits below the tracing API and relies on immutable context objects, whereas baggage in OpenTracing is stored in a `span` which is mutable (and may occasionally lead to tricky race conditions when starting children spans).
|
||||
|
||||
#### We need your help!
|
||||
|
||||
If you find inaccuracies or have information that can be added, please open an issue or a PR to the [documentation repo](https://github.com/jaegertracing/documentation). If some features are missing and you need them, please open tickets in the respective OpenTelemetry repos or contibute. For example, Jaeger's remote samplers are not yet implemented in every OpenTelemetry SDK, but porting them from the Jaeger codebase is a fairly straightforward task.
|
||||
|
||||
#### Copying Jaeger code
|
||||
|
||||
We encourage OpenTelemetry SDK authors to copy relevant pieces of the Jaeger clients instead of depending on Jaeger modules directly. This is why we use a liberal APL2 license. When copying code, the correct way to respect the license requirements is to keep the copyright notices. For example, Jaeger authors did the same with the code originally written at Uber:
|
||||
|
||||
```
|
||||
// Copyright (c) 2019 The Jaeger Authors.
|
||||
// Copyright (c) 2017 Uber Technologies, Inc.
|
||||
// ... <rest of Apache notice> ...
|
||||
```
|
||||
|
||||
#### Java
|
||||
* OpenTelemetry SDK: https://github.com/open-telemetry/opentelemetry-java
|
||||
* Remote sampling: [sdk-extensions/jaeger-remote-sampler](https://github.com/open-telemetry/opentelemetry-java/tree/main/sdk-extensions/jaeger-remote-sampler)
|
||||
* Internal SDK metrics: n/a
|
||||
* OpenTracing Bridge: [opentracing-shim](https://github.com/open-telemetry/opentelemetry-java/tree/main/opentracing-shim)
|
||||
* Migration Guide: ["Migrating from Jaeger client to OpenTelemetry SDK"][blog-otel-java]
|
||||
|
||||
#### Python
|
||||
* OpenTelemetry SDK: https://github.com/open-telemetry/opentelemetry-python
|
||||
* Remote sampling: ?
|
||||
* Internal SDK metrics: n/a
|
||||
* OpenTracing Bridge: [opentelemetry-opentracing-shim](https://github.com/open-telemetry/opentelemetry-python/tree/main/shim/opentelemetry-opentracing-shim)
|
||||
* Migration Guide: available in the shim's README
|
||||
|
||||
#### Node.js
|
||||
* OpenTelemetry SDK: https://github.com/open-telemetry/opentelemetry-js
|
||||
* Remote sampling: ?
|
||||
* Internal SDK metrics: n/a
|
||||
* OpenTracing Bridge: [opentelemetry-shim-opentracing](https://github.com/open-telemetry/opentelemetry-js/tree/main/packages/opentelemetry-shim-opentracing)
|
||||
* Migration Guide: available in the shim's README and the corresponding readthedocs
|
||||
|
||||
#### Go
|
||||
* OpenTelemetry SDK: https://github.com/open-telemetry/opentelemetry-go
|
||||
* Remote sampling: https://github.com/open-telemetry/opentelemetry-go-contrib/pull/936
|
||||
* Internal SDK metrics: n/a
|
||||
* OpenTracing Bridge: [bridge/opentracing](https://github.com/open-telemetry/opentelemetry-go/tree/main/bridge/opentracing)
|
||||
* Migration Guide: ?
|
||||
|
||||
#### C# / .NET
|
||||
* OpenTelemetry SDK: https://github.com/open-telemetry/opentelemetry-dotnet
|
||||
* Remote sampling: ?
|
||||
* Internal SDK metrics: n/a
|
||||
* OpenTracing Bridge: [OpenTelemetry.Shims.OpenTracing](https://github.com/open-telemetry/opentelemetry-dotnet/tree/main/src/OpenTelemetry.Shims.OpenTracing)
|
||||
* Migration Guide: available in the shim's README, but is very terse
|
||||
|
||||
#### C++
|
||||
* OpenTelemetry SDK: https://github.com/open-telemetry/opentelemetry-cpp
|
||||
* Remote sampling: ?
|
||||
* Internal SDK metrics: n/a
|
||||
* OpenTracing Bridge: n/a
|
||||
* Migration Guide: n/a
|
||||
|
||||
## Intro
|
||||
|
||||
All Jaeger client libraries support the [OpenTracing APIs](http://opentracing.io). The following resources provide more information about instrumenting your application with OpenTracing:
|
||||
|
||||
* [OpenTracing tutorials](https://github.com/yurishkuro/opentracing-tutorial) for Java, Go, Python, Node.js and C#
|
||||
* A deep dive blog post [Tracing HTTP request latency in Go][http-latency-medium]
|
||||
* The official OpenTracing documentation and other materials at [opentracing.io](http://opentracing.io)
|
||||
* The [`opentracing-contrib` org on GitHub](https://github.com/opentracing-contrib) contains many repositories with off-the-shelf instrumentation for many popular frameworks, including JAXRS & Dropwizard (Java), Flask & Django (Python), Go std library, etc.
|
||||
|
||||
The rest of this page contains information about configuring and instantiating a Jaeger tracer in an application that is already instrumented with OpenTracing API.
|
||||
|
||||
## Terminology
|
||||
|
||||
We use the terms *client library*, *instrumentation library*, and *tracer* interchangeably in this document.
|
||||
|
||||
## Supported Libraries
|
||||
|
||||
The following client libraries are officially supported:
|
||||
|
||||
{{< clientsTable >}}
|
||||
|
||||
Libraries in other languages are currently under development, please see [issue #366](https://github.com/jaegertracing/jaeger/issues/366).
|
||||
|
||||
## Initializing Jaeger Tracer
|
||||
|
||||
The initialization syntax is slightly different in each languages, please refer to the README's in the respective repositories.
|
||||
The general pattern is to not create the Tracer explicitly, but use a Configuration class to do that. Configuration allows
|
||||
simpler parameterization of the Tracer, such as changing the default sampler or the location of Jaeger agent.
|
||||
|
||||
## Tracer Internals
|
||||
|
||||
### Sampling
|
||||
|
||||
See [Architecture | Sampling](../sampling#client-sampling-configuration).
|
||||
|
||||
### Reporters
|
||||
|
||||
Jaeger tracers use **reporters** to process finished {{< tip "spans" "span" >}}. Typically Jaeger libraries ship with the following reporters:
|
||||
|
||||
* **NullReporter** does nothing with the span. It can be useful in unit tests.
|
||||
* **LoggingReporter** simply logs the fact that a span was finished, usually by printing the trace and span ID and the operation name.
|
||||
* **CompositeReporter** takes a list of other reporters and invokes them one by one.
|
||||
* **RemoteReporter** (default) buffers a certain number of finished spans in memory and uses a **sender** to submit a batch of spans out of process to Jaeger backend. The sender is responsible for serializing the span to the wire format (e.g. Thrift or JSON) and communicating with the backend components (e.g. over UDP or HTTP).
|
||||
|
||||
#### EMSGSIZE and UDP buffer limits
|
||||
|
||||
By default Jaeger libraries use a UDP sender to report finished {{< tip "spans" "span" >}} to the `jaeger-agent` daemon.
|
||||
The default max packet size is 65,000 bytes, which can be transmitted without segmentation when
|
||||
connecting to the agent via loopback interface. However, some OSs (in particular, MacOS), limit
|
||||
the max buffer size for UDP packets, as raised in [this GitHub issue](https://github.com/uber/jaeger-client-node/issues/124).
|
||||
If you run into issue with `EMSGSIZE` errors, consider raising the limits in your kernel (see the issue for examples).
|
||||
You can also configure the client libraries to use a smaller max packet size, but that may cause
|
||||
issues if you have large spans, e.g. if you log big chunks of data. Spans that exceed max packet size
|
||||
are dropped by the clients (with metrics emitted to indicate that). Another alternative is
|
||||
to use non-UDP transports, such as [HttpSender in Java][HttpSender] (not currently available for all languages).
|
||||
|
||||
### Metrics
|
||||
|
||||
Jaeger tracers emit various metrics about how many spans or traces they have started and finished, how many of them were sampled or not sampled, if there were any errors in decoding trace context from inbound requests or reporting spans to the backend.
|
||||
|
||||
TODO standardize and describe the metric names and labels (issues [#572](https://github.com/jaegertracing/jaeger/issues/572), [#611](https://github.com/jaegertracing/jaeger/issues/611)).
|
||||
|
||||
## Propagation Format
|
||||
|
||||
When `SpanContext` is encoded on the wire as part of the request to another service, Jaeger client libraries default to the Jaeger native propagation format specified below. In addition, Jaeger clients support [Zipkin B3 format](https://github.com/openzipkin/b3-propagation) and [W3C Trace-Context](https://github.com/w3c/distributed-tracing).
|
||||
|
||||
### Trace/Span Identity
|
||||
|
||||
#### Key
|
||||
|
||||
`uber-trace-id`
|
||||
|
||||
* Case-insensitive in HTTP
|
||||
* Lower-case in protocols that preserve header case
|
||||
|
||||
#### Value
|
||||
|
||||
`{trace-id}:{span-id}:{parent-span-id}:{flags}`
|
||||
|
||||
* `{trace-id}`
|
||||
* 64-bit or 128-bit random number in base16 format
|
||||
* Can be variable length, shorter values are 0-padded on the left
|
||||
* Receivers MUST accept hex-strings shorter than 32 characters and 0-pad them on the left
|
||||
* Senders SHOULD generate hex strings of exactly 16 or 32 characters in length
|
||||
* Clients in some languages support 128-bit, migration pending
|
||||
* Value of 0 is not valid
|
||||
* `{span-id}`
|
||||
* 64-bit random number in base16 format
|
||||
* Can be variable length, shorter values are 0-padded on the left
|
||||
* Receivers MUST accept hex-strings shorter than 16 characters and 0-pad them on the left
|
||||
* Senders SHOULD generate hex strings of exactly 16 characters in length
|
||||
* Value of 0 is not valid
|
||||
* `{parent-span-id}`
|
||||
* 64-bit value in base16 format representing parent span id
|
||||
* Deprecated, most Jaeger clients ignore on the receiving side, but still include it on the sending side
|
||||
* 0 value is valid and means “root span” (when not ignored)
|
||||
* `{flags}`
|
||||
* One byte bitmap, as one or two hex digits (leading zero may be omitted)
|
||||
* Bit 1 (right-most, least significant, bit mask `0x01`) is "sampled" flag
|
||||
* 1 means the trace is sampled and all downstream services are advised to respect that
|
||||
* 0 means the trace is not sampled and all downstream services are advised to respect that
|
||||
* We’re considering a new feature that allows downstream services to upsample if they find their tracing level is too low
|
||||
* Bit 2 (bit mask `0x02` ) is "debug" flag
|
||||
* Debug flag should only be set when the sampled flag is set
|
||||
* Instructs the backend to try really hard not to drop this trace
|
||||
* Bit 3 (bit mask `0x04` ) is not used
|
||||
* Bit 4 (bit mask `0x08` ) is "firehose" flag
|
||||
* Spans tagged as "firehose" are excluded from being indexed in the storage
|
||||
* The traces can only be retrieved by trace ID (usually available from other sources, like logs)
|
||||
* Other bits are unused
|
||||
|
||||
### Baggage
|
||||
|
||||
* Key: `uberctx-{baggage-key}`
|
||||
* Value: `{baggage-value}` as a string (see Value Encoding below)
|
||||
* Limitation: since HTTP headers don’t preserve the case, Jaeger recommends baggage keys to be lowercase-kebab-case,
|
||||
e.g. `my-baggage-key-1`.
|
||||
|
||||
Example: the following code sequence:
|
||||
|
||||
```
|
||||
span.SetBaggageItem("key1", "value1")
|
||||
span.SetBaggageItem("key2", "value2")
|
||||
```
|
||||
|
||||
will result in the following HTTP headers:
|
||||
|
||||
```
|
||||
uberctx-key1: value1
|
||||
uberctx-key2: value2
|
||||
```
|
||||
|
||||
#### Value Encoding
|
||||
|
||||
OpenTracing defines two formats for plain text headers: `HTTP_HEADERS` and `TEXT_MAP`. The former was introduced to deal with restrictions imposed by the HTTP protocol on the context of the headers, whereas the latter does not impose any restrictions, e.g. it can be used with Kafka Record Headers. The main difference between these two formats in the Jaeger SDKs is that the baggage values are URL-encoded when using the `HTTP_HEADERS` propagation format.
|
||||
|
||||
Example: when using the `HTTP_HEADERS` propagation format, the following code sequence:
|
||||
|
||||
```
|
||||
span.SetBaggageItem("key1", "value 1 / blah")
|
||||
```
|
||||
|
||||
will result in the following HTTP header:
|
||||
|
||||
```
|
||||
uberctx-key1: value%201%20%2F%20blah
|
||||
```
|
||||
|
||||
|
||||
[HttpSender]: https://github.com/jaegertracing/jaeger-client-java/blob/master/jaeger-thrift/src/main/java/io/jaegertracing/thrift/internal/senders/HttpSender.java
|
||||
[http-latency-medium]: https://medium.com/@YuriShkuro/tracing-http-request-latency-in-go-with-opentracing-7cc1282a100a
|
||||
[blog-otel-java]: https://medium.com/jaegertracing/migrating-from-jaeger-client-to-opentelemetry-sdk-bd337d796759
|
|
@ -0,0 +1,3 @@
|
|||
---
|
||||
title: 'Jaeger: open source, distributed tracing platform'
|
||||
---
|
|
@ -1,2 +0,0 @@
|
|||
*/client-libraries.md
|
||||
*/client-features.md
|
|
@ -1,66 +0,0 @@
|
|||
---
|
||||
title: Introduction
|
||||
weight: 1
|
||||
---
|
||||
|
||||
Welcome to Jaeger's documentation portal! Below, you'll find information for beginners and experienced Jaeger users.
|
||||
|
||||
If you can't find what you are looking for, or have an issue not covered here, we'd love to [hear from you](/get-in-touch).
|
||||
|
||||
## About
|
||||
|
||||
Jaeger, inspired by [Dapper][dapper] and [OpenZipkin](http://zipkin.io),
|
||||
is a distributed tracing system released as open source by [Uber Technologies][ubeross].
|
||||
It is used for monitoring and troubleshooting microservices-based distributed systems, including:
|
||||
|
||||
* Distributed context propagation
|
||||
* Distributed transaction monitoring
|
||||
* Root cause analysis
|
||||
* Service dependency analysis
|
||||
* Performance / latency optimization
|
||||
|
||||
Uber published a blog post, [Evolving Distributed Tracing at Uber](https://eng.uber.com/distributed-tracing/), where they explain the history and reasons for the architectural choices made in Jaeger.
|
||||
|
||||
## Features
|
||||
|
||||
* [OpenTracing](http://opentracing.io/) compatible data model and instrumentation libraries
|
||||
* in [Go](https://github.com/jaegertracing/jaeger-client-go), [Java](https://github.com/jaegertracing/jaeger-client-java), [Node](https://github.com/jaegertracing/jaeger-client-node), [Python](https://github.com/jaegertracing/jaeger-client-python)
|
||||
and [C++](https://github.com/jaegertracing/cpp-client)
|
||||
* Uses consistent upfront sampling with individual per service/endpoint probabilities
|
||||
* Multiple storage backends: Cassandra, Elasticsearch, memory.
|
||||
* Adaptive sampling (coming soon)
|
||||
* Post-collection data processing pipeline (coming soon)
|
||||
|
||||
See [Features](./features/) page for more details.
|
||||
|
||||
## Technical Specs
|
||||
|
||||
* Backend components implemented in Go
|
||||
* React/Javascript UI
|
||||
* Supported storage backends:
|
||||
* [Cassandra 3.4+](./deployment/#cassandra)
|
||||
* [Elasticsearch 5.x, 6.x](./deployment/#elasticsearch)
|
||||
* [Kafka](./deployment/#kafka)
|
||||
* memory storage
|
||||
|
||||
## Quick Start
|
||||
See [running a docker all in one image](getting-started#all-in-one).
|
||||
|
||||
## Screenshots
|
||||
|
||||
### Traces View
|
||||
[](/img/traces-ss.png)
|
||||
|
||||
### Trace Detail View
|
||||
[](/img/trace-detail-ss.png)
|
||||
|
||||
## Related links
|
||||
- [Evolving Distributed tracing At Uber Engineering](https://eng.uber.com/distributed-tracing/)
|
||||
- [Tracing HTTP request latency in Go with OpenTracing](https://medium.com/opentracing/tracing-http-request-latency-in-go-with-opentracing-7cc1282a100a)
|
||||
- [Distributed Tracing with Jaeger & Prometheus on Kubernetes](https://blog.openshift.com/openshift-commons-briefing-82-distributed-tracing-with-jaeger-prometheus-on-kubernetes/)
|
||||
- [Using Jaeger with Istio](https://istio.io/docs/tasks/telemetry/distributed-tracing.html)
|
||||
- [Using Jaeger with Envoy](https://www.envoyproxy.io/docs/envoy/latest/start/sandboxes/jaeger_tracing.html)
|
||||
|
||||
[dapper]: https://research.google.com/pubs/pub36356.html
|
||||
[ubeross]: http://uber.github.io
|
||||
|
|
@ -1,355 +0,0 @@
|
|||
---
|
||||
title: Deployment
|
||||
weight: 7
|
||||
children:
|
||||
- title: Frontend/UI
|
||||
url: frontend-ui
|
||||
---
|
||||
|
||||
The main Jaeger backend components are released as Docker images on Docker Hub:
|
||||
|
||||
Component | Repository
|
||||
--------------------- | ---
|
||||
**jaeger-agent** | [hub.docker.com/r/jaegertracing/jaeger-agent/](https://hub.docker.com/r/jaegertracing/jaeger-agent/)
|
||||
**jaeger-collector** | [hub.docker.com/r/jaegertracing/jaeger-collector/](https://hub.docker.com/r/jaegertracing/jaeger-collector/)
|
||||
**jaeger-query** | [hub.docker.com/r/jaegertracing/jaeger-query/](https://hub.docker.com/r/jaegertracing/jaeger-query/)
|
||||
**jaeger-ingester** | [hub.docker.com/r/jaegertracing/jaeger-ingester/](https://hub.docker.com/r/jaegertracing/jaeger-ingester/)
|
||||
|
||||
There are orchestration templates for running Jaeger with:
|
||||
|
||||
* Kubernetes: [github.com/jaegertracing/jaeger-kubernetes](https://github.com/jaegertracing/jaeger-kubernetes),
|
||||
* OpenShift: [github.com/jaegertracing/jaeger-openshift](https://github.com/jaegertracing/jaeger-openshift).
|
||||
|
||||
## Configuration Options
|
||||
|
||||
Jaeger binaries can be configured in a number of ways (in the order of decreasing priority):
|
||||
|
||||
* command line arguments,
|
||||
* environment variables,
|
||||
* configuration files in JSON, TOML, YAML, HCL, or Java properties formats.
|
||||
|
||||
To see the complete list of options, run the binary with `help` command. Options that are specific to a certain storage backend are only listed if the storage type is selected. For example, to see all available options in the Collector with Cassandra storage:
|
||||
|
||||
```sh
|
||||
$ docker run --rm \
|
||||
-e SPAN_STORAGE_TYPE=cassandra \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}} \
|
||||
help
|
||||
```
|
||||
|
||||
In order to provide configuration parameters via environment variables, find the respective command line option and convert its name to UPPER_SNAKE_CASE, for example:
|
||||
|
||||
Command line option | Environment variable
|
||||
-----------------------------------|-------------------------------
|
||||
`--cassandra.connections-per-host` | `CASSANDRA_CONNECTIONS_PER_HOST`
|
||||
`--metrics-backend` | `METRICS_BACKEND`
|
||||
|
||||
## Agent
|
||||
|
||||
Jaeger client libraries expect **jaeger-agent** process to run locally on each host.
|
||||
The agent exposes the following ports:
|
||||
|
||||
Port | Protocol | Function
|
||||
---- | ------- | ---
|
||||
5775 | UDP | accept zipkin.thrift over compact thrift protocol
|
||||
6831 | UDP | accept jaeger.thrift over compact thrift protocol
|
||||
6832 | UDP | accept jaeger.thrift over binary thrift protocol
|
||||
5778 | HTTP | serve configs, sampling strategies
|
||||
|
||||
It can be executed directly on the host or via Docker, as follows:
|
||||
|
||||
```sh
|
||||
## make sure to expose only the ports you use in your deployment scenario!
|
||||
docker run \
|
||||
--rm \
|
||||
-p5775:5775/udp \
|
||||
-p6831:6831/udp \
|
||||
-p6832:6832/udp \
|
||||
-p5778:5778/tcp \
|
||||
jaegertracing/jaeger-agent:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
### Discovery System Integration
|
||||
|
||||
The agents can connect point to point to a single collector address, which could be
|
||||
load balanced by another infrastructure component (e.g. DNS) across multiple collectors.
|
||||
The agent can also be configured with a static list of collector addresses.
|
||||
|
||||
On Docker, a command like the following can be used:
|
||||
|
||||
```sh
|
||||
docker run \
|
||||
--rm \
|
||||
-p5775:5775/udp \
|
||||
-p6831:6831/udp \
|
||||
-p6832:6832/udp \
|
||||
-p5778:5778/tcp \
|
||||
jaegertracing/jaeger-agent:{{< currentVersion >}} \
|
||||
--reporter.tchannel.host-port=jaeger-collector.jaeger-infra.svc:14267
|
||||
```
|
||||
|
||||
Or add `--reporter.type=grpc` and `--reporter.grpc.host-port=jaeger-collector.jaeger-infra.svc:14250` to use gRPC
|
||||
communication with the collector. Then the `tchannel` option can be removed.
|
||||
|
||||
When using gRPC, you have several options for load balancing and name resolution:
|
||||
|
||||
* Single connection and no load balancing. This is the default if you specify a single `host:port`. (example: `--reporter.grpc.host-port=jaeger-collector.jaeger-infra.svc:14250`)
|
||||
* Static list of hostnames and round-robin load balancing. This is what you get with a comma-separated list of addresses. (example: `reporter.grpc.host-port=jaeger-collector1:14250,jaeger-collector2:14250,jaeger-collector3:14250`)
|
||||
* Dynamic DNS resolution and round-robin load balancing. To get this behaviour, prefix the address with `dns:///` and gRPC will attempt to resolve the hostname using SRV records (for [external load balancing](https://github.com/grpc/grpc/blob/master/doc/load-balancing.md)), TXT records (for [service configs](https://github.com/grpc/grpc/blob/master/doc/service_config.md)), and A records. Refer to the [gRPC Name Resolution docs](https://github.com/grpc/grpc/blob/master/doc/naming.md) and the [dns_resolver.go implementation](https://github.com/grpc/grpc-go/blob/master/resolver/dns/dns_resolver.go) for more info. (example: `--reporter.grpc.host-port=dns:///jaeger-collector.jaeger-infra.svc:14250`)
|
||||
|
||||
In the future we will support different service discovery systems to dynamically load balance
|
||||
across several collectors ([issue 213](https://github.com/jaegertracing/jaeger/issues/213)).
|
||||
|
||||
|
||||
## Collectors
|
||||
|
||||
The collectors are stateless and thus many instances of **jaeger-collector** can be run in parallel.
|
||||
Collectors require almost no configuration, except for the location of Cassandra cluster,
|
||||
via `--cassandra.keyspace` and `--cassandra.servers` options, or the location of Elasticsearch cluster, via
|
||||
`--es.server-urls`, depending on which storage is specified. To see all command line options run
|
||||
|
||||
```sh
|
||||
go run ./cmd/collector/main.go -h
|
||||
```
|
||||
|
||||
or, if you don't have the source code
|
||||
|
||||
```sh
|
||||
docker run -it --rm jaegertracing/jaeger-collector:{{< currentVersion >}} -h
|
||||
```
|
||||
|
||||
At default settings the collector exposes the following ports:
|
||||
|
||||
Port | Protocol | Function
|
||||
----- | ------- | ---
|
||||
14267 | TChannel | used by **jaeger-agent** to send spans in jaeger.thrift format
|
||||
14250 | gRPC | used by **jaeger-agent** to send spans in model.proto format
|
||||
14268 | HTTP | can accept spans directly from clients in jaeger.thrift format over binary thrift protocol
|
||||
9411 | HTTP | can accept Zipkin spans in JSON or Thrift (disabled by default)
|
||||
14269 | HTTP | Health check at **/**
|
||||
|
||||
|
||||
## Storage Backends
|
||||
|
||||
Collectors require a persistent storage backend. Cassandra and Elasticsearch are the primary supported storage backends. Additional backends are [discussed here](https://github.com/jaegertracing/jaeger/issues/638).
|
||||
|
||||
The storage type can be passed via `SPAN_STORAGE_TYPE` environment variable. Valid values are `cassandra`, `elasticsearch`, `kafka` (only as a buffer) and `memory` (only for all-in-one binary).
|
||||
As of version 1.6.0, it's possible to use multiple storage types at the same time by providing a comma-separated list of valid types to the `SPAN_STORAGE_TYPE` environment variable.
|
||||
It's important to note that all listed storage types are used for writing, but only the first type in the list will be used for reading and archiving.
|
||||
|
||||
### Memory
|
||||
|
||||
The in-memory storage is not intended for production workloads. It's intended as a simple solution to get started quickly and
|
||||
data will be lost once the process is gone.
|
||||
|
||||
By default, there's no limit in the amount of traces stored in memory but a limit can be established by passing an
|
||||
integer value via `--memory.max-traces`.
|
||||
|
||||
### Cassandra
|
||||
Supported versions: 3.4+
|
||||
|
||||
Deploying Cassandra itself is out of scope for our documentation. One good
|
||||
source of documentation is the [Apache Cassandra Docs](https://cassandra.apache.org/doc/latest/).
|
||||
|
||||
#### Configuration
|
||||
##### Minimal
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=cassandra \
|
||||
-e CASSANDRA_SERVERS=<...> \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
##### All options
|
||||
To view the full list of configuration options, you can run the following command:
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=cassandra \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}} \
|
||||
--help
|
||||
```
|
||||
|
||||
#### Schema script
|
||||
|
||||
A script is provided to initialize Cassandra keyspace and schema
|
||||
using Cassandra's interactive shell [`cqlsh`][cqlsh]:
|
||||
|
||||
```sh
|
||||
MODE=test sh ./plugin/storage/cassandra/schema/create.sh | cqlsh
|
||||
```
|
||||
|
||||
For production deployment, pass `MODE=prod DATACENTER={datacenter}` arguments to the script,
|
||||
where `{datacenter}` is the name used in the Cassandra configuration / network topology.
|
||||
|
||||
The script also allows overriding TTL, keyspace name, replication factor, etc.
|
||||
Run the script without arguments to see the full list of recognized parameters.
|
||||
|
||||
#### TLS support
|
||||
|
||||
Jaeger supports TLS client to node connections as long as you've configured
|
||||
your Cassandra cluster correctly. After verifying with e.g. `cqlsh`, you can
|
||||
configure the collector and query like so:
|
||||
|
||||
```sh
|
||||
docker run \
|
||||
-e CASSANDRA_SERVERS=<...> \
|
||||
-e CASSANDRA_TLS=true \
|
||||
-e CASSANDRA_TLS_SERVER_NAME="CN-in-certificate" \
|
||||
-e CASSANDRA_TLS_KEY=<path to client key file> \
|
||||
-e CASSANDRA_TLS_CERT=<path to client cert file> \
|
||||
-e CASSANDRA_TLS_CA=<path to your CA cert file> \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
The schema tool also supports TLS. You need to make a custom cqlshrc file like
|
||||
so:
|
||||
|
||||
```
|
||||
# Creating schema in a cassandra cluster requiring client TLS certificates.
|
||||
#
|
||||
# Create a volume for the schema docker container containing four files:
|
||||
# cqlshrc: this file
|
||||
# ca-cert: the cert authority for your keys
|
||||
# client-key: the keyfile for your client
|
||||
# client-cert: the cert file matching client-key
|
||||
#
|
||||
# if there is any sort of DNS mismatch and you want to ignore server validation
|
||||
# issues, then uncomment validate = false below.
|
||||
#
|
||||
# When running the container, map this volume to /root/.cassandra and set the
|
||||
# environment variable CQLSH_SSL=--ssl
|
||||
[ssl]
|
||||
certfile = ~/.cassandra/ca-cert
|
||||
userkey = ~/.cassandra/client-key
|
||||
usercert = ~/.cassandra/client-cert
|
||||
# validate = false
|
||||
```
|
||||
|
||||
### Elasticsearch
|
||||
Supported in Jaeger since 0.6.0
|
||||
Supported versions: 5.x, 6.x
|
||||
|
||||
Elasticsearch does not require initialization other than
|
||||
[installing and running Elasticsearch](https://www.elastic.co/downloads/elasticsearch).
|
||||
Once it is running, pass the correct configuration values to the Jaeger collector and query service.
|
||||
|
||||
#### Configuration
|
||||
##### Minimal
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=elasticsearch \
|
||||
-e ES_SERVER_URLS=<...> \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
##### All options
|
||||
To view the full list of configuration options, you can run the following command:
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=elasticsearch \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}} \
|
||||
--help
|
||||
```
|
||||
|
||||
See the [README](https://github.com/jaegertracing/jaeger/tree/master/plugin/storage/es/README.md) for an in-depth overview of how Jaeger uses Elasticsearch for storage.
|
||||
|
||||
#### Shards and Replicas for Elasticsearch indices
|
||||
|
||||
Shards and replicas are some configuration values to take special attention to, because this is decided upon
|
||||
index creation. [This article](https://qbox.io/blog/optimizing-elasticsearch-how-many-shards-per-index) goes into
|
||||
more information about choosing how many shards should be chosen for optimization.
|
||||
|
||||
### Kafka
|
||||
Supported in Jaeger since 1.6.0
|
||||
Supported Kafka versions: 0.9+
|
||||
|
||||
Starting from version 1.7.0, a new component [Ingester](#ingester) was added to support reading from Kafka and storing it in another storage backend (Elasticsearch or Cassandra).
|
||||
|
||||
Writing to Kafka is particularly useful for building post-processing data pipelines.
|
||||
|
||||
#### Configuration
|
||||
##### Minimal
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=kafka \
|
||||
-e KAFKA_BROKERS=<...> \
|
||||
-e KAFKA_TOPIC=<...> \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
##### All options
|
||||
To view the full list of configuration options, you can run the following command:
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=kafka \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}} \
|
||||
--help
|
||||
```
|
||||
|
||||
#### Topic & partitions
|
||||
Unless your Kafka cluster is configured to automatically create topics, you will need to create it ahead of time. You can refer to [the Kafka quickstart documentation](https://kafka.apache.org/documentation/#quickstart_createtopic) to learn how.
|
||||
|
||||
You can find more information about topics and partitions in general in the [official documentation](https://kafka.apache.org/documentation/#intro_topics). [This article](https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/) provide more details about how to choose the number of partitions.
|
||||
|
||||
## Ingester
|
||||
**jaeger-ingester** is a service which reads span data from Kafka topic and writes it to another storage backend (Elasticsearch or Cassandra).
|
||||
|
||||
Port | Protocol | Function
|
||||
----- | ------- | ---
|
||||
14270 | HTTP | Health check at **/**
|
||||
14271 | HTTP | Metrics endpoint
|
||||
|
||||
To view all exposed configuration options run the following command:
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=cassandra \
|
||||
jaegertracing/jaeger-ingester:{{< currentVersion >}}
|
||||
--help
|
||||
```
|
||||
|
||||
## Query Service & UI
|
||||
|
||||
**jaeger-query** serves the API endpoints and a React/Javascript UI.
|
||||
The service is stateless and is typically run behind a load balancer, such as **nginx**.
|
||||
|
||||
At default settings the query service exposes the following port(s):
|
||||
|
||||
Port | Protocol | Function
|
||||
----- | ------- | ---
|
||||
16686 | HTTP | **/api/*** endpoints and Jaeger UI at **/**
|
||||
16687 | HTTP | Health check at **/**
|
||||
|
||||
### Minimal deployment example (Elasticsearch backend):
|
||||
```sh
|
||||
docker run -d --rm \
|
||||
-p 16686:16686 \
|
||||
-p 16687:16687 \
|
||||
-e SPAN_STORAGE_TYPE=elasticsearch \
|
||||
-e ES_SERVER_URLS=http://<ES_SERVER_IP>:<ES_SERVER_PORT> \
|
||||
jaegertracing/jaeger-query:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
### UI Base Path
|
||||
|
||||
The base path for all **jaeger-query** HTTP routes can be set to a non-root value, e.g. `/jaeger` would cause all UI URLs to start with `/jaeger`. This can be useful when running **jaeger-query** behind a reverse proxy.
|
||||
|
||||
The base path can be configured via the `--query.base-path` command line parameter or the `QUERY_BASE_PATH` environment variable.
|
||||
|
||||
### UI Customization and Embedding
|
||||
|
||||
Please refer to the [dedicated Frontend/UI page](../frontend-ui/).
|
||||
|
||||
## Aggregation Jobs for Service Dependencies
|
||||
|
||||
Production deployments need an external process which aggregates data and creates dependency links between services.
|
||||
Project [spark-dependencies](https://github.com/jaegertracing/spark-dependencies) is a Spark job which derives
|
||||
dependency links and stores them directly to the storage.
|
||||
|
||||
## Configuration
|
||||
|
||||
All binaries accepts command line properties and environmental variables which are managed by
|
||||
by [viper](https://github.com/spf13/viper) and [cobra](https://github.com/spf13/cobra).
|
||||
The names of environmental properties are capital letters and characters `-` and `.` are replaced with `_`.
|
||||
To list all configuration properties call `jaeger-binary -h`.
|
||||
|
||||
[cqlsh]: http://cassandra.apache.org/doc/latest/tools/cqlsh.html
|
|
@ -1,57 +0,0 @@
|
|||
---
|
||||
title: Features
|
||||
weight: 3
|
||||
---
|
||||
|
||||
Jaeger is used for monitoring and troubleshooting microservices-based distributed systems, including:
|
||||
|
||||
* Distributed context propagation
|
||||
* Distributed transaction monitoring
|
||||
* Root cause analysis
|
||||
* Service dependency analysis
|
||||
* Performance / latency optimization
|
||||
|
||||
## High Scalability
|
||||
|
||||
Jaeger backend is designed to have no single points of failure and to scale with the business needs.
|
||||
For example, any given Jaeger installation at Uber is typically processing several billion {{< tip "spans" "span" >}} per day.
|
||||
|
||||
## Native support for OpenTracing
|
||||
|
||||
Jaeger backend, Web UI, and instrumentation libraries have been designed from ground up to support the OpenTracing standard.
|
||||
|
||||
* Represent {{< tip "traces" "trace" >}} as {{< tip "directed acyclic graphs" "directed acyclic graph" >}} (not just trees) via [span references](https://github.com/opentracing/specification/blob/master/specification.md#references-between-spans)
|
||||
* Support strongly typed span _tags_ and _structured logs_
|
||||
* Support general distributed context propagation mechanism via _baggage_
|
||||
|
||||
## Multiple storage backends
|
||||
|
||||
Jaeger supports two popular open source NoSQL databases as trace storage backends: Cassandra 3.4+ and Elasticsearch 5.x/6.x.
|
||||
There are ongoing community experiments using other databases, such as ScyllaDB, InfluxDB, Amazon DynamoDB. Jaeger also ships
|
||||
with a simple in-memory storage for testing setups.
|
||||
|
||||
## Modern Web UI
|
||||
|
||||
Jaeger Web UI is implemented in Javascript using popular open source frameworks like React. Several performance
|
||||
improvements have been released in v1.0 to allow the UI to efficiently deal with large volumes of data, and to display
|
||||
{{< tip "traces" "trace" >}} with tens of thousands of {{< tip "spans" "span" >}} (e.g. we tried a trace with 80,000 spans).
|
||||
|
||||
## Cloud Native Deployment
|
||||
|
||||
Jaeger backend is distributed as a collection of Docker images. The binaries support various configuration methods,
|
||||
including command line options, environment variables, and configuration files in multiple formats (yaml, toml, etc.)
|
||||
Deployment to Kubernetes clusters is assisted by [Kubernetes templates](https://github.com/jaegertracing/jaeger-kubernetes)
|
||||
and a [Helm chart](https://github.com/kubernetes/charts/tree/master/incubator/jaeger).
|
||||
|
||||
## Observability
|
||||
|
||||
All Jaeger backend components expose [Prometheus](https://prometheus.io/) metrics by default (other metrics backends are
|
||||
also supported). Logs are written to standard out using the structured logging library [zap](https://github.com/uber-go/zap).
|
||||
|
||||
## Backwards compatibility with Zipkin
|
||||
|
||||
Although we recommend instrumenting applications with OpenTracing API and binding to Jaeger client libraries to benefit
|
||||
from advanced features not available elsewhere, if your organization has already invested in the instrumentation
|
||||
using Zipkin libraries, you do not have to rewrite all that code. Jaeger provides backwards compatibility with Zipkin
|
||||
by accepting spans in Zipkin formats (Thrift or JSON v1/v2) over HTTP. Switching from Zipkin backend is just a matter
|
||||
of routing the traffic from Zipkin libraries to the Jaeger backend.
|
|
@ -1,196 +0,0 @@
|
|||
---
|
||||
title: Frontend/UI
|
||||
hasparent: true
|
||||
weight: 10
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
Several aspects of the UI can be configured:
|
||||
|
||||
* The Dependencies section can be enabled / configured
|
||||
* Google Analytics tracking can be enabled / configured
|
||||
* Additional menu options can be added to the global nav
|
||||
|
||||
These options can be configured by a JSON configuration file. The `--query.ui-config` command line parameter of the query service must then be set to the path to the JSON file when the query service is started.
|
||||
|
||||
An example configuration file:
|
||||
|
||||
```json
|
||||
{
|
||||
"dependencies": {
|
||||
"dagMaxNumServices": 200,
|
||||
"menuEnabled": true
|
||||
},
|
||||
"archiveEnabled": true,
|
||||
"tracking": {
|
||||
"gaID": "UA-000000-2",
|
||||
"trackErrors": true
|
||||
},
|
||||
"menu": [
|
||||
{
|
||||
"label": "About Jaeger",
|
||||
"items": [
|
||||
{
|
||||
"label": "GitHub",
|
||||
"url": "https://github.com/jaegertracing/jaeger"
|
||||
},
|
||||
{
|
||||
"label": "Docs",
|
||||
"url": "http://jaeger.readthedocs.io/en/latest/"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
`dependencies.dagMaxNumServices` defines the maximum number of services allowed before the DAG dependency view is disabled. Default: `200`.
|
||||
|
||||
`dependencies.menuEnabled` enables (`true`) or disables (`false`) the dependencies menu button. Default: `true`.
|
||||
|
||||
`archiveEnabled` enables (`true`) or disables (`false`) the archive traces button. Default: `false`. It requires a configuration of an archive storage in Query service. Archived traces are only accessible directly by ID, they are not searchable.
|
||||
|
||||
`tracking.gaID` defines the Google Analytics tracking ID. This is required for Google Analytics tracking, and setting it to a non-`null` value enables Google Analytics tracking. Default: `null`.
|
||||
|
||||
`tracking.trackErrors` enables (`true`) or disables (`false`) error tracking via Google Analytics. Errors can only be tracked if a valid Google Analytics ID is provided. For additional details on error tracking via Google Analytics see the [tracking README](https://github.com/jaegertracing/jaeger-ui/blob/c622330546afc1be59a42f874bcc1c2fadf7e69a/src/utils/tracking/README.md) in the UI repo. Default: `true`.
|
||||
|
||||
`menu` allows additional links to be added to the global nav. The additional links are right-aligned.
|
||||
|
||||
In the sample JSON config above, the configured menu will have a dropdown labeled "About Jaeger" with sub-options for "GitHub" and "Docs". The format for a link in the top right menu is as follows:
|
||||
|
||||
```json
|
||||
{
|
||||
"label": "Some text here",
|
||||
"url": "https://example.com"
|
||||
}
|
||||
```
|
||||
|
||||
Links can either be members of the `menu` Array, directly, or they can be grouped into a dropdown menu option. The format for a group of links is:
|
||||
|
||||
```json
|
||||
{
|
||||
"label": "Dropdown button",
|
||||
"items": [ ]
|
||||
}
|
||||
```
|
||||
|
||||
The `items` Array should contain one or more link configurations.
|
||||
|
||||
## Embedded Mode
|
||||
|
||||
Starting with version 1.9, Jaeger UI provides an "embedded" layout mode which is intended to support integrating Jaeger UI into other applications. Currently (as of `v0`), the approach taken is to remove various UI elements from the page to make the UI better suited for space-constrained layouts.
|
||||
|
||||
The embedded mode is induced and configured via URL query parameters.
|
||||
|
||||
To enter embedded mode, the `uiEmbed=v0` query parameter and value must be added to the URL. For example, the following URL will show the trace with ID `abc123` in embedded mode:
|
||||
|
||||
```
|
||||
http://localhost:16686/trace/abc123?uiEmbed=v0
|
||||
```
|
||||
|
||||
`uiEmbed=v0` is required.
|
||||
|
||||
Further, each page supported has an <img src="/img/frontend-ui/embed-open-icon.png" style="width: 20px; height:20px;display:inline;" alt="Embed open window"> button added that will open the non-embedded page in a new tab.
|
||||
|
||||
The following pages support embedded mode:
|
||||
|
||||
* Search Page
|
||||
* Trace Page
|
||||
|
||||
### Search Page
|
||||
|
||||
To integrate the Search Trace Page to our application we have to indicate to the Jaeger UI that we want to use the embed mode with `uiEmbed=v0`.
|
||||
|
||||
For example:
|
||||
|
||||
```
|
||||
http://localhost:16686/search?
|
||||
service=my-service&
|
||||
start=1543917759557000&
|
||||
end=1543921359557000&
|
||||
limit=20&
|
||||
lookback=1h&
|
||||
maxDuration&
|
||||
minDuration&
|
||||
uiEmbed=v0
|
||||
```
|
||||
|
||||

|
||||
|
||||
#### Configuration options
|
||||
|
||||
The following query parameter can be used to configure the layout of the search page :
|
||||
|
||||
* `uiSearchHideGraph=1` - disables the display of the scatter plot above the search results
|
||||
|
||||
```
|
||||
http://localhost:16686/search?
|
||||
service=my-service&
|
||||
start=1543917759557000&
|
||||
end=1543921359557000&
|
||||
limit=20&
|
||||
lookback=1h&
|
||||
maxDuration&
|
||||
minDuration&
|
||||
uiEmbed=v0&
|
||||
uiSearchHideGraph=1
|
||||
```
|
||||
|
||||

|
||||
|
||||
### Trace Page
|
||||
|
||||
|
||||
To integrate the Trace Page to our application we have to indicate to the Jaeger UI that we want to use the embed mode with `uiEmbed=v0`.
|
||||
|
||||
For example:
|
||||
|
||||
```sh
|
||||
http://localhost:16686/trace/{trace-id}?uiEmbed=v0
|
||||
```
|
||||

|
||||
|
||||
If we have navigated to this view from the search traces page we'll have a button to go back to the results page.
|
||||
|
||||

|
||||
|
||||
#### Configuration options
|
||||
|
||||
The following query parameters can be used to configure the layout of the trace page :
|
||||
|
||||
* `uiTimelineCollapseTitle=1` causes the trace header to start out collapsed, which hides the summary and the minimap.
|
||||
|
||||
```
|
||||
http://localhost:16686/trace/{trace-id}?
|
||||
uiEmbed=v0&
|
||||
uiTimelineCollapseTitle=1
|
||||
```
|
||||

|
||||
|
||||
* `uiTimelineHideMinimap=1` removes the minimap, entirely, regardless of whether the trace header is expanded or not.
|
||||
|
||||
```
|
||||
http://localhost:16686/trace/{trace-id}?
|
||||
uiEmbed=v0&
|
||||
uiTimelineHideMinimap=1
|
||||
```
|
||||

|
||||
|
||||
* `uiTimelineHideSummary=1` - removes the trace summary information (number of services, etc.) entirely, regardless of whether the trace header is expanded or not.
|
||||
|
||||
```
|
||||
http://localhost:16686/trace/{trace-id}?
|
||||
uiEmbed=v0&
|
||||
uiTimelineHideSummary=1
|
||||
```
|
||||

|
||||
|
||||
We can also combine the options:
|
||||
```
|
||||
http://localhost:16686/trace/{trace-id}?
|
||||
uiEmbed=v0&
|
||||
uiTimelineHideMinimap=1&
|
||||
uiTimelineHideSummary=1
|
||||
```
|
||||

|
|
@ -1,129 +0,0 @@
|
|||
---
|
||||
title: Getting started
|
||||
description: Get up and running with Jaeger in your local environment
|
||||
weight: 2
|
||||
---
|
||||
|
||||
## Instrumentation
|
||||
|
||||
Your applications must be instrumented before they can send tracing data to Jaeger backend. Check the [Client Libraries](../client-libraries) section for information about how to use the OpenTracing API and how to initialize and configure Jaeger tracers.
|
||||
|
||||
## All in One
|
||||
|
||||
All-in-one is an executable designed for quick local testing, launches the Jaeger UI, collector, query, and agent, with an in memory storage component.
|
||||
|
||||
The simplest way to start the all-in-one is to use the pre-built image published to DockerHub (a single command line).
|
||||
|
||||
```bash
|
||||
$ docker run -d --name jaeger \
|
||||
-e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
|
||||
-p 5775:5775/udp \
|
||||
-p 6831:6831/udp \
|
||||
-p 6832:6832/udp \
|
||||
-p 5778:5778 \
|
||||
-p 16686:16686 \
|
||||
-p 14268:14268 \
|
||||
-p 9411:9411 \
|
||||
jaegertracing/all-in-one:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
Or run the `jaeger-all-in-one(.exe)` executable from the [binary distribution archives][download]:
|
||||
|
||||
```bash
|
||||
$ jaeger-all-in-one --collector.zipkin.http-port=9411
|
||||
```
|
||||
|
||||
You can then navigate to `http://localhost:16686` to access the Jaeger UI.
|
||||
|
||||
The container exposes the following ports:
|
||||
|
||||
Port | Protocol | Component | Function
|
||||
----- | ------- | --------- | ---
|
||||
5775 | UDP | agent | accept `zipkin.thrift` over compact thrift protocol (deprecated, used by legacy clients only)
|
||||
6831 | UDP | agent | accept `jaeger.thrift` over compact thrift protocol
|
||||
6832 | UDP | agent | accept `jaeger.thrift` over binary thrift protocol
|
||||
5778 | HTTP | agent | serve configs
|
||||
16686 | HTTP | query | serve frontend
|
||||
14268 | HTTP | collector | accept `jaeger.thrift` directly from clients
|
||||
14250 | HTTP | collector | accept `model.proto`
|
||||
9411 | HTTP | collector | Zipkin compatible endpoint (optional)
|
||||
|
||||
|
||||
## Kubernetes and OpenShift
|
||||
|
||||
* Kubernetes templates: https://github.com/jaegertracing/jaeger-kubernetes
|
||||
* Kubernetes Operator: https://github.com/jaegertracing/jaeger-operator
|
||||
* OpenShift templates: https://github.com/jaegertracing/jaeger-openshift
|
||||
|
||||
## Sample App: HotROD
|
||||
|
||||
HotROD (Rides on Demand) is a demo application that consists of several microservices and
|
||||
illustrates the use of the [OpenTracing API](http://opentracing.io).
|
||||
A tutorial / walkthrough is available in the blog post:
|
||||
[Take OpenTracing for a HotROD ride][hotrod-tutorial].
|
||||
|
||||
It can be run standalone, but requires Jaeger backend to view the traces.
|
||||
|
||||
### Features
|
||||
|
||||
- Discover architecture of the whole system via data-driven dependency
|
||||
diagram.
|
||||
- View request timeline and errors; understand how the app works.
|
||||
- Find sources of latency and lack of concurrency.
|
||||
- Highly contextualized logging.
|
||||
- Use baggage propagation to:
|
||||
|
||||
- Diagnose inter-request contention (queueing).
|
||||
- Attribute time spent in a service.
|
||||
|
||||
- Use open source libraries with OpenTracing integration to get
|
||||
vendor-neutral instrumentation for free.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- You need Go 1.11 or higher installed on your machine to run from source.
|
||||
- Requires a [running Jaeger backend](#all-in-one) to view the traces.
|
||||
|
||||
### Running
|
||||
|
||||
#### From Source
|
||||
|
||||
```bash
|
||||
mkdir -p $GOPATH/src/github.com/jaegertracing
|
||||
cd $GOPATH/src/github.com/jaegertracing
|
||||
git clone git@github.com:jaegertracing/jaeger.git jaeger
|
||||
cd jaeger
|
||||
make install
|
||||
go run ./examples/hotrod/main.go all
|
||||
```
|
||||
#### From docker
|
||||
|
||||
```bash
|
||||
$ docker run --rm -it \
|
||||
--link jaeger \
|
||||
-p8080-8083:8080-8083 \
|
||||
-e JAEGER_AGENT_HOST="jaeger" \
|
||||
jaegertracing/example-hotrod:{{< currentVersion >}} \
|
||||
all
|
||||
```
|
||||
|
||||
#### From binary distribution
|
||||
|
||||
Run `example-hotrod(.exe)` executable from the [binary distribution archives][download]:
|
||||
```bash
|
||||
$ example-hotrod all
|
||||
```
|
||||
|
||||
Then navigate to `http://localhost:8080`.
|
||||
|
||||
|
||||
## Migrating from Zipkin
|
||||
|
||||
Collector service exposes Zipkin compatible REST API `/api/v1/spans` which accepts both Thrift and JSON. Also there is `/api/v2/spans` for JSON only.
|
||||
By default it's disabled. It can be enabled with `--collector.zipkin.http-port=9411`.
|
||||
|
||||
Zipkin Thrift IDL file can be found in [jaegertracing/jaeger-idl](https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/zipkincore.thrift).
|
||||
It's compatible with [openzipkin/zipkin-api](https://github.com/openzipkin/zipkin-api/blob/master/thrift/zipkinCore.thrift)
|
||||
|
||||
[hotrod-tutorial]: https://medium.com/@YuriShkuro/take-opentracing-for-a-hotrod-ride-f6e3141f7941
|
||||
[download]: ../../../download/
|
|
@ -1,66 +0,0 @@
|
|||
---
|
||||
title: Introduction
|
||||
weight: 1
|
||||
---
|
||||
|
||||
Welcome to Jaeger's documentation portal! Below, you'll find information for beginners and experienced Jaeger users.
|
||||
|
||||
If you can't find what you are looking for, or have an issue not covered here, we'd love to [hear from you](/get-in-touch).
|
||||
|
||||
## About
|
||||
|
||||
Jaeger, inspired by [Dapper][dapper] and [OpenZipkin](http://zipkin.io),
|
||||
is a distributed tracing system released as open source by [Uber Technologies][ubeross].
|
||||
It is used for monitoring and troubleshooting microservices-based distributed systems, including:
|
||||
|
||||
* Distributed context propagation
|
||||
* Distributed transaction monitoring
|
||||
* Root cause analysis
|
||||
* Service dependency analysis
|
||||
* Performance / latency optimization
|
||||
|
||||
Uber published a blog post, [Evolving Distributed Tracing at Uber](https://eng.uber.com/distributed-tracing/), where they explain the history and reasons for the architectural choices made in Jaeger.
|
||||
|
||||
## Features
|
||||
|
||||
* [OpenTracing](http://opentracing.io/) compatible data model and instrumentation libraries
|
||||
* in [Go](https://github.com/jaegertracing/jaeger-client-go), [Java](https://github.com/jaegertracing/jaeger-client-java), [Node](https://github.com/jaegertracing/jaeger-client-node), [Python](https://github.com/jaegertracing/jaeger-client-python)
|
||||
and [C++](https://github.com/jaegertracing/cpp-client)
|
||||
* Uses consistent upfront sampling with individual per service/endpoint probabilities
|
||||
* Multiple storage backends: Cassandra, Elasticsearch, memory.
|
||||
* Adaptive sampling (coming soon)
|
||||
* Post-collection data processing pipeline (coming soon)
|
||||
|
||||
See [Features](./features/) page for more details.
|
||||
|
||||
## Technical Specs
|
||||
|
||||
* Backend components implemented in Go
|
||||
* React/Javascript UI
|
||||
* Supported storage backends:
|
||||
* [Cassandra 3.4+](./deployment/#cassandra)
|
||||
* [Elasticsearch 5.x, 6.x](./deployment/#elasticsearch)
|
||||
* [Kafka](./deployment/#kafka)
|
||||
* memory storage
|
||||
|
||||
## Quick Start
|
||||
See [running a docker all in one image](getting-started#all-in-one).
|
||||
|
||||
## Screenshots
|
||||
|
||||
### Traces View
|
||||
[](/img/traces-ss.png)
|
||||
|
||||
### Trace Detail View
|
||||
[](/img/trace-detail-ss.png)
|
||||
|
||||
## Related links
|
||||
- [Evolving Distributed tracing At Uber Engineering](https://eng.uber.com/distributed-tracing/)
|
||||
- [Tracing HTTP request latency in Go with OpenTracing](https://medium.com/opentracing/tracing-http-request-latency-in-go-with-opentracing-7cc1282a100a)
|
||||
- [Distributed Tracing with Jaeger & Prometheus on Kubernetes](https://blog.openshift.com/openshift-commons-briefing-82-distributed-tracing-with-jaeger-prometheus-on-kubernetes/)
|
||||
- [Using Jaeger with Istio](https://istio.io/docs/tasks/telemetry/distributed-tracing.html)
|
||||
- [Using Jaeger with Envoy](https://www.envoyproxy.io/docs/envoy/latest/start/sandboxes/jaeger_tracing.html)
|
||||
|
||||
[dapper]: https://research.google.com/pubs/pub36356.html
|
||||
[ubeross]: http://uber.github.io
|
||||
|
|
@ -1,355 +0,0 @@
|
|||
---
|
||||
title: Deployment
|
||||
weight: 7
|
||||
children:
|
||||
- title: Frontend/UI
|
||||
url: frontend-ui
|
||||
---
|
||||
|
||||
The main Jaeger backend components are released as Docker images on Docker Hub:
|
||||
|
||||
Component | Repository
|
||||
--------------------- | ---
|
||||
**jaeger-agent** | [hub.docker.com/r/jaegertracing/jaeger-agent/](https://hub.docker.com/r/jaegertracing/jaeger-agent/)
|
||||
**jaeger-collector** | [hub.docker.com/r/jaegertracing/jaeger-collector/](https://hub.docker.com/r/jaegertracing/jaeger-collector/)
|
||||
**jaeger-query** | [hub.docker.com/r/jaegertracing/jaeger-query/](https://hub.docker.com/r/jaegertracing/jaeger-query/)
|
||||
**jaeger-ingester** | [hub.docker.com/r/jaegertracing/jaeger-ingester/](https://hub.docker.com/r/jaegertracing/jaeger-ingester/)
|
||||
|
||||
There are orchestration templates for running Jaeger with:
|
||||
|
||||
* Kubernetes: [github.com/jaegertracing/jaeger-kubernetes](https://github.com/jaegertracing/jaeger-kubernetes),
|
||||
* OpenShift: [github.com/jaegertracing/jaeger-openshift](https://github.com/jaegertracing/jaeger-openshift).
|
||||
|
||||
## Configuration Options
|
||||
|
||||
Jaeger binaries can be configured in a number of ways (in the order of decreasing priority):
|
||||
|
||||
* command line arguments,
|
||||
* environment variables,
|
||||
* configuration files in JSON, TOML, YAML, HCL, or Java properties formats.
|
||||
|
||||
To see the complete list of options, run the binary with `help` command. Options that are specific to a certain storage backend are only listed if the storage type is selected. For example, to see all available options in the Collector with Cassandra storage:
|
||||
|
||||
```sh
|
||||
$ docker run --rm \
|
||||
-e SPAN_STORAGE_TYPE=cassandra \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}} \
|
||||
help
|
||||
```
|
||||
|
||||
In order to provide configuration parameters via environment variables, find the respective command line option and convert its name to UPPER_SNAKE_CASE, for example:
|
||||
|
||||
Command line option | Environment variable
|
||||
-----------------------------------|-------------------------------
|
||||
`--cassandra.connections-per-host` | `CASSANDRA_CONNECTIONS_PER_HOST`
|
||||
`--metrics-backend` | `METRICS_BACKEND`
|
||||
|
||||
## Agent
|
||||
|
||||
Jaeger client libraries expect **jaeger-agent** process to run locally on each host.
|
||||
The agent exposes the following ports:
|
||||
|
||||
Port | Protocol | Function
|
||||
---- | ------- | ---
|
||||
5775 | UDP | accept zipkin.thrift over compact thrift protocol
|
||||
6831 | UDP | accept jaeger.thrift over compact thrift protocol
|
||||
6832 | UDP | accept jaeger.thrift over binary thrift protocol
|
||||
5778 | HTTP | serve configs, sampling strategies
|
||||
|
||||
It can be executed directly on the host or via Docker, as follows:
|
||||
|
||||
```sh
|
||||
## make sure to expose only the ports you use in your deployment scenario!
|
||||
docker run \
|
||||
--rm \
|
||||
-p5775:5775/udp \
|
||||
-p6831:6831/udp \
|
||||
-p6832:6832/udp \
|
||||
-p5778:5778/tcp \
|
||||
jaegertracing/jaeger-agent:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
### Discovery System Integration
|
||||
|
||||
The agents can connect point to point to a single collector address, which could be
|
||||
load balanced by another infrastructure component (e.g. DNS) across multiple collectors.
|
||||
The agent can also be configured with a static list of collector addresses.
|
||||
|
||||
On Docker, a command like the following can be used:
|
||||
|
||||
```sh
|
||||
docker run \
|
||||
--rm \
|
||||
-p5775:5775/udp \
|
||||
-p6831:6831/udp \
|
||||
-p6832:6832/udp \
|
||||
-p5778:5778/tcp \
|
||||
jaegertracing/jaeger-agent:{{< currentVersion >}} \
|
||||
--reporter.grpc.host-port=jaeger-collector.jaeger-infra.svc:14250
|
||||
```
|
||||
|
||||
Or use `--reporter.tchannel.host-port=jaeger-collector.jaeger-infra.svc:14267` to use
|
||||
legacy tchannel reporter.
|
||||
|
||||
When using gRPC, you have several options for load balancing and name resolution:
|
||||
|
||||
* Single connection and no load balancing. This is the default if you specify a single `host:port`. (example: `--reporter.grpc.host-port=jaeger-collector.jaeger-infra.svc:14250`)
|
||||
* Static list of hostnames and round-robin load balancing. This is what you get with a comma-separated list of addresses. (example: `reporter.grpc.host-port=jaeger-collector1:14250,jaeger-collector2:14250,jaeger-collector3:14250`)
|
||||
* Dynamic DNS resolution and round-robin load balancing. To get this behaviour, prefix the address with `dns:///` and gRPC will attempt to resolve the hostname using SRV records (for [external load balancing](https://github.com/grpc/grpc/blob/master/doc/load-balancing.md)), TXT records (for [service configs](https://github.com/grpc/grpc/blob/master/doc/service_config.md)), and A records. Refer to the [gRPC Name Resolution docs](https://github.com/grpc/grpc/blob/master/doc/naming.md) and the [dns_resolver.go implementation](https://github.com/grpc/grpc-go/blob/master/resolver/dns/dns_resolver.go) for more info. (example: `--reporter.grpc.host-port=dns:///jaeger-collector.jaeger-infra.svc:14250`)
|
||||
|
||||
|
||||
## Collectors
|
||||
|
||||
The collectors are stateless and thus many instances of **jaeger-collector** can be run in parallel.
|
||||
Collectors require almost no configuration, except for the location of Cassandra cluster,
|
||||
via `--cassandra.keyspace` and `--cassandra.servers` options, or the location of Elasticsearch cluster, via
|
||||
`--es.server-urls`, depending on which storage is specified. To see all command line options run
|
||||
|
||||
```sh
|
||||
go run ./cmd/collector/main.go -h
|
||||
```
|
||||
|
||||
or, if you don't have the source code
|
||||
|
||||
```sh
|
||||
docker run -it --rm jaegertracing/jaeger-collector:{{< currentVersion >}} -h
|
||||
```
|
||||
|
||||
At default settings the collector exposes the following ports:
|
||||
|
||||
Port | Protocol | Function
|
||||
----- | ------- | ---
|
||||
14267 | TChannel | used by **jaeger-agent** to send spans in jaeger.thrift format
|
||||
14250 | gRPC | used by **jaeger-agent** to send spans in model.proto format
|
||||
14268 | HTTP | can accept spans directly from clients in jaeger.thrift format over binary thrift protocol
|
||||
9411 | HTTP | can accept Zipkin spans in JSON or Thrift (disabled by default)
|
||||
14269 | HTTP | Health check at **/**
|
||||
|
||||
|
||||
## Storage Backends
|
||||
|
||||
Collectors require a persistent storage backend. Cassandra and Elasticsearch are the primary supported storage backends. Additional backends are [discussed here](https://github.com/jaegertracing/jaeger/issues/638).
|
||||
|
||||
The storage type can be passed via `SPAN_STORAGE_TYPE` environment variable. Valid values are `cassandra`, `elasticsearch`, `kafka` (only as a buffer) and `memory` (only for all-in-one binary).
|
||||
As of version 1.6.0, it's possible to use multiple storage types at the same time by providing a comma-separated list of valid types to the `SPAN_STORAGE_TYPE` environment variable.
|
||||
It's important to note that all listed storage types are used for writing, but only the first type in the list will be used for reading and archiving.
|
||||
|
||||
### Memory
|
||||
|
||||
The in-memory storage is not intended for production workloads. It's intended as a simple solution to get started quickly and
|
||||
data will be lost once the process is gone.
|
||||
|
||||
By default, there's no limit in the amount of traces stored in memory but a limit can be established by passing an
|
||||
integer value via `--memory.max-traces`.
|
||||
|
||||
### Cassandra
|
||||
Supported versions: 3.4+
|
||||
|
||||
Deploying Cassandra itself is out of scope for our documentation. One good
|
||||
source of documentation is the [Apache Cassandra Docs](https://cassandra.apache.org/doc/latest/).
|
||||
|
||||
#### Configuration
|
||||
##### Minimal
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=cassandra \
|
||||
-e CASSANDRA_SERVERS=<...> \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
##### All options
|
||||
To view the full list of configuration options, you can run the following command:
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=cassandra \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}} \
|
||||
--help
|
||||
```
|
||||
|
||||
#### Schema script
|
||||
|
||||
A script is provided to initialize Cassandra keyspace and schema
|
||||
using Cassandra's interactive shell [`cqlsh`][cqlsh]:
|
||||
|
||||
```sh
|
||||
MODE=test sh ./plugin/storage/cassandra/schema/create.sh | cqlsh
|
||||
```
|
||||
|
||||
For production deployment, pass `MODE=prod DATACENTER={datacenter}` arguments to the script,
|
||||
where `{datacenter}` is the name used in the Cassandra configuration / network topology.
|
||||
|
||||
The script also allows overriding TTL, keyspace name, replication factor, etc.
|
||||
Run the script without arguments to see the full list of recognized parameters.
|
||||
|
||||
#### TLS support
|
||||
|
||||
Jaeger supports TLS client to node connections as long as you've configured
|
||||
your Cassandra cluster correctly. After verifying with e.g. `cqlsh`, you can
|
||||
configure the collector and query like so:
|
||||
|
||||
```sh
|
||||
docker run \
|
||||
-e CASSANDRA_SERVERS=<...> \
|
||||
-e CASSANDRA_TLS=true \
|
||||
-e CASSANDRA_TLS_SERVER_NAME="CN-in-certificate" \
|
||||
-e CASSANDRA_TLS_KEY=<path to client key file> \
|
||||
-e CASSANDRA_TLS_CERT=<path to client cert file> \
|
||||
-e CASSANDRA_TLS_CA=<path to your CA cert file> \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
The schema tool also supports TLS. You need to make a custom cqlshrc file like
|
||||
so:
|
||||
|
||||
```
|
||||
# Creating schema in a cassandra cluster requiring client TLS certificates.
|
||||
#
|
||||
# Create a volume for the schema docker container containing four files:
|
||||
# cqlshrc: this file
|
||||
# ca-cert: the cert authority for your keys
|
||||
# client-key: the keyfile for your client
|
||||
# client-cert: the cert file matching client-key
|
||||
#
|
||||
# if there is any sort of DNS mismatch and you want to ignore server validation
|
||||
# issues, then uncomment validate = false below.
|
||||
#
|
||||
# When running the container, map this volume to /root/.cassandra and set the
|
||||
# environment variable CQLSH_SSL=--ssl
|
||||
[ssl]
|
||||
certfile = ~/.cassandra/ca-cert
|
||||
userkey = ~/.cassandra/client-key
|
||||
usercert = ~/.cassandra/client-cert
|
||||
# validate = false
|
||||
```
|
||||
|
||||
### Elasticsearch
|
||||
Supported in Jaeger since 0.6.0
|
||||
Supported versions: 5.x, 6.x
|
||||
|
||||
Elasticsearch does not require initialization other than
|
||||
[installing and running Elasticsearch](https://www.elastic.co/downloads/elasticsearch).
|
||||
Once it is running, pass the correct configuration values to the Jaeger collector and query service.
|
||||
|
||||
#### Configuration
|
||||
##### Minimal
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=elasticsearch \
|
||||
-e ES_SERVER_URLS=<...> \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
##### All options
|
||||
To view the full list of configuration options, you can run the following command:
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=elasticsearch \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}} \
|
||||
--help
|
||||
```
|
||||
|
||||
See the [README](https://github.com/jaegertracing/jaeger/tree/master/plugin/storage/es/README.md) for an in-depth overview of how Jaeger uses Elasticsearch for storage.
|
||||
|
||||
#### Shards and Replicas for Elasticsearch indices
|
||||
|
||||
Shards and replicas are some configuration values to take special attention to, because this is decided upon
|
||||
index creation. [This article](https://qbox.io/blog/optimizing-elasticsearch-how-many-shards-per-index) goes into
|
||||
more information about choosing how many shards should be chosen for optimization.
|
||||
|
||||
### Kafka
|
||||
Supported in Jaeger since 1.6.0
|
||||
Supported Kafka versions: 0.9+
|
||||
|
||||
Kafka can be used as an intermediary buffer between collector and an actual storage.
|
||||
The collector is configured with `SPAN_STORAGE_TYPE=kafka` that makes it write all received spans
|
||||
into a Kafka topic. A new component [Ingester](#ingester), added in version 1.7.0, is used to read from
|
||||
Kafka and store spans in another storage backend (Elasticsearch or Cassandra).
|
||||
|
||||
Writing to Kafka is particularly useful for building post-processing data pipelines.
|
||||
|
||||
#### Configuration
|
||||
##### Minimal
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=kafka \
|
||||
-e KAFKA_BROKERS=<...> \
|
||||
-e KAFKA_TOPIC=<...> \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
##### All options
|
||||
To view the full list of configuration options, you can run the following command:
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=kafka \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}} \
|
||||
--help
|
||||
```
|
||||
|
||||
#### Topic & partitions
|
||||
Unless your Kafka cluster is configured to automatically create topics, you will need to create it ahead of time. You can refer to [the Kafka quickstart documentation](https://kafka.apache.org/documentation/#quickstart_createtopic) to learn how.
|
||||
|
||||
You can find more information about topics and partitions in general in the [official documentation](https://kafka.apache.org/documentation/#intro_topics). [This article](https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/) provide more details about how to choose the number of partitions.
|
||||
|
||||
## Ingester
|
||||
**jaeger-ingester** is a service which reads span data from Kafka topic and writes it to another storage backend (Elasticsearch or Cassandra).
|
||||
|
||||
Port | Protocol | Function
|
||||
----- | ------- | ---
|
||||
14270 | HTTP | Health check at **/**
|
||||
14271 | HTTP | Metrics endpoint
|
||||
|
||||
To view all exposed configuration options run the following command:
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=cassandra \
|
||||
jaegertracing/jaeger-ingester:{{< currentVersion >}}
|
||||
--help
|
||||
```
|
||||
|
||||
## Query Service & UI
|
||||
|
||||
**jaeger-query** serves the API endpoints and a React/Javascript UI.
|
||||
The service is stateless and is typically run behind a load balancer, such as **nginx**.
|
||||
|
||||
At default settings the query service exposes the following port(s):
|
||||
|
||||
Port | Protocol | Function
|
||||
----- | ------- | ---
|
||||
16686 | HTTP | **/api/*** endpoints and Jaeger UI at **/**
|
||||
16687 | HTTP | Health check at **/**
|
||||
|
||||
### Minimal deployment example (Elasticsearch backend):
|
||||
```sh
|
||||
docker run -d --rm \
|
||||
-p 16686:16686 \
|
||||
-p 16687:16687 \
|
||||
-e SPAN_STORAGE_TYPE=elasticsearch \
|
||||
-e ES_SERVER_URLS=http://<ES_SERVER_IP>:<ES_SERVER_PORT> \
|
||||
jaegertracing/jaeger-query:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
### UI Base Path
|
||||
|
||||
The base path for all **jaeger-query** HTTP routes can be set to a non-root value, e.g. `/jaeger` would cause all UI URLs to start with `/jaeger`. This can be useful when running **jaeger-query** behind a reverse proxy.
|
||||
|
||||
The base path can be configured via the `--query.base-path` command line parameter or the `QUERY_BASE_PATH` environment variable.
|
||||
|
||||
### UI Customization and Embedding
|
||||
|
||||
Please refer to the [dedicated Frontend/UI page](../frontend-ui/).
|
||||
|
||||
## Aggregation Jobs for Service Dependencies
|
||||
|
||||
Production deployments need an external process which aggregates data and creates dependency links between services.
|
||||
Project [spark-dependencies](https://github.com/jaegertracing/spark-dependencies) is a Spark job which derives
|
||||
dependency links and stores them directly to the storage.
|
||||
|
||||
## Configuration
|
||||
|
||||
All binaries accepts command line properties and environmental variables which are managed by
|
||||
by [viper](https://github.com/spf13/viper) and [cobra](https://github.com/spf13/cobra).
|
||||
The names of environmental properties are capital letters and characters `-` and `.` are replaced with `_`.
|
||||
To list all configuration properties call `jaeger-binary -h`.
|
||||
|
||||
[cqlsh]: http://cassandra.apache.org/doc/latest/tools/cqlsh.html
|
|
@ -1,16 +0,0 @@
|
|||
---
|
||||
title: Frequently Asked Questions
|
||||
description: Answers to some frequently asked questions about Jaeger.
|
||||
weight: 11
|
||||
---
|
||||
|
||||
## Why is the Dependencies page empty?
|
||||
|
||||
The Dependencies page shows a graph of services traced by Jaeger and connections between them. When you are using `all-in-one` binary with in-memory storage, the graph is calculated on-demand from all the traces stored in memory. However, if you are using a real distributed storage like Cassandra or Elasticsearch, it is too expensive to scan all the data in the database to build the service graph. Instead, the Jaeger project provides "big data" jobs that can be used to extract the service graph data from traces:
|
||||
|
||||
* https://github.com/jaegertracing/spark-dependencies - the older Spark job that can be run periodically
|
||||
* https://github.com/jaegertracing/jaeger-analytics - the new (experimental) streaming Flink jobs that run continuously and build the service graph in smaller time intervals
|
||||
|
||||
## Why do I not see any spans in Jaeger?
|
||||
|
||||
Please refer to the [Troubleshooting](../troubleshooting/) guide.
|
|
@ -1,57 +0,0 @@
|
|||
---
|
||||
title: Features
|
||||
weight: 3
|
||||
---
|
||||
|
||||
Jaeger is used for monitoring and troubleshooting microservices-based distributed systems, including:
|
||||
|
||||
* Distributed context propagation
|
||||
* Distributed transaction monitoring
|
||||
* Root cause analysis
|
||||
* Service dependency analysis
|
||||
* Performance / latency optimization
|
||||
|
||||
## High Scalability
|
||||
|
||||
Jaeger backend is designed to have no single points of failure and to scale with the business needs.
|
||||
For example, any given Jaeger installation at Uber is typically processing several billion {{< tip "spans" "span" >}} per day.
|
||||
|
||||
## Native support for OpenTracing
|
||||
|
||||
Jaeger backend, Web UI, and instrumentation libraries have been designed from ground up to support the OpenTracing standard.
|
||||
|
||||
* Represent {{< tip "traces" "trace" >}} as {{< tip "directed acyclic graphs" "directed acyclic graph" >}} (not just trees) via [span references](https://github.com/opentracing/specification/blob/master/specification.md#references-between-spans)
|
||||
* Support strongly typed span _tags_ and _structured logs_
|
||||
* Support general distributed context propagation mechanism via _baggage_
|
||||
|
||||
## Multiple storage backends
|
||||
|
||||
Jaeger supports two popular open source NoSQL databases as trace storage backends: Cassandra 3.4+ and Elasticsearch 5.x/6.x.
|
||||
There are ongoing community experiments using other databases, such as ScyllaDB, InfluxDB, Amazon DynamoDB. Jaeger also ships
|
||||
with a simple in-memory storage for testing setups.
|
||||
|
||||
## Modern Web UI
|
||||
|
||||
Jaeger Web UI is implemented in Javascript using popular open source frameworks like React. Several performance
|
||||
improvements have been released in v1.0 to allow the UI to efficiently deal with large volumes of data, and to display
|
||||
{{< tip "traces" "trace" >}} with tens of thousands of {{< tip "spans" "span" >}} (e.g. we tried a trace with 80,000 spans).
|
||||
|
||||
## Cloud Native Deployment
|
||||
|
||||
Jaeger backend is distributed as a collection of Docker images. The binaries support various configuration methods,
|
||||
including command line options, environment variables, and configuration files in multiple formats (yaml, toml, etc.).
|
||||
Deployment to Kubernetes clusters is assisted by a [Kubernetes operator](https://github.com/jaegertracing/jaeger-operator), [Kubernetes templates](https://github.com/jaegertracing/jaeger-kubernetes)
|
||||
and a [Helm chart](https://github.com/kubernetes/charts/tree/master/incubator/jaeger).
|
||||
|
||||
## Observability
|
||||
|
||||
All Jaeger backend components expose [Prometheus](https://prometheus.io/) metrics by default (other metrics backends are
|
||||
also supported). Logs are written to standard out using the structured logging library [zap](https://github.com/uber-go/zap).
|
||||
|
||||
## Backwards compatibility with Zipkin
|
||||
|
||||
Although we recommend instrumenting applications with OpenTracing API and binding to Jaeger client libraries to benefit
|
||||
from advanced features not available elsewhere, if your organization has already invested in the instrumentation
|
||||
using Zipkin libraries, you do not have to rewrite all that code. Jaeger provides backwards compatibility with Zipkin
|
||||
by accepting spans in Zipkin formats (Thrift or JSON v1/v2) over HTTP. Switching from Zipkin backend is just a matter
|
||||
of routing the traffic from Zipkin libraries to the Jaeger backend.
|
|
@ -1,196 +0,0 @@
|
|||
---
|
||||
title: Frontend/UI
|
||||
hasparent: true
|
||||
weight: 10
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
Several aspects of the UI can be configured:
|
||||
|
||||
* The Dependencies section can be enabled / configured
|
||||
* Google Analytics tracking can be enabled / configured
|
||||
* Additional menu options can be added to the global nav
|
||||
|
||||
These options can be configured by a JSON configuration file. The `--query.ui-config` command line parameter of the query service must then be set to the path to the JSON file when the query service is started.
|
||||
|
||||
An example configuration file:
|
||||
|
||||
```json
|
||||
{
|
||||
"dependencies": {
|
||||
"dagMaxNumServices": 200,
|
||||
"menuEnabled": true
|
||||
},
|
||||
"archiveEnabled": true,
|
||||
"tracking": {
|
||||
"gaID": "UA-000000-2",
|
||||
"trackErrors": true
|
||||
},
|
||||
"menu": [
|
||||
{
|
||||
"label": "About Jaeger",
|
||||
"items": [
|
||||
{
|
||||
"label": "GitHub",
|
||||
"url": "https://github.com/jaegertracing/jaeger"
|
||||
},
|
||||
{
|
||||
"label": "Docs",
|
||||
"url": "http://jaeger.readthedocs.io/en/latest/"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
`dependencies.dagMaxNumServices` defines the maximum number of services allowed before the DAG dependency view is disabled. Default: `200`.
|
||||
|
||||
`dependencies.menuEnabled` enables (`true`) or disables (`false`) the dependencies menu button. Default: `true`.
|
||||
|
||||
`archiveEnabled` enables (`true`) or disables (`false`) the archive traces button. Default: `false`. It requires a configuration of an archive storage in Query service. Archived traces are only accessible directly by ID, they are not searchable.
|
||||
|
||||
`tracking.gaID` defines the Google Analytics tracking ID. This is required for Google Analytics tracking, and setting it to a non-`null` value enables Google Analytics tracking. Default: `null`.
|
||||
|
||||
`tracking.trackErrors` enables (`true`) or disables (`false`) error tracking via Google Analytics. Errors can only be tracked if a valid Google Analytics ID is provided. For additional details on error tracking via Google Analytics see the [tracking README](https://github.com/jaegertracing/jaeger-ui/blob/c622330546afc1be59a42f874bcc1c2fadf7e69a/src/utils/tracking/README.md) in the UI repo. Default: `true`.
|
||||
|
||||
`menu` allows additional links to be added to the global nav. The additional links are right-aligned.
|
||||
|
||||
In the sample JSON config above, the configured menu will have a dropdown labeled "About Jaeger" with sub-options for "GitHub" and "Docs". The format for a link in the top right menu is as follows:
|
||||
|
||||
```json
|
||||
{
|
||||
"label": "Some text here",
|
||||
"url": "https://example.com"
|
||||
}
|
||||
```
|
||||
|
||||
Links can either be members of the `menu` Array, directly, or they can be grouped into a dropdown menu option. The format for a group of links is:
|
||||
|
||||
```json
|
||||
{
|
||||
"label": "Dropdown button",
|
||||
"items": [ ]
|
||||
}
|
||||
```
|
||||
|
||||
The `items` Array should contain one or more link configurations.
|
||||
|
||||
## Embedded Mode
|
||||
|
||||
Starting with version 1.9, Jaeger UI provides an "embedded" layout mode which is intended to support integrating Jaeger UI into other applications. Currently (as of `v0`), the approach taken is to remove various UI elements from the page to make the UI better suited for space-constrained layouts.
|
||||
|
||||
The embedded mode is induced and configured via URL query parameters.
|
||||
|
||||
To enter embedded mode, the `uiEmbed=v0` query parameter and value must be added to the URL. For example, the following URL will show the trace with ID `abc123` in embedded mode:
|
||||
|
||||
```
|
||||
http://localhost:16686/trace/abc123?uiEmbed=v0
|
||||
```
|
||||
|
||||
`uiEmbed=v0` is required.
|
||||
|
||||
Further, each page supported has an <img src="/img/frontend-ui/embed-open-icon.png" style="width: 20px; height:20px;display:inline;" alt="Embed open window"> button added that will open the non-embedded page in a new tab.
|
||||
|
||||
The following pages support embedded mode:
|
||||
|
||||
* Search Page
|
||||
* Trace Page
|
||||
|
||||
### Search Page
|
||||
|
||||
To integrate the Search Trace Page to our application we have to indicate to the Jaeger UI that we want to use the embed mode with `uiEmbed=v0`.
|
||||
|
||||
For example:
|
||||
|
||||
```
|
||||
http://localhost:16686/search?
|
||||
service=my-service&
|
||||
start=1543917759557000&
|
||||
end=1543921359557000&
|
||||
limit=20&
|
||||
lookback=1h&
|
||||
maxDuration&
|
||||
minDuration&
|
||||
uiEmbed=v0
|
||||
```
|
||||
|
||||

|
||||
|
||||
#### Configuration options
|
||||
|
||||
The following query parameter can be used to configure the layout of the search page :
|
||||
|
||||
* `uiSearchHideGraph=1` - disables the display of the scatter plot above the search results
|
||||
|
||||
```
|
||||
http://localhost:16686/search?
|
||||
service=my-service&
|
||||
start=1543917759557000&
|
||||
end=1543921359557000&
|
||||
limit=20&
|
||||
lookback=1h&
|
||||
maxDuration&
|
||||
minDuration&
|
||||
uiEmbed=v0&
|
||||
uiSearchHideGraph=1
|
||||
```
|
||||
|
||||

|
||||
|
||||
### Trace Page
|
||||
|
||||
|
||||
To integrate the Trace Page to our application we have to indicate to the Jaeger UI that we want to use the embed mode with `uiEmbed=v0`.
|
||||
|
||||
For example:
|
||||
|
||||
```sh
|
||||
http://localhost:16686/trace/{trace-id}?uiEmbed=v0
|
||||
```
|
||||

|
||||
|
||||
If we have navigated to this view from the search traces page we'll have a button to go back to the results page.
|
||||
|
||||

|
||||
|
||||
#### Configuration options
|
||||
|
||||
The following query parameters can be used to configure the layout of the trace page :
|
||||
|
||||
* `uiTimelineCollapseTitle=1` causes the trace header to start out collapsed, which hides the summary and the minimap.
|
||||
|
||||
```
|
||||
http://localhost:16686/trace/{trace-id}?
|
||||
uiEmbed=v0&
|
||||
uiTimelineCollapseTitle=1
|
||||
```
|
||||

|
||||
|
||||
* `uiTimelineHideMinimap=1` removes the minimap, entirely, regardless of whether the trace header is expanded or not.
|
||||
|
||||
```
|
||||
http://localhost:16686/trace/{trace-id}?
|
||||
uiEmbed=v0&
|
||||
uiTimelineHideMinimap=1
|
||||
```
|
||||

|
||||
|
||||
* `uiTimelineHideSummary=1` - removes the trace summary information (number of services, etc.) entirely, regardless of whether the trace header is expanded or not.
|
||||
|
||||
```
|
||||
http://localhost:16686/trace/{trace-id}?
|
||||
uiEmbed=v0&
|
||||
uiTimelineHideSummary=1
|
||||
```
|
||||

|
||||
|
||||
We can also combine the options:
|
||||
```
|
||||
http://localhost:16686/trace/{trace-id}?
|
||||
uiEmbed=v0&
|
||||
uiTimelineHideMinimap=1&
|
||||
uiTimelineHideSummary=1
|
||||
```
|
||||

|
|
@ -1,129 +0,0 @@
|
|||
---
|
||||
title: Getting started
|
||||
description: Get up and running with Jaeger in your local environment
|
||||
weight: 2
|
||||
---
|
||||
|
||||
## Instrumentation
|
||||
|
||||
Your applications must be instrumented before they can send tracing data to Jaeger backend. Check the [Client Libraries](../client-libraries) section for information about how to use the OpenTracing API and how to initialize and configure Jaeger tracers.
|
||||
|
||||
## All in One
|
||||
|
||||
All-in-one is an executable designed for quick local testing, launches the Jaeger UI, collector, query, and agent, with an in memory storage component.
|
||||
|
||||
The simplest way to start the all-in-one is to use the pre-built image published to DockerHub (a single command line).
|
||||
|
||||
```bash
|
||||
$ docker run -d --name jaeger \
|
||||
-e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
|
||||
-p 5775:5775/udp \
|
||||
-p 6831:6831/udp \
|
||||
-p 6832:6832/udp \
|
||||
-p 5778:5778 \
|
||||
-p 16686:16686 \
|
||||
-p 14268:14268 \
|
||||
-p 9411:9411 \
|
||||
jaegertracing/all-in-one:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
Or run the `jaeger-all-in-one(.exe)` executable from the [binary distribution archives][download]:
|
||||
|
||||
```bash
|
||||
$ jaeger-all-in-one --collector.zipkin.http-port=9411
|
||||
```
|
||||
|
||||
You can then navigate to `http://localhost:16686` to access the Jaeger UI.
|
||||
|
||||
The container exposes the following ports:
|
||||
|
||||
Port | Protocol | Component | Function
|
||||
----- | ------- | --------- | ---
|
||||
5775 | UDP | agent | accept `zipkin.thrift` over compact thrift protocol (deprecated, used by legacy clients only)
|
||||
6831 | UDP | agent | accept `jaeger.thrift` over compact thrift protocol
|
||||
6832 | UDP | agent | accept `jaeger.thrift` over binary thrift protocol
|
||||
5778 | HTTP | agent | serve configs
|
||||
16686 | HTTP | query | serve frontend
|
||||
14268 | HTTP | collector | accept `jaeger.thrift` directly from clients
|
||||
14250 | HTTP | collector | accept `model.proto`
|
||||
9411 | HTTP | collector | Zipkin compatible endpoint (optional)
|
||||
|
||||
|
||||
## Kubernetes and OpenShift
|
||||
|
||||
* Kubernetes templates: https://github.com/jaegertracing/jaeger-kubernetes
|
||||
* Kubernetes Operator: https://github.com/jaegertracing/jaeger-operator
|
||||
* OpenShift templates: https://github.com/jaegertracing/jaeger-openshift
|
||||
|
||||
## Sample App: HotROD
|
||||
|
||||
HotROD (Rides on Demand) is a demo application that consists of several microservices and
|
||||
illustrates the use of the [OpenTracing API](http://opentracing.io).
|
||||
A tutorial / walkthrough is available in the blog post:
|
||||
[Take OpenTracing for a HotROD ride][hotrod-tutorial].
|
||||
|
||||
It can be run standalone, but requires Jaeger backend to view the traces.
|
||||
|
||||
### Features
|
||||
|
||||
- Discover architecture of the whole system via data-driven dependency
|
||||
diagram.
|
||||
- View request timeline and errors; understand how the app works.
|
||||
- Find sources of latency and lack of concurrency.
|
||||
- Highly contextualized logging.
|
||||
- Use baggage propagation to:
|
||||
|
||||
- Diagnose inter-request contention (queueing).
|
||||
- Attribute time spent in a service.
|
||||
|
||||
- Use open source libraries with OpenTracing integration to get
|
||||
vendor-neutral instrumentation for free.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- You need Go 1.11 or higher installed on your machine to run from source.
|
||||
- Requires a [running Jaeger backend](#all-in-one) to view the traces.
|
||||
|
||||
### Running
|
||||
|
||||
#### From Source
|
||||
|
||||
```bash
|
||||
mkdir -p $GOPATH/src/github.com/jaegertracing
|
||||
cd $GOPATH/src/github.com/jaegertracing
|
||||
git clone git@github.com:jaegertracing/jaeger.git jaeger
|
||||
cd jaeger
|
||||
make install
|
||||
go run ./examples/hotrod/main.go all
|
||||
```
|
||||
#### From docker
|
||||
|
||||
```bash
|
||||
$ docker run --rm -it \
|
||||
--link jaeger \
|
||||
-p8080-8083:8080-8083 \
|
||||
-e JAEGER_AGENT_HOST="jaeger" \
|
||||
jaegertracing/example-hotrod:{{< currentVersion >}} \
|
||||
all
|
||||
```
|
||||
|
||||
#### From binary distribution
|
||||
|
||||
Run `example-hotrod(.exe)` executable from the [binary distribution archives][download]:
|
||||
```bash
|
||||
$ example-hotrod all
|
||||
```
|
||||
|
||||
Then navigate to `http://localhost:8080`.
|
||||
|
||||
|
||||
## Migrating from Zipkin
|
||||
|
||||
Collector service exposes Zipkin compatible REST API `/api/v1/spans` which accepts both Thrift and JSON. Also there is `/api/v2/spans` for JSON only.
|
||||
By default it's disabled. It can be enabled with `--collector.zipkin.http-port=9411`.
|
||||
|
||||
Zipkin Thrift IDL file can be found in [jaegertracing/jaeger-idl](https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/zipkincore.thrift).
|
||||
It's compatible with [openzipkin/zipkin-api](https://github.com/openzipkin/zipkin-api/blob/master/thrift/zipkinCore.thrift)
|
||||
|
||||
[hotrod-tutorial]: https://medium.com/@YuriShkuro/take-opentracing-for-a-hotrod-ride-f6e3141f7941
|
||||
[download]: ../../../download/
|
|
@ -1,66 +0,0 @@
|
|||
---
|
||||
title: Introduction
|
||||
weight: 1
|
||||
---
|
||||
|
||||
Welcome to Jaeger's documentation portal! Below, you'll find information for beginners and experienced Jaeger users.
|
||||
|
||||
If you can't find what you are looking for, or have an issue not covered here, we'd love to [hear from you](/get-in-touch).
|
||||
|
||||
## About
|
||||
|
||||
Jaeger, inspired by [Dapper][dapper] and [OpenZipkin](http://zipkin.io),
|
||||
is a distributed tracing system released as open source by [Uber Technologies][ubeross].
|
||||
It is used for monitoring and troubleshooting microservices-based distributed systems, including:
|
||||
|
||||
* Distributed context propagation
|
||||
* Distributed transaction monitoring
|
||||
* Root cause analysis
|
||||
* Service dependency analysis
|
||||
* Performance / latency optimization
|
||||
|
||||
Uber published a blog post, [Evolving Distributed Tracing at Uber](https://eng.uber.com/distributed-tracing/), where they explain the history and reasons for the architectural choices made in Jaeger.
|
||||
|
||||
## Features
|
||||
|
||||
* [OpenTracing](http://opentracing.io/) compatible data model and instrumentation libraries
|
||||
* in [Go](https://github.com/jaegertracing/jaeger-client-go), [Java](https://github.com/jaegertracing/jaeger-client-java), [Node](https://github.com/jaegertracing/jaeger-client-node), [Python](https://github.com/jaegertracing/jaeger-client-python)
|
||||
and [C++](https://github.com/jaegertracing/cpp-client)
|
||||
* Uses consistent upfront sampling with individual per service/endpoint probabilities
|
||||
* Multiple storage backends: Cassandra, Elasticsearch, memory.
|
||||
* Adaptive sampling (coming soon)
|
||||
* Post-collection data processing pipeline (coming soon)
|
||||
|
||||
See [Features](./features/) page for more details.
|
||||
|
||||
## Technical Specs
|
||||
|
||||
* Backend components implemented in Go
|
||||
* React/Javascript UI
|
||||
* Supported storage backends:
|
||||
* [Cassandra 3.4+](./deployment/#cassandra)
|
||||
* [Elasticsearch 5.x, 6.x](./deployment/#elasticsearch)
|
||||
* [Kafka](./deployment/#kafka)
|
||||
* memory storage
|
||||
|
||||
## Quick Start
|
||||
See [running a docker all in one image](getting-started#all-in-one).
|
||||
|
||||
## Screenshots
|
||||
|
||||
### Traces View
|
||||
[](/img/traces-ss.png)
|
||||
|
||||
### Trace Detail View
|
||||
[](/img/trace-detail-ss.png)
|
||||
|
||||
## Related links
|
||||
- [Evolving Distributed tracing At Uber Engineering](https://eng.uber.com/distributed-tracing/)
|
||||
- [Tracing HTTP request latency in Go with OpenTracing](https://medium.com/opentracing/tracing-http-request-latency-in-go-with-opentracing-7cc1282a100a)
|
||||
- [Distributed Tracing with Jaeger & Prometheus on Kubernetes](https://blog.openshift.com/openshift-commons-briefing-82-distributed-tracing-with-jaeger-prometheus-on-kubernetes/)
|
||||
- [Using Jaeger with Istio](https://istio.io/docs/tasks/telemetry/distributed-tracing.html)
|
||||
- [Using Jaeger with Envoy](https://www.envoyproxy.io/docs/envoy/latest/start/sandboxes/jaeger_tracing.html)
|
||||
|
||||
[dapper]: https://research.google.com/pubs/pub36356.html
|
||||
[ubeross]: http://uber.github.io
|
||||
|
|
@ -1,396 +0,0 @@
|
|||
---
|
||||
title: Deployment
|
||||
weight: 7
|
||||
children:
|
||||
- title: Frontend/UI
|
||||
url: frontend-ui
|
||||
---
|
||||
|
||||
The main Jaeger backend components are released as Docker images on Docker Hub:
|
||||
|
||||
Component | Repository
|
||||
--------------------- | ---
|
||||
**jaeger-agent** | [hub.docker.com/r/jaegertracing/jaeger-agent/](https://hub.docker.com/r/jaegertracing/jaeger-agent/)
|
||||
**jaeger-collector** | [hub.docker.com/r/jaegertracing/jaeger-collector/](https://hub.docker.com/r/jaegertracing/jaeger-collector/)
|
||||
**jaeger-query** | [hub.docker.com/r/jaegertracing/jaeger-query/](https://hub.docker.com/r/jaegertracing/jaeger-query/)
|
||||
**jaeger-ingester** | [hub.docker.com/r/jaegertracing/jaeger-ingester/](https://hub.docker.com/r/jaegertracing/jaeger-ingester/)
|
||||
|
||||
There are orchestration templates for running Jaeger with:
|
||||
|
||||
* Kubernetes: [github.com/jaegertracing/jaeger-kubernetes](https://github.com/jaegertracing/jaeger-kubernetes),
|
||||
* OpenShift: [github.com/jaegertracing/jaeger-openshift](https://github.com/jaegertracing/jaeger-openshift).
|
||||
|
||||
## Configuration Options
|
||||
|
||||
Jaeger binaries can be configured in a number of ways (in the order of decreasing priority):
|
||||
|
||||
* command line arguments,
|
||||
* environment variables,
|
||||
* configuration files in JSON, TOML, YAML, HCL, or Java properties formats.
|
||||
|
||||
To see the complete list of options, run the binary with `help` command. Options that are specific to a certain storage backend are only listed if the storage type is selected. For example, to see all available options in the Collector with Cassandra storage:
|
||||
|
||||
```sh
|
||||
$ docker run --rm \
|
||||
-e SPAN_STORAGE_TYPE=cassandra \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}} \
|
||||
help
|
||||
```
|
||||
|
||||
In order to provide configuration parameters via environment variables, find the respective command line option and convert its name to UPPER_SNAKE_CASE, for example:
|
||||
|
||||
Command line option | Environment variable
|
||||
-----------------------------------|-------------------------------
|
||||
`--cassandra.connections-per-host` | `CASSANDRA_CONNECTIONS_PER_HOST`
|
||||
`--metrics-backend` | `METRICS_BACKEND`
|
||||
|
||||
## Agent
|
||||
|
||||
Jaeger client libraries expect **jaeger-agent** process to run locally on each host.
|
||||
The agent exposes the following ports:
|
||||
|
||||
Port | Protocol | Function
|
||||
----- | ------- | ---
|
||||
6831 | UDP | accept [jaeger.thrift][jaeger-thrift] in `compact` Thrift protocol used by most current Jaeger clients
|
||||
6832 | UDP | accept [jaeger.thrift][jaeger-thrift] in `binary` Thrift protocol used by Node.js Jaeger client (because [thriftrw][thriftrw] npm package does not support `compact` protocol)
|
||||
5778 | HTTP | serve configs, sampling strategies
|
||||
5775 | UDP | accept [zipkin.thrift][zipkin-thrift] in `compact` Thrift protocol (deprecated; only used by very old Jaeger clients, circa 2016)
|
||||
14271 | HTTP | Healthcheck at `/` and metrics at `/metrics`
|
||||
|
||||
It can be executed directly on the host or via Docker, as follows:
|
||||
|
||||
```sh
|
||||
## make sure to expose only the ports you use in your deployment scenario!
|
||||
docker run \
|
||||
--rm \
|
||||
-p6831:6831/udp \
|
||||
-p6832:6832/udp \
|
||||
-p5778:5778/tcp \
|
||||
-p5775:5775/udp \
|
||||
jaegertracing/jaeger-agent:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
### Discovery System Integration
|
||||
|
||||
The agents can connect point to point to a single collector address, which could be
|
||||
load balanced by another infrastructure component (e.g. DNS) across multiple collectors.
|
||||
The agent can also be configured with a static list of collector addresses.
|
||||
|
||||
On Docker, a command like the following can be used:
|
||||
|
||||
```sh
|
||||
docker run \
|
||||
--rm \
|
||||
-p5775:5775/udp \
|
||||
-p6831:6831/udp \
|
||||
-p6832:6832/udp \
|
||||
-p5778:5778/tcp \
|
||||
jaegertracing/jaeger-agent:{{< currentVersion >}} \
|
||||
--reporter.grpc.host-port=jaeger-collector.jaeger-infra.svc:14250
|
||||
```
|
||||
|
||||
Or use `--reporter.tchannel.host-port=jaeger-collector.jaeger-infra.svc:14267` to use
|
||||
legacy tchannel reporter.
|
||||
|
||||
When using gRPC, you have several options for load balancing and name resolution:
|
||||
|
||||
* Single connection and no load balancing. This is the default if you specify a single `host:port`. (example: `--reporter.grpc.host-port=jaeger-collector.jaeger-infra.svc:14250`)
|
||||
* Static list of hostnames and round-robin load balancing. This is what you get with a comma-separated list of addresses. (example: `reporter.grpc.host-port=jaeger-collector1:14250,jaeger-collector2:14250,jaeger-collector3:14250`)
|
||||
* Dynamic DNS resolution and round-robin load balancing. To get this behaviour, prefix the address with `dns:///` and gRPC will attempt to resolve the hostname using SRV records (for [external load balancing](https://github.com/grpc/grpc/blob/master/doc/load-balancing.md)), TXT records (for [service configs](https://github.com/grpc/grpc/blob/master/doc/service_config.md)), and A records. Refer to the [gRPC Name Resolution docs](https://github.com/grpc/grpc/blob/master/doc/naming.md) and the [dns_resolver.go implementation](https://github.com/grpc/grpc-go/blob/master/resolver/dns/dns_resolver.go) for more info. (example: `--reporter.grpc.host-port=dns:///jaeger-collector.jaeger-infra.svc:14250`)
|
||||
|
||||
### Agent level tags
|
||||
|
||||
Jaeger supports agent level tags, that can be added to the process tags of all spans passing through the agent. This is supported through the command line flag `--jaeger.tags=key1=value1,key2=value2,...,keyn=valuen`. Tags can also be set through an environment flag like so - `--jaeger.tags=key=${envFlag:defaultValue}` - The tag value will be set to the value of the `envFlag` environment key and `defaultValue` if not set. This feature is not supported for the tchannel reporter, enabled using the flags `--collector.host-port` or `--reporter.tchannel.host-port`.
|
||||
|
||||
|
||||
## Collectors
|
||||
|
||||
The collectors are stateless and thus many instances of **jaeger-collector** can be run in parallel.
|
||||
Collectors require almost no configuration, except for the location of Cassandra cluster,
|
||||
via `--cassandra.keyspace` and `--cassandra.servers` options, or the location of Elasticsearch cluster, via
|
||||
`--es.server-urls`, depending on which storage is specified. To see all command line options run
|
||||
|
||||
```sh
|
||||
go run ./cmd/collector/main.go -h
|
||||
```
|
||||
|
||||
or, if you don't have the source code
|
||||
|
||||
```sh
|
||||
docker run -it --rm jaegertracing/jaeger-collector:{{< currentVersion >}} -h
|
||||
```
|
||||
|
||||
At default settings the collector exposes the following ports:
|
||||
|
||||
Port | Protocol | Function
|
||||
----- | ------- | ---
|
||||
14267 | TChannel | used by **jaeger-agent** to send spans in jaeger.thrift format
|
||||
14250 | gRPC | used by **jaeger-agent** to send spans in model.proto format
|
||||
14268 | HTTP | can accept spans directly from clients in jaeger.thrift format over binary thrift protocol
|
||||
9411 | HTTP | can accept Zipkin spans in JSON or Thrift (disabled by default)
|
||||
14269 | HTTP | Healthcheck at `/` and metrics at `/metrics`
|
||||
|
||||
|
||||
## Storage Backends
|
||||
|
||||
Collectors require a persistent storage backend. Cassandra and Elasticsearch are the primary supported storage backends. Additional backends are [discussed here](https://github.com/jaegertracing/jaeger/issues/638).
|
||||
|
||||
The storage type can be passed via `SPAN_STORAGE_TYPE` environment variable. Valid values are `cassandra`, `elasticsearch`, `kafka` (only as a buffer), `grpc-plugin`, `badger` (only with all-in-one) and `memory` (only with all-in-one).
|
||||
As of version 1.6.0, it's possible to use multiple storage types at the same time by providing a comma-separated list of valid types to the `SPAN_STORAGE_TYPE` environment variable.
|
||||
It's important to note that all listed storage types are used for writing, but only the first type in the list will be used for reading and archiving.
|
||||
|
||||
### Memory
|
||||
|
||||
The in-memory storage is not intended for production workloads. It's intended as a simple solution to get started quickly and
|
||||
data will be lost once the process is gone.
|
||||
|
||||
By default, there's no limit in the amount of traces stored in memory but a limit can be established by passing an
|
||||
integer value via `--memory.max-traces`.
|
||||
|
||||
### Badger - local storage
|
||||
Experimental since Jaeger 1.9
|
||||
|
||||
[Badger](https://github.com/dgraph-io/badger) is an embedded local storage, only available
|
||||
with **all-in-one** distribution. By default it acts as an ephemeral storage using a temporary filesystem.
|
||||
This can be overridden by using the `--badger.ephemeral=false` option.
|
||||
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=badger \
|
||||
-e BADGER_EPHEMERAL=false \
|
||||
-e BADGER_DIRECTORY_VALUE=/badger/data \
|
||||
-e BADGER_DIRECTORY_KEY=/badger/key \
|
||||
-v <storage_dir_on_host>:/badger \
|
||||
-p 16686:16686 \
|
||||
jaegertracing/all-in-one:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
### Cassandra
|
||||
Supported versions: 3.4+
|
||||
|
||||
Deploying Cassandra itself is out of scope for our documentation. One good
|
||||
source of documentation is the [Apache Cassandra Docs](https://cassandra.apache.org/doc/latest/).
|
||||
|
||||
#### Configuration
|
||||
##### Minimal
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=cassandra \
|
||||
-e CASSANDRA_SERVERS=<...> \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
##### All options
|
||||
To view the full list of configuration options, you can run the following command:
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=cassandra \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}} \
|
||||
--help
|
||||
```
|
||||
|
||||
#### Schema script
|
||||
|
||||
A script is provided to initialize Cassandra keyspace and schema
|
||||
using Cassandra's interactive shell [`cqlsh`][cqlsh]:
|
||||
|
||||
```sh
|
||||
MODE=test sh ./plugin/storage/cassandra/schema/create.sh | cqlsh
|
||||
```
|
||||
|
||||
For production deployment, pass `MODE=prod DATACENTER={datacenter}` arguments to the script,
|
||||
where `{datacenter}` is the name used in the Cassandra configuration / network topology.
|
||||
|
||||
The script also allows overriding TTL, keyspace name, replication factor, etc.
|
||||
Run the script without arguments to see the full list of recognized parameters.
|
||||
|
||||
#### TLS support
|
||||
|
||||
Jaeger supports TLS client to node connections as long as you've configured
|
||||
your Cassandra cluster correctly. After verifying with e.g. `cqlsh`, you can
|
||||
configure the collector and query like so:
|
||||
|
||||
```sh
|
||||
docker run \
|
||||
-e CASSANDRA_SERVERS=<...> \
|
||||
-e CASSANDRA_TLS=true \
|
||||
-e CASSANDRA_TLS_SERVER_NAME="CN-in-certificate" \
|
||||
-e CASSANDRA_TLS_KEY=<path to client key file> \
|
||||
-e CASSANDRA_TLS_CERT=<path to client cert file> \
|
||||
-e CASSANDRA_TLS_CA=<path to your CA cert file> \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
The schema tool also supports TLS. You need to make a custom cqlshrc file like
|
||||
so:
|
||||
|
||||
```
|
||||
# Creating schema in a cassandra cluster requiring client TLS certificates.
|
||||
#
|
||||
# Create a volume for the schema docker container containing four files:
|
||||
# cqlshrc: this file
|
||||
# ca-cert: the cert authority for your keys
|
||||
# client-key: the keyfile for your client
|
||||
# client-cert: the cert file matching client-key
|
||||
#
|
||||
# if there is any sort of DNS mismatch and you want to ignore server validation
|
||||
# issues, then uncomment validate = false below.
|
||||
#
|
||||
# When running the container, map this volume to /root/.cassandra and set the
|
||||
# environment variable CQLSH_SSL=--ssl
|
||||
[ssl]
|
||||
certfile = ~/.cassandra/ca-cert
|
||||
userkey = ~/.cassandra/client-key
|
||||
usercert = ~/.cassandra/client-cert
|
||||
# validate = false
|
||||
```
|
||||
|
||||
### Elasticsearch
|
||||
Supported in Jaeger since 0.6.0
|
||||
Supported versions: 5.x, 6.x
|
||||
|
||||
Elasticsearch does not require initialization other than
|
||||
[installing and running Elasticsearch](https://www.elastic.co/downloads/elasticsearch).
|
||||
Once it is running, pass the correct configuration values to the Jaeger collector and query service.
|
||||
|
||||
#### Configuration
|
||||
##### Minimal
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=elasticsearch \
|
||||
-e ES_SERVER_URLS=<...> \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
##### All options
|
||||
To view the full list of configuration options, you can run the following command:
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=elasticsearch \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}} \
|
||||
--help
|
||||
```
|
||||
|
||||
See the [README](https://github.com/jaegertracing/jaeger/tree/master/plugin/storage/es/README.md) for an in-depth overview of how Jaeger uses Elasticsearch for storage.
|
||||
|
||||
#### Shards and Replicas for Elasticsearch indices
|
||||
|
||||
Shards and replicas are some configuration values to take special attention to, because this is decided upon
|
||||
index creation. [This article](https://qbox.io/blog/optimizing-elasticsearch-how-many-shards-per-index) goes into
|
||||
more information about choosing how many shards should be chosen for optimization.
|
||||
|
||||
### Kafka
|
||||
Supported in Jaeger since 1.6.0
|
||||
Supported Kafka versions: 0.9+
|
||||
|
||||
Kafka can be used as an intermediary buffer between collector and an actual storage.
|
||||
The collector is configured with `SPAN_STORAGE_TYPE=kafka` that makes it write all received spans
|
||||
into a Kafka topic. A new component [Ingester](#ingester), added in version 1.7.0, is used to read from
|
||||
Kafka and store spans in another storage backend (Elasticsearch or Cassandra).
|
||||
|
||||
Writing to Kafka is particularly useful for building post-processing data pipelines.
|
||||
|
||||
#### Configuration
|
||||
##### Minimal
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=kafka \
|
||||
-e KAFKA_BROKERS=<...> \
|
||||
-e KAFKA_TOPIC=<...> \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
##### All options
|
||||
To view the full list of configuration options, you can run the following command:
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=kafka \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}} \
|
||||
--help
|
||||
```
|
||||
|
||||
#### Topic & partitions
|
||||
Unless your Kafka cluster is configured to automatically create topics, you will need to create it ahead of time. You can refer to [the Kafka quickstart documentation](https://kafka.apache.org/documentation/#quickstart_createtopic) to learn how.
|
||||
|
||||
You can find more information about topics and partitions in general in the [official documentation](https://kafka.apache.org/documentation/#intro_topics). [This article](https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/) provide more details about how to choose the number of partitions.
|
||||
|
||||
### Storage plugin
|
||||
|
||||
Jaeger supports gRPC based storage plugins. For more information refer to [jaeger/plugin/storage/grpc](https://github.com/jaegertracing/jaeger/tree/master/plugin/storage/grpc)
|
||||
|
||||
Available plugins:
|
||||
|
||||
* [InfluxDB](https://github.com/influxdata/jaeger-influxdb/)
|
||||
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=grpc-plugin \
|
||||
-e GRPC_STORAGE_PLUGIN_BINARY=<...> \
|
||||
-e GRPC_STORAGE_PLUGIN_CONFIGURATION_FILE=<...> \
|
||||
jaegertracing/all-in-one:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
## Ingester
|
||||
**jaeger-ingester** is a service which reads span data from Kafka topic and writes it to another storage backend (Elasticsearch or Cassandra).
|
||||
|
||||
Port | Protocol | Function
|
||||
----- | ------- | ---
|
||||
14270 | HTTP | Healthcheck at `/` and metrics at `/metrics`
|
||||
|
||||
To view all exposed configuration options run the following command:
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=cassandra \
|
||||
jaegertracing/jaeger-ingester:{{< currentVersion >}}
|
||||
--help
|
||||
```
|
||||
|
||||
## Query Service & UI
|
||||
|
||||
**jaeger-query** serves the API endpoints and a React/Javascript UI.
|
||||
The service is stateless and is typically run behind a load balancer, such as **nginx**.
|
||||
|
||||
At default settings the query service exposes the following port(s):
|
||||
|
||||
Port | Protocol | Function
|
||||
----- | ------- | ---
|
||||
16686 | HTTP | `/api/*` endpoints and Jaeger UI at `/`
|
||||
16687 | HTTP | Healthcheck at `/` and metrics at `/metrics`
|
||||
|
||||
### Minimal deployment example (Elasticsearch backend):
|
||||
```sh
|
||||
docker run -d --rm \
|
||||
-p 16686:16686 \
|
||||
-p 16687:16687 \
|
||||
-e SPAN_STORAGE_TYPE=elasticsearch \
|
||||
-e ES_SERVER_URLS=http://<ES_SERVER_IP>:<ES_SERVER_PORT> \
|
||||
jaegertracing/jaeger-query:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
### UI Base Path
|
||||
|
||||
The base path for all **jaeger-query** HTTP routes can be set to a non-root value, e.g. `/jaeger` would cause all UI URLs to start with `/jaeger`. This can be useful when running **jaeger-query** behind a reverse proxy.
|
||||
|
||||
The base path can be configured via the `--query.base-path` command line parameter or the `QUERY_BASE_PATH` environment variable.
|
||||
|
||||
### UI Customization and Embedding
|
||||
|
||||
Please refer to the [dedicated Frontend/UI page](../frontend-ui/).
|
||||
|
||||
## Aggregation Jobs for Service Dependencies
|
||||
|
||||
Production deployments need an external process which aggregates data and creates dependency links between services.
|
||||
Project [spark-dependencies](https://github.com/jaegertracing/spark-dependencies) is a Spark job which derives
|
||||
dependency links and stores them directly to the storage.
|
||||
|
||||
## Configuration
|
||||
|
||||
All binaries accepts command line properties and environmental variables which are managed by
|
||||
by [viper](https://github.com/spf13/viper) and [cobra](https://github.com/spf13/cobra).
|
||||
The names of environmental properties are capital letters and characters `-` and `.` are replaced with `_`.
|
||||
To list all configuration properties call `jaeger-binary -h`.
|
||||
|
||||
[cqlsh]: http://cassandra.apache.org/doc/latest/tools/cqlsh.html
|
||||
[zipkin-thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/zipkincore.thrift
|
||||
[jaeger-thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/jaeger.thrift
|
||||
[thriftrw]: https://www.npmjs.com/package/thriftrw
|
|
@ -1,16 +0,0 @@
|
|||
---
|
||||
title: Frequently Asked Questions
|
||||
description: Answers to some frequently asked questions about Jaeger.
|
||||
weight: 11
|
||||
---
|
||||
|
||||
## Why is the Dependencies page empty?
|
||||
|
||||
The Dependencies page shows a graph of services traced by Jaeger and connections between them. When you are using `all-in-one` binary with in-memory storage, the graph is calculated on-demand from all the traces stored in memory. However, if you are using a real distributed storage like Cassandra or Elasticsearch, it is too expensive to scan all the data in the database to build the service graph. Instead, the Jaeger project provides "big data" jobs that can be used to extract the service graph data from traces:
|
||||
|
||||
* https://github.com/jaegertracing/spark-dependencies - the older Spark job that can be run periodically
|
||||
* https://github.com/jaegertracing/jaeger-analytics - the new (experimental) streaming Flink jobs that run continuously and build the service graph in smaller time intervals
|
||||
|
||||
## Why do I not see any spans in Jaeger?
|
||||
|
||||
Please refer to the [Troubleshooting](../troubleshooting/) guide.
|
|
@ -1,57 +0,0 @@
|
|||
---
|
||||
title: Features
|
||||
weight: 3
|
||||
---
|
||||
|
||||
Jaeger is used for monitoring and troubleshooting microservices-based distributed systems, including:
|
||||
|
||||
* Distributed context propagation
|
||||
* Distributed transaction monitoring
|
||||
* Root cause analysis
|
||||
* Service dependency analysis
|
||||
* Performance / latency optimization
|
||||
|
||||
## High Scalability
|
||||
|
||||
Jaeger backend is designed to have no single points of failure and to scale with the business needs.
|
||||
For example, any given Jaeger installation at Uber is typically processing several billion {{< tip "spans" "span" >}} per day.
|
||||
|
||||
## Native support for OpenTracing
|
||||
|
||||
Jaeger backend, Web UI, and instrumentation libraries have been designed from ground up to support the OpenTracing standard.
|
||||
|
||||
* Represent {{< tip "traces" "trace" >}} as {{< tip "directed acyclic graphs" "directed acyclic graph" >}} (not just trees) via [span references](https://github.com/opentracing/specification/blob/master/specification.md#references-between-spans)
|
||||
* Support strongly typed span _tags_ and _structured logs_
|
||||
* Support general distributed context propagation mechanism via _baggage_
|
||||
|
||||
## Multiple storage backends
|
||||
|
||||
Jaeger supports two popular open source NoSQL databases as trace storage backends: Cassandra 3.4+ and Elasticsearch 5.x/6.x.
|
||||
There are ongoing community experiments using other databases, such as ScyllaDB, InfluxDB, Amazon DynamoDB. Jaeger also ships
|
||||
with a simple in-memory storage for testing setups.
|
||||
|
||||
## Modern Web UI
|
||||
|
||||
Jaeger Web UI is implemented in Javascript using popular open source frameworks like React. Several performance
|
||||
improvements have been released in v1.0 to allow the UI to efficiently deal with large volumes of data, and to display
|
||||
{{< tip "traces" "trace" >}} with tens of thousands of {{< tip "spans" "span" >}} (e.g. we tried a trace with 80,000 spans).
|
||||
|
||||
## Cloud Native Deployment
|
||||
|
||||
Jaeger backend is distributed as a collection of Docker images. The binaries support various configuration methods,
|
||||
including command line options, environment variables, and configuration files in multiple formats (yaml, toml, etc.).
|
||||
Deployment to Kubernetes clusters is assisted by a [Kubernetes operator](https://github.com/jaegertracing/jaeger-operator), [Kubernetes templates](https://github.com/jaegertracing/jaeger-kubernetes)
|
||||
and a [Helm chart](https://github.com/kubernetes/charts/tree/master/incubator/jaeger).
|
||||
|
||||
## Observability
|
||||
|
||||
All Jaeger backend components expose [Prometheus](https://prometheus.io/) metrics by default (other metrics backends are
|
||||
also supported). Logs are written to standard out using the structured logging library [zap](https://github.com/uber-go/zap).
|
||||
|
||||
## Backwards compatibility with Zipkin
|
||||
|
||||
Although we recommend instrumenting applications with OpenTracing API and binding to Jaeger client libraries to benefit
|
||||
from advanced features not available elsewhere, if your organization has already invested in the instrumentation
|
||||
using Zipkin libraries, you do not have to rewrite all that code. Jaeger provides backwards compatibility with Zipkin
|
||||
by accepting spans in Zipkin formats (Thrift or JSON v1/v2) over HTTP. Switching from Zipkin backend is just a matter
|
||||
of routing the traffic from Zipkin libraries to the Jaeger backend.
|
|
@ -1,196 +0,0 @@
|
|||
---
|
||||
title: Frontend/UI
|
||||
hasparent: true
|
||||
weight: 10
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
Several aspects of the UI can be configured:
|
||||
|
||||
* The Dependencies section can be enabled / configured
|
||||
* Google Analytics tracking can be enabled / configured
|
||||
* Additional menu options can be added to the global nav
|
||||
|
||||
These options can be configured by a JSON configuration file. The `--query.ui-config` command line parameter of the query service must then be set to the path to the JSON file when the query service is started.
|
||||
|
||||
An example configuration file:
|
||||
|
||||
```json
|
||||
{
|
||||
"dependencies": {
|
||||
"dagMaxNumServices": 200,
|
||||
"menuEnabled": true
|
||||
},
|
||||
"archiveEnabled": true,
|
||||
"tracking": {
|
||||
"gaID": "UA-000000-2",
|
||||
"trackErrors": true
|
||||
},
|
||||
"menu": [
|
||||
{
|
||||
"label": "About Jaeger",
|
||||
"items": [
|
||||
{
|
||||
"label": "GitHub",
|
||||
"url": "https://github.com/jaegertracing/jaeger"
|
||||
},
|
||||
{
|
||||
"label": "Docs",
|
||||
"url": "http://jaeger.readthedocs.io/en/latest/"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
`dependencies.dagMaxNumServices` defines the maximum number of services allowed before the DAG dependency view is disabled. Default: `200`.
|
||||
|
||||
`dependencies.menuEnabled` enables (`true`) or disables (`false`) the dependencies menu button. Default: `true`.
|
||||
|
||||
`archiveEnabled` enables (`true`) or disables (`false`) the archive traces button. Default: `false`. It requires a configuration of an archive storage in Query service. Archived traces are only accessible directly by ID, they are not searchable.
|
||||
|
||||
`tracking.gaID` defines the Google Analytics tracking ID. This is required for Google Analytics tracking, and setting it to a non-`null` value enables Google Analytics tracking. Default: `null`.
|
||||
|
||||
`tracking.trackErrors` enables (`true`) or disables (`false`) error tracking via Google Analytics. Errors can only be tracked if a valid Google Analytics ID is provided. For additional details on error tracking via Google Analytics see the [tracking README](https://github.com/jaegertracing/jaeger-ui/blob/c622330546afc1be59a42f874bcc1c2fadf7e69a/src/utils/tracking/README.md) in the UI repo. Default: `true`.
|
||||
|
||||
`menu` allows additional links to be added to the global nav. The additional links are right-aligned.
|
||||
|
||||
In the sample JSON config above, the configured menu will have a dropdown labeled "About Jaeger" with sub-options for "GitHub" and "Docs". The format for a link in the top right menu is as follows:
|
||||
|
||||
```json
|
||||
{
|
||||
"label": "Some text here",
|
||||
"url": "https://example.com"
|
||||
}
|
||||
```
|
||||
|
||||
Links can either be members of the `menu` Array, directly, or they can be grouped into a dropdown menu option. The format for a group of links is:
|
||||
|
||||
```json
|
||||
{
|
||||
"label": "Dropdown button",
|
||||
"items": [ ]
|
||||
}
|
||||
```
|
||||
|
||||
The `items` Array should contain one or more link configurations.
|
||||
|
||||
## Embedded Mode
|
||||
|
||||
Starting with version 1.9, Jaeger UI provides an "embedded" layout mode which is intended to support integrating Jaeger UI into other applications. Currently (as of `v0`), the approach taken is to remove various UI elements from the page to make the UI better suited for space-constrained layouts.
|
||||
|
||||
The embedded mode is induced and configured via URL query parameters.
|
||||
|
||||
To enter embedded mode, the `uiEmbed=v0` query parameter and value must be added to the URL. For example, the following URL will show the trace with ID `abc123` in embedded mode:
|
||||
|
||||
```
|
||||
http://localhost:16686/trace/abc123?uiEmbed=v0
|
||||
```
|
||||
|
||||
`uiEmbed=v0` is required.
|
||||
|
||||
Further, each page supported has an <img src="/img/frontend-ui/embed-open-icon.png" style="width: 20px; height:20px;display:inline;" alt="Embed open window"> button added that will open the non-embedded page in a new tab.
|
||||
|
||||
The following pages support embedded mode:
|
||||
|
||||
* Search Page
|
||||
* Trace Page
|
||||
|
||||
### Search Page
|
||||
|
||||
To integrate the Search Trace Page to our application we have to indicate to the Jaeger UI that we want to use the embed mode with `uiEmbed=v0`.
|
||||
|
||||
For example:
|
||||
|
||||
```
|
||||
http://localhost:16686/search?
|
||||
service=my-service&
|
||||
start=1543917759557000&
|
||||
end=1543921359557000&
|
||||
limit=20&
|
||||
lookback=1h&
|
||||
maxDuration&
|
||||
minDuration&
|
||||
uiEmbed=v0
|
||||
```
|
||||
|
||||

|
||||
|
||||
#### Configuration options
|
||||
|
||||
The following query parameter can be used to configure the layout of the search page :
|
||||
|
||||
* `uiSearchHideGraph=1` - disables the display of the scatter plot above the search results
|
||||
|
||||
```
|
||||
http://localhost:16686/search?
|
||||
service=my-service&
|
||||
start=1543917759557000&
|
||||
end=1543921359557000&
|
||||
limit=20&
|
||||
lookback=1h&
|
||||
maxDuration&
|
||||
minDuration&
|
||||
uiEmbed=v0&
|
||||
uiSearchHideGraph=1
|
||||
```
|
||||
|
||||

|
||||
|
||||
### Trace Page
|
||||
|
||||
|
||||
To integrate the Trace Page to our application we have to indicate to the Jaeger UI that we want to use the embed mode with `uiEmbed=v0`.
|
||||
|
||||
For example:
|
||||
|
||||
```sh
|
||||
http://localhost:16686/trace/{trace-id}?uiEmbed=v0
|
||||
```
|
||||

|
||||
|
||||
If we have navigated to this view from the search traces page we'll have a button to go back to the results page.
|
||||
|
||||

|
||||
|
||||
#### Configuration options
|
||||
|
||||
The following query parameters can be used to configure the layout of the trace page :
|
||||
|
||||
* `uiTimelineCollapseTitle=1` causes the trace header to start out collapsed, which hides the summary and the minimap.
|
||||
|
||||
```
|
||||
http://localhost:16686/trace/{trace-id}?
|
||||
uiEmbed=v0&
|
||||
uiTimelineCollapseTitle=1
|
||||
```
|
||||

|
||||
|
||||
* `uiTimelineHideMinimap=1` removes the minimap, entirely, regardless of whether the trace header is expanded or not.
|
||||
|
||||
```
|
||||
http://localhost:16686/trace/{trace-id}?
|
||||
uiEmbed=v0&
|
||||
uiTimelineHideMinimap=1
|
||||
```
|
||||

|
||||
|
||||
* `uiTimelineHideSummary=1` - removes the trace summary information (number of services, etc.) entirely, regardless of whether the trace header is expanded or not.
|
||||
|
||||
```
|
||||
http://localhost:16686/trace/{trace-id}?
|
||||
uiEmbed=v0&
|
||||
uiTimelineHideSummary=1
|
||||
```
|
||||

|
||||
|
||||
We can also combine the options:
|
||||
```
|
||||
http://localhost:16686/trace/{trace-id}?
|
||||
uiEmbed=v0&
|
||||
uiTimelineHideMinimap=1&
|
||||
uiTimelineHideSummary=1
|
||||
```
|
||||

|
|
@ -1,129 +0,0 @@
|
|||
---
|
||||
title: Getting started
|
||||
description: Get up and running with Jaeger in your local environment
|
||||
weight: 2
|
||||
---
|
||||
|
||||
## Instrumentation
|
||||
|
||||
Your applications must be instrumented before they can send tracing data to Jaeger backend. Check the [Client Libraries](../client-libraries) section for information about how to use the OpenTracing API and how to initialize and configure Jaeger tracers.
|
||||
|
||||
## All in One
|
||||
|
||||
All-in-one is an executable designed for quick local testing, launches the Jaeger UI, collector, query, and agent, with an in memory storage component.
|
||||
|
||||
The simplest way to start the all-in-one is to use the pre-built image published to DockerHub (a single command line).
|
||||
|
||||
```bash
|
||||
$ docker run -d --name jaeger \
|
||||
-e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
|
||||
-p 5775:5775/udp \
|
||||
-p 6831:6831/udp \
|
||||
-p 6832:6832/udp \
|
||||
-p 5778:5778 \
|
||||
-p 16686:16686 \
|
||||
-p 14268:14268 \
|
||||
-p 9411:9411 \
|
||||
jaegertracing/all-in-one:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
Or run the `jaeger-all-in-one(.exe)` executable from the [binary distribution archives][download]:
|
||||
|
||||
```bash
|
||||
$ jaeger-all-in-one --collector.zipkin.http-port=9411
|
||||
```
|
||||
|
||||
You can then navigate to `http://localhost:16686` to access the Jaeger UI.
|
||||
|
||||
The container exposes the following ports:
|
||||
|
||||
Port | Protocol | Component | Function
|
||||
----- | ------- | --------- | ---
|
||||
5775 | UDP | agent | accept `zipkin.thrift` over compact thrift protocol (deprecated, used by legacy clients only)
|
||||
6831 | UDP | agent | accept `jaeger.thrift` over compact thrift protocol
|
||||
6832 | UDP | agent | accept `jaeger.thrift` over binary thrift protocol
|
||||
5778 | HTTP | agent | serve configs
|
||||
16686 | HTTP | query | serve frontend
|
||||
14268 | HTTP | collector | accept `jaeger.thrift` directly from clients
|
||||
14250 | HTTP | collector | accept `model.proto`
|
||||
9411 | HTTP | collector | Zipkin compatible endpoint (optional)
|
||||
|
||||
|
||||
## Kubernetes and OpenShift
|
||||
|
||||
* Kubernetes templates: https://github.com/jaegertracing/jaeger-kubernetes
|
||||
* Kubernetes Operator: https://github.com/jaegertracing/jaeger-operator
|
||||
* OpenShift templates: https://github.com/jaegertracing/jaeger-openshift
|
||||
|
||||
## Sample App: HotROD
|
||||
|
||||
HotROD (Rides on Demand) is a demo application that consists of several microservices and
|
||||
illustrates the use of the [OpenTracing API](http://opentracing.io).
|
||||
A tutorial / walkthrough is available in the blog post:
|
||||
[Take OpenTracing for a HotROD ride][hotrod-tutorial].
|
||||
|
||||
It can be run standalone, but requires Jaeger backend to view the traces.
|
||||
|
||||
### Features
|
||||
|
||||
- Discover architecture of the whole system via data-driven dependency
|
||||
diagram.
|
||||
- View request timeline and errors; understand how the app works.
|
||||
- Find sources of latency and lack of concurrency.
|
||||
- Highly contextualized logging.
|
||||
- Use baggage propagation to:
|
||||
|
||||
- Diagnose inter-request contention (queueing).
|
||||
- Attribute time spent in a service.
|
||||
|
||||
- Use open source libraries with OpenTracing integration to get
|
||||
vendor-neutral instrumentation for free.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- You need Go 1.11 or higher installed on your machine to run from source.
|
||||
- Requires a [running Jaeger backend](#all-in-one) to view the traces.
|
||||
|
||||
### Running
|
||||
|
||||
#### From Source
|
||||
|
||||
```bash
|
||||
mkdir -p $GOPATH/src/github.com/jaegertracing
|
||||
cd $GOPATH/src/github.com/jaegertracing
|
||||
git clone git@github.com:jaegertracing/jaeger.git jaeger
|
||||
cd jaeger
|
||||
make install
|
||||
go run ./examples/hotrod/main.go all
|
||||
```
|
||||
#### From docker
|
||||
|
||||
```bash
|
||||
$ docker run --rm -it \
|
||||
--link jaeger \
|
||||
-p8080-8083:8080-8083 \
|
||||
-e JAEGER_AGENT_HOST="jaeger" \
|
||||
jaegertracing/example-hotrod:{{< currentVersion >}} \
|
||||
all
|
||||
```
|
||||
|
||||
#### From binary distribution
|
||||
|
||||
Run `example-hotrod(.exe)` executable from the [binary distribution archives][download]:
|
||||
```bash
|
||||
$ example-hotrod all
|
||||
```
|
||||
|
||||
Then navigate to `http://localhost:8080`.
|
||||
|
||||
|
||||
## Migrating from Zipkin
|
||||
|
||||
Collector service exposes Zipkin compatible REST API `/api/v1/spans` which accepts both Thrift and JSON. Also there is `/api/v2/spans` for JSON only.
|
||||
By default it's disabled. It can be enabled with `--collector.zipkin.http-port=9411`.
|
||||
|
||||
Zipkin Thrift IDL file can be found in [jaegertracing/jaeger-idl](https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/zipkincore.thrift).
|
||||
It's compatible with [openzipkin/zipkin-api](https://github.com/openzipkin/zipkin-api/blob/master/thrift/zipkinCore.thrift)
|
||||
|
||||
[hotrod-tutorial]: https://medium.com/@YuriShkuro/take-opentracing-for-a-hotrod-ride-f6e3141f7941
|
||||
[download]: ../../../download/
|
|
@ -1,71 +0,0 @@
|
|||
---
|
||||
title: Introduction
|
||||
weight: 1
|
||||
children:
|
||||
- title: Features
|
||||
url: features
|
||||
---
|
||||
|
||||
Welcome to Jaeger's documentation portal! Below, you'll find information for beginners and experienced Jaeger users.
|
||||
|
||||
If you can't find what you are looking for, or have an issue not covered here, we'd love to [hear from you](/get-in-touch).
|
||||
|
||||
## About
|
||||
|
||||
Jaeger, inspired by [Dapper][dapper] and [OpenZipkin](http://zipkin.io),
|
||||
is a distributed tracing system released as open source by [Uber Technologies][ubeross].
|
||||
It is used for monitoring and troubleshooting microservices-based distributed systems, including:
|
||||
|
||||
* Distributed context propagation
|
||||
* Distributed transaction monitoring
|
||||
* Root cause analysis
|
||||
* Service dependency analysis
|
||||
* Performance / latency optimization
|
||||
|
||||
Uber published a blog post, [Evolving Distributed Tracing at Uber](https://eng.uber.com/distributed-tracing/), where they explain the history and reasons for the architectural choices made in Jaeger. [Yuri Shkuro](https://shkuro.com), creator of Jaeger, also published a book [Mastering Distributed Tracing](https://shkuro.com/books/2019-mastering-distributed-tracing/) that covers in-depth many aspects of Jaeger design and operation, as well as distributed tracing in general.
|
||||
|
||||
## Features
|
||||
|
||||
* [OpenTracing](http://opentracing.io/) compatible data model and instrumentation libraries
|
||||
* in [Go](https://github.com/jaegertracing/jaeger-client-go), [Java](https://github.com/jaegertracing/jaeger-client-java), [Node](https://github.com/jaegertracing/jaeger-client-node), [Python](https://github.com/jaegertracing/jaeger-client-python)
|
||||
and [C++](https://github.com/jaegertracing/cpp-client)
|
||||
* Uses consistent upfront sampling with individual per service/endpoint probabilities
|
||||
* Multiple storage backends: Cassandra, Elasticsearch, memory.
|
||||
* Adaptive sampling (coming soon)
|
||||
* Post-collection data processing pipeline (coming soon)
|
||||
|
||||
See [Features](./features/) page for more details.
|
||||
|
||||
## Technical Specs
|
||||
|
||||
* Backend components implemented in Go
|
||||
* React/Javascript UI
|
||||
* Supported storage backends:
|
||||
* [Cassandra 3.4+](./deployment/#cassandra)
|
||||
* [Elasticsearch 5.x, 6.x](./deployment/#elasticsearch)
|
||||
* [Kafka](./deployment/#kafka)
|
||||
* memory storage
|
||||
|
||||
## Quick Start
|
||||
See [running a docker all in one image](getting-started#all-in-one).
|
||||
|
||||
## Screenshots
|
||||
|
||||
### Traces View
|
||||
[](/img/traces-ss.png)
|
||||
|
||||
### Trace Detail View
|
||||
[](/img/trace-detail-ss.png)
|
||||
|
||||
## Related links
|
||||
|
||||
- [Evolving Distributed tracing At Uber Engineering](https://eng.uber.com/distributed-tracing/)
|
||||
- [Mastering Distributed Tracing](https://shkuro.com/books/2019-mastering-distributed-tracing/)
|
||||
- [Tracing HTTP request latency in Go with OpenTracing](https://medium.com/opentracing/tracing-http-request-latency-in-go-with-opentracing-7cc1282a100a)
|
||||
- [Distributed Tracing with Jaeger & Prometheus on Kubernetes](https://blog.openshift.com/openshift-commons-briefing-82-distributed-tracing-with-jaeger-prometheus-on-kubernetes/)
|
||||
- [Using Jaeger with Istio](https://istio.io/docs/tasks/telemetry/distributed-tracing.html)
|
||||
- [Using Jaeger with Envoy](https://www.envoyproxy.io/docs/envoy/latest/start/sandboxes/jaeger_tracing.html)
|
||||
|
||||
[dapper]: https://research.google.com/pubs/pub36356.html
|
||||
[ubeross]: http://uber.github.io
|
||||
|
|
@ -1,87 +0,0 @@
|
|||
---
|
||||
title: APIs
|
||||
hasparent: true
|
||||
---
|
||||
|
||||
Jaeger components implement various APIs for saving or retrieving trace data.
|
||||
|
||||
The following labels are used to describe API compatibility guarantees.
|
||||
|
||||
* **stable** - the API guarantees backwards compatibility. If breaking changes are going to be made in the future, they will result in a new API version, e.g. `/api/v2` URL prefix or a different namespace in the IDL.
|
||||
* **internal** - the APIs intended for internal communications between Jaeger components and not recommended for use by external components.
|
||||
* **deprecated** - the APIs that are only maintained for legacy reasons and will be phased out in the future.
|
||||
|
||||
## Span reporting APIs
|
||||
|
||||
Agent and Collector are the two components of the Jaeger backend that can receive spans. At this time they support two sets of non-overlapping APIs.
|
||||
|
||||
### Thrift over UDP (stable)
|
||||
|
||||
The Agent can only receive spans over UDP in Thrift format. The primary API is a UDP packet that contains a Thrift-encoded `Batch` struct defined in [jaeger.thrift][jaeger.thrift] IDL file, located in the [jaeger-idl][jaeger-idl] repository. Most Jaeger Clients use Thrift's `compact` encoding, however some client libraries do not support it (notably, Node.js) and use Thrift's `binary` encoding (sent to a different UDP port). The Agent's API is defined by [agent.thrift][agent.thrift] IDL file.
|
||||
|
||||
For legacy reasons, the Agent also accepts spans in Zipkin format, however, only very old versions of Jaeger clients can send data in that format and it is officially deprecated.
|
||||
|
||||
### Protobuf via gRPC (stable)
|
||||
|
||||
In a typical Jaeger deployment, Agents receive spans from Clients and forward them to Collectors. Since Jaeger version 1.11 the official and recommended protocol between Agents and Collectors is gRPC with Protobuf.
|
||||
|
||||
The Protobuf IDL [collector.proto][collector.proto] is currently located in the main Jaeger repository, under [model/proto/api_v2][collector.proto]. In the future it will be moved to [jaeger-idl][jaeger-idl] repository ([jaeger-idl/issues/55](https://github.com/jaegertracing/jaeger-idl/issues/55)).
|
||||
|
||||
### Thrift over HTTP (stable)
|
||||
|
||||
In some cases it is not feasible to deploy Jaeger Agent next to the application, for example, when the application code is running as AWS Lambda function. In these scenarios the Jaeger Clients can be configured to submit spans directly to the Collectors over HTTP/HTTPS.
|
||||
|
||||
The same [jaeger.thrift][jaeger.thrift] payload can be submitted in HTTP POST request to `/api/traces` endpoint, for example, `https://jaeger-collector:14268/api/traces`. The `Batch` struct needs to be encoded using Thrift's `binary` encoding, and the HTTP request should specify the content type header:
|
||||
|
||||
```
|
||||
Content-Type: application/vnd.apache.thrift.binary
|
||||
```
|
||||
|
||||
### JSON over HTTP (n/a)
|
||||
|
||||
There is no official Jaeger JSON format that can be accepted by the collector. In the future the Protobuf-generated JSON may be supported.
|
||||
|
||||
### Thrift via TChannel (deprecated)
|
||||
|
||||
Agent and Collector can communicate using TChannel protocol. This protocol is generally not supported by the routing infrastructure and has been deprecated. It will be eventually removed from Jaeger.
|
||||
|
||||
### Zipkin Formats (stable)
|
||||
|
||||
Jaeger Collector can also accept spans in several Zipkin data format, namely JSON v1/v2 and Thrift. The Collector needs to be configured to enable Zipkin HTTP server, e.g. on port 9411 used by Zipkin collectors. The server enables two endpoints that expect POST requests:
|
||||
|
||||
* `/api/v1/spans` for submitting spans in Zipkin JSON v1 or Zipkin Thrift format.
|
||||
* `/api/v2/spans` for submitting spans in Zipkin JSON v2.
|
||||
|
||||
## Trace retrieval APIs
|
||||
|
||||
Traces saved in the storage can be retrieved by calling Jaeger Query Service.
|
||||
|
||||
### gRPC/Protobuf (stable)
|
||||
|
||||
The recommended way for programmatically retrieving traces and other data is via gRPC endpoint defined in [query.proto][query.proto] IDL file (located in the main Jaeger repository, similar to [collector.proto][collector.proto]).
|
||||
|
||||
### HTTP JSON (internal)
|
||||
|
||||
Jaeger UI communicates with Jaeger Query Service via JSON API. For example, a trace can be retrieved via GET request to `https://jaeger-query:16686/api/traces/{trace-id-hex-string}`. This JSON API is intentionally undocumented and subject to change.
|
||||
|
||||
## Clients configuration (internal)
|
||||
|
||||
Client libraries not only submit finished spans to Jaeger backend, but also periodically poll the Agents for various configurations, such as sampling strategies. The schema for the payload is defined by [sampling.thrift][sampling.thrift], encoded as JSON using Thrift's built-in JSON generation capabilities.
|
||||
|
||||
## Service dependencies graph (internal)
|
||||
|
||||
Can be retrieved from Query Service at `/api/dependencies` endpoint. The GET request expects two parameters:
|
||||
|
||||
* `endTs` (number of milliseconds since epoch) - the end of the time interval
|
||||
* `lookback` (in milliseconds) - the length the time interval (i.e. start-time + lookback = end-time).
|
||||
|
||||
The returned JSON is a list of edges represented as tuples `(caller, callee, count)`.
|
||||
|
||||
For programmatic access to service graph, the recommended API is gRPC/Protobuf described above.
|
||||
|
||||
[jaeger-idl]: https://github.com/jaegertracing/jaeger-idl/
|
||||
[jaeger.thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/jaeger.thrift
|
||||
[agent.thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/agent.thrift
|
||||
[sampling.thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/sampling.thrift
|
||||
[collector.proto]: https://github.com/jaegertracing/jaeger/blob/master/model/proto/api_v2/collector.proto
|
||||
[query.proto]: https://github.com/jaegertracing/jaeger/blob/master/model/proto/api_v2/query.proto
|
|
@ -1,13 +0,0 @@
|
|||
---
|
||||
title: CLI flags
|
||||
widescreen: true
|
||||
hasparent: true
|
||||
---
|
||||
|
||||
This is auto-generated documentation for CLI flags supported by Jaeger binaries.
|
||||
|
||||
* CLI flags for some binaries change depending on the `SPAN_STORAGE_TYPE` environment variable. Relevant variations are included below.
|
||||
* Some binaries support _commands_ (mostly informational), such as `env`, `docs`, and `version`. These commands are not included here.
|
||||
* All parameters can be also provided via environment variables, by changing all letters to upper-case and replacing all punctuation characters with underscore `_`. For example, the value for the flag `--cassandra.connections-per-host` can be provided via `CASSANDRA_CONNECTIONS_PER_HOST` environment variable.
|
||||
|
||||
{{< cli/tools-list >}}
|
|
@ -1,397 +0,0 @@
|
|||
---
|
||||
title: Deployment
|
||||
weight: 4
|
||||
children:
|
||||
- title: Operator for Kubernetes
|
||||
navtitle: Kubernetes
|
||||
url: operator
|
||||
- title: Frontend/UI
|
||||
url: frontend-ui
|
||||
- title: CLI Flags
|
||||
url: cli
|
||||
---
|
||||
|
||||
The main Jaeger backend components are released as Docker images on Docker Hub:
|
||||
|
||||
Component | Repository
|
||||
--------------------- | ---
|
||||
**jaeger-agent** | [hub.docker.com/r/jaegertracing/jaeger-agent/](https://hub.docker.com/r/jaegertracing/jaeger-agent/)
|
||||
**jaeger-collector** | [hub.docker.com/r/jaegertracing/jaeger-collector/](https://hub.docker.com/r/jaegertracing/jaeger-collector/)
|
||||
**jaeger-query** | [hub.docker.com/r/jaegertracing/jaeger-query/](https://hub.docker.com/r/jaegertracing/jaeger-query/)
|
||||
**jaeger-ingester** | [hub.docker.com/r/jaegertracing/jaeger-ingester/](https://hub.docker.com/r/jaegertracing/jaeger-ingester/)
|
||||
|
||||
There are orchestration templates for running Jaeger with:
|
||||
|
||||
* Kubernetes: [github.com/jaegertracing/jaeger-kubernetes](https://github.com/jaegertracing/jaeger-kubernetes),
|
||||
* OpenShift: [github.com/jaegertracing/jaeger-openshift](https://github.com/jaegertracing/jaeger-openshift).
|
||||
|
||||
## Configuration Options
|
||||
|
||||
Jaeger binaries can be configured in a number of ways (in the order of decreasing priority):
|
||||
|
||||
* command line arguments,
|
||||
* environment variables,
|
||||
* configuration files in JSON, TOML, YAML, HCL, or Java properties formats.
|
||||
|
||||
To see the complete list of options, run the binary with `help` command. Options that are specific to a certain storage backend are only listed if the storage type is selected. For example, to see all available options in the Collector with Cassandra storage:
|
||||
|
||||
```sh
|
||||
$ docker run --rm \
|
||||
-e SPAN_STORAGE_TYPE=cassandra \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}} \
|
||||
help
|
||||
```
|
||||
|
||||
In order to provide configuration parameters via environment variables, find the respective command line option and convert its name to UPPER_SNAKE_CASE, for example:
|
||||
|
||||
Command line option | Environment variable
|
||||
-----------------------------------|-------------------------------
|
||||
`--cassandra.connections-per-host` | `CASSANDRA_CONNECTIONS_PER_HOST`
|
||||
`--metrics-backend` | `METRICS_BACKEND`
|
||||
|
||||
## Agent
|
||||
|
||||
Jaeger client libraries expect **jaeger-agent** process to run locally on each host.
|
||||
The agent exposes the following ports:
|
||||
|
||||
Port | Protocol | Function
|
||||
----- | ------- | ---
|
||||
6831 | UDP | accept [jaeger.thrift][jaeger-thrift] in `compact` Thrift protocol used by most current Jaeger clients
|
||||
6832 | UDP | accept [jaeger.thrift][jaeger-thrift] in `binary` Thrift protocol used by Node.js Jaeger client (because [thriftrw][thriftrw] npm package does not support `compact` protocol)
|
||||
5778 | HTTP | serve configs, sampling strategies
|
||||
5775 | UDP | accept [zipkin.thrift][zipkin-thrift] in `compact` Thrift protocol (deprecated; only used by very old Jaeger clients, circa 2016)
|
||||
14271 | HTTP | Healthcheck at `/` and metrics at `/metrics`
|
||||
|
||||
It can be executed directly on the host or via Docker, as follows:
|
||||
|
||||
```sh
|
||||
## make sure to expose only the ports you use in your deployment scenario!
|
||||
docker run \
|
||||
--rm \
|
||||
-p6831:6831/udp \
|
||||
-p6832:6832/udp \
|
||||
-p5778:5778/tcp \
|
||||
-p5775:5775/udp \
|
||||
jaegertracing/jaeger-agent:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
### Discovery System Integration
|
||||
|
||||
The agents can connect point to point to a single collector address, which could be
|
||||
load balanced by another infrastructure component (e.g. DNS) across multiple collectors.
|
||||
The agent can also be configured with a static list of collector addresses.
|
||||
|
||||
On Docker, a command like the following can be used:
|
||||
|
||||
```sh
|
||||
docker run \
|
||||
--rm \
|
||||
-p5775:5775/udp \
|
||||
-p6831:6831/udp \
|
||||
-p6832:6832/udp \
|
||||
-p5778:5778/tcp \
|
||||
jaegertracing/jaeger-agent:{{< currentVersion >}} \
|
||||
--reporter.grpc.host-port=jaeger-collector.jaeger-infra.svc:14250
|
||||
```
|
||||
|
||||
Or use `--reporter.tchannel.host-port=jaeger-collector.jaeger-infra.svc:14267` to use
|
||||
legacy tchannel reporter.
|
||||
|
||||
When using gRPC, you have several options for load balancing and name resolution:
|
||||
|
||||
* Single connection and no load balancing. This is the default if you specify a single `host:port`. (example: `--reporter.grpc.host-port=jaeger-collector.jaeger-infra.svc:14250`)
|
||||
* Static list of hostnames and round-robin load balancing. This is what you get with a comma-separated list of addresses. (example: `reporter.grpc.host-port=jaeger-collector1:14250,jaeger-collector2:14250,jaeger-collector3:14250`)
|
||||
* Dynamic DNS resolution and round-robin load balancing. To get this behaviour, prefix the address with `dns:///` and gRPC will attempt to resolve the hostname using SRV records (for [external load balancing](https://github.com/grpc/grpc/blob/master/doc/load-balancing.md)), TXT records (for [service configs](https://github.com/grpc/grpc/blob/master/doc/service_config.md)), and A records. Refer to the [gRPC Name Resolution docs](https://github.com/grpc/grpc/blob/master/doc/naming.md) and the [dns_resolver.go implementation](https://github.com/grpc/grpc-go/blob/master/resolver/dns/dns_resolver.go) for more info. (example: `--reporter.grpc.host-port=dns:///jaeger-collector.jaeger-infra.svc:14250`)
|
||||
|
||||
### Agent level tags
|
||||
|
||||
Jaeger supports agent level tags, that can be added to the process tags of all spans passing through the agent. This is supported through the command line flag `--jaeger.tags=key1=value1,key2=value2,...,keyn=valuen`. Tags can also be set through an environment flag like so - `--jaeger.tags=key=${envFlag:defaultValue}` - The tag value will be set to the value of the `envFlag` environment key and `defaultValue` if not set.
|
||||
|
||||
|
||||
## Collectors
|
||||
|
||||
The collectors are stateless and thus many instances of **jaeger-collector** can be run in parallel.
|
||||
Collectors require almost no configuration, except for the location of Cassandra cluster,
|
||||
via `--cassandra.keyspace` and `--cassandra.servers` options, or the location of Elasticsearch cluster, via
|
||||
`--es.server-urls`, depending on which storage is specified. To see all command line options run
|
||||
|
||||
```sh
|
||||
go run ./cmd/collector/main.go -h
|
||||
```
|
||||
|
||||
or, if you don't have the source code
|
||||
|
||||
```sh
|
||||
docker run -it --rm jaegertracing/jaeger-collector:{{< currentVersion >}} -h
|
||||
```
|
||||
|
||||
At default settings the collector exposes the following ports:
|
||||
|
||||
Port | Protocol | Function
|
||||
----- | ------- | ---
|
||||
14267 | TChannel | used by **jaeger-agent** to send spans in jaeger.thrift format
|
||||
14250 | gRPC | used by **jaeger-agent** to send spans in model.proto format
|
||||
14268 | HTTP | can accept spans directly from clients in jaeger.thrift format over binary thrift protocol
|
||||
9411 | HTTP | can accept Zipkin spans in JSON or Thrift (disabled by default)
|
||||
14269 | HTTP | Healthcheck at `/` and metrics at `/metrics`
|
||||
|
||||
|
||||
## Storage Backends
|
||||
|
||||
Collectors require a persistent storage backend. Cassandra and Elasticsearch are the primary supported storage backends. Additional backends are [discussed here](https://github.com/jaegertracing/jaeger/issues/638).
|
||||
|
||||
The storage type can be passed via `SPAN_STORAGE_TYPE` environment variable. Valid values are `cassandra`, `elasticsearch`, `kafka` (only as a buffer), `grpc-plugin`, `badger` (only with all-in-one) and `memory` (only with all-in-one).
|
||||
|
||||
As of version 1.6.0, it's possible to use multiple storage types at the same time by providing a comma-separated list of valid types to the `SPAN_STORAGE_TYPE` environment variable.
|
||||
It's important to note that all listed storage types are used for writing, but only the first type in the list will be used for reading and archiving.
|
||||
|
||||
### Memory
|
||||
|
||||
The in-memory storage is not intended for production workloads. It's intended as a simple solution to get started quickly and
|
||||
data will be lost once the process is gone.
|
||||
|
||||
By default, there's no limit in the amount of traces stored in memory but a limit can be established by passing an
|
||||
integer value via `--memory.max-traces`.
|
||||
|
||||
### Badger - local storage
|
||||
Experimental since Jaeger 1.9
|
||||
|
||||
[Badger](https://github.com/dgraph-io/badger) is an embedded local storage, only available
|
||||
with **all-in-one** distribution. By default it acts as an ephemeral storage using a temporary filesystem.
|
||||
This can be overridden by using the `--badger.ephemeral=false` option.
|
||||
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=badger \
|
||||
-e BADGER_EPHEMERAL=false \
|
||||
-e BADGER_DIRECTORY_VALUE=/badger/data \
|
||||
-e BADGER_DIRECTORY_KEY=/badger/key \
|
||||
-v <storage_dir_on_host>:/badger \
|
||||
-p 16686:16686 \
|
||||
jaegertracing/all-in-one:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
### Cassandra
|
||||
Supported versions: 3.4+
|
||||
|
||||
Deploying Cassandra itself is out of scope for our documentation. One good
|
||||
source of documentation is the [Apache Cassandra Docs](https://cassandra.apache.org/doc/latest/).
|
||||
|
||||
#### Configuration
|
||||
##### Minimal
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=cassandra \
|
||||
-e CASSANDRA_SERVERS=<...> \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
##### All options
|
||||
To view the full list of configuration options, you can run the following command:
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=cassandra \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}} \
|
||||
--help
|
||||
```
|
||||
|
||||
#### Schema script
|
||||
|
||||
A script is provided to initialize Cassandra keyspace and schema
|
||||
using Cassandra's interactive shell [`cqlsh`][cqlsh]:
|
||||
|
||||
```sh
|
||||
MODE=test sh ./plugin/storage/cassandra/schema/create.sh | cqlsh
|
||||
```
|
||||
|
||||
For production deployment, pass `MODE=prod DATACENTER={datacenter}` arguments to the script,
|
||||
where `{datacenter}` is the name used in the Cassandra configuration / network topology.
|
||||
|
||||
The script also allows overriding TTL, keyspace name, replication factor, etc.
|
||||
Run the script without arguments to see the full list of recognized parameters.
|
||||
|
||||
#### TLS support
|
||||
|
||||
Jaeger supports TLS client to node connections as long as you've configured
|
||||
your Cassandra cluster correctly. After verifying with e.g. `cqlsh`, you can
|
||||
configure the collector and query like so:
|
||||
|
||||
```sh
|
||||
docker run \
|
||||
-e CASSANDRA_SERVERS=<...> \
|
||||
-e CASSANDRA_TLS=true \
|
||||
-e CASSANDRA_TLS_SERVER_NAME="CN-in-certificate" \
|
||||
-e CASSANDRA_TLS_KEY=<path to client key file> \
|
||||
-e CASSANDRA_TLS_CERT=<path to client cert file> \
|
||||
-e CASSANDRA_TLS_CA=<path to your CA cert file> \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
The schema tool also supports TLS. You need to make a custom cqlshrc file like
|
||||
so:
|
||||
|
||||
```
|
||||
# Creating schema in a cassandra cluster requiring client TLS certificates.
|
||||
#
|
||||
# Create a volume for the schema docker container containing four files:
|
||||
# cqlshrc: this file
|
||||
# ca-cert: the cert authority for your keys
|
||||
# client-key: the keyfile for your client
|
||||
# client-cert: the cert file matching client-key
|
||||
#
|
||||
# if there is any sort of DNS mismatch and you want to ignore server validation
|
||||
# issues, then uncomment validate = false below.
|
||||
#
|
||||
# When running the container, map this volume to /root/.cassandra and set the
|
||||
# environment variable CQLSH_SSL=--ssl
|
||||
[ssl]
|
||||
certfile = ~/.cassandra/ca-cert
|
||||
userkey = ~/.cassandra/client-key
|
||||
usercert = ~/.cassandra/client-cert
|
||||
# validate = false
|
||||
```
|
||||
|
||||
### Elasticsearch
|
||||
Supported in Jaeger since 0.6.0
|
||||
Supported versions: 5.x, 6.x
|
||||
|
||||
Elasticsearch does not require initialization other than
|
||||
[installing and running Elasticsearch](https://www.elastic.co/downloads/elasticsearch).
|
||||
Once it is running, pass the correct configuration values to the Jaeger collector and query service.
|
||||
|
||||
#### Configuration
|
||||
##### Minimal
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=elasticsearch \
|
||||
-e ES_SERVER_URLS=<...> \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
##### All options
|
||||
To view the full list of configuration options, you can run the following command:
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=elasticsearch \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}} \
|
||||
--help
|
||||
```
|
||||
|
||||
See the [README](https://github.com/jaegertracing/jaeger/tree/master/plugin/storage/es/README.md) for an in-depth overview of how Jaeger uses Elasticsearch for storage.
|
||||
|
||||
#### Shards and Replicas for Elasticsearch indices
|
||||
|
||||
Shards and replicas are some configuration values to take special attention to, because this is decided upon
|
||||
index creation. [This article](https://qbox.io/blog/optimizing-elasticsearch-how-many-shards-per-index) goes into
|
||||
more information about choosing how many shards should be chosen for optimization.
|
||||
|
||||
### Kafka
|
||||
Supported in Jaeger since 1.6.0
|
||||
Supported Kafka versions: 0.9+
|
||||
|
||||
Kafka can be used as an intermediary buffer between collector and an actual storage.
|
||||
The collector is configured with `SPAN_STORAGE_TYPE=kafka` that makes it write all received spans
|
||||
into a Kafka topic. A new component [Ingester](#ingester), added in version 1.7.0, is used to read from
|
||||
Kafka and store spans in another storage backend (Elasticsearch or Cassandra).
|
||||
|
||||
Writing to Kafka is particularly useful for building post-processing data pipelines.
|
||||
|
||||
#### Configuration
|
||||
##### Minimal
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=kafka \
|
||||
-e KAFKA_BROKERS=<...> \
|
||||
-e KAFKA_TOPIC=<...> \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
##### All options
|
||||
To view the full list of configuration options, you can run the following command:
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=kafka \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}} \
|
||||
--help
|
||||
```
|
||||
|
||||
#### Topic & partitions
|
||||
Unless your Kafka cluster is configured to automatically create topics, you will need to create it ahead of time. You can refer to [the Kafka quickstart documentation](https://kafka.apache.org/documentation/#quickstart_createtopic) to learn how.
|
||||
|
||||
You can find more information about topics and partitions in general in the [official documentation](https://kafka.apache.org/documentation/#intro_topics). [This article](https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/) provide more details about how to choose the number of partitions.
|
||||
|
||||
### Storage plugin
|
||||
|
||||
Jaeger supports gRPC based storage plugins. For more information refer to [jaeger/plugin/storage/grpc](https://github.com/jaegertracing/jaeger/tree/master/plugin/storage/grpc)
|
||||
|
||||
Available plugins:
|
||||
|
||||
* [InfluxDB](https://github.com/influxdata/jaeger-influxdb/)
|
||||
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=grpc-plugin \
|
||||
-e GRPC_STORAGE_PLUGIN_BINARY=<...> \
|
||||
-e GRPC_STORAGE_PLUGIN_CONFIGURATION_FILE=<...> \
|
||||
jaegertracing/all-in-one:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
## Ingester
|
||||
**jaeger-ingester** is a service which reads span data from Kafka topic and writes it to another storage backend (Elasticsearch or Cassandra).
|
||||
|
||||
Port | Protocol | Function
|
||||
----- | ------- | ---
|
||||
14270 | HTTP | Healthcheck at `/` and metrics at `/metrics`
|
||||
|
||||
To view all exposed configuration options run the following command:
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=cassandra \
|
||||
jaegertracing/jaeger-ingester:{{< currentVersion >}}
|
||||
--help
|
||||
```
|
||||
|
||||
## Query Service & UI
|
||||
|
||||
**jaeger-query** serves the API endpoints and a React/Javascript UI.
|
||||
The service is stateless and is typically run behind a load balancer, such as [**NGINX**](https://www.nginx.com/).
|
||||
|
||||
At default settings the query service exposes the following port(s):
|
||||
|
||||
Port | Protocol | Function
|
||||
----- | ------- | ---
|
||||
16686 | HTTP | `/api/*` endpoints and Jaeger UI at `/`
|
||||
16687 | HTTP | Healthcheck at `/` and metrics at `/metrics`
|
||||
|
||||
### Minimal deployment example (Elasticsearch backend):
|
||||
```sh
|
||||
docker run -d --rm \
|
||||
-p 16686:16686 \
|
||||
-p 16687:16687 \
|
||||
-e SPAN_STORAGE_TYPE=elasticsearch \
|
||||
-e ES_SERVER_URLS=http://<ES_SERVER_IP>:<ES_SERVER_PORT> \
|
||||
jaegertracing/jaeger-query:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
### UI Base Path
|
||||
|
||||
The base path for all **jaeger-query** HTTP routes can be set to a non-root value, e.g. `/jaeger` would cause all UI URLs to start with `/jaeger`. This can be useful when running **jaeger-query** behind a reverse proxy.
|
||||
|
||||
The base path can be configured via the `--query.base-path` command line parameter or the `QUERY_BASE_PATH` environment variable.
|
||||
|
||||
### UI Customization and Embedding
|
||||
|
||||
Please refer to the [dedicated Frontend/UI page](../frontend-ui/).
|
||||
|
||||
## Aggregation Jobs for Service Dependencies
|
||||
|
||||
Production deployments need an external process which aggregates data and creates dependency links between services. Project [spark-dependencies](https://github.com/jaegertracing/spark-dependencies) is a Spark job which derives dependency links and stores them directly to the storage.
|
||||
|
||||
## Configuration
|
||||
|
||||
All binaries accepts command line properties and environmental variables, power by [viper](https://github.com/spf13/viper) and [cobra](https://github.com/spf13/cobra) libraries. Please refer to the [CLI Flags](../cli/) page for more information.
|
||||
|
||||
[cqlsh]: http://cassandra.apache.org/doc/latest/tools/cqlsh.html
|
||||
[zipkin-thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/zipkincore.thrift
|
||||
[jaeger-thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/jaeger.thrift
|
||||
[thriftrw]: https://www.npmjs.com/package/thriftrw
|
|
@ -1,17 +0,0 @@
|
|||
---
|
||||
title: Frequently Asked Questions
|
||||
navtitle: FAQs
|
||||
description: Answers to some frequently asked questions about Jaeger.
|
||||
weight: 11
|
||||
---
|
||||
|
||||
## Why is the Dependencies page empty?
|
||||
|
||||
The Dependencies page shows a graph of services traced by Jaeger and connections between them. When you are using `all-in-one` binary with in-memory storage, the graph is calculated on-demand from all the traces stored in memory. However, if you are using a real distributed storage like Cassandra or Elasticsearch, it is too expensive to scan all the data in the database to build the service graph. Instead, the Jaeger project provides "big data" jobs that can be used to extract the service graph data from traces:
|
||||
|
||||
* https://github.com/jaegertracing/spark-dependencies - the older Spark job that can be run periodically
|
||||
* https://github.com/jaegertracing/jaeger-analytics - the new (experimental) streaming Flink jobs that run continuously and builds the service graph in smaller time intervals
|
||||
|
||||
## Why do I not see any spans in Jaeger?
|
||||
|
||||
Please refer to the [Troubleshooting](../troubleshooting/) guide.
|
|
@ -1,57 +0,0 @@
|
|||
---
|
||||
title: Features
|
||||
hasparent: true
|
||||
---
|
||||
|
||||
Jaeger is used for monitoring and troubleshooting microservices-based distributed systems, including:
|
||||
|
||||
* Distributed context propagation
|
||||
* Distributed transaction monitoring
|
||||
* Root cause analysis
|
||||
* Service dependency analysis
|
||||
* Performance / latency optimization
|
||||
|
||||
## High Scalability
|
||||
|
||||
Jaeger backend is designed to have no single points of failure and to scale with the business needs.
|
||||
For example, any given Jaeger installation at Uber is typically processing several billion {{< tip "spans" "span" >}} per day.
|
||||
|
||||
## Native support for OpenTracing
|
||||
|
||||
Jaeger backend, Web UI, and instrumentation libraries have been designed from ground up to support the OpenTracing standard.
|
||||
|
||||
* Represent {{< tip "traces" "trace" >}} as {{< tip "directed acyclic graphs" "directed acyclic graph" >}} (not just trees) via [span references](https://github.com/opentracing/specification/blob/master/specification.md#references-between-spans)
|
||||
* Support strongly typed span _tags_ and _structured logs_
|
||||
* Support general distributed context propagation mechanism via _baggage_
|
||||
|
||||
## Multiple storage backends
|
||||
|
||||
Jaeger supports two popular open source NoSQL databases as trace storage backends: Cassandra 3.4+ and Elasticsearch 5.x/6.x.
|
||||
There are ongoing community experiments using other databases, such as ScyllaDB, InfluxDB, Amazon DynamoDB. Jaeger also ships
|
||||
with a simple in-memory storage for testing setups.
|
||||
|
||||
## Modern Web UI
|
||||
|
||||
Jaeger Web UI is implemented in Javascript using popular open source frameworks like React. Several performance
|
||||
improvements have been released in v1.0 to allow the UI to efficiently deal with large volumes of data, and to display
|
||||
{{< tip "traces" "trace" >}} with tens of thousands of {{< tip "spans" "span" >}} (e.g. we tried a trace with 80,000 spans).
|
||||
|
||||
## Cloud Native Deployment
|
||||
|
||||
Jaeger backend is distributed as a collection of Docker images. The binaries support various configuration methods,
|
||||
including command line options, environment variables, and configuration files in multiple formats (yaml, toml, etc.).
|
||||
Deployment to Kubernetes clusters is assisted by a [Kubernetes operator](https://github.com/jaegertracing/jaeger-operator), [Kubernetes templates](https://github.com/jaegertracing/jaeger-kubernetes)
|
||||
and a [Helm chart](https://github.com/kubernetes/charts/tree/master/incubator/jaeger).
|
||||
|
||||
## Observability
|
||||
|
||||
All Jaeger backend components expose [Prometheus](https://prometheus.io/) metrics by default (other metrics backends are
|
||||
also supported). Logs are written to standard out using the structured logging library [zap](https://github.com/uber-go/zap).
|
||||
|
||||
## Backwards compatibility with Zipkin
|
||||
|
||||
Although we recommend instrumenting applications with OpenTracing API and binding to Jaeger client libraries to benefit
|
||||
from advanced features not available elsewhere, if your organization has already invested in the instrumentation
|
||||
using Zipkin libraries, you do not have to rewrite all that code. Jaeger provides backwards compatibility with Zipkin
|
||||
by accepting spans in Zipkin formats (Thrift or JSON v1/v2) over HTTP. Switching from Zipkin backend is just a matter
|
||||
of routing the traffic from Zipkin libraries to the Jaeger backend.
|
|
@ -1,224 +0,0 @@
|
|||
---
|
||||
title: Frontend/UI Configuration
|
||||
navtitle: Frontend/UI
|
||||
hasparent: true
|
||||
weight: 7
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
Several aspects of the UI can be configured:
|
||||
|
||||
* The Dependencies section can be enabled / configured
|
||||
* Google Analytics tracking can be enabled / configured
|
||||
* Additional menu options can be added to the global nav
|
||||
|
||||
These options can be configured by a JSON configuration file. The `--query.ui-config` command line parameter of the query service must then be set to the path to the JSON file when the query service is started.
|
||||
|
||||
An example configuration file:
|
||||
|
||||
```json
|
||||
{
|
||||
"dependencies": {
|
||||
"dagMaxNumServices": 200,
|
||||
"menuEnabled": true
|
||||
},
|
||||
"archiveEnabled": true,
|
||||
"tracking": {
|
||||
"gaID": "UA-000000-2",
|
||||
"trackErrors": true
|
||||
},
|
||||
"menu": [
|
||||
{
|
||||
"label": "About Jaeger",
|
||||
"items": [
|
||||
{
|
||||
"label": "GitHub",
|
||||
"url": "https://github.com/jaegertracing/jaeger"
|
||||
},
|
||||
{
|
||||
"label": "Docs",
|
||||
"url": "http://jaeger.readthedocs.io/en/latest/"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"linkPatterns": [{
|
||||
"type": "process",
|
||||
"key": "jaeger.version",
|
||||
"url": "https://github.com/jaegertracing/jaeger-client-java/releases/tag/#{jaeger.version}",
|
||||
"text": "Information about Jaeger release #{jaeger.version}"
|
||||
}]
|
||||
}
|
||||
```
|
||||
|
||||
### Dependencies
|
||||
|
||||
`dependencies.dagMaxNumServices` defines the maximum number of services allowed before the DAG dependency view is disabled. Default: `200`.
|
||||
|
||||
`dependencies.menuEnabled` enables (`true`) or disables (`false`) the dependencies menu button. Default: `true`.
|
||||
|
||||
### Archive Support
|
||||
|
||||
`archiveEnabled` enables (`true`) or disables (`false`) the archive traces button. Default: `false`. It requires a configuration of an archive storage in Query service. Archived traces are only accessible directly by ID, they are not searchable.
|
||||
|
||||
### Google Analytics Tracking
|
||||
|
||||
`tracking.gaID` defines the Google Analytics tracking ID. This is required for Google Analytics tracking, and setting it to a non-`null` value enables Google Analytics tracking. Default: `null`.
|
||||
|
||||
`tracking.trackErrors` enables (`true`) or disables (`false`) error tracking via Google Analytics. Errors can only be tracked if a valid Google Analytics ID is provided. For additional details on error tracking via Google Analytics see the [tracking README](https://github.com/jaegertracing/jaeger-ui/blob/c622330546afc1be59a42f874bcc1c2fadf7e69a/src/utils/tracking/README.md) in the UI repo. Default: `true`.
|
||||
|
||||
### Custom Menu Items
|
||||
|
||||
`menu` allows additional links to be added to the global nav. The additional links are right-aligned.
|
||||
|
||||
In the sample JSON config above, the configured menu will have a dropdown labeled "About Jaeger" with sub-options for "GitHub" and "Docs". The format for a link in the top right menu is as follows:
|
||||
|
||||
```json
|
||||
{
|
||||
"label": "Some text here",
|
||||
"url": "https://example.com"
|
||||
}
|
||||
```
|
||||
|
||||
Links can either be members of the `menu` Array, directly, or they can be grouped into a dropdown menu option. The format for a group of links is:
|
||||
|
||||
```json
|
||||
{
|
||||
"label": "Dropdown button",
|
||||
"items": [ ]
|
||||
}
|
||||
```
|
||||
|
||||
The `items` Array should contain one or more link configurations.
|
||||
|
||||
### Link Patterns
|
||||
|
||||
The `linkPatterns` node can be used to create links from fields displayed in the Jaeger UI.
|
||||
|
||||
Field | Description
|
||||
------|------------
|
||||
type | The metadata section in which your link will be added: process, tags, logs
|
||||
key | The name of tag/process/log attribute which value will be displayed as a link
|
||||
url | The URL where the link should point to, it can be an external site or relative path in Jaeger UI
|
||||
text | The text displayed in the tooltip for the link
|
||||
|
||||
Both `url` and `text` can be defined as templates (i.e. using `#{field-name}`) where Jaeger UI will dynamically substitute values based on tags/logs data.
|
||||
|
||||
## Embedded Mode
|
||||
|
||||
Starting with version 1.9, Jaeger UI provides an "embedded" layout mode which is intended to support integrating Jaeger UI into other applications. Currently (as of `v0`), the approach taken is to remove various UI elements from the page to make the UI better suited for space-constrained layouts.
|
||||
|
||||
The embedded mode is induced and configured via URL query parameters.
|
||||
|
||||
To enter embedded mode, the `uiEmbed=v0` query parameter and value must be added to the URL. For example, the following URL will show the trace with ID `abc123` in embedded mode:
|
||||
|
||||
```
|
||||
http://localhost:16686/trace/abc123?uiEmbed=v0
|
||||
```
|
||||
|
||||
`uiEmbed=v0` is required.
|
||||
|
||||
Further, each page supported has an <img src="/img/frontend-ui/embed-open-icon.png" style="width: 20px; height:20px;display:inline;" alt="Embed open window"> button added that will open the non-embedded page in a new tab.
|
||||
|
||||
The following pages support embedded mode:
|
||||
|
||||
* Search Page
|
||||
* Trace Page
|
||||
|
||||
### Search Page
|
||||
|
||||
To integrate the Search Trace Page to our application we have to indicate to the Jaeger UI that we want to use the embed mode with `uiEmbed=v0`.
|
||||
|
||||
For example:
|
||||
|
||||
```
|
||||
http://localhost:16686/search?
|
||||
service=my-service&
|
||||
start=1543917759557000&
|
||||
end=1543921359557000&
|
||||
limit=20&
|
||||
lookback=1h&
|
||||
maxDuration&
|
||||
minDuration&
|
||||
uiEmbed=v0
|
||||
```
|
||||
|
||||

|
||||
|
||||
#### Configuration options
|
||||
|
||||
The following query parameter can be used to configure the layout of the search page :
|
||||
|
||||
* `uiSearchHideGraph=1` - disables the display of the scatter plot above the search results
|
||||
|
||||
```
|
||||
http://localhost:16686/search?
|
||||
service=my-service&
|
||||
start=1543917759557000&
|
||||
end=1543921359557000&
|
||||
limit=20&
|
||||
lookback=1h&
|
||||
maxDuration&
|
||||
minDuration&
|
||||
uiEmbed=v0&
|
||||
uiSearchHideGraph=1
|
||||
```
|
||||
|
||||

|
||||
|
||||
### Trace Page
|
||||
|
||||
|
||||
To integrate the Trace Page to our application we have to indicate to the Jaeger UI that we want to use the embed mode with `uiEmbed=v0`.
|
||||
|
||||
For example:
|
||||
|
||||
```sh
|
||||
http://localhost:16686/trace/{trace-id}?uiEmbed=v0
|
||||
```
|
||||

|
||||
|
||||
If we have navigated to this view from the search traces page we'll have a button to go back to the results page.
|
||||
|
||||

|
||||
|
||||
#### Configuration options
|
||||
|
||||
The following query parameters can be used to configure the layout of the trace page :
|
||||
|
||||
* `uiTimelineCollapseTitle=1` causes the trace header to start out collapsed, which hides the summary and the minimap.
|
||||
|
||||
```
|
||||
http://localhost:16686/trace/{trace-id}?
|
||||
uiEmbed=v0&
|
||||
uiTimelineCollapseTitle=1
|
||||
```
|
||||

|
||||
|
||||
* `uiTimelineHideMinimap=1` removes the minimap, entirely, regardless of whether the trace header is expanded or not.
|
||||
|
||||
```
|
||||
http://localhost:16686/trace/{trace-id}?
|
||||
uiEmbed=v0&
|
||||
uiTimelineHideMinimap=1
|
||||
```
|
||||

|
||||
|
||||
* `uiTimelineHideSummary=1` - removes the trace summary information (number of services, etc.) entirely, regardless of whether the trace header is expanded or not.
|
||||
|
||||
```
|
||||
http://localhost:16686/trace/{trace-id}?
|
||||
uiEmbed=v0&
|
||||
uiTimelineHideSummary=1
|
||||
```
|
||||

|
||||
|
||||
We can also combine the options:
|
||||
```
|
||||
http://localhost:16686/trace/{trace-id}?
|
||||
uiEmbed=v0&
|
||||
uiTimelineHideMinimap=1&
|
||||
uiTimelineHideSummary=1
|
||||
```
|
||||

|
|
@ -1,129 +0,0 @@
|
|||
---
|
||||
title: Getting Started
|
||||
description: Get up and running with Jaeger in your local environment
|
||||
weight: 2
|
||||
---
|
||||
|
||||
## Instrumentation
|
||||
|
||||
Your applications must be instrumented before they can send tracing data to Jaeger backend. Check the [Client Libraries](../client-libraries) section for information about how to use the OpenTracing API and how to initialize and configure Jaeger tracers.
|
||||
|
||||
## All in One
|
||||
|
||||
All-in-one is an executable designed for quick local testing, launches the Jaeger UI, collector, query, and agent, with an in memory storage component.
|
||||
|
||||
The simplest way to start the all-in-one is to use the pre-built image published to DockerHub (a single command line).
|
||||
|
||||
```bash
|
||||
$ docker run -d --name jaeger \
|
||||
-e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
|
||||
-p 5775:5775/udp \
|
||||
-p 6831:6831/udp \
|
||||
-p 6832:6832/udp \
|
||||
-p 5778:5778 \
|
||||
-p 16686:16686 \
|
||||
-p 14268:14268 \
|
||||
-p 9411:9411 \
|
||||
jaegertracing/all-in-one:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
Or run the `jaeger-all-in-one(.exe)` executable from the [binary distribution archives][download]:
|
||||
|
||||
```bash
|
||||
$ jaeger-all-in-one --collector.zipkin.http-port=9411
|
||||
```
|
||||
|
||||
You can then navigate to `http://localhost:16686` to access the Jaeger UI.
|
||||
|
||||
The container exposes the following ports:
|
||||
|
||||
Port | Protocol | Component | Function
|
||||
----- | ------- | --------- | ---
|
||||
5775 | UDP | agent | accept `zipkin.thrift` over compact thrift protocol (deprecated, used by legacy clients only)
|
||||
6831 | UDP | agent | accept `jaeger.thrift` over compact thrift protocol
|
||||
6832 | UDP | agent | accept `jaeger.thrift` over binary thrift protocol
|
||||
5778 | HTTP | agent | serve configs
|
||||
16686 | HTTP | query | serve frontend
|
||||
14268 | HTTP | collector | accept `jaeger.thrift` directly from clients
|
||||
14250 | HTTP | collector | accept `model.proto`
|
||||
9411 | HTTP | collector | Zipkin compatible endpoint (optional)
|
||||
|
||||
|
||||
## Kubernetes and OpenShift
|
||||
|
||||
* Kubernetes templates: https://github.com/jaegertracing/jaeger-kubernetes
|
||||
* Kubernetes Operator: https://github.com/jaegertracing/jaeger-operator
|
||||
* OpenShift templates: https://github.com/jaegertracing/jaeger-openshift
|
||||
|
||||
## Sample App: HotROD
|
||||
|
||||
HotROD (Rides on Demand) is a demo application that consists of several microservices and
|
||||
illustrates the use of the [OpenTracing API](http://opentracing.io).
|
||||
A tutorial / walkthrough is available in the blog post:
|
||||
[Take OpenTracing for a HotROD ride][hotrod-tutorial].
|
||||
|
||||
It can be run standalone, but requires Jaeger backend to view the traces.
|
||||
|
||||
### Features
|
||||
|
||||
- Discover architecture of the whole system via data-driven dependency
|
||||
diagram.
|
||||
- View request timeline and errors; understand how the app works.
|
||||
- Find sources of latency and lack of concurrency.
|
||||
- Highly contextualized logging.
|
||||
- Use baggage propagation to:
|
||||
|
||||
- Diagnose inter-request contention (queueing).
|
||||
- Attribute time spent in a service.
|
||||
|
||||
- Use open source libraries with OpenTracing integration to get
|
||||
vendor-neutral instrumentation for free.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- You need Go 1.11 or higher installed on your machine to run from source.
|
||||
- Requires a [running Jaeger backend](#all-in-one) to view the traces.
|
||||
|
||||
### Running
|
||||
|
||||
#### From Source
|
||||
|
||||
```bash
|
||||
mkdir -p $GOPATH/src/github.com/jaegertracing
|
||||
cd $GOPATH/src/github.com/jaegertracing
|
||||
git clone git@github.com:jaegertracing/jaeger.git jaeger
|
||||
cd jaeger
|
||||
make install
|
||||
go run ./examples/hotrod/main.go all
|
||||
```
|
||||
#### From docker
|
||||
|
||||
```bash
|
||||
$ docker run --rm -it \
|
||||
--link jaeger \
|
||||
-p8080-8083:8080-8083 \
|
||||
-e JAEGER_AGENT_HOST="jaeger" \
|
||||
jaegertracing/example-hotrod:{{< currentVersion >}} \
|
||||
all
|
||||
```
|
||||
|
||||
#### From binary distribution
|
||||
|
||||
Run `example-hotrod(.exe)` executable from the [binary distribution archives][download]:
|
||||
```bash
|
||||
$ example-hotrod all
|
||||
```
|
||||
|
||||
Then navigate to `http://localhost:8080`.
|
||||
|
||||
|
||||
## Migrating from Zipkin
|
||||
|
||||
Collector service exposes Zipkin compatible REST API `/api/v1/spans` which accepts both Thrift and JSON. Also there is `/api/v2/spans` for JSON only.
|
||||
By default it's disabled. It can be enabled with `--collector.zipkin.http-port=9411`.
|
||||
|
||||
Zipkin Thrift IDL file can be found in [jaegertracing/jaeger-idl](https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/zipkincore.thrift).
|
||||
It's compatible with [openzipkin/zipkin-api](https://github.com/openzipkin/zipkin-api/blob/master/thrift/zipkinCore.thrift)
|
||||
|
||||
[hotrod-tutorial]: https://medium.com/@YuriShkuro/take-opentracing-for-a-hotrod-ride-f6e3141f7941
|
||||
[download]: ../../../download/
|
|
@ -1,755 +0,0 @@
|
|||
---
|
||||
title: Operator for Kubernetes
|
||||
hasparent: true
|
||||
---
|
||||
|
||||
# Understanding Operators
|
||||
|
||||
The Jaeger Operator is an implementation of a [Kubernetes Operator](https://coreos.com/operators/). Operators are pieces of software that ease the operational complexity of running another piece of software. More technically, _Operators_ are a method of packaging, deploying, and managing a Kubernetes application.
|
||||
|
||||
A Kubernetes application is an application that is both deployed on Kubernetes and managed using the Kubernetes APIs and `kubectl` (kubernetes) or `oc` (OKD) tooling. To be able to make the most of Kubernetes, you need a set of cohesive APIs to extend in order to service and manage your apps that run on Kubernetes. Think of Operators as the runtime that manages this type of app on Kubernetes.
|
||||
|
||||
# Installing the Operator
|
||||
|
||||
{{< info >}}
|
||||
The Jaeger Operator version tracks one version of the Jaeger components (Query, Collector, Agent). When a new version of the Jaeger components is released, a new version of the operator will be released that understands how running instances of the previous version can be upgraded to the new version.
|
||||
{{< /info >}}
|
||||
|
||||
## Installing the Operator on Kubernetes
|
||||
|
||||
The following instructions will create the `observability` namespace and install the Jaeger Operator.
|
||||
|
||||
{{< info >}}
|
||||
Make sure your `kubectl` command is properly configured to talk to a valid Kubernetes cluster. If you don't have a cluster, you can create one locally using [`minikube`](https://kubernetes.io/docs/tasks/tools/install-minikube/).
|
||||
{{< /info >}}
|
||||
|
||||
To install the operator, run:
|
||||
<!--TODO - Does Kubernetes have privileged users? Needs to be run as a system:admin on OKD/OpenShift.-->
|
||||
|
||||
```bash
|
||||
kubectl create namespace observability # <1>
|
||||
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/crds/jaegertracing.io_jaegers_crd.yaml # <2>
|
||||
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/service_account.yaml
|
||||
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role.yaml
|
||||
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role_binding.yaml
|
||||
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/operator.yaml
|
||||
```
|
||||
<1> This creates the namespace used by default in the deployment files. If you want to install the Jaeger operator in a different namespace, you must edit the deployment files to change `observability` to the desired namespace value.
|
||||
|
||||
<2> This installs the "Custom Resource Definition" for the `apiVersion: jaegertracing.io/v1`
|
||||
|
||||
At this point, there should be a `jaeger-operator` deployment available. You can view it by running the following command:
|
||||
|
||||
```bash
|
||||
$ kubectl get deployment jaeger-operator -n observability
|
||||
|
||||
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
|
||||
jaeger-operator 1 1 1 1 48s
|
||||
```
|
||||
|
||||
The operator is now ready to create Jaeger instances.
|
||||
|
||||
## Installing the Operator on OKD/OpenShift
|
||||
|
||||
<!-- TODO: Add instructions for installing via the operatorhub? -->
|
||||
|
||||
The instructions from the previous section also work for installing the operator on OKD or OpenShift. Make sure you are logged in as a privileged user, when you install the role based acces control (RBAC) rules, the custom resource definition, and the operator.
|
||||
|
||||
```bash
|
||||
oc login -u <privileged user>
|
||||
|
||||
oc new-project observability # <1>
|
||||
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/crds/jaegertracing.io_jaegers_crd.yaml # <2>
|
||||
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/service_account.yaml
|
||||
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role.yaml
|
||||
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role_binding.yaml
|
||||
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/operator.yaml
|
||||
```
|
||||
<1> This creates the namespace used by default in the deployment files. If you want to install the Jaeger operator in a different namespace, you must edit the deployment files to change `observability` to the desired namespace value.
|
||||
|
||||
<2> This installs the "Custom Resource Definition" for the `apiVersion: jaegertracing.io/v1`
|
||||
|
||||
Once the operator is installed, grant the role `jaeger-operator` to users who should be able to install individual Jaeger instances. The following example creates a role binding allowing the user `developer` to create Jaeger instances:
|
||||
|
||||
```bash
|
||||
oc create \
|
||||
rolebinding developer-jaeger-operator \
|
||||
--role=jaeger-operator \
|
||||
--user=developer
|
||||
```
|
||||
|
||||
After the role is granted, switch back to a non-privileged user.
|
||||
|
||||
Jaeger Agent can be configured to be deployed as a `DaemonSet` using a `HostPort` to allow Jaeger clients in the same node to discover the agent. In OpenShift, a `HostPort` can only be set when a special security context is set. A separate service account can be used by the Jaeger Agent with the permission to bind to `HostPort`, as follows:
|
||||
|
||||
```bash
|
||||
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/examples/openshift/hostport-scc-daemonset.yaml # <1>
|
||||
oc new-project myappnamespace
|
||||
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/examples/openshift/service_account_jaeger-agent-daemonset.yaml # <2>
|
||||
oc adm policy add-scc-to-user daemonset-with-hostport -z jaeger-agent-daemonset # <3>
|
||||
oc apply -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/examples/openshift/agent-as-daemonset.yaml # <4>
|
||||
```
|
||||
<1> The `SecurityContextConstraints` with the `allowHostPorts` policy
|
||||
|
||||
<2> The `ServiceAccount` to be used by the Jaeger Agent
|
||||
|
||||
<3> Adds the security policy to the service account
|
||||
|
||||
<4> Creates the Jaeger Instance using the `serviceAccount` created in the steps above
|
||||
|
||||
{{< warning >}}
|
||||
Without such a policy, errors like the following will prevent a `DaemonSet` to be created: `Warning FailedCreate 4s (x14 over 45s) daemonset-controller Error creating: pods "agent-as-daemonset-agent-daemonset-" is forbidden: unable to validate against any security context constraint: [spec.containers[0].securityContext.containers[0].hostPort: Invalid value: 5775: Host ports are not allowed to be used`
|
||||
{{< /warning >}}
|
||||
|
||||
After a few seconds, the `DaemonSet` should be up and running:
|
||||
|
||||
```bash
|
||||
$ oc get daemonset agent-as-daemonset-agent-daemonset
|
||||
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE
|
||||
agent-as-daemonset-agent-daemonset 1 1 1 1 1
|
||||
```
|
||||
|
||||
# Quick Start - Deploying the AllInOne image
|
||||
|
||||
The simplest possible way to create a Jaeger instance is by creating a YAML file like the following example. This will install the default AllInOne strategy, which deploys the "all-in-one" image (agent, collector, query, ingester, Jaeger UI) in a single pod, using in-memory storage by default.
|
||||
|
||||
{{< info >}}
|
||||
This default strategy is intended for development, testing, and demo purposes, not for production.
|
||||
{{< /info >}}
|
||||
|
||||
```yaml
|
||||
apiVersion: jaegertracing.io/v1
|
||||
kind: Jaeger
|
||||
metadata:
|
||||
name: simplest
|
||||
```
|
||||
|
||||
The YAML file can then be used with `kubectl`:
|
||||
<!-- TODO - Add OKD commands and tabs shortcode. -->
|
||||
|
||||
```bash
|
||||
kubectl apply -f simplest.yaml
|
||||
```
|
||||
|
||||
In a few seconds, a new in-memory all-in-one instance of Jaeger will be available, suitable for quick demos and development purposes. To check the instances that were created, list the `jaeger` objects:
|
||||
|
||||
```bash
|
||||
$ kubectl get jaegers
|
||||
NAME CREATED AT
|
||||
simplest 28s
|
||||
```
|
||||
|
||||
To get the pod name, query for the pods belonging to the `simplest` Jaeger instance:
|
||||
|
||||
|
||||
```bash
|
||||
$ kubectl get pods -l app.kubernetes.io/instance=simplest
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
simplest-6499bb6cdd-kqx75 1/1 Running 0 2m
|
||||
```
|
||||
|
||||
Similarly, the logs can be queried either from the pod directly using the pod name obtained from the previous example, or from all pods belonging to our instance:
|
||||
|
||||
```bash
|
||||
$ kubectl logs -l app.kubernetes.io/instance=simplest
|
||||
...
|
||||
{"level":"info","ts":1535385688.0951214,"caller":"healthcheck/handler.go:133","msg":"Health Check state change","status":"ready"}
|
||||
```
|
||||
|
||||
{{< info >}}
|
||||
On OKD/OpenShift the container name must be specified.
|
||||
{{< /info >}}
|
||||
|
||||
```bash
|
||||
$ kubectl logs -l app.kubernetes.io/instance=simplest -c jaeger
|
||||
...
|
||||
{"level":"info","ts":1535385688.0951214,"caller":"healthcheck/handler.go:133","msg":"Health Check state change","status":"ready"}
|
||||
```
|
||||
|
||||
# Deployment Strategies
|
||||
|
||||
When you create a Jaeger instance, it is associated with a strategy. The strategy is defined in the custom resource file, and determines the architecture to be used for the Jaeger backend. The default strategy is `allInOne`. The other possible values are `production` and `streaming`.
|
||||
|
||||
The available strategies are described in the following sections.
|
||||
|
||||
## AllInOne (Default) strategy
|
||||
|
||||
This strategy is intended for development, testing, and demo purposes.
|
||||
|
||||
The main backend components, agent, collector and query service, are all packaged into a single executable which is configured (by default) to use in-memory storage.
|
||||
|
||||
## Production strategy
|
||||
|
||||
The `production` strategy is intended (as the name suggests) for production environments, where long term storage of trace data is important, as well as a more scalable and highly available architecture is required. Each of the backend components is therefore separately deployed.
|
||||
|
||||
The agent can be injected as a sidecar on the instrumented application or as a daemonset.
|
||||
|
||||
The query and collector services are configured with a supported storage type - currently Cassandra or Elasticsearch. Multiple instances of each of these components can be provisioned as required for performance and resilience purposes.
|
||||
|
||||
The main additional requirement is to provide the details of the storage type and options, for example:
|
||||
|
||||
```yaml
|
||||
storage:
|
||||
type: elasticsearch
|
||||
options:
|
||||
es:
|
||||
server-urls: http://elasticsearch:9200
|
||||
```
|
||||
|
||||
## Streaming strategy
|
||||
|
||||
The `streaming` strategy is designed to augment the `production` strategy by providing a streaming capability that effectively sits between the collector and the backend storage (Cassandra or Elasticsearch). This provides the benefit of reducing the pressure on the backend storage, under high load situations, and enables other trace post-processing capabilities to tap into the real time span data directly from the streaming platform (Kafka).
|
||||
|
||||
The only additional information required is to provide the details for accessing the Kafka platform, which is configured in the `collector` component (as producer) and `ingester` component (as consumer):
|
||||
|
||||
```yaml
|
||||
apiVersion: jaegertracing.io/v1
|
||||
kind: Jaeger
|
||||
metadata:
|
||||
name: simple-streaming
|
||||
spec:
|
||||
strategy: streaming
|
||||
collector:
|
||||
options:
|
||||
kafka: # <1>
|
||||
producer:
|
||||
topic: jaeger-spans
|
||||
brokers: my-cluster-kafka-brokers.kafka:9092
|
||||
ingester:
|
||||
options:
|
||||
kafka: # <1>
|
||||
consumer:
|
||||
topic: jaeger-spans
|
||||
brokers: my-cluster-kafka-brokers.kafka:9092
|
||||
ingester:
|
||||
deadlockInterval: 0 # <2>
|
||||
storage:
|
||||
type: elasticsearch
|
||||
options:
|
||||
es:
|
||||
server-urls: http://elasticsearch:9200
|
||||
```
|
||||
<1> Identifies the Kafka configuration used by the collector, to produce the messages, and the ingester to consume the messages.
|
||||
|
||||
<2> The deadlock interval can be disabled to avoid the ingester being terminated when no messages arrive within the default 1 minute period
|
||||
|
||||
{{< info >}}
|
||||
A Kafka environment can be configured using [Strimzi's Kafka operator](https://strimzi.io/).
|
||||
{{< /info >}}
|
||||
|
||||
# Understanding Custom Resource Definitions
|
||||
|
||||
In the Kubernetes API, a resource is an endpoint that stores a collection of API objects of a certain kind. For example, the built-in Pods resource contains a collection of Pod objects. A _Custom Resource Definition_ (CRD) object defines a new, unique object `Kind` in the cluster and lets the Kubernetes API server handle its entire lifecycle.
|
||||
|
||||
To create _Custom Resource_ (CR) objects, cluster administrators must first create a Custom Resource Definition (CRD). The CRDs allow cluster users to create CRs to add the new resource types into their projects. An Operator watches for custom resource objects to be created, and when it sees a custom resource being created, it creates the application based on the parameters defined in the custom resource object.
|
||||
|
||||
{{< info >}}
|
||||
While only cluster administrators can create CRDs, developers can create the CR from an existing CRD if they have read and write permission to it.
|
||||
{{< /info >}}
|
||||
|
||||
<!--
|
||||
## Jaeger Custom Resource Parameters
|
||||
|
||||
TODO Create a TABLE with all the parameters, descriptions/notes, valid values, and defaults.
|
||||
Figure out if we can generate the options? Can we filter them in any way?
|
||||
https://github.com/jaegertracing/jaeger/issues/1537
|
||||
https://github.com/jaegertracing/documentation/issues/250-->
|
||||
|
||||
For reference, here's how you can create a more complex all-in-one instance:
|
||||
|
||||
```yaml
|
||||
apiVersion: jaegertracing.io/v1
|
||||
kind: Jaeger
|
||||
metadata:
|
||||
name: my-jaeger
|
||||
spec:
|
||||
strategy: allInOne # <1>
|
||||
allInOne:
|
||||
image: jaegertracing/all-in-one:latest # <2>
|
||||
options: # <3>
|
||||
log-level: debug # <4>
|
||||
storage:
|
||||
type: memory # <5>
|
||||
options: # <6>
|
||||
memory: # <7>
|
||||
max-traces: 100000
|
||||
ingress:
|
||||
enabled: false # <8>
|
||||
agent:
|
||||
strategy: DaemonSet # <9>
|
||||
annotations:
|
||||
scheduler.alpha.kubernetes.io/critical-pod: "" # <10>
|
||||
```
|
||||
|
||||
<1> The default strategy is `allInOne`. The other possible values are `production` and `streaming`.
|
||||
|
||||
<2> The image to use, in a regular Docker syntax.
|
||||
|
||||
<3> The (non-storage related) options to be passed verbatim to the underlying binary. Refer to the Jaeger documentation and/or to the `--help` option from the related binary for all the available options.
|
||||
|
||||
<4> The option is a simple `key: value` map. In this case, we want the option `--log-level=debug` to be passed to the binary.
|
||||
|
||||
<5> The storage type to be used. By default it will be `memory`, but can be any other supported storage type (Cassandra, Elasticsearch, Kafka).
|
||||
|
||||
<6> All storage related options should be placed here, rather than under the 'allInOne' or other component options.
|
||||
|
||||
<7> Some options are namespaced and we can alternatively break them into nested objects. We could have specified `memory.max-traces: 100000`.
|
||||
|
||||
<8> By default, an ingress object is created for the query service. It can be disabled by setting its `enabled` option to `false`. If deploying on OpenShift, this will be represented by a Route object.
|
||||
|
||||
<9> By default, the operator assumes that agents are deployed as sidecars within the target pods. Specifying the strategy as "DaemonSet" changes that and makes the operator deploy the agent as DaemonSet. Note that your tracer client will probably have to override the "JAEGER_AGENT_HOST" environment variable to use the node's IP.
|
||||
|
||||
<10> Define annotations to be applied to all deployments (not services). These can be overridden by annotations defined on the individual components.
|
||||
|
||||
You can view example custom resources for different Jaeger configurations [on GitHub](https://github.com/jaegertracing/jaeger-operator/tree/master/examples).
|
||||
|
||||
# Configuring the Custom Resource
|
||||
|
||||
<!--TODO
|
||||
esIndexCleaner
|
||||
Spark dependencies
|
||||
-->
|
||||
|
||||
You can use the simplest example (shown above) and create a Jaeger instance using the defaults, or you can create your own custom resource file.
|
||||
|
||||
## Storage options
|
||||
|
||||
### Cassandra storage
|
||||
|
||||
When the storage type is set to Cassandra, the operator will automatically create a batch job that creates the required schema for Jaeger to run. This batch job will block the Jaeger installation, so that it starts only after the schema is successfuly created. The creation of this batch job can be disabled by setting the `enabled` property to `false`:
|
||||
|
||||
```yaml
|
||||
apiVersion: jaegertracing.io/v1
|
||||
kind: Jaeger
|
||||
metadata:
|
||||
name: cassandra-without-create-schema
|
||||
spec:
|
||||
strategy: allInOne
|
||||
storage:
|
||||
type: cassandra
|
||||
cassandraCreateSchema:
|
||||
enabled: false # <1>
|
||||
```
|
||||
<1> Defaults to `true`
|
||||
|
||||
Further aspects of the batch job can be configured as well. An example with all the possible options is shown below:
|
||||
|
||||
```yaml
|
||||
apiVersion: jaegertracing.io/v1
|
||||
kind: Jaeger
|
||||
metadata:
|
||||
name: cassandra-with-create-schema
|
||||
spec:
|
||||
strategy: allInOne # <1>
|
||||
storage:
|
||||
type: cassandra
|
||||
options: # <2>
|
||||
cassandra:
|
||||
servers: cassandra
|
||||
keyspace: jaeger_v1_datacenter3
|
||||
cassandraCreateSchema: # <3>
|
||||
datacenter: "datacenter3"
|
||||
mode: "test"
|
||||
```
|
||||
<1> The same works for `production` and `streaming`.
|
||||
|
||||
<2> These options are for the regular Jaeger components, like `collector` and `query`.
|
||||
|
||||
<3> The options for the `create-schema` job.
|
||||
|
||||
{{< info >}}
|
||||
The default create-schema job uses `MODE=prod`, which implies a replication factor of `2`, using `NetworkTopologyStrategy` as the class, effectively meaning that at least 3 nodes are required in the Cassandra cluster. If a `SimpleStrategy` is desired, set the mode to `test`, which then sets the replication factor of `1`. Refer to the [create-schema script](https://github.com/jaegertracing/jaeger/blob/master/plugin/storage/cassandra/schema/create.sh) for more details.
|
||||
{{< /info >}}
|
||||
|
||||
### Elasticsearch storage
|
||||
|
||||
Under some circumstances, the Jaeger Operator can make use of the [Elasticsearch Operator](https://github.com/openshift/elasticsearch-operator) to provision a suitable Elasticsearch cluster.
|
||||
|
||||
{{< warning >}}
|
||||
This feature is only tested on OpenShift clusters. Spark dependencies are not supported with this feature [Issue #294](https://github.com/jaegertracing/jaeger-operator/issues/294).
|
||||
{{< /warning >}}
|
||||
|
||||
When there are no `es.server-urls` options as part of a Jaeger `production` instance and `elasticsearch` is set as the storage type, the Jaeger Operator creates an Elasticsearch cluster via the Elasticsearch Operator by creating a Custom Resource based on the configuration provided in storage section. The Elasticsearch cluster is meant to be dedicated for a single Jaeger instance.
|
||||
|
||||
The self-provision of an Elasticsearch cluster can be disabled by setting the flag `--es-provision` to `false`. The default value is `auto`, which will make the Jaeger Operator query the Kubernetes cluster for its ability to handle a `Elasticsearch` custom resource. This is usually set by the Elasticsearch Operator during its installation process, so, if the Elasticsearch Operator is expected to run *after* the Jaeger Operator, the flag can be set to `true`.
|
||||
|
||||
{{< danger >}}
|
||||
At the moment there can be only one Jaeger with self-provisioned Elasticsearch instance per namespace.
|
||||
{{< /danger >}}
|
||||
|
||||
#### Elasticsearch index cleaner job
|
||||
|
||||
When using `elasticsearch` storage by default a job is created to clean old traces from it, the options for it are listed below so you can configure it to your use case
|
||||
|
||||
```yaml
|
||||
storage:
|
||||
type: elasticsearch
|
||||
esIndexCleaner:
|
||||
enabled: false # turn the job deployment on and off
|
||||
numberOfDays: 7 # number of days to wait before deleting a record
|
||||
schedule: "55 23 * * *" # cron expression for it to run
|
||||
image: jaegertracing/jaeger-es-index-cleaner # image of the job
|
||||
```
|
||||
|
||||
## Auto-injecting Jaeger Agent Sidecars
|
||||
|
||||
The operator can inject Jaeger Agent sidecars in `Deployment` workloads, provided that the deployment has the annotation `sidecar.jaegertracing.io/inject` with a suitable value. The values can be either `"true"` (as string), or the Jaeger instance name, as returned by `kubectl get jaegers`. When `"true"` is used, there should be exactly *one* Jaeger instance for the same namespace as the deployment, otherwise, the operator can't figure out automatically which Jaeger instance to use.
|
||||
|
||||
The following snippet shows a simple application that will get a sidecar injected, with the Jaeger Agent pointing to the single Jaeger instance available in the same namespace:
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: myapp
|
||||
annotations:
|
||||
"sidecar.jaegertracing.io/inject": "true" # <1>
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app: myapp
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: myapp
|
||||
spec:
|
||||
containers:
|
||||
- name: myapp
|
||||
image: acme/myapp:myversion
|
||||
```
|
||||
<1> Either `"true"` (as string) or the Jaeger instance name.
|
||||
|
||||
A complete sample deployment is available at [`deploy/examples/business-application-injected-sidecar.yaml`](https://github.com/jaegertracing/jaeger-operator/blob/master/examples/business-application-injected-sidecar.yaml).
|
||||
|
||||
When the sidecar is injected, the Jaeger Agent can then be accessed at its default location on `localhost`.
|
||||
|
||||
## Installing the Agent as DaemonSet
|
||||
By default, the Operator expects the agents to be deployed as sidecars to the target applications. This is convenient for several purposes, like in a multi-tenant scenario or to have better load balancing, but there are scenarios where you might want to install the agent as a `DaemonSet`. In that case, specify the Agent's strategy to `DaemonSet`, as follows:
|
||||
|
||||
```yaml
|
||||
apiVersion: jaegertracing.io/v1
|
||||
kind: Jaeger
|
||||
metadata:
|
||||
name: my-jaeger
|
||||
spec:
|
||||
agent:
|
||||
strategy: DaemonSet
|
||||
```
|
||||
|
||||
{{< danger >}}
|
||||
If you attempt to install two Jaeger instances on the same cluster with `DaemonSet` as the strategy, only *one* will end up deploying a `DaemonSet`, as the agent is required to bind to well-known ports on the node. Because of that, the second daemon set will fail to bind to those ports.
|
||||
{{< /danger >}}
|
||||
|
||||
Your tracer client will then most likely need to be told where the agent is located. This is usually done by setting the environment variable `JAEGER_AGENT_HOST` to the value of the Kubernetes node's IP, for example:
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: myapp
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app: myapp
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: myapp
|
||||
spec:
|
||||
containers:
|
||||
- name: myapp
|
||||
image: acme/myapp:myversion
|
||||
env:
|
||||
- name: JAEGER_AGENT_HOST
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
fieldPath: status.hostIP
|
||||
```
|
||||
|
||||
## Secrets Support
|
||||
|
||||
The Operator supports passing secrets to the Collector, Query and All-In-One deployments. This can be used for example, to pass credentials (username/password) to access the underlying storage backend (for example: Elasticsearch).
|
||||
The secrets are available as environment variables in the (Collector/Query/All-In-One) nodes.
|
||||
|
||||
```yaml
|
||||
storage:
|
||||
type: elasticsearch
|
||||
options:
|
||||
es:
|
||||
server-urls: http://elasticsearch:9200
|
||||
secretName: jaeger-secrets
|
||||
```
|
||||
|
||||
The secret itself would be managed outside of the `jaeger-operator` custom resource.
|
||||
|
||||
## Configuring the UI
|
||||
|
||||
Information on various configuration options for the UI can be found [here](../frontend-ui/#configuration), defined in json format.
|
||||
|
||||
To apply UI configuration changes within the Custom Resource, the same information can be included in yaml format as shown below:
|
||||
|
||||
```yaml
|
||||
ui:
|
||||
options:
|
||||
dependencies:
|
||||
menuEnabled: false
|
||||
tracking:
|
||||
gaID: UA-000000-2
|
||||
menu:
|
||||
- label: "About Jaeger"
|
||||
items:
|
||||
- label: "Documentation"
|
||||
url: "https://www.jaegertracing.io/docs/latest"
|
||||
linkPatterns:
|
||||
- type: "logs"
|
||||
key: "customer_id"
|
||||
url: /search?limit=20&lookback=1h&service=frontend&tags=%7B%22customer_id%22%3A%22#{customer_id}%22%7D
|
||||
text: "Search for other traces for customer_id=#{customer_id}"
|
||||
```
|
||||
|
||||
## Defining Sampling Strategies
|
||||
|
||||
{{< info >}}
|
||||
This is not relevant if a trace was started by the Istio proxy as the sampling decision is made there. And the Jaeger sampling decisions are only relevant when you are using the Jaeger tracer (client).
|
||||
{{< /info >}}
|
||||
|
||||
The operator can be used to define sampling strategies that will be supplied to tracers that have been configured to use a remote sampler:
|
||||
|
||||
```yaml
|
||||
apiVersion: jaegertracing.io/v1
|
||||
kind: Jaeger
|
||||
metadata:
|
||||
name: with-sampling
|
||||
spec:
|
||||
strategy: allInOne
|
||||
sampling:
|
||||
options:
|
||||
default_strategy:
|
||||
type: probabilistic
|
||||
param: 0.5
|
||||
```
|
||||
|
||||
This example defines a default sampling strategy that is probabilistic, with a 50% chance of the trace instances being sampled.
|
||||
|
||||
Refer to the Jaeger documentation on [Collector Sampling Configuration](https://www.jaegertracing.io/docs/latest/sampling/#collector-sampling-configuration) to see how service and endpoint sampling can be configured. The JSON representation described in that documentation can be used in the operator by converting to YAML.
|
||||
|
||||
## Finer grained configuration
|
||||
|
||||
The custom resource can be used to define finer grained Kubernetes configuration applied to all Jaeger components or at the individual component level.
|
||||
|
||||
When a common definition (for all Jaeger components) is required, it is defined under the `spec` node. When the definition relates to an individual component, it is placed under the `spec/<component>` node.
|
||||
|
||||
The types of supported configuration include:
|
||||
|
||||
* [affinity](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity) to determine which nodes a pod can be allocated to
|
||||
|
||||
* [annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/)
|
||||
|
||||
* [labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/)
|
||||
|
||||
* [resources](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container) to limit cpu and memory
|
||||
|
||||
* [tolerations](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/) in conjunction with `taints` to enable pods to avoid being repelled from a node
|
||||
|
||||
* [volumes](https://kubernetes.io/docs/concepts/storage/volumes/) and volume mounts
|
||||
|
||||
* [serviceAccount](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/) to run each component with separate identity
|
||||
|
||||
* [securityContext](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/) to define privileges of running components
|
||||
|
||||
```yaml
|
||||
apiVersion: jaegertracing.io/v1
|
||||
kind: Jaeger
|
||||
metadata:
|
||||
name: simple-prod
|
||||
spec:
|
||||
strategy: production
|
||||
storage:
|
||||
type: elasticsearch
|
||||
options:
|
||||
es:
|
||||
server-urls: http://elasticsearch:9200
|
||||
annotations:
|
||||
key1: value1
|
||||
labels:
|
||||
key2: value2
|
||||
resources:
|
||||
requests:
|
||||
memory: "64Mi"
|
||||
cpu: "250m"
|
||||
limits:
|
||||
memory: "128Mi"
|
||||
cpu: "500m"
|
||||
affinity:
|
||||
nodeAffinity:
|
||||
requiredDuringSchedulingIgnoredDuringExecution:
|
||||
nodeSelectorTerms:
|
||||
- matchExpressions:
|
||||
- key: kubernetes.io/e2e-az-name
|
||||
operator: In
|
||||
values:
|
||||
- e2e-az1
|
||||
- e2e-az2
|
||||
preferredDuringSchedulingIgnoredDuringExecution:
|
||||
- weight: 1
|
||||
preference:
|
||||
matchExpressions:
|
||||
- key: another-node-label-key
|
||||
operator: In
|
||||
values:
|
||||
- another-node-label-value
|
||||
tolerations:
|
||||
- key: "key1"
|
||||
operator: "Equal"
|
||||
value: "value1"
|
||||
effect: "NoSchedule"
|
||||
- key: "key1"
|
||||
operator: "Equal"
|
||||
value: "value1"
|
||||
effect: "NoExecute"
|
||||
serviceAccount: nameOfServiceAccount
|
||||
securityContext:
|
||||
runAsUser: 1000
|
||||
volumeMounts:
|
||||
- name: config-vol
|
||||
mountPath: /etc/config
|
||||
volumes:
|
||||
- name: config-vol
|
||||
configMap:
|
||||
name: log-config
|
||||
items:
|
||||
- key: log_level
|
||||
path: log_level
|
||||
```
|
||||
|
||||
# Accessing the Jaeger Console (UI)
|
||||
<!-- TODO Add tabs shortcode -->
|
||||
|
||||
## Kubernetes
|
||||
|
||||
The operator creates a Kubernetes [`ingress`](https://kubernetes.io/docs/concepts/services-networking/ingress/) route, which is the Kubernetes' standard for exposing a service to the outside world, but by default it does not come with Ingress providers.
|
||||
Check the [Kubernetes documentation](https://kubernetes.github.io/ingress-nginx/deploy/#verify-installation) for the most appropriate way to achieve an Ingress provider for your platform. The following command enables the Ingress provider on `minikube`:
|
||||
|
||||
```bash
|
||||
minikube addons enable ingress
|
||||
```
|
||||
|
||||
Once Ingress is enabled, the address for the Jaeger console can be found by querying the Ingress object:
|
||||
|
||||
```bash
|
||||
$ kubectl get ingress
|
||||
NAME HOSTS ADDRESS PORTS AGE
|
||||
simplest-query * 192.168.122.34 80 3m
|
||||
```
|
||||
|
||||
In this example, the Jaeger UI is available at http://192.168.122.34.
|
||||
|
||||
## OpenShift
|
||||
|
||||
When the Operator is running on OpenShift, the Operator will automatically create a `Route` object for the query services. Use the following command to check the hostname/port:
|
||||
|
||||
```bash
|
||||
oc get routes
|
||||
```
|
||||
|
||||
{{< info >}}
|
||||
Make sure to use `https` with the hostname/port you get from the command above, otherwise you'll see a message like: "Application is not available".
|
||||
{{< /info >}}
|
||||
|
||||
By default, the Jaeger UI is protected with OpenShift's OAuth service and any valid user is able to login. To disable this feature and leave the Jaeger UI unsecured, set the Ingress property `security` to `none` in the custom resource file:
|
||||
|
||||
```yaml
|
||||
apiVersion: jaegertracing.io/v1
|
||||
kind: Jaeger
|
||||
metadata:
|
||||
name: disable-oauth-proxy
|
||||
spec:
|
||||
ingress:
|
||||
security: none
|
||||
```
|
||||
|
||||
Custom `SAR` and `Delegate URL` values can be specified as part of the `.Spec.Ingress.OpenShift.SAR` and `.Spec.Ingress.Openshift.DelegateURLs`, as follows:
|
||||
|
||||
```yaml
|
||||
apiVersion: jaegertracing.io/v1
|
||||
kind: Jaeger
|
||||
metadata:
|
||||
name: custom-sar-oauth-proxy
|
||||
spec:
|
||||
ingress:
|
||||
openshift:
|
||||
sar: '{"namespace": "default", "resource": "pods", "verb": "get"}'
|
||||
delegate-urls: '{"/":{"namespace": "default", "resource": "pods", "verb": "get"}}'
|
||||
```
|
||||
|
||||
When the `delegate-urls` is set, the Jaeger Operator needs to create a new `ClusterRoleBinding` between the service account used by the UI Proxy (`{InstanceName}-ui-proxy`) and the role `system:auth-delegator`, as required by the OpenShift OAuth Proxy. Because of that, the service account used by the operator itself needs to have the same cluster role binding. To accomplish that, a `ClusterRoleBinding` such as the following has to be created:
|
||||
|
||||
```yaml
|
||||
kind: ClusterRoleBinding
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
metadata:
|
||||
name: jaeger-operator-with-auth-delegator
|
||||
namespace: observability
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: jaeger-operator
|
||||
namespace: observability
|
||||
roleRef:
|
||||
kind: ClusterRole
|
||||
name: system:auth-delegator
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
```
|
||||
|
||||
Cluster administrators not comfortable in letting users deploy Jaeger instances with this cluster role are free to not add this cluster role to the operator's service account. In that case, the Operator will auto-detect that the required permissions are missing and will log a message similar to: `the requested instance specifies the delegate-urls option for the OAuth Proxy, but this operator cannot assign the proper cluster role to it (system:auth-delegator). Create a cluster role binding between the operator's service account and the cluster role 'system:auth-delegator' in order to allow instances to use 'delegate-urls'`.
|
||||
|
||||
# Updating a Jaeger instance (experimental)
|
||||
|
||||
A Jaeger instance can be updated by changing the `CustomResource`, either via `kubectl edit jaeger simplest`, where `simplest` is the Jaeger's instance name, or by applying the updated YAML file via `kubectl apply -f simplest.yaml`.
|
||||
|
||||
{{< danger >}}
|
||||
The name of the Jaeger instance cannot be updated, as it is part of the identifying information for the resource.
|
||||
{{< /danger >}}
|
||||
|
||||
Simpler changes such as changing the replica sizes can be applied without much concern, whereas changes to the strategy should be watched closely and might potentially cause an outage for individual components (collector/query/agent).
|
||||
|
||||
While changing the backing storage is supported, migration of the data is not.
|
||||
|
||||
# Removing a Jaeger instance
|
||||
<!-- TODO Add OKD/OpenShift commands and tabs shortcode-->
|
||||
|
||||
To remove an instance, use the `delete` command with the custom resource file used when you created the instance:
|
||||
|
||||
```bash
|
||||
kubectl delete -f simplest.yaml
|
||||
```
|
||||
|
||||
Alternatively, you can remove a Jaeger instance by running:
|
||||
|
||||
```bash
|
||||
kubectl delete jaeger simplest
|
||||
```
|
||||
|
||||
{{< info >}}
|
||||
Deleting the instance will not remove the data from any permanent storage used with this instance. Data from in-memory instances, however, will be lost.
|
||||
{{< /info >}}
|
||||
|
||||
# Monitoring the operator
|
||||
|
||||
The Jaeger Operator starts a Prometheus-compatible endpoint on `0.0.0.0:8383/metrics` with internal metrics that can be used to monitor the process.
|
||||
|
||||
{{< info >}}
|
||||
The Jaeger Operator does not yet publish its own metrics. Rather, it makes available metrics reported by the components it uses, such as the Operator SDK.
|
||||
{{< /info >}}
|
||||
|
||||
# Uninstalling the operator
|
||||
<!-- TODO Add OKD/OpenShift commands and tabs shortcode -->
|
||||
|
||||
To uninstall the operator, run the following commands:
|
||||
|
||||
```bash
|
||||
kubectl delete -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/operator.yaml
|
||||
kubectl delete -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role_binding.yaml
|
||||
kubectl delete -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role.yaml
|
||||
kubectl delete -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/service_account.yaml
|
||||
kubectl delete -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/crds/jaegertracing.io_jaegers_crd.yaml
|
||||
```
|
|
@ -1,95 +0,0 @@
|
|||
---
|
||||
title: Performance Tuning Guide
|
||||
description: Tweaking your Jaeger instance to achieve a better performance
|
||||
weight: 10
|
||||
---
|
||||
|
||||
Jaeger was built from day 1 to be able to ingest huge amounts of data in a resilient way. To better utilize resources that might cause delays, such as storage or network communications, Jaeger buffers and batches data. When more spans are generated than Jaeger is able to safely process, spans might get dropped. However, the defaults might not fit all scenarios: for instance, Agents running as a sidecar might have more memory constraints than Agents running as a daemon in bare metal.
|
||||
|
||||
## Deployment considerations
|
||||
|
||||
Although performance tuning the individual components is important, the way Jaeger is deployed can be decisive in obtaining optimal performance.
|
||||
|
||||
### Scale the Collector up and down
|
||||
|
||||
Use the auto-scaling capabilities of your platform: the Collector is nearly horizontally scalable so that more instances can be added and removed on-demand. A good way to scale up and down is by checking the `jaeger_collector_queue_length` metric: add instances when the length is higher than 50% of the maximum size for extended periods of time. Another metric that can be taken into consideration is `jaeger_collector_in_queue_latency_bucket`, which is a histogram indicating how long spans have been waiting in the queue before a worker picked it up. When the queue latency gets higher over time, it’s a good indication to increase the number of the workers, or to improve the storage performance.
|
||||
|
||||
Adding Collector instances is recommended when your platform provides auto-scaling capabilities, or when it's easier to start/stop Collector instances than changing existing, running instances. Scaling horizontally is also indicated when the CPU usage should be spread across nodes.
|
||||
|
||||
### Make sure the storage can keep up
|
||||
|
||||
Each span is written to the storage by the Collector using one worker, blocking it until the span has been stored. When the storage is too slow, the number of workers blocked by the storage might be too high, causing spans to be dropped. To help diagnose this situation, the histogram `jaeger_collector_save_latency_bucket` can be analyzed. Ideally, the latency should remain the same over time. When the histogram shows that most spans are taking longer and longer over time, it’s a good indication that your storage might need some attention.
|
||||
|
||||
### Place the Agents close to your applications
|
||||
|
||||
The Agent is meant to be placed on the same host as the instrumented application, in order to avoid UDP packet loss over the network. This is typically accomplished by having one Agent per bare metal host for traditional applications, or as a sidecar in container environments like Kubernetes, as this helps spread the load handled by Agents with the additional advantage of allowing each Agent to be tweaked individually, according to the application’s needs and importance.
|
||||
|
||||
### Consider using Apache Kafka as intermediate buffer
|
||||
|
||||
Jaeger [can use Apache Kafka](../architecture/) as a buffer between the Collector and the actual backing storage (Elasticsearch, Apache Cassandra). This is ideal for cases where the traffic spikes are relatively frequent (prime time traffic) but the storage can eventually catch up once the traffic normalizes. For that, the `SPAN_STORAGE_TYPE` environment variable should be set to `kafka` in the Collector and the Jaeger Ingester component can be used, reading data from Kafka and writing it to the storage.
|
||||
|
||||
In addition to the performance aspects, having spans written to Kafka is useful for building real time data pipeline for aggregations and feature extraction from traces.
|
||||
|
||||
## Client (Tracer) settings
|
||||
|
||||
The Jaeger Clients are built to have minimal effect to the instrumented application. As such, it has conservative defaults that might not be suitable for all cases. Note that Jaeger Clients can be configured programmatically or via [environment variables](../client-features/).
|
||||
|
||||
### Adjust the sampling configuration
|
||||
|
||||
Together, the `JAEGER_SAMPLER_TYPE` and `JAEGER_SAMPLER_PARAM` specify how often traces should be "sampled", ie, recorded and sent to the Jaeger backend. For applications generating a large number of spans, setting the sampling type to `probabilistic` and the value to `0.001` (the default) will cause traces to be reported with a 1/1000th chance. Note that the sampling decision is made at the root span and propagated down to all child spans.
|
||||
|
||||
For applications with low to medium traffic, setting the sampling type to `const` and value to `1` will cause all spans to be reported. Similarly, tracing can be disabled by setting the value to `0`, while context propagation will continue to work.
|
||||
|
||||
Some Clients support the setting `JAEGER_DISABLED` to completely disable the Jaeger Tracer. This is recommended only if the Tracer is behaving in a way that causes problems to the instrumented application, as it will not propagate the context to the downstream services.
|
||||
|
||||
{{< info >}}
|
||||
We recommend to set your clients to use the [`remote` sampling strategy](../sampling/#collector-sampling-configuration), so that admins can centrally set the concrete sampling strategy for each service.
|
||||
{{< /info >}}
|
||||
|
||||
### Increase in-memory queue size
|
||||
|
||||
Most of the Jaeger clients, such as the Java, Go, and C# clients, buffer spans in memory before sending them to the Jaeger Agent/Collector. The maximum size of this buffer is defined by the environment variable `JAEGER_REPORTER_MAX_QUEUE_SIZE` (default value: about `100` spans): the larger the size, the higher the potential memory consumption. When the instrumented application is generating a large number of spans, it’s possible that the queue will be full causing the Client to discard the new spans (metric `jaeger_tracer_reporter_spans_total{result="dropped",}`).
|
||||
|
||||
In most common scenarios, the queue will be close to empty (metric: `jaeger_tracer_reporter_queue_length`), as spans are flushed to the Agent or Collector at regular intervals or when a certain size of the batch is reached. The detailed behavior of this queue is described in this [GitHub issue](https://github.com/jaegertracing/jaeger-client-java/issues/607).
|
||||
|
||||
### Modify the batched spans flush interval
|
||||
|
||||
The Java, Go, NodeJS, Python and C# Clients allow the customization of the flush interval (default value: `1000` milliseconds, or 1 second) used by the reporters, such as the `RemoteReporter`, to trigger a `flush` operation, sending all in-memory spans to the Agent or Collector. The lower the flush interval is set to, the more frequent the flush operations happen. As most reporters will wait until enough data is in the queue, this setting will force a flush operation at periodic intervals, so that spans are sent to the backend in a timely fashion.
|
||||
|
||||
When the instrumented application is generating a large number of spans and the Agent/Collector is close to the application, the networking overhead might be low, justifying a higher number of flush operations. When the `HttpSender` is being used and the Collector is not close enough to the application, the networking overhead might be too high so that a higher value for this property makes sense.
|
||||
|
||||
## Agent settings
|
||||
|
||||
Jaeger Agents receive data from Clients, sending them in batches to the Collector. When not properly configured, it might end up discarding data even if the host machine has plenty of resources.
|
||||
|
||||
### Adjust server queue sizes
|
||||
|
||||
The set of “server queue size” properties ( `processor.jaeger-binary.server-queue-size`, `processor.jaeger-compact.server-queue-size`, `processor.zipkin-compact.server-queue-size`) indicate the maximum number of span batches that the Agent can accept and store in memory. It’s safe to assume that `jaeger-compact` is the most important processor in your Agent setup, as it’s the only one available in most Clients, such as the Java and Go Clients.
|
||||
|
||||
The default value for each queue is `1000` span batches. Given that each span batch has up to 64KiB worth of spans, each queue can hold up to 64MiB worth of spans.
|
||||
|
||||
In typical scenarios, the queue will be close to empty (metric `jaeger_agent_thrift_udp_server_queue_size`) as span batches should be quickly picked up and processed by a worker. However, sudden spikes in the number of span batches submitted by Clients might occur, causing the batches to be queued. When the queue is full, the older batches are overridden causing spans to be discarded (metric `jaeger_agent_thrift_udp_server_packets_dropped_total`).
|
||||
|
||||
### Adjust processor workers
|
||||
|
||||
The set of “processor workers” properties ( `processor.jaeger-binary.workers`, `processor.jaeger-compact.workers`, `processor.zipkin-compact.workers`) indicate the number of parallel span batch processors to start. Each worker type has a default size of `10`. In general, span batches are processed as soon as they are placed in the server queue and will block a worker until the whole packet is sent to the Collector. For Agents processing data from multiple Clients, the number of workers should be increased. Given that the cost of each worker is low, a good rule of thumb is 10 workers per Client with moderate traffic: given that each span batch might contain up to 64KiB worth of spans, it means that 10 workers are able to send about 640KiB concurrently to a Collector.
|
||||
|
||||
## Collector settings
|
||||
|
||||
The Collector receives data from Clients and Agents. When not properly configured, it might process less data than what would be possible on the same host, or it might overload the host by consuming more memory than permitted.
|
||||
|
||||
### Adjust queue size
|
||||
|
||||
Similar to the Agent, the Collector is able to receive spans and place them in an internal queue for processing. This allows the Collector to return immediately to the Client/Agent instead of waiting for the span to make its way to the storage.
|
||||
|
||||
The setting `collector.queue-size` (default: `2000`) dictates how many spans the queue should support. In the typical scenario, the queue will be close to empty, as enough workers should exist picking up spans from the queue and sending them to the storage. When the number of items in the queue (metric `jaeger_collector_queue_length`) is permanently high, it’s an indication that either the number of workers should be increased or that the storage cannot keep up with the volume of data that it’s receiving. When the queue is full, the older items in the queue are overridden, causing spans to be discarded (metric `jaeger_collector_spans_dropped_total`).
|
||||
|
||||
{{< warning >}}
|
||||
The queue size for the Agent is about _span batches_, whereas the queue size for the Collector is about _individual spans_.
|
||||
{{< /warning >}}
|
||||
|
||||
Given that the queue size should be close to empty most of the time, this setting should be as high as the available memory for the Collector, to provide maximum protection against sudden traffic spikes. However, if your storage layer is under-provisioned and cannot keep up, even a large queue will quickly fill up and start dropping data.
|
||||
|
||||
### Adjust processor workers
|
||||
|
||||
Items from the span queue in the Collector are picked up by workers. Each worker picks one span from the queue and persists it to the storage. The number of workers can be specified by the setting `collector.num-workers` (default: `50`) and should be as high as needed to keep the queue close to zero. The general rule is: the faster the backing storage, the lower the number of workers can be. Given that workers are relatively cheap, this number can be increased at will. As a general rule, one worker per 50 items in the queue should be sufficient when the storage is fast. With a `collector.queue-size` of `2000`, having about `40` workers should be sufficient. For slower storage mechanisms, this ratio should be adjusted accordingly, having more workers per queue item.
|
|
@ -1,66 +0,0 @@
|
|||
---
|
||||
title: Sampling
|
||||
hasparent: true
|
||||
---
|
||||
|
||||
Jaeger libraries implement consistent upfront (or head-based) sampling. For example, assume we have a simple call graph where service A calls service B, and B calls service C: `A -> B -> C`. When service A receives a request that contains no tracing information, Jaeger tracer will start a new {{< tip "trace" >}}, assign it a random trace ID, and make a sampling decision based on the currently installed sampling strategy. The sampling decision will be propagated with the requests to B and to C, so those services will not be making the sampling decision again but instead will respect the decision made by the top service A. This approach guarantees that if a trace is sampled, all its {{< tip "spans" "span" >}} will be recorded in the backend. If each service was making its own sampling decision we would rarely get complete traces in the backend.
|
||||
|
||||
## Client Sampling Configuration
|
||||
|
||||
When using configuration object to instantiate the tracer, the type of sampling can be selected via `sampler.type` and `sampler.param` properties. Jaeger libraries support the following samplers:
|
||||
|
||||
* **Constant** (`sampler.type=const`) sampler always makes the same decision for all traces. It either samples all traces (`sampler.param=1`) or none of them (`sampler.param=0`).
|
||||
* **Probabilistic** (`sampler.type=probabilistic`) sampler makes a random sampling decision with the probability of sampling equal to the value of `sampler.param` property. For example, with `sampler.param=0.1` approximately 1 in 10 traces will be sampled.
|
||||
* **Rate Limiting** (`sampler.type=ratelimiting`) sampler uses a leaky bucket rate limiter to ensure that traces are sampled with a certain constant rate. For example, when `sampler.param=2.0` it will sample requests with the rate of 2 traces per second.
|
||||
* **Remote** (`sampler.type=remote`, which is also the default) sampler consults Jaeger agent for the appropriate sampling strategy to use in the current service. This allows controlling the sampling strategies in the services from a [central configuration](#collector-sampling-configuration) in Jaeger backend, or even dynamically (see [Adaptive Sampling](https://github.com/jaegertracing/jaeger/issues/365)).
|
||||
|
||||
### Adaptive Sampler
|
||||
|
||||
Adaptive sampler is a composite sampler that combines two functions:
|
||||
|
||||
* It makes sampling decisions on a per-operation basis, i.e. based on {{< tip "span" >}} operation name. This is especially useful in the API services whose endpoints may have very different traffic volumes and using a single probabilistic sampler for the whole service might starve (never sample) some of the low QPS endpoints.
|
||||
* It supports a minimum guaranteed rate of sampling, such as always allowing up to N {{< tip "traces" "trace" >}} per seconds and then sampling anything above that with a certain probability (everything is per-operation, not per-service).
|
||||
|
||||
Per-operation parameters can be configured statically or pulled periodically from Jaeger backend with the help of Remote sampler. Adaptive sampler is designed to work with the upcoming [Adaptive Sampling](https://github.com/jaegertracing/jaeger/issues/365) feature of the Jaeger backend.
|
||||
|
||||
## Collector Sampling Configuration
|
||||
|
||||
Collectors can be instantiated with static sampling strategies (which are propagated to the respective service if configured with Remote sampler) via the `--sampling.strategies-file` option. This option requires a path to a json file which have the sampling strategies defined.
|
||||
|
||||
Example `strategies.json`
|
||||
```json
|
||||
{
|
||||
"service_strategies": [
|
||||
{
|
||||
"service": "foo",
|
||||
"type": "probabilistic",
|
||||
"param": 0.8,
|
||||
"operation_strategies": [
|
||||
{
|
||||
"operation": "op1",
|
||||
"type": "probabilistic",
|
||||
"param": 0.2
|
||||
},
|
||||
{
|
||||
"operation": "op2",
|
||||
"type": "probabilistic",
|
||||
"param": 0.4
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "bar",
|
||||
"type": "ratelimiting",
|
||||
"param": 5
|
||||
}
|
||||
],
|
||||
"default_strategy": {
|
||||
"type": "probabilistic",
|
||||
"param": 0.5
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`service_strategies` defines service specific sampling strategies and `operation_strategies` defines operation specific sampling strategies. There are 2 types of strategies possible: `probabilistic` and `ratelimiting` which are described [above](#client-sampling-configuration) (NOTE: `ratelimiting` is not supported for `operation_strategies`). `default_strategy` defines the catch-all sampling strategy that is propagated if the service is not included as part of `service_strategies`.
|
||||
|
||||
In the above example, all service `foo` operations are sampled probabilistically with a probability of 0.8 except `op1` and `op2` which are probabilistically sampled with a probability of 0.2 and 0.4 respectively. All operations for service `bar` are ratelimited at 5 traces per second. Any other service is probabilistically sampled with a probability of 0.5.
|
|
@ -1,70 +0,0 @@
|
|||
---
|
||||
title: Introduction
|
||||
weight: 1
|
||||
children:
|
||||
- title: Features
|
||||
url: features
|
||||
---
|
||||
|
||||
Welcome to Jaeger's documentation portal! Below, you'll find information for beginners and experienced Jaeger users.
|
||||
|
||||
If you can't find what you are looking for, or have an issue not covered here, we'd love to [hear from you](/get-in-touch).
|
||||
|
||||
## About
|
||||
|
||||
Jaeger, inspired by [Dapper][dapper] and [OpenZipkin](http://zipkin.io),
|
||||
is a distributed tracing system released as open source by [Uber Technologies][ubeross].
|
||||
It is used for monitoring and troubleshooting microservices-based distributed systems, including:
|
||||
|
||||
* Distributed context propagation
|
||||
* Distributed transaction monitoring
|
||||
* Root cause analysis
|
||||
* Service dependency analysis
|
||||
* Performance / latency optimization
|
||||
|
||||
Uber published a blog post, [Evolving Distributed Tracing at Uber](https://eng.uber.com/distributed-tracing/), where they explain the history and reasons for the architectural choices made in Jaeger. [Yuri Shkuro](https://shkuro.com), creator of Jaeger, also published a book [Mastering Distributed Tracing](https://shkuro.com/books/2019-mastering-distributed-tracing/) that covers in-depth many aspects of Jaeger design and operation, as well as distributed tracing in general.
|
||||
|
||||
## Features
|
||||
|
||||
* [OpenTracing](http://opentracing.io/) compatible data model and instrumentation libraries
|
||||
* in [Go](https://github.com/jaegertracing/jaeger-client-go), [Java](https://github.com/jaegertracing/jaeger-client-java), [Node](https://github.com/jaegertracing/jaeger-client-node), [Python](https://github.com/jaegertracing/jaeger-client-python)
|
||||
and [C++](https://github.com/jaegertracing/cpp-client)
|
||||
* Uses consistent upfront sampling with individual per service/endpoint probabilities
|
||||
* Multiple storage backends: Cassandra, Elasticsearch, memory.
|
||||
* Adaptive sampling (coming soon)
|
||||
* Post-collection data processing pipeline (coming soon)
|
||||
|
||||
See [Features](./features/) page for more details.
|
||||
|
||||
## Technical Specs
|
||||
|
||||
* Backend components implemented in Go
|
||||
* React/Javascript UI
|
||||
* Supported storage backends:
|
||||
* [Cassandra 3.4+](./deployment/#cassandra)
|
||||
* [Elasticsearch 5.x, 6.x, 7.x](./deployment/#elasticsearch)
|
||||
* [Kafka](./deployment/#kafka)
|
||||
* memory storage
|
||||
|
||||
## Quick Start
|
||||
See [running a docker all in one image](getting-started#all-in-one).
|
||||
|
||||
## Screenshots
|
||||
|
||||
### Traces View
|
||||
[](/img/traces-ss.png)
|
||||
|
||||
### Trace Detail View
|
||||
[](/img/trace-detail-ss.png)
|
||||
|
||||
## Related links
|
||||
- [Evolving Distributed tracing At Uber Engineering](https://eng.uber.com/distributed-tracing/)
|
||||
- [Mastering Distributed Tracing](https://shkuro.com/books/2019-mastering-distributed-tracing/)
|
||||
- [Tracing HTTP request latency in Go with OpenTracing](https://medium.com/opentracing/tracing-http-request-latency-in-go-with-opentracing-7cc1282a100a)
|
||||
- [Distributed Tracing with Jaeger & Prometheus on Kubernetes](https://blog.openshift.com/openshift-commons-briefing-82-distributed-tracing-with-jaeger-prometheus-on-kubernetes/)
|
||||
- [Using Jaeger with Istio](https://istio.io/docs/tasks/telemetry/distributed-tracing.html)
|
||||
- [Using Jaeger with Envoy](https://www.envoyproxy.io/docs/envoy/latest/start/sandboxes/jaeger_tracing.html)
|
||||
|
||||
[dapper]: https://research.google.com/pubs/pub36356.html
|
||||
[ubeross]: http://uber.github.io
|
||||
|
|
@ -1,87 +0,0 @@
|
|||
---
|
||||
title: APIs
|
||||
hasparent: true
|
||||
---
|
||||
|
||||
Jaeger components implement various APIs for saving or retrieving trace data.
|
||||
|
||||
The following labels are used to describe API compatibility guarantees.
|
||||
|
||||
* **stable** - the API guarantees backwards compatibility. If breaking changes are going to be made in the future, they will result in a new API version, e.g. `/api/v2` URL prefix or a different namespace in the IDL.
|
||||
* **internal** - the APIs intended for internal communications between Jaeger components and not recommended for use by external components.
|
||||
* **deprecated** - the APIs that are only maintained for legacy reasons and will be phased out in the future.
|
||||
|
||||
## Span reporting APIs
|
||||
|
||||
Agent and Collector are the two components of the Jaeger backend that can receive spans. At this time they support two sets of non-overlapping APIs.
|
||||
|
||||
### Thrift over UDP (stable)
|
||||
|
||||
The Agent can only receive spans over UDP in Thrift format. The primary API is a UDP packet that contains a Thrift-encoded `Batch` struct defined in [jaeger.thrift][jaeger.thrift] IDL file, located in the [jaeger-idl][jaeger-idl] repository. Most Jaeger Clients use Thrift's `compact` encoding, however some client libraries do not support it (notably, Node.js) and use Thrift's `binary` encoding (sent to a different UDP port). The Agent's API is defined by [agent.thrift][agent.thrift] IDL file.
|
||||
|
||||
For legacy reasons, the Agent also accepts spans in Zipkin format, however, only very old versions of Jaeger clients can send data in that format and it is officially deprecated.
|
||||
|
||||
### Protobuf via gRPC (stable)
|
||||
|
||||
In a typical Jaeger deployment, Agents receive spans from Clients and forward them to Collectors. Since Jaeger version 1.11 the official and recommended protocol between Agents and Collectors is gRPC with Protobuf.
|
||||
|
||||
The Protobuf IDL [collector.proto][collector.proto] is currently located in the main Jaeger repository, under [model/proto/api_v2][collector.proto]. In the future it will be moved to [jaeger-idl][jaeger-idl] repository ([jaeger-idl/issues/55](https://github.com/jaegertracing/jaeger-idl/issues/55)).
|
||||
|
||||
### Thrift over HTTP (stable)
|
||||
|
||||
In some cases it is not feasible to deploy Jaeger Agent next to the application, for example, when the application code is running as AWS Lambda function. In these scenarios the Jaeger Clients can be configured to submit spans directly to the Collectors over HTTP/HTTPS.
|
||||
|
||||
The same [jaeger.thrift][jaeger.thrift] payload can be submitted in HTTP POST request to `/api/traces` endpoint, for example, `https://jaeger-collector:14268/api/traces`. The `Batch` struct needs to be encoded using Thrift's `binary` encoding, and the HTTP request should specify the content type header:
|
||||
|
||||
```
|
||||
Content-Type: application/vnd.apache.thrift.binary
|
||||
```
|
||||
|
||||
### JSON over HTTP (n/a)
|
||||
|
||||
There is no official Jaeger JSON format that can be accepted by the collector. In the future the Protobuf-generated JSON may be supported.
|
||||
|
||||
### Thrift via TChannel (deprecated)
|
||||
|
||||
Agent and Collector can communicate using TChannel protocol. This protocol is generally not supported by the routing infrastructure and has been deprecated. It will be eventually removed from Jaeger.
|
||||
|
||||
### Zipkin Formats (stable)
|
||||
|
||||
Jaeger Collector can also accept spans in several Zipkin data format, namely JSON v1/v2 and Thrift. The Collector needs to be configured to enable Zipkin HTTP server, e.g. on port 9411 used by Zipkin collectors. The server enables two endpoints that expect POST requests:
|
||||
|
||||
* `/api/v1/spans` for submitting spans in Zipkin JSON v1 or Zipkin Thrift format.
|
||||
* `/api/v2/spans` for submitting spans in Zipkin JSON v2.
|
||||
|
||||
## Trace retrieval APIs
|
||||
|
||||
Traces saved in the storage can be retrieved by calling Jaeger Query Service.
|
||||
|
||||
### gRPC/Protobuf (stable)
|
||||
|
||||
The recommended way for programmatically retrieving traces and other data is via gRPC endpoint defined in [query.proto][query.proto] IDL file (located in the main Jaeger repository, similar to [collector.proto][collector.proto]).
|
||||
|
||||
### HTTP JSON (internal)
|
||||
|
||||
Jaeger UI communicates with Jaeger Query Service via JSON API. For example, a trace can be retrieved via GET request to `https://jaeger-query:16686/api/traces/{trace-id-hex-string}`. This JSON API is intentionally undocumented and subject to change.
|
||||
|
||||
## Clients configuration (internal)
|
||||
|
||||
Client libraries not only submit finished spans to Jaeger backend, but also periodically poll the Agents for various configurations, such as sampling strategies. The schema for the payload is defined by [sampling.thrift][sampling.thrift], encoded as JSON using Thrift's built-in JSON generation capabilities.
|
||||
|
||||
## Service dependencies graph (internal)
|
||||
|
||||
Can be retrieved from Query Service at `/api/dependencies` endpoint. The GET request expects two parameters:
|
||||
|
||||
* `endTs` (number of milliseconds since epoch) - the end of the time interval
|
||||
* `lookback` (in milliseconds) - the length the time interval (i.e. start-time + lookback = end-time).
|
||||
|
||||
The returned JSON is a list of edges represented as tuples `(caller, callee, count)`.
|
||||
|
||||
For programmatic access to service graph, the recommended API is gRPC/Protobuf described above.
|
||||
|
||||
[jaeger-idl]: https://github.com/jaegertracing/jaeger-idl/
|
||||
[jaeger.thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/jaeger.thrift
|
||||
[agent.thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/agent.thrift
|
||||
[sampling.thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/sampling.thrift
|
||||
[collector.proto]: https://github.com/jaegertracing/jaeger/blob/master/model/proto/api_v2/collector.proto
|
||||
[query.proto]: https://github.com/jaegertracing/jaeger/blob/master/model/proto/api_v2/query.proto
|
|
@ -1,13 +0,0 @@
|
|||
---
|
||||
title: CLI flags
|
||||
widescreen: true
|
||||
hasparent: true
|
||||
---
|
||||
|
||||
This is auto-generated documentation for CLI flags supported by Jaeger binaries.
|
||||
|
||||
* CLI flags for some binaries change depending on the `SPAN_STORAGE_TYPE` environment variable. Relevant variations are included below.
|
||||
* Some binaries support _commands_ (mostly informational), such as `env`, `docs`, and `version`. These commands are not included here.
|
||||
* All parameters can be also provided via environment variables, by changing all letters to upper-case and replacing all punctuation characters with underscore `_`. For example, the value for the flag `--cassandra.connections-per-host` can be provided via `CASSANDRA_CONNECTIONS_PER_HOST` environment variable.
|
||||
|
||||
{{< cli/tools-list >}}
|
|
@ -1,467 +0,0 @@
|
|||
---
|
||||
title: Deployment
|
||||
weight: 4
|
||||
children:
|
||||
- title: Operator for Kubernetes
|
||||
navtitle: Kubernetes
|
||||
url: operator
|
||||
- title: Frontend/UI
|
||||
url: frontend-ui
|
||||
- title: CLI Flags
|
||||
url: cli
|
||||
---
|
||||
|
||||
The main Jaeger backend components are released as Docker images on Docker Hub:
|
||||
|
||||
Component | Repository
|
||||
--------------------- | ---
|
||||
**jaeger-agent** | [hub.docker.com/r/jaegertracing/jaeger-agent/](https://hub.docker.com/r/jaegertracing/jaeger-agent/)
|
||||
**jaeger-collector** | [hub.docker.com/r/jaegertracing/jaeger-collector/](https://hub.docker.com/r/jaegertracing/jaeger-collector/)
|
||||
**jaeger-query** | [hub.docker.com/r/jaegertracing/jaeger-query/](https://hub.docker.com/r/jaegertracing/jaeger-query/)
|
||||
**jaeger-ingester** | [hub.docker.com/r/jaegertracing/jaeger-ingester/](https://hub.docker.com/r/jaegertracing/jaeger-ingester/)
|
||||
|
||||
There are orchestration templates for running Jaeger with:
|
||||
|
||||
* Kubernetes: [github.com/jaegertracing/jaeger-kubernetes](https://github.com/jaegertracing/jaeger-kubernetes),
|
||||
* OpenShift: [github.com/jaegertracing/jaeger-openshift](https://github.com/jaegertracing/jaeger-openshift).
|
||||
|
||||
## Configuration Options
|
||||
|
||||
Jaeger binaries can be configured in a number of ways (in the order of decreasing priority):
|
||||
|
||||
* command line arguments,
|
||||
* environment variables,
|
||||
* configuration files in JSON, TOML, YAML, HCL, or Java properties formats.
|
||||
|
||||
To see the complete list of options, run the binary with `help` command. Options that are specific to a certain storage backend are only listed if the storage type is selected. For example, to see all available options in the Collector with Cassandra storage:
|
||||
|
||||
```sh
|
||||
$ docker run --rm \
|
||||
-e SPAN_STORAGE_TYPE=cassandra \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}} \
|
||||
help
|
||||
```
|
||||
|
||||
In order to provide configuration parameters via environment variables, find the respective command line option and convert its name to UPPER_SNAKE_CASE, for example:
|
||||
|
||||
Command line option | Environment variable
|
||||
-----------------------------------|-------------------------------
|
||||
`--cassandra.connections-per-host` | `CASSANDRA_CONNECTIONS_PER_HOST`
|
||||
`--metrics-backend` | `METRICS_BACKEND`
|
||||
|
||||
## Agent
|
||||
|
||||
Jaeger client libraries expect **jaeger-agent** process to run locally on each host.
|
||||
The agent exposes the following ports:
|
||||
|
||||
Port | Protocol | Function
|
||||
----- | ------- | ---
|
||||
6831 | UDP | accept [jaeger.thrift][jaeger-thrift] in `compact` Thrift protocol used by most current Jaeger clients
|
||||
6832 | UDP | accept [jaeger.thrift][jaeger-thrift] in `binary` Thrift protocol used by Node.js Jaeger client (because [thriftrw][thriftrw] npm package does not support `compact` protocol)
|
||||
5778 | HTTP | serve configs, sampling strategies
|
||||
5775 | UDP | accept [zipkin.thrift][zipkin-thrift] in `compact` Thrift protocol (deprecated; only used by very old Jaeger clients, circa 2016)
|
||||
14271 | HTTP | Healthcheck at `/` and metrics at `/metrics`
|
||||
|
||||
It can be executed directly on the host or via Docker, as follows:
|
||||
|
||||
```sh
|
||||
## make sure to expose only the ports you use in your deployment scenario!
|
||||
docker run \
|
||||
--rm \
|
||||
-p6831:6831/udp \
|
||||
-p6832:6832/udp \
|
||||
-p5778:5778/tcp \
|
||||
-p5775:5775/udp \
|
||||
jaegertracing/jaeger-agent:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
### Discovery System Integration
|
||||
|
||||
The agents can connect point to point to a single collector address, which could be
|
||||
load balanced by another infrastructure component (e.g. DNS) across multiple collectors.
|
||||
The agent can also be configured with a static list of collector addresses.
|
||||
|
||||
On Docker, a command like the following can be used:
|
||||
|
||||
```sh
|
||||
docker run \
|
||||
--rm \
|
||||
-p5775:5775/udp \
|
||||
-p6831:6831/udp \
|
||||
-p6832:6832/udp \
|
||||
-p5778:5778/tcp \
|
||||
jaegertracing/jaeger-agent:{{< currentVersion >}} \
|
||||
--reporter.grpc.host-port=jaeger-collector.jaeger-infra.svc:14250
|
||||
```
|
||||
|
||||
Or use `--reporter.tchannel.host-port=jaeger-collector.jaeger-infra.svc:14267` to use
|
||||
legacy tchannel reporter.
|
||||
|
||||
When using gRPC, you have several options for load balancing and name resolution:
|
||||
|
||||
* Single connection and no load balancing. This is the default if you specify a single `host:port`. (example: `--reporter.grpc.host-port=jaeger-collector.jaeger-infra.svc:14250`)
|
||||
* Static list of hostnames and round-robin load balancing. This is what you get with a comma-separated list of addresses. (example: `reporter.grpc.host-port=jaeger-collector1:14250,jaeger-collector2:14250,jaeger-collector3:14250`)
|
||||
* Dynamic DNS resolution and round-robin load balancing. To get this behaviour, prefix the address with `dns:///` and gRPC will attempt to resolve the hostname using SRV records (for [external load balancing](https://github.com/grpc/grpc/blob/master/doc/load-balancing.md)), TXT records (for [service configs](https://github.com/grpc/grpc/blob/master/doc/service_config.md)), and A records. Refer to the [gRPC Name Resolution docs](https://github.com/grpc/grpc/blob/master/doc/naming.md) and the [dns_resolver.go implementation](https://github.com/grpc/grpc-go/blob/master/resolver/dns/dns_resolver.go) for more info. (example: `--reporter.grpc.host-port=dns:///jaeger-collector.jaeger-infra.svc:14250`)
|
||||
|
||||
### Agent level tags
|
||||
|
||||
Jaeger supports agent level tags, that can be added to the process tags of all spans passing through the agent. This is supported through the command line flag `--jaeger.tags=key1=value1,key2=value2,...,keyn=valuen`. Tags can also be set through an environment flag like so - `--jaeger.tags=key=${envFlag:defaultValue}` - The tag value will be set to the value of the `envFlag` environment key and `defaultValue` if not set. This feature is not supported for the tchannel reporter, enabled using the flags `--collector.host-port` or `--reporter.tchannel.host-port`.
|
||||
|
||||
## Collectors
|
||||
|
||||
The collectors are stateless and thus many instances of **jaeger-collector** can be run in parallel.
|
||||
Collectors require almost no configuration, except for the location of Cassandra cluster,
|
||||
via `--cassandra.keyspace` and `--cassandra.servers` options, or the location of Elasticsearch cluster, via
|
||||
`--es.server-urls`, depending on which storage is specified. To see all command line options run
|
||||
|
||||
```sh
|
||||
go run ./cmd/collector/main.go -h
|
||||
```
|
||||
|
||||
or, if you don't have the source code
|
||||
|
||||
```sh
|
||||
docker run -it --rm jaegertracing/jaeger-collector:{{< currentVersion >}} -h
|
||||
```
|
||||
|
||||
At default settings the collector exposes the following ports:
|
||||
|
||||
Port | Protocol | Function
|
||||
----- | ------- | ---
|
||||
14267 | TChannel | used by **jaeger-agent** to send spans in jaeger.thrift format
|
||||
14250 | gRPC | used by **jaeger-agent** to send spans in model.proto format
|
||||
14268 | HTTP | can accept spans directly from clients in jaeger.thrift format over binary thrift protocol
|
||||
9411 | HTTP | can accept Zipkin spans in Thrift, JSON and Proto (disabled by default)
|
||||
14269 | HTTP | Healthcheck at `/` and metrics at `/metrics`
|
||||
|
||||
|
||||
## Storage Backends
|
||||
|
||||
Collectors require a persistent storage backend. Cassandra and Elasticsearch are the primary supported storage backends. Additional backends are [discussed here](https://github.com/jaegertracing/jaeger/issues/638).
|
||||
|
||||
The storage type can be passed via `SPAN_STORAGE_TYPE` environment variable. Valid values are `cassandra`, `elasticsearch`, `kafka` (only as a buffer), `grpc-plugin`, `badger` (only with all-in-one) and `memory` (only with all-in-one).
|
||||
|
||||
As of version 1.6.0, it's possible to use multiple storage types at the same time by providing a comma-separated list of valid types to the `SPAN_STORAGE_TYPE` environment variable.
|
||||
It's important to note that all listed storage types are used for writing, but only the first type in the list will be used for reading and archiving.
|
||||
|
||||
### Memory
|
||||
|
||||
The in-memory storage is not intended for production workloads. It's intended as a simple solution to get started quickly and
|
||||
data will be lost once the process is gone.
|
||||
|
||||
By default, there's no limit in the amount of traces stored in memory but a limit can be established by passing an
|
||||
integer value via `--memory.max-traces`.
|
||||
|
||||
### Badger - local storage
|
||||
Experimental since Jaeger 1.9
|
||||
|
||||
[Badger](https://github.com/dgraph-io/badger) is an embedded local storage, only available
|
||||
with **all-in-one** distribution. By default it acts as an ephemeral storage using a temporary filesystem.
|
||||
This can be overridden by using the `--badger.ephemeral=false` option.
|
||||
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=badger \
|
||||
-e BADGER_EPHEMERAL=false \
|
||||
-e BADGER_DIRECTORY_VALUE=/badger/data \
|
||||
-e BADGER_DIRECTORY_KEY=/badger/key \
|
||||
-v <storage_dir_on_host>:/badger \
|
||||
-p 16686:16686 \
|
||||
jaegertracing/all-in-one:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
### Cassandra
|
||||
Supported versions: 3.4+
|
||||
|
||||
Deploying Cassandra itself is out of scope for our documentation. One good
|
||||
source of documentation is the [Apache Cassandra Docs](https://cassandra.apache.org/doc/latest/).
|
||||
|
||||
#### Configuration
|
||||
##### Minimal
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=cassandra \
|
||||
-e CASSANDRA_SERVERS=<...> \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
##### All options
|
||||
To view the full list of configuration options, you can run the following command:
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=cassandra \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}} \
|
||||
--help
|
||||
```
|
||||
|
||||
#### Schema script
|
||||
|
||||
A script is provided to initialize Cassandra keyspace and schema
|
||||
using Cassandra's interactive shell [`cqlsh`][cqlsh]:
|
||||
|
||||
```sh
|
||||
MODE=test sh ./plugin/storage/cassandra/schema/create.sh | cqlsh
|
||||
```
|
||||
|
||||
For production deployment, pass `MODE=prod DATACENTER={datacenter}` arguments to the script,
|
||||
where `{datacenter}` is the name used in the Cassandra configuration / network topology.
|
||||
|
||||
The script also allows overriding TTL, keyspace name, replication factor, etc.
|
||||
Run the script without arguments to see the full list of recognized parameters.
|
||||
|
||||
#### TLS support
|
||||
|
||||
Jaeger supports TLS client to node connections as long as you've configured
|
||||
your Cassandra cluster correctly. After verifying with e.g. `cqlsh`, you can
|
||||
configure the collector and query like so:
|
||||
|
||||
```sh
|
||||
docker run \
|
||||
-e CASSANDRA_SERVERS=<...> \
|
||||
-e CASSANDRA_TLS=true \
|
||||
-e CASSANDRA_TLS_SERVER_NAME="CN-in-certificate" \
|
||||
-e CASSANDRA_TLS_KEY=<path to client key file> \
|
||||
-e CASSANDRA_TLS_CERT=<path to client cert file> \
|
||||
-e CASSANDRA_TLS_CA=<path to your CA cert file> \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
The schema tool also supports TLS. You need to make a custom cqlshrc file like
|
||||
so:
|
||||
|
||||
```
|
||||
# Creating schema in a cassandra cluster requiring client TLS certificates.
|
||||
#
|
||||
# Create a volume for the schema docker container containing four files:
|
||||
# cqlshrc: this file
|
||||
# ca-cert: the cert authority for your keys
|
||||
# client-key: the keyfile for your client
|
||||
# client-cert: the cert file matching client-key
|
||||
#
|
||||
# if there is any sort of DNS mismatch and you want to ignore server validation
|
||||
# issues, then uncomment validate = false below.
|
||||
#
|
||||
# When running the container, map this volume to /root/.cassandra and set the
|
||||
# environment variable CQLSH_SSL=--ssl
|
||||
[ssl]
|
||||
certfile = ~/.cassandra/ca-cert
|
||||
userkey = ~/.cassandra/client-key
|
||||
usercert = ~/.cassandra/client-cert
|
||||
# validate = false
|
||||
```
|
||||
|
||||
### Elasticsearch
|
||||
Supported in Jaeger since 0.6.0
|
||||
Supported versions: 5.x, 6.x, 7.x
|
||||
|
||||
Elasticsearch version is automatically retrieved from root/ping endpoint.
|
||||
Based on this version Jaeger uses compatible index mappings and Elasticsearch REST API.
|
||||
The version can be explicitly provided via `--es.version=` flag.
|
||||
|
||||
Elasticsearch does not require initialization other than
|
||||
[installing and running Elasticsearch](https://www.elastic.co/downloads/elasticsearch).
|
||||
Once it is running, pass the correct configuration values to the Jaeger collector and query service.
|
||||
|
||||
#### Configuration
|
||||
##### Minimal
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=elasticsearch \
|
||||
-e ES_SERVER_URLS=<...> \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
##### All options
|
||||
To view the full list of configuration options, you can run the following command:
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=elasticsearch \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}} \
|
||||
--help
|
||||
```
|
||||
|
||||
#### Shards and Replicas for Elasticsearch indices
|
||||
|
||||
Shards and replicas are some configuration values to take special attention to, because this is decided upon
|
||||
index creation. [This article](https://qbox.io/blog/optimizing-elasticsearch-how-many-shards-per-index) goes into
|
||||
more information about choosing how many shards should be chosen for optimization.
|
||||
|
||||
#### Upgrade Elasticsearch version
|
||||
|
||||
Elasticsearch defines wire and index compatibility versions. The index compatibility defines
|
||||
the minimal version a node can read data from. For example Elasticsearch 7 can read indices
|
||||
created by Elasticsearch 6, however it cannot read indices created by Elasticsearch 5 even
|
||||
though they use the same index mappings. Therefore upgrade from Elasticsearch 6 to 7 does not require any
|
||||
data migration. However, upgrade from Elasticsearch 5 to 7 has to be done through Elasticsearch 6 and wait
|
||||
until indices created by ES 5.x are removed or explicitly reindexed.
|
||||
|
||||
Refer to the Elasticsearch [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current//setup-upgrade.html)
|
||||
for wire and index compatibility versions. Generally this information can be retrieved from root/ping REST endpoint.
|
||||
|
||||
##### Reindex
|
||||
|
||||
Manual reindexing can be used when upgrading from Elasticsearch 5 to 7 (through Elasticsearch 6)
|
||||
without waiting until indices created by Elasticsearch 5 are removed.
|
||||
|
||||
1. Reindex all span indices to new indices with suffix `-1`:
|
||||
|
||||
```bash
|
||||
curl -ivX POST -H "Content-Type: application/json" http://localhost:9200/_reindex -d @reindex.json
|
||||
{
|
||||
"source": {
|
||||
"index": "jaeger-span-*"
|
||||
},
|
||||
"dest": {
|
||||
"index": "jaeger-span"
|
||||
},
|
||||
"script": {
|
||||
"lang": "painless",
|
||||
"source": "ctx._index = 'jaeger-span-' + (ctx._index.substring('jaeger-span-'.length(), ctx._index.length())) + '-1'"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
2. Delete indices with old mapping:
|
||||
|
||||
```bash
|
||||
curl -ivX DELETE -H "Content-Type: application/json" http://localhost:9200/jaeger-span-\*,-\*-1
|
||||
```
|
||||
|
||||
3. Create indices without `-1` suffix:
|
||||
|
||||
```bash
|
||||
curl -ivX POST -H "Content-Type: application/json" http://localhost:9200/_reindex -d @reindex.json
|
||||
{
|
||||
"source": {
|
||||
"index": "jaeger-span-*"
|
||||
},
|
||||
"dest": {
|
||||
"index": "jaeger-span"
|
||||
},
|
||||
"script": {
|
||||
"lang": "painless",
|
||||
"source": "ctx._index = 'jaeger-span-' + (ctx._index.substring('jaeger-span-'.length(), ctx._index.length() - 2))"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
4. Remove suffixed indices:
|
||||
|
||||
```bash
|
||||
curl -ivX DELETE -H "Content-Type: application/json" http://localhost:9200/jaeger-span-\*-1
|
||||
```
|
||||
|
||||
Run the commands analogically for other Jaeger indices.
|
||||
|
||||
There might exist more effective migration procedure. Please share with the community any findings.
|
||||
|
||||
### Kafka
|
||||
Supported in Jaeger since 1.6.0
|
||||
Supported Kafka versions: 0.9+
|
||||
|
||||
Kafka can be used as an intermediary buffer between collector and an actual storage.
|
||||
The collector is configured with `SPAN_STORAGE_TYPE=kafka` that makes it write all received spans
|
||||
into a Kafka topic. A new component [Ingester](#ingester), added in version 1.7.0, is used to read from
|
||||
Kafka and store spans in another storage backend (Elasticsearch or Cassandra).
|
||||
|
||||
Writing to Kafka is particularly useful for building post-processing data pipelines.
|
||||
|
||||
#### Configuration
|
||||
##### Minimal
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=kafka \
|
||||
-e KAFKA_BROKERS=<...> \
|
||||
-e KAFKA_TOPIC=<...> \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
##### All options
|
||||
To view the full list of configuration options, you can run the following command:
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=kafka \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}} \
|
||||
--help
|
||||
```
|
||||
|
||||
#### Topic & partitions
|
||||
Unless your Kafka cluster is configured to automatically create topics, you will need to create it ahead of time. You can refer to [the Kafka quickstart documentation](https://kafka.apache.org/documentation/#quickstart_createtopic) to learn how.
|
||||
|
||||
You can find more information about topics and partitions in general in the [official documentation](https://kafka.apache.org/documentation/#intro_topics). [This article](https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/) provide more details about how to choose the number of partitions.
|
||||
|
||||
### Storage plugin
|
||||
|
||||
Jaeger supports gRPC based storage plugins. For more information refer to [jaeger/plugin/storage/grpc](https://github.com/jaegertracing/jaeger/tree/master/plugin/storage/grpc)
|
||||
|
||||
Available plugins:
|
||||
|
||||
* [InfluxDB](https://github.com/influxdata/jaeger-influxdb/)
|
||||
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=grpc-plugin \
|
||||
-e GRPC_STORAGE_PLUGIN_BINARY=<...> \
|
||||
-e GRPC_STORAGE_PLUGIN_CONFIGURATION_FILE=<...> \
|
||||
jaegertracing/all-in-one:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
## Ingester
|
||||
**jaeger-ingester** is a service which reads span data from Kafka topic and writes it to another storage backend (Elasticsearch or Cassandra).
|
||||
|
||||
Port | Protocol | Function
|
||||
----- | ------- | ---
|
||||
14270 | HTTP | Healthcheck at `/` and metrics at `/metrics`
|
||||
|
||||
To view all exposed configuration options run the following command:
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=cassandra \
|
||||
jaegertracing/jaeger-ingester:{{< currentVersion >}}
|
||||
--help
|
||||
```
|
||||
|
||||
## Query Service & UI
|
||||
|
||||
**jaeger-query** serves the API endpoints and a React/Javascript UI.
|
||||
The service is stateless and is typically run behind a load balancer, such as [**NGINX**](https://www.nginx.com/).
|
||||
|
||||
At default settings the query service exposes the following port(s):
|
||||
|
||||
Port | Protocol | Function
|
||||
----- | ------- | ---
|
||||
16686 | HTTP | `/api/*` endpoints and Jaeger UI at `/`
|
||||
16687 | HTTP | Healthcheck at `/` and metrics at `/metrics`
|
||||
|
||||
### Minimal deployment example (Elasticsearch backend):
|
||||
```sh
|
||||
docker run -d --rm \
|
||||
-p 16686:16686 \
|
||||
-p 16687:16687 \
|
||||
-e SPAN_STORAGE_TYPE=elasticsearch \
|
||||
-e ES_SERVER_URLS=http://<ES_SERVER_IP>:<ES_SERVER_PORT> \
|
||||
jaegertracing/jaeger-query:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
### UI Base Path
|
||||
|
||||
The base path for all **jaeger-query** HTTP routes can be set to a non-root value, e.g. `/jaeger` would cause all UI URLs to start with `/jaeger`. This can be useful when running **jaeger-query** behind a reverse proxy.
|
||||
|
||||
The base path can be configured via the `--query.base-path` command line parameter or the `QUERY_BASE_PATH` environment variable.
|
||||
|
||||
### UI Customization and Embedding
|
||||
|
||||
Please refer to the [dedicated Frontend/UI page](../frontend-ui/).
|
||||
|
||||
## Aggregation Jobs for Service Dependencies
|
||||
|
||||
Production deployments need an external process which aggregates data and creates dependency links between services. Project [spark-dependencies](https://github.com/jaegertracing/spark-dependencies) is a Spark job which derives dependency links and stores them directly to the storage.
|
||||
|
||||
## Configuration
|
||||
|
||||
All binaries accepts command line properties and environmental variables, power by [viper](https://github.com/spf13/viper) and [cobra](https://github.com/spf13/cobra) libraries. Please refer to the [CLI Flags](../cli/) page for more information.
|
||||
|
||||
[cqlsh]: http://cassandra.apache.org/doc/latest/tools/cqlsh.html
|
||||
[zipkin-thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/zipkincore.thrift
|
||||
[jaeger-thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/jaeger.thrift
|
||||
[thriftrw]: https://www.npmjs.com/package/thriftrw
|
|
@ -1,17 +0,0 @@
|
|||
---
|
||||
title: Frequently Asked Questions
|
||||
navtitle: FAQs
|
||||
description: Answers to some frequently asked questions about Jaeger.
|
||||
weight: 11
|
||||
---
|
||||
|
||||
## Why is the Dependencies page empty?
|
||||
|
||||
The Dependencies page shows a graph of services traced by Jaeger and connections between them. When you are using `all-in-one` binary with in-memory storage, the graph is calculated on-demand from all the traces stored in memory. However, if you are using a real distributed storage like Cassandra or Elasticsearch, it is too expensive to scan all the data in the database to build the service graph. Instead, the Jaeger project provides "big data" jobs that can be used to extract the service graph data from traces:
|
||||
|
||||
* https://github.com/jaegertracing/spark-dependencies - the older Spark job that can be run periodically
|
||||
* https://github.com/jaegertracing/jaeger-analytics - the new (experimental) streaming Flink jobs that run continuously and builds the service graph in smaller time intervals
|
||||
|
||||
## Why do I not see any spans in Jaeger?
|
||||
|
||||
Please refer to the [Troubleshooting](../troubleshooting/) guide.
|
|
@ -1,57 +0,0 @@
|
|||
---
|
||||
title: Features
|
||||
hasparent: true
|
||||
---
|
||||
|
||||
Jaeger is used for monitoring and troubleshooting microservices-based distributed systems, including:
|
||||
|
||||
* Distributed context propagation
|
||||
* Distributed transaction monitoring
|
||||
* Root cause analysis
|
||||
* Service dependency analysis
|
||||
* Performance / latency optimization
|
||||
|
||||
## High Scalability
|
||||
|
||||
Jaeger backend is designed to have no single points of failure and to scale with the business needs.
|
||||
For example, any given Jaeger installation at Uber is typically processing several billion {{< tip "spans" "span" >}} per day.
|
||||
|
||||
## Native support for OpenTracing
|
||||
|
||||
Jaeger backend, Web UI, and instrumentation libraries have been designed from ground up to support the OpenTracing standard.
|
||||
|
||||
* Represent {{< tip "traces" "trace" >}} as {{< tip "directed acyclic graphs" "directed acyclic graph" >}} (not just trees) via [span references](https://github.com/opentracing/specification/blob/master/specification.md#references-between-spans)
|
||||
* Support strongly typed span _tags_ and _structured logs_
|
||||
* Support general distributed context propagation mechanism via _baggage_
|
||||
|
||||
## Multiple storage backends
|
||||
|
||||
Jaeger supports two popular open source NoSQL databases as trace storage backends: Cassandra 3.4+ and Elasticsearch 5.x/6.x/7.x.
|
||||
There are ongoing community experiments using other databases, such as ScyllaDB, InfluxDB, Amazon DynamoDB. Jaeger also ships
|
||||
with a simple in-memory storage for testing setups.
|
||||
|
||||
## Modern Web UI
|
||||
|
||||
Jaeger Web UI is implemented in Javascript using popular open source frameworks like React. Several performance
|
||||
improvements have been released in v1.0 to allow the UI to efficiently deal with large volumes of data, and to display
|
||||
{{< tip "traces" "trace" >}} with tens of thousands of {{< tip "spans" "span" >}} (e.g. we tried a trace with 80,000 spans).
|
||||
|
||||
## Cloud Native Deployment
|
||||
|
||||
Jaeger backend is distributed as a collection of Docker images. The binaries support various configuration methods,
|
||||
including command line options, environment variables, and configuration files in multiple formats (yaml, toml, etc.).
|
||||
Deployment to Kubernetes clusters is assisted by a [Kubernetes operator](https://github.com/jaegertracing/jaeger-operator), [Kubernetes templates](https://github.com/jaegertracing/jaeger-kubernetes)
|
||||
and a [Helm chart](https://github.com/kubernetes/charts/tree/master/incubator/jaeger).
|
||||
|
||||
## Observability
|
||||
|
||||
All Jaeger backend components expose [Prometheus](https://prometheus.io/) metrics by default (other metrics backends are
|
||||
also supported). Logs are written to standard out using the structured logging library [zap](https://github.com/uber-go/zap).
|
||||
|
||||
## Backwards compatibility with Zipkin
|
||||
|
||||
Although we recommend instrumenting applications with OpenTracing API and binding to Jaeger client libraries to benefit
|
||||
from advanced features not available elsewhere, if your organization has already invested in the instrumentation
|
||||
using Zipkin libraries, you do not have to rewrite all that code. Jaeger provides backwards compatibility with Zipkin
|
||||
by accepting spans in Zipkin formats (Thrift, JSON v1/v2 and Protobuf) over HTTP. Switching from Zipkin backend is just a matter
|
||||
of routing the traffic from Zipkin libraries to the Jaeger backend.
|
|
@ -1,224 +0,0 @@
|
|||
---
|
||||
title: Frontend/UI Configuration
|
||||
navtitle: Frontend/UI
|
||||
hasparent: true
|
||||
weight: 7
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
Several aspects of the UI can be configured:
|
||||
|
||||
* The Dependencies section can be enabled / configured
|
||||
* Google Analytics tracking can be enabled / configured
|
||||
* Additional menu options can be added to the global nav
|
||||
|
||||
These options can be configured by a JSON configuration file. The `--query.ui-config` command line parameter of the query service must then be set to the path to the JSON file when the query service is started.
|
||||
|
||||
An example configuration file:
|
||||
|
||||
```json
|
||||
{
|
||||
"dependencies": {
|
||||
"dagMaxNumServices": 200,
|
||||
"menuEnabled": true
|
||||
},
|
||||
"archiveEnabled": true,
|
||||
"tracking": {
|
||||
"gaID": "UA-000000-2",
|
||||
"trackErrors": true
|
||||
},
|
||||
"menu": [
|
||||
{
|
||||
"label": "About Jaeger",
|
||||
"items": [
|
||||
{
|
||||
"label": "GitHub",
|
||||
"url": "https://github.com/jaegertracing/jaeger"
|
||||
},
|
||||
{
|
||||
"label": "Docs",
|
||||
"url": "http://jaeger.readthedocs.io/en/latest/"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"linkPatterns": [{
|
||||
"type": "process",
|
||||
"key": "jaeger.version",
|
||||
"url": "https://github.com/jaegertracing/jaeger-client-java/releases/tag/#{jaeger.version}",
|
||||
"text": "Information about Jaeger release #{jaeger.version}"
|
||||
}]
|
||||
}
|
||||
```
|
||||
|
||||
### Dependencies
|
||||
|
||||
`dependencies.dagMaxNumServices` defines the maximum number of services allowed before the DAG dependency view is disabled. Default: `200`.
|
||||
|
||||
`dependencies.menuEnabled` enables (`true`) or disables (`false`) the dependencies menu button. Default: `true`.
|
||||
|
||||
### Archive Support
|
||||
|
||||
`archiveEnabled` enables (`true`) or disables (`false`) the archive traces button. Default: `false`. It requires a configuration of an archive storage in Query service. Archived traces are only accessible directly by ID, they are not searchable.
|
||||
|
||||
### Google Analytics Tracking
|
||||
|
||||
`tracking.gaID` defines the Google Analytics tracking ID. This is required for Google Analytics tracking, and setting it to a non-`null` value enables Google Analytics tracking. Default: `null`.
|
||||
|
||||
`tracking.trackErrors` enables (`true`) or disables (`false`) error tracking via Google Analytics. Errors can only be tracked if a valid Google Analytics ID is provided. For additional details on error tracking via Google Analytics see the [tracking README](https://github.com/jaegertracing/jaeger-ui/blob/c622330546afc1be59a42f874bcc1c2fadf7e69a/src/utils/tracking/README.md) in the UI repo. Default: `true`.
|
||||
|
||||
### Custom Menu Items
|
||||
|
||||
`menu` allows additional links to be added to the global nav. The additional links are right-aligned.
|
||||
|
||||
In the sample JSON config above, the configured menu will have a dropdown labeled "About Jaeger" with sub-options for "GitHub" and "Docs". The format for a link in the top right menu is as follows:
|
||||
|
||||
```json
|
||||
{
|
||||
"label": "Some text here",
|
||||
"url": "https://example.com"
|
||||
}
|
||||
```
|
||||
|
||||
Links can either be members of the `menu` Array, directly, or they can be grouped into a dropdown menu option. The format for a group of links is:
|
||||
|
||||
```json
|
||||
{
|
||||
"label": "Dropdown button",
|
||||
"items": [ ]
|
||||
}
|
||||
```
|
||||
|
||||
The `items` Array should contain one or more link configurations.
|
||||
|
||||
### Link Patterns
|
||||
|
||||
The `linkPatterns` node can be used to create links from fields displayed in the Jaeger UI.
|
||||
|
||||
Field | Description
|
||||
------|------------
|
||||
type | The metadata section in which your link will be added: process, tags, logs
|
||||
key | The name of tag/process/log attribute which value will be displayed as a link
|
||||
url | The URL where the link should point to, it can be an external site or relative path in Jaeger UI
|
||||
text | The text displayed in the tooltip for the link
|
||||
|
||||
Both `url` and `text` can be defined as templates (i.e. using `#{field-name}`) where Jaeger UI will dynamically substitute values based on tags/logs data.
|
||||
|
||||
## Embedded Mode
|
||||
|
||||
Starting with version 1.9, Jaeger UI provides an "embedded" layout mode which is intended to support integrating Jaeger UI into other applications. Currently (as of `v0`), the approach taken is to remove various UI elements from the page to make the UI better suited for space-constrained layouts.
|
||||
|
||||
The embedded mode is induced and configured via URL query parameters.
|
||||
|
||||
To enter embedded mode, the `uiEmbed=v0` query parameter and value must be added to the URL. For example, the following URL will show the trace with ID `abc123` in embedded mode:
|
||||
|
||||
```
|
||||
http://localhost:16686/trace/abc123?uiEmbed=v0
|
||||
```
|
||||
|
||||
`uiEmbed=v0` is required.
|
||||
|
||||
Further, each page supported has an <img src="/img/frontend-ui/embed-open-icon.png" style="width: 20px; height:20px;display:inline;" alt="Embed open window"> button added that will open the non-embedded page in a new tab.
|
||||
|
||||
The following pages support embedded mode:
|
||||
|
||||
* Search Page
|
||||
* Trace Page
|
||||
|
||||
### Search Page
|
||||
|
||||
To integrate the Search Trace Page to our application we have to indicate to the Jaeger UI that we want to use the embed mode with `uiEmbed=v0`.
|
||||
|
||||
For example:
|
||||
|
||||
```
|
||||
http://localhost:16686/search?
|
||||
service=my-service&
|
||||
start=1543917759557000&
|
||||
end=1543921359557000&
|
||||
limit=20&
|
||||
lookback=1h&
|
||||
maxDuration&
|
||||
minDuration&
|
||||
uiEmbed=v0
|
||||
```
|
||||
|
||||

|
||||
|
||||
#### Configuration options
|
||||
|
||||
The following query parameter can be used to configure the layout of the search page :
|
||||
|
||||
* `uiSearchHideGraph=1` - disables the display of the scatter plot above the search results
|
||||
|
||||
```
|
||||
http://localhost:16686/search?
|
||||
service=my-service&
|
||||
start=1543917759557000&
|
||||
end=1543921359557000&
|
||||
limit=20&
|
||||
lookback=1h&
|
||||
maxDuration&
|
||||
minDuration&
|
||||
uiEmbed=v0&
|
||||
uiSearchHideGraph=1
|
||||
```
|
||||
|
||||

|
||||
|
||||
### Trace Page
|
||||
|
||||
|
||||
To integrate the Trace Page to our application we have to indicate to the Jaeger UI that we want to use the embed mode with `uiEmbed=v0`.
|
||||
|
||||
For example:
|
||||
|
||||
```sh
|
||||
http://localhost:16686/trace/{trace-id}?uiEmbed=v0
|
||||
```
|
||||

|
||||
|
||||
If we have navigated to this view from the search traces page we'll have a button to go back to the results page.
|
||||
|
||||

|
||||
|
||||
#### Configuration options
|
||||
|
||||
The following query parameters can be used to configure the layout of the trace page :
|
||||
|
||||
* `uiTimelineCollapseTitle=1` causes the trace header to start out collapsed, which hides the summary and the minimap.
|
||||
|
||||
```
|
||||
http://localhost:16686/trace/{trace-id}?
|
||||
uiEmbed=v0&
|
||||
uiTimelineCollapseTitle=1
|
||||
```
|
||||

|
||||
|
||||
* `uiTimelineHideMinimap=1` removes the minimap, entirely, regardless of whether the trace header is expanded or not.
|
||||
|
||||
```
|
||||
http://localhost:16686/trace/{trace-id}?
|
||||
uiEmbed=v0&
|
||||
uiTimelineHideMinimap=1
|
||||
```
|
||||

|
||||
|
||||
* `uiTimelineHideSummary=1` - removes the trace summary information (number of services, etc.) entirely, regardless of whether the trace header is expanded or not.
|
||||
|
||||
```
|
||||
http://localhost:16686/trace/{trace-id}?
|
||||
uiEmbed=v0&
|
||||
uiTimelineHideSummary=1
|
||||
```
|
||||

|
||||
|
||||
We can also combine the options:
|
||||
```
|
||||
http://localhost:16686/trace/{trace-id}?
|
||||
uiEmbed=v0&
|
||||
uiTimelineHideMinimap=1&
|
||||
uiTimelineHideSummary=1
|
||||
```
|
||||

|
|
@ -1,129 +0,0 @@
|
|||
---
|
||||
title: Getting Started
|
||||
description: Get up and running with Jaeger in your local environment
|
||||
weight: 2
|
||||
---
|
||||
|
||||
## Instrumentation
|
||||
|
||||
Your applications must be instrumented before they can send tracing data to Jaeger backend. Check the [Client Libraries](../client-libraries) section for information about how to use the OpenTracing API and how to initialize and configure Jaeger tracers.
|
||||
|
||||
## All in One
|
||||
|
||||
All-in-one is an executable designed for quick local testing, launches the Jaeger UI, collector, query, and agent, with an in memory storage component.
|
||||
|
||||
The simplest way to start the all-in-one is to use the pre-built image published to DockerHub (a single command line).
|
||||
|
||||
```bash
|
||||
$ docker run -d --name jaeger \
|
||||
-e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
|
||||
-p 5775:5775/udp \
|
||||
-p 6831:6831/udp \
|
||||
-p 6832:6832/udp \
|
||||
-p 5778:5778 \
|
||||
-p 16686:16686 \
|
||||
-p 14268:14268 \
|
||||
-p 9411:9411 \
|
||||
jaegertracing/all-in-one:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
Or run the `jaeger-all-in-one(.exe)` executable from the [binary distribution archives][download]:
|
||||
|
||||
```bash
|
||||
$ jaeger-all-in-one --collector.zipkin.http-port=9411
|
||||
```
|
||||
|
||||
You can then navigate to `http://localhost:16686` to access the Jaeger UI.
|
||||
|
||||
The container exposes the following ports:
|
||||
|
||||
Port | Protocol | Component | Function
|
||||
----- | ------- | --------- | ---
|
||||
5775 | UDP | agent | accept `zipkin.thrift` over compact thrift protocol (deprecated, used by legacy clients only)
|
||||
6831 | UDP | agent | accept `jaeger.thrift` over compact thrift protocol
|
||||
6832 | UDP | agent | accept `jaeger.thrift` over binary thrift protocol
|
||||
5778 | HTTP | agent | serve configs
|
||||
16686 | HTTP | query | serve frontend
|
||||
14268 | HTTP | collector | accept `jaeger.thrift` directly from clients
|
||||
14250 | HTTP | collector | accept `model.proto`
|
||||
9411 | HTTP | collector | Zipkin compatible endpoint (optional)
|
||||
|
||||
|
||||
## Kubernetes and OpenShift
|
||||
|
||||
* Kubernetes templates: https://github.com/jaegertracing/jaeger-kubernetes
|
||||
* Kubernetes Operator: https://github.com/jaegertracing/jaeger-operator
|
||||
* OpenShift templates: https://github.com/jaegertracing/jaeger-openshift
|
||||
|
||||
## Sample App: HotROD
|
||||
|
||||
HotROD (Rides on Demand) is a demo application that consists of several microservices and
|
||||
illustrates the use of the [OpenTracing API](http://opentracing.io).
|
||||
A tutorial / walkthrough is available in the blog post:
|
||||
[Take OpenTracing for a HotROD ride][hotrod-tutorial].
|
||||
|
||||
It can be run standalone, but requires Jaeger backend to view the traces.
|
||||
|
||||
### Features
|
||||
|
||||
- Discover architecture of the whole system via data-driven dependency
|
||||
diagram.
|
||||
- View request timeline and errors; understand how the app works.
|
||||
- Find sources of latency and lack of concurrency.
|
||||
- Highly contextualized logging.
|
||||
- Use baggage propagation to:
|
||||
|
||||
- Diagnose inter-request contention (queueing).
|
||||
- Attribute time spent in a service.
|
||||
|
||||
- Use open source libraries with OpenTracing integration to get
|
||||
vendor-neutral instrumentation for free.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- You need Go 1.11 or higher installed on your machine to run from source.
|
||||
- Requires a [running Jaeger backend](#all-in-one) to view the traces.
|
||||
|
||||
### Running
|
||||
|
||||
#### From Source
|
||||
|
||||
```bash
|
||||
mkdir -p $GOPATH/src/github.com/jaegertracing
|
||||
cd $GOPATH/src/github.com/jaegertracing
|
||||
git clone git@github.com:jaegertracing/jaeger.git jaeger
|
||||
cd jaeger
|
||||
make install
|
||||
go run ./examples/hotrod/main.go all
|
||||
```
|
||||
#### From docker
|
||||
|
||||
```bash
|
||||
$ docker run --rm -it \
|
||||
--link jaeger \
|
||||
-p8080-8083:8080-8083 \
|
||||
-e JAEGER_AGENT_HOST="jaeger" \
|
||||
jaegertracing/example-hotrod:{{< currentVersion >}} \
|
||||
all
|
||||
```
|
||||
|
||||
#### From binary distribution
|
||||
|
||||
Run `example-hotrod(.exe)` executable from the [binary distribution archives][download]:
|
||||
```bash
|
||||
$ example-hotrod all
|
||||
```
|
||||
|
||||
Then navigate to `http://localhost:8080`.
|
||||
|
||||
|
||||
## Migrating from Zipkin
|
||||
|
||||
Collector service exposes Zipkin compatible REST API `/api/v1/spans` which accepts both Thrift and JSON. Also there is `/api/v2/spans` for JSON and Proto.
|
||||
By default it's disabled. It can be enabled with `--collector.zipkin.http-port=9411`.
|
||||
|
||||
Zipkin [Thrift](https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/zipkincore.thrift) IDL and Zipkin [Proto](https://github.com/jaegertracing/jaeger-idl/blob/master/proto/zipkin.proto) IDL files can be found in [jaegertracing/jaeger-idl](https://github.com/jaegertracing/jaeger-idl) repository.
|
||||
They're compatible with [openzipkin/zipkin-api](https://github.com/openzipkin/zipkin-api) [Thrift](https://github.com/openzipkin/zipkin-api/blob/master/thrift/zipkinCore.thrift) and [Proto](https://github.com/openzipkin/zipkin-api/blob/master/zipkin.proto).
|
||||
|
||||
[hotrod-tutorial]: https://medium.com/@YuriShkuro/take-opentracing-for-a-hotrod-ride-f6e3141f7941
|
||||
[download]: ../../../download/
|
|
@ -1,816 +0,0 @@
|
|||
---
|
||||
title: Operator for Kubernetes
|
||||
hasparent: true
|
||||
---
|
||||
|
||||
# Understanding Operators
|
||||
|
||||
The Jaeger Operator is an implementation of a [Kubernetes Operator](https://coreos.com/operators/). Operators are pieces of software that ease the operational complexity of running another piece of software. More technically, _Operators_ are a method of packaging, deploying, and managing a Kubernetes application.
|
||||
|
||||
A Kubernetes application is an application that is both deployed on Kubernetes and managed using the Kubernetes APIs and `kubectl` (kubernetes) or `oc` (OKD) tooling. To be able to make the most of Kubernetes, you need a set of cohesive APIs to extend in order to service and manage your apps that run on Kubernetes. Think of Operators as the runtime that manages this type of app on Kubernetes.
|
||||
|
||||
# Installing the Operator
|
||||
|
||||
{{< info >}}
|
||||
The Jaeger Operator version tracks one version of the Jaeger components (Query, Collector, Agent). When a new version of the Jaeger components is released, a new version of the operator will be released that understands how running instances of the previous version can be upgraded to the new version.
|
||||
{{< /info >}}
|
||||
|
||||
## Installing the Operator on Kubernetes
|
||||
|
||||
The following instructions will create the `observability` namespace and install the Jaeger Operator.
|
||||
|
||||
{{< info >}}
|
||||
Make sure your `kubectl` command is properly configured to talk to a valid Kubernetes cluster. If you don't have a cluster, you can create one locally using [`minikube`](https://kubernetes.io/docs/tasks/tools/install-minikube/).
|
||||
{{< /info >}}
|
||||
|
||||
To install the operator, run:
|
||||
<!--TODO - Does Kubernetes have privileged users? Needs to be run as a system:admin on OKD/OpenShift.-->
|
||||
|
||||
```bash
|
||||
kubectl create namespace observability # <1>
|
||||
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/crds/jaegertracing.io_jaegers_crd.yaml # <2>
|
||||
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/service_account.yaml
|
||||
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role.yaml
|
||||
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role_binding.yaml
|
||||
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/operator.yaml
|
||||
```
|
||||
<1> This creates the namespace used by default in the deployment files. If you want to install the Jaeger operator in a different namespace, you must edit the deployment files to change `observability` to the desired namespace value.
|
||||
|
||||
<2> This installs the "Custom Resource Definition" for the `apiVersion: jaegertracing.io/v1`
|
||||
|
||||
At this point, there should be a `jaeger-operator` deployment available. You can view it by running the following command:
|
||||
|
||||
```bash
|
||||
$ kubectl get deployment jaeger-operator -n observability
|
||||
|
||||
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
|
||||
jaeger-operator 1 1 1 1 48s
|
||||
```
|
||||
|
||||
The operator is now ready to create Jaeger instances.
|
||||
|
||||
## Installing the Operator on OKD/OpenShift
|
||||
|
||||
<!-- TODO: Add instructions for installing via the operatorhub? -->
|
||||
|
||||
The instructions from the previous section also work for installing the operator on OKD or OpenShift. Make sure you are logged in as a privileged user, when you install the role based acces control (RBAC) rules, the custom resource definition, and the operator.
|
||||
|
||||
```bash
|
||||
oc login -u <privileged user>
|
||||
|
||||
oc new-project observability # <1>
|
||||
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/crds/jaegertracing.io_jaegers_crd.yaml # <2>
|
||||
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/service_account.yaml
|
||||
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role.yaml
|
||||
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role_binding.yaml
|
||||
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/operator.yaml
|
||||
```
|
||||
<1> This creates the namespace used by default in the deployment files. If you want to install the Jaeger operator in a different namespace, you must edit the deployment files to change `observability` to the desired namespace value.
|
||||
|
||||
<2> This installs the "Custom Resource Definition" for the `apiVersion: jaegertracing.io/v1`
|
||||
|
||||
Once the operator is installed, grant the role `jaeger-operator` to users who should be able to install individual Jaeger instances. The following example creates a role binding allowing the user `developer` to create Jaeger instances:
|
||||
|
||||
```bash
|
||||
oc create \
|
||||
rolebinding developer-jaeger-operator \
|
||||
--role=jaeger-operator \
|
||||
--user=developer
|
||||
```
|
||||
|
||||
After the role is granted, switch back to a non-privileged user.
|
||||
|
||||
# Quick Start - Deploying the AllInOne image
|
||||
|
||||
The simplest possible way to create a Jaeger instance is by creating a YAML file like the following example. This will install the default AllInOne strategy, which deploys the "all-in-one" image (agent, collector, query, ingester, Jaeger UI) in a single pod, using in-memory storage by default.
|
||||
|
||||
{{< info >}}
|
||||
This default strategy is intended for development, testing, and demo purposes, not for production.
|
||||
{{< /info >}}
|
||||
|
||||
```yaml
|
||||
apiVersion: jaegertracing.io/v1
|
||||
kind: Jaeger
|
||||
metadata:
|
||||
name: simplest
|
||||
```
|
||||
|
||||
The YAML file can then be used with `kubectl`:
|
||||
<!-- TODO - Add OKD commands and tabs shortcode. -->
|
||||
|
||||
```bash
|
||||
kubectl apply -f simplest.yaml
|
||||
```
|
||||
|
||||
In a few seconds, a new in-memory all-in-one instance of Jaeger will be available, suitable for quick demos and development purposes. To check the instances that were created, list the `jaeger` objects:
|
||||
|
||||
```bash
|
||||
$ kubectl get jaegers
|
||||
NAME CREATED AT
|
||||
simplest 28s
|
||||
```
|
||||
|
||||
To get the pod name, query for the pods belonging to the `simplest` Jaeger instance:
|
||||
|
||||
|
||||
```bash
|
||||
$ kubectl get pods -l app.kubernetes.io/instance=simplest
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
simplest-6499bb6cdd-kqx75 1/1 Running 0 2m
|
||||
```
|
||||
|
||||
Similarly, the logs can be queried either from the pod directly using the pod name obtained from the previous example, or from all pods belonging to our instance:
|
||||
|
||||
```bash
|
||||
$ kubectl logs -l app.kubernetes.io/instance=simplest
|
||||
...
|
||||
{"level":"info","ts":1535385688.0951214,"caller":"healthcheck/handler.go:133","msg":"Health Check state change","status":"ready"}
|
||||
```
|
||||
|
||||
{{< info >}}
|
||||
On OKD/OpenShift the container name must be specified.
|
||||
{{< /info >}}
|
||||
|
||||
```bash
|
||||
$ kubectl logs -l app.kubernetes.io/instance=simplest -c jaeger
|
||||
...
|
||||
{"level":"info","ts":1535385688.0951214,"caller":"healthcheck/handler.go:133","msg":"Health Check state change","status":"ready"}
|
||||
```
|
||||
|
||||
# Deployment Strategies
|
||||
|
||||
When you create a Jaeger instance, it is associated with a strategy. The strategy is defined in the custom resource file, and determines the architecture to be used for the Jaeger backend. The default strategy is `allInOne`. The other possible values are `production` and `streaming`.
|
||||
|
||||
The available strategies are described in the following sections.
|
||||
|
||||
## AllInOne (Default) strategy
|
||||
|
||||
This strategy is intended for development, testing, and demo purposes.
|
||||
|
||||
The main backend components, agent, collector and query service, are all packaged into a single executable which is configured (by default) to use in-memory storage.
|
||||
|
||||
## Production strategy
|
||||
|
||||
The `production` strategy is intended (as the name suggests) for production environments, where long term storage of trace data is important, as well as a more scalable and highly available architecture is required. Each of the backend components is therefore separately deployed.
|
||||
|
||||
The agent can be injected as a sidecar on the instrumented application or as a daemonset.
|
||||
|
||||
The query and collector services are configured with a supported storage type - currently Cassandra or Elasticsearch. Multiple instances of each of these components can be provisioned as required for performance and resilience purposes.
|
||||
|
||||
The main additional requirement is to provide the details of the storage type and options, for example:
|
||||
|
||||
```yaml
|
||||
storage:
|
||||
type: elasticsearch
|
||||
options:
|
||||
es:
|
||||
server-urls: http://elasticsearch:9200
|
||||
```
|
||||
|
||||
## Streaming strategy
|
||||
|
||||
The `streaming` strategy is designed to augment the `production` strategy by providing a streaming capability that effectively sits between the collector and the backend storage (Cassandra or Elasticsearch). This provides the benefit of reducing the pressure on the backend storage, under high load situations, and enables other trace post-processing capabilities to tap into the real time span data directly from the streaming platform (Kafka).
|
||||
|
||||
The only additional information required is to provide the details for accessing the Kafka platform, which is configured in the `collector` component (as producer) and `ingester` component (as consumer):
|
||||
|
||||
```yaml
|
||||
apiVersion: jaegertracing.io/v1
|
||||
kind: Jaeger
|
||||
metadata:
|
||||
name: simple-streaming
|
||||
spec:
|
||||
strategy: streaming
|
||||
collector:
|
||||
options:
|
||||
kafka: # <1>
|
||||
producer:
|
||||
topic: jaeger-spans
|
||||
brokers: my-cluster-kafka-brokers.kafka:9092
|
||||
ingester:
|
||||
options:
|
||||
kafka: # <1>
|
||||
consumer:
|
||||
topic: jaeger-spans
|
||||
brokers: my-cluster-kafka-brokers.kafka:9092
|
||||
ingester:
|
||||
deadlockInterval: 0 # <2>
|
||||
storage:
|
||||
type: elasticsearch
|
||||
options:
|
||||
es:
|
||||
server-urls: http://elasticsearch:9200
|
||||
```
|
||||
<1> Identifies the Kafka configuration used by the collector, to produce the messages, and the ingester to consume the messages.
|
||||
|
||||
<2> The deadlock interval can be disabled to avoid the ingester being terminated when no messages arrive within the default 1 minute period
|
||||
|
||||
{{< info >}}
|
||||
A Kafka environment can be configured using [Strimzi's Kafka operator](https://strimzi.io/).
|
||||
{{< /info >}}
|
||||
|
||||
# Understanding Custom Resource Definitions
|
||||
|
||||
In the Kubernetes API, a resource is an endpoint that stores a collection of API objects of a certain kind. For example, the built-in Pods resource contains a collection of Pod objects. A _Custom Resource Definition_ (CRD) object defines a new, unique object `Kind` in the cluster and lets the Kubernetes API server handle its entire lifecycle.
|
||||
|
||||
To create _Custom Resource_ (CR) objects, cluster administrators must first create a Custom Resource Definition (CRD). The CRDs allow cluster users to create CRs to add the new resource types into their projects. An Operator watches for custom resource objects to be created, and when it sees a custom resource being created, it creates the application based on the parameters defined in the custom resource object.
|
||||
|
||||
{{< info >}}
|
||||
While only cluster administrators can create CRDs, developers can create the CR from an existing CRD if they have read and write permission to it.
|
||||
{{< /info >}}
|
||||
|
||||
<!--
|
||||
## Jaeger Custom Resource Parameters
|
||||
|
||||
TODO Create a TABLE with all the parameters, descriptions/notes, valid values, and defaults.
|
||||
Figure out if we can generate the options? Can we filter them in any way?
|
||||
https://github.com/jaegertracing/jaeger/issues/1537
|
||||
https://github.com/jaegertracing/documentation/issues/250-->
|
||||
|
||||
For reference, here's how you can create a more complex all-in-one instance:
|
||||
|
||||
```yaml
|
||||
apiVersion: jaegertracing.io/v1
|
||||
kind: Jaeger
|
||||
metadata:
|
||||
name: my-jaeger
|
||||
spec:
|
||||
strategy: allInOne # <1>
|
||||
allInOne:
|
||||
image: jaegertracing/all-in-one:latest # <2>
|
||||
options: # <3>
|
||||
log-level: debug # <4>
|
||||
storage:
|
||||
type: memory # <5>
|
||||
options: # <6>
|
||||
memory: # <7>
|
||||
max-traces: 100000
|
||||
ingress:
|
||||
enabled: false # <8>
|
||||
agent:
|
||||
strategy: DaemonSet # <9>
|
||||
annotations:
|
||||
scheduler.alpha.kubernetes.io/critical-pod: "" # <10>
|
||||
```
|
||||
|
||||
<1> The default strategy is `allInOne`. The other possible values are `production` and `streaming`.
|
||||
|
||||
<2> The image to use, in a regular Docker syntax.
|
||||
|
||||
<3> The (non-storage related) options to be passed verbatim to the underlying binary. Refer to the Jaeger documentation and/or to the `--help` option from the related binary for all the available options.
|
||||
|
||||
<4> The option is a simple `key: value` map. In this case, we want the option `--log-level=debug` to be passed to the binary.
|
||||
|
||||
<5> The storage type to be used. By default it will be `memory`, but can be any other supported storage type (Cassandra, Elasticsearch, Kafka).
|
||||
|
||||
<6> All storage related options should be placed here, rather than under the 'allInOne' or other component options.
|
||||
|
||||
<7> Some options are namespaced and we can alternatively break them into nested objects. We could have specified `memory.max-traces: 100000`.
|
||||
|
||||
<8> By default, an ingress object is created for the query service. It can be disabled by setting its `enabled` option to `false`. If deploying on OpenShift, this will be represented by a Route object.
|
||||
|
||||
<9> By default, the operator assumes that agents are deployed as sidecars within the target pods. Specifying the strategy as "DaemonSet" changes that and makes the operator deploy the agent as DaemonSet. Note that your tracer client will probably have to override the "JAEGER_AGENT_HOST" environment variable to use the node's IP.
|
||||
|
||||
<10> Define annotations to be applied to all deployments (not services). These can be overridden by annotations defined on the individual components.
|
||||
|
||||
You can view example custom resources for different Jaeger configurations [on GitHub](https://github.com/jaegertracing/jaeger-operator/tree/master/examples).
|
||||
|
||||
# Configuring the Custom Resource
|
||||
|
||||
<!--TODO
|
||||
esIndexCleaner
|
||||
Spark dependencies
|
||||
-->
|
||||
|
||||
You can use the simplest example (shown above) and create a Jaeger instance using the defaults, or you can create your own custom resource file.
|
||||
|
||||
## Storage options
|
||||
|
||||
### Cassandra storage
|
||||
|
||||
When the storage type is set to Cassandra, the operator will automatically create a batch job that creates the required schema for Jaeger to run. This batch job will block the Jaeger installation, so that it starts only after the schema is successfuly created. The creation of this batch job can be disabled by setting the `enabled` property to `false`:
|
||||
|
||||
```yaml
|
||||
apiVersion: jaegertracing.io/v1
|
||||
kind: Jaeger
|
||||
metadata:
|
||||
name: cassandra-without-create-schema
|
||||
spec:
|
||||
strategy: allInOne
|
||||
storage:
|
||||
type: cassandra
|
||||
cassandraCreateSchema:
|
||||
enabled: false # <1>
|
||||
```
|
||||
<1> Defaults to `true`
|
||||
|
||||
Further aspects of the batch job can be configured as well. An example with all the possible options is shown below:
|
||||
|
||||
```yaml
|
||||
apiVersion: jaegertracing.io/v1
|
||||
kind: Jaeger
|
||||
metadata:
|
||||
name: cassandra-with-create-schema
|
||||
spec:
|
||||
strategy: allInOne # <1>
|
||||
storage:
|
||||
type: cassandra
|
||||
options: # <2>
|
||||
cassandra:
|
||||
servers: cassandra
|
||||
keyspace: jaeger_v1_datacenter3
|
||||
cassandraCreateSchema: # <3>
|
||||
datacenter: "datacenter3"
|
||||
mode: "test"
|
||||
```
|
||||
<1> The same works for `production` and `streaming`.
|
||||
|
||||
<2> These options are for the regular Jaeger components, like `collector` and `query`.
|
||||
|
||||
<3> The options for the `create-schema` job.
|
||||
|
||||
{{< info >}}
|
||||
The default create-schema job uses `MODE=prod`, which implies a replication factor of `2`, using `NetworkTopologyStrategy` as the class, effectively meaning that at least 3 nodes are required in the Cassandra cluster. If a `SimpleStrategy` is desired, set the mode to `test`, which then sets the replication factor of `1`. Refer to the [create-schema script](https://github.com/jaegertracing/jaeger/blob/master/plugin/storage/cassandra/schema/create.sh) for more details.
|
||||
{{< /info >}}
|
||||
|
||||
### Elasticsearch storage
|
||||
|
||||
Under some circumstances, the Jaeger Operator can make use of the [Elasticsearch Operator](https://github.com/openshift/elasticsearch-operator) to provision a suitable Elasticsearch cluster.
|
||||
|
||||
{{< warning >}}
|
||||
This feature is only tested on OpenShift clusters. Spark dependencies are not supported with this feature [Issue #294](https://github.com/jaegertracing/jaeger-operator/issues/294).
|
||||
{{< /warning >}}
|
||||
|
||||
When there are no `es.server-urls` options as part of a Jaeger `production` instance and `elasticsearch` is set as the storage type, the Jaeger Operator creates an Elasticsearch cluster via the Elasticsearch Operator by creating a Custom Resource based on the configuration provided in storage section. The Elasticsearch cluster is meant to be dedicated for a single Jaeger instance.
|
||||
|
||||
The self-provision of an Elasticsearch cluster can be disabled by setting the flag `--es-provision` to `false`. The default value is `auto`, which will make the Jaeger Operator query the Kubernetes cluster for its ability to handle a `Elasticsearch` custom resource. This is usually set by the Elasticsearch Operator during its installation process, so, if the Elasticsearch Operator is expected to run *after* the Jaeger Operator, the flag can be set to `true`.
|
||||
|
||||
{{< danger >}}
|
||||
At the moment there can be only one Jaeger with self-provisioned Elasticsearch instance per namespace.
|
||||
{{< /danger >}}
|
||||
|
||||
#### Elasticsearch index cleaner job
|
||||
|
||||
When using `elasticsearch` storage by default a job is created to clean old traces from it, the options for it are listed below so you can configure it to your use case
|
||||
|
||||
```yaml
|
||||
storage:
|
||||
type: elasticsearch
|
||||
esIndexCleaner:
|
||||
enabled: false # turn the job deployment on and off
|
||||
numberOfDays: 7 # number of days to wait before deleting a record
|
||||
schedule: "55 23 * * *" # cron expression for it to run
|
||||
image: jaegertracing/jaeger-es-index-cleaner # image of the job
|
||||
```
|
||||
|
||||
## Deriving dependencies
|
||||
|
||||
The processing to derive dependencies will collect spans from storage, analyzes links between services and store them for later presentation in the UI.
|
||||
This job can only be used with the `production` strategy and storage type `cassandra` or `elasticsearch`.
|
||||
|
||||
```yaml
|
||||
storage:
|
||||
type: elasticsearch
|
||||
dependencies:
|
||||
enabled: true # turn the job deployment on and off
|
||||
schedule: "55 23 * * *" # cron expression for it to run
|
||||
sparkMaster: # spark master connection string, when empty spark runs in embedded local mode
|
||||
```
|
||||
|
||||
The connection configuration to storage is derived from storage options.
|
||||
|
||||
## Auto-injecting Jaeger Agent Sidecars
|
||||
|
||||
The operator can inject Jaeger Agent sidecars in `Deployment` workloads, provided that the deployment has the annotation `sidecar.jaegertracing.io/inject` with a suitable value. The values can be either `"true"` (as string), or the Jaeger instance name, as returned by `kubectl get jaegers`. When `"true"` is used, there should be exactly *one* Jaeger instance for the same namespace as the deployment, otherwise, the operator can't figure out automatically which Jaeger instance to use.
|
||||
|
||||
The following snippet shows a simple application that will get a sidecar injected, with the Jaeger Agent pointing to the single Jaeger instance available in the same namespace:
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: myapp
|
||||
annotations:
|
||||
"sidecar.jaegertracing.io/inject": "true" # <1>
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app: myapp
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: myapp
|
||||
spec:
|
||||
containers:
|
||||
- name: myapp
|
||||
image: acme/myapp:myversion
|
||||
```
|
||||
<1> Either `"true"` (as string) or the Jaeger instance name.
|
||||
|
||||
A complete sample deployment is available at [`deploy/examples/business-application-injected-sidecar.yaml`](https://github.com/jaegertracing/jaeger-operator/blob/master/examples/business-application-injected-sidecar.yaml).
|
||||
|
||||
When the sidecar is injected, the Jaeger Agent can be accessed at its default location on `localhost`.
|
||||
|
||||
## Installing the Agent as DaemonSet
|
||||
By default, the Operator expects the agents to be deployed as sidecars to the target applications. This is convenient for several purposes, like in a multi-tenant scenario or to have better load balancing, but there are scenarios where you might want to install the agent as a `DaemonSet`. In that case, specify the Agent's strategy to `DaemonSet`, as follows:
|
||||
|
||||
```yaml
|
||||
apiVersion: jaegertracing.io/v1
|
||||
kind: Jaeger
|
||||
metadata:
|
||||
name: my-jaeger
|
||||
spec:
|
||||
agent:
|
||||
strategy: DaemonSet
|
||||
```
|
||||
|
||||
{{< danger >}}
|
||||
If you attempt to install two Jaeger instances on the same cluster with `DaemonSet` as the strategy, only *one* will end up deploying a `DaemonSet`, as the agent is required to bind to well-known ports on the node. Because of that, the second daemon set will fail to bind to those ports.
|
||||
{{< /danger >}}
|
||||
|
||||
Your tracer client will then most likely need to be told where the agent is located. This is usually done by setting the environment variable `JAEGER_AGENT_HOST` to the value of the Kubernetes node's IP, for example:
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: myapp
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app: myapp
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: myapp
|
||||
spec:
|
||||
containers:
|
||||
- name: myapp
|
||||
image: acme/myapp:myversion
|
||||
env:
|
||||
- name: JAEGER_AGENT_HOST
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
fieldPath: status.hostIP
|
||||
```
|
||||
|
||||
### OpenShift
|
||||
|
||||
In OpenShift, a `HostPort` can only be set when a special security context is set. A separate service account can be used by the Jaeger Agent with the permission to bind to `HostPort`, as follows:
|
||||
|
||||
```bash
|
||||
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/examples/openshift/hostport-scc-daemonset.yaml # <1>
|
||||
oc new-project myappnamespace
|
||||
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/examples/openshift/service_account_jaeger-agent-daemonset.yaml # <2>
|
||||
oc adm policy add-scc-to-user daemonset-with-hostport -z jaeger-agent-daemonset # <3>
|
||||
oc apply -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/examples/openshift/agent-as-daemonset.yaml # <4>
|
||||
```
|
||||
<1> The `SecurityContextConstraints` with the `allowHostPorts` policy
|
||||
|
||||
<2> The `ServiceAccount` to be used by the Jaeger Agent
|
||||
|
||||
<3> Adds the security policy to the service account
|
||||
|
||||
<4> Creates the Jaeger Instance using the `serviceAccount` created in the steps above
|
||||
|
||||
{{< warning >}}
|
||||
Without such a policy, errors like the following will prevent a `DaemonSet` to be created: `Warning FailedCreate 4s (x14 over 45s) daemonset-controller Error creating: pods "agent-as-daemonset-agent-daemonset-" is forbidden: unable to validate against any security context constraint: [spec.containers[0].securityContext.containers[0].hostPort: Invalid value: 5775: Host ports are not allowed to be used`
|
||||
{{< /warning >}}
|
||||
|
||||
After a few seconds, the `DaemonSet` should be up and running:
|
||||
|
||||
```bash
|
||||
$ oc get daemonset agent-as-daemonset-agent-daemonset
|
||||
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE
|
||||
agent-as-daemonset-agent-daemonset 1 1 1 1 1
|
||||
```
|
||||
|
||||
## Secrets Support
|
||||
|
||||
The Operator supports passing secrets to the Collector, Query and All-In-One deployments. This can be used for example, to pass credentials (username/password) to access the underlying storage backend (for example: Elasticsearch).
|
||||
The secrets are available as environment variables in the (Collector/Query/All-In-One) nodes.
|
||||
|
||||
```yaml
|
||||
storage:
|
||||
type: elasticsearch
|
||||
options:
|
||||
es:
|
||||
server-urls: http://elasticsearch:9200
|
||||
secretName: jaeger-secrets
|
||||
```
|
||||
|
||||
The secret itself would be managed outside of the `jaeger-operator` custom resource.
|
||||
|
||||
## Configuring the UI
|
||||
|
||||
Information on various configuration options for the UI can be found [here](../frontend-ui/#configuration), defined in json format.
|
||||
|
||||
To apply UI configuration changes within the Custom Resource, the same information can be included in yaml format as shown below:
|
||||
|
||||
```yaml
|
||||
ui:
|
||||
options:
|
||||
dependencies:
|
||||
menuEnabled: false
|
||||
tracking:
|
||||
gaID: UA-000000-2
|
||||
menu:
|
||||
- label: "About Jaeger"
|
||||
items:
|
||||
- label: "Documentation"
|
||||
url: "https://www.jaegertracing.io/docs/latest"
|
||||
linkPatterns:
|
||||
- type: "logs"
|
||||
key: "customer_id"
|
||||
url: /search?limit=20&lookback=1h&service=frontend&tags=%7B%22customer_id%22%3A%22#{customer_id}%22%7D
|
||||
text: "Search for other traces for customer_id=#{customer_id}"
|
||||
```
|
||||
|
||||
## Defining Sampling Strategies
|
||||
|
||||
{{< info >}}
|
||||
This is not relevant if a trace was started by the Istio proxy as the sampling decision is made there. And the Jaeger sampling decisions are only relevant when you are using the Jaeger tracer (client).
|
||||
{{< /info >}}
|
||||
|
||||
The operator can be used to define sampling strategies that will be supplied to tracers that have been configured to use a remote sampler:
|
||||
|
||||
```yaml
|
||||
apiVersion: jaegertracing.io/v1
|
||||
kind: Jaeger
|
||||
metadata:
|
||||
name: with-sampling
|
||||
spec:
|
||||
strategy: allInOne
|
||||
sampling:
|
||||
options:
|
||||
default_strategy:
|
||||
type: probabilistic
|
||||
param: 0.5
|
||||
```
|
||||
|
||||
This example defines a default sampling strategy that is probabilistic, with a 50% chance of the trace instances being sampled.
|
||||
|
||||
Refer to the Jaeger documentation on [Collector Sampling Configuration](https://www.jaegertracing.io/docs/latest/sampling/#collector-sampling-configuration) to see how service and endpoint sampling can be configured. The JSON representation described in that documentation can be used in the operator by converting to YAML.
|
||||
|
||||
## Finer grained configuration
|
||||
|
||||
The custom resource can be used to define finer grained Kubernetes configuration applied to all Jaeger components or at the individual component level.
|
||||
|
||||
When a common definition (for all Jaeger components) is required, it is defined under the `spec` node. When the definition relates to an individual component, it is placed under the `spec/<component>` node.
|
||||
|
||||
The types of supported configuration include:
|
||||
|
||||
* [affinity](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity) to determine which nodes a pod can be allocated to
|
||||
|
||||
* [annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/)
|
||||
|
||||
* [labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/)
|
||||
|
||||
* [resources](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container) to limit cpu and memory
|
||||
|
||||
* [tolerations](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/) in conjunction with `taints` to enable pods to avoid being repelled from a node
|
||||
|
||||
* [volumes](https://kubernetes.io/docs/concepts/storage/volumes/) and volume mounts
|
||||
|
||||
* [serviceAccount](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/) to run each component with separate identity
|
||||
|
||||
* [securityContext](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/) to define privileges of running components
|
||||
|
||||
```yaml
|
||||
apiVersion: jaegertracing.io/v1
|
||||
kind: Jaeger
|
||||
metadata:
|
||||
name: simple-prod
|
||||
spec:
|
||||
strategy: production
|
||||
storage:
|
||||
type: elasticsearch
|
||||
options:
|
||||
es:
|
||||
server-urls: http://elasticsearch:9200
|
||||
annotations:
|
||||
key1: value1
|
||||
labels:
|
||||
key2: value2
|
||||
resources:
|
||||
requests:
|
||||
memory: "64Mi"
|
||||
cpu: "250m"
|
||||
limits:
|
||||
memory: "128Mi"
|
||||
cpu: "500m"
|
||||
affinity:
|
||||
nodeAffinity:
|
||||
requiredDuringSchedulingIgnoredDuringExecution:
|
||||
nodeSelectorTerms:
|
||||
- matchExpressions:
|
||||
- key: kubernetes.io/e2e-az-name
|
||||
operator: In
|
||||
values:
|
||||
- e2e-az1
|
||||
- e2e-az2
|
||||
preferredDuringSchedulingIgnoredDuringExecution:
|
||||
- weight: 1
|
||||
preference:
|
||||
matchExpressions:
|
||||
- key: another-node-label-key
|
||||
operator: In
|
||||
values:
|
||||
- another-node-label-value
|
||||
tolerations:
|
||||
- key: "key1"
|
||||
operator: "Equal"
|
||||
value: "value1"
|
||||
effect: "NoSchedule"
|
||||
- key: "key1"
|
||||
operator: "Equal"
|
||||
value: "value1"
|
||||
effect: "NoExecute"
|
||||
serviceAccount: nameOfServiceAccount
|
||||
securityContext:
|
||||
runAsUser: 1000
|
||||
volumeMounts:
|
||||
- name: config-vol
|
||||
mountPath: /etc/config
|
||||
volumes:
|
||||
- name: config-vol
|
||||
configMap:
|
||||
name: log-config
|
||||
items:
|
||||
- key: log_level
|
||||
path: log_level
|
||||
```
|
||||
|
||||
# Accessing the Jaeger Console (UI)
|
||||
<!-- TODO Add tabs shortcode -->
|
||||
|
||||
## Kubernetes
|
||||
|
||||
The operator creates a Kubernetes [`ingress`](https://kubernetes.io/docs/concepts/services-networking/ingress/) route, which is the Kubernetes' standard for exposing a service to the outside world, but by default it does not come with Ingress providers.
|
||||
Check the [Kubernetes documentation](https://kubernetes.github.io/ingress-nginx/deploy/#verify-installation) for the most appropriate way to achieve an Ingress provider for your platform. The following command enables the Ingress provider on `minikube`:
|
||||
|
||||
```bash
|
||||
minikube addons enable ingress
|
||||
```
|
||||
|
||||
Once Ingress is enabled, the address for the Jaeger console can be found by querying the Ingress object:
|
||||
|
||||
```bash
|
||||
$ kubectl get ingress
|
||||
NAME HOSTS ADDRESS PORTS AGE
|
||||
simplest-query * 192.168.122.34 80 3m
|
||||
```
|
||||
|
||||
In this example, the Jaeger UI is available at http://192.168.122.34.
|
||||
|
||||
## OpenShift
|
||||
|
||||
When the Operator is running on OpenShift, the Operator will automatically create a `Route` object for the query services. Use the following command to check the hostname/port:
|
||||
|
||||
```bash
|
||||
oc get routes
|
||||
```
|
||||
|
||||
{{< info >}}
|
||||
Make sure to use `https` with the hostname/port you get from the command above, otherwise you'll see a message like: "Application is not available".
|
||||
{{< /info >}}
|
||||
|
||||
By default, the Jaeger UI is protected with OpenShift's OAuth service and any valid user is able to login. To disable this feature and leave the Jaeger UI unsecured, set the Ingress property `security` to `none` in the custom resource file:
|
||||
|
||||
```yaml
|
||||
apiVersion: jaegertracing.io/v1
|
||||
kind: Jaeger
|
||||
metadata:
|
||||
name: disable-oauth-proxy
|
||||
spec:
|
||||
ingress:
|
||||
security: none
|
||||
```
|
||||
|
||||
Custom `SAR` and `Delegate URL` values can be specified as part of the `.Spec.Ingress.OpenShift.SAR` and `.Spec.Ingress.Openshift.DelegateURLs`, as follows:
|
||||
|
||||
```yaml
|
||||
apiVersion: jaegertracing.io/v1
|
||||
kind: Jaeger
|
||||
metadata:
|
||||
name: custom-sar-oauth-proxy
|
||||
spec:
|
||||
ingress:
|
||||
openshift:
|
||||
sar: '{"namespace": "default", "resource": "pods", "verb": "get"}'
|
||||
delegateUrls: '{"/":{"namespace": "default", "resource": "pods", "verb": "get"}}'
|
||||
```
|
||||
|
||||
When the `delegateUrls` is set, the Jaeger Operator needs to create a new `ClusterRoleBinding` between the service account used by the UI Proxy (`{InstanceName}-ui-proxy`) and the role `system:auth-delegator`, as required by the OpenShift OAuth Proxy. Because of that, the service account used by the operator itself needs to have the same cluster role binding. To accomplish that, a `ClusterRoleBinding` such as the following has to be created:
|
||||
|
||||
```yaml
|
||||
kind: ClusterRoleBinding
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
metadata:
|
||||
name: jaeger-operator-with-auth-delegator
|
||||
namespace: observability
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: jaeger-operator
|
||||
namespace: observability
|
||||
roleRef:
|
||||
kind: ClusterRole
|
||||
name: system:auth-delegator
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
```
|
||||
|
||||
Cluster administrators not comfortable in letting users deploy Jaeger instances with this cluster role are free to not add this cluster role to the operator's service account. In that case, the Operator will auto-detect that the required permissions are missing and will log a message similar to: `the requested instance specifies the delegateUrls option for the OAuth Proxy, but this operator cannot assign the proper cluster role to it (system:auth-delegator). Create a cluster role binding between the operator's service account and the cluster role 'system:auth-delegator' in order to allow instances to use 'delegateUrls'`.
|
||||
|
||||
The Jaeger Operator also supports authentication using `htpasswd` files via the OpenShift OAuth Proxy. To make use of that, specify the `htpasswdFile` option within the OpenShift-specific entries, pointing to the file `htpasswd` file location in the local disk. The `htpasswd` file can be created using the `htpasswd` utility:
|
||||
|
||||
```console
|
||||
$ htpasswd -cs /tmp/htpasswd jdoe
|
||||
New password:
|
||||
Re-type new password:
|
||||
Adding password for user jdoe
|
||||
```
|
||||
|
||||
This file can then be used as the input for the `kubectl create secret` command:
|
||||
|
||||
```console
|
||||
$ kubectl create secret generic htpasswd --from-file=htpasswd=/tmp/htpasswd
|
||||
secret/htpasswd created
|
||||
```
|
||||
|
||||
Once the secret is created, it can be specified in the Jaeger CR as a volume/volume mount:
|
||||
|
||||
```yaml
|
||||
apiVersion: jaegertracing.io/v1
|
||||
kind: Jaeger
|
||||
metadata:
|
||||
name: with-htpasswd
|
||||
spec:
|
||||
ingress:
|
||||
openshift:
|
||||
sar: '{"namespace": "default", "resource": "pods", "verb": "get"}'
|
||||
htpasswdFile: /usr/local/data/htpasswd
|
||||
volumeMounts:
|
||||
- name: htpasswd-volume
|
||||
mountPath: /usr/local/data
|
||||
volumes:
|
||||
- name: htpasswd-volume
|
||||
secret:
|
||||
secretName: htpasswd
|
||||
```
|
||||
|
||||
# Upgrading the Operator and its managed instances
|
||||
|
||||
Each version of the Jaeger Operator follows one Jaeger version. Whenever a new version of the Jaeger Operator is installed, all the Jaeger instances managed by the operator will be upgraded to the Operator's supported version. For example, an instance named `simplest` that was created with Jaeger Operator 1.12.0 will be running Jaeger 1.12.0. Once the Jaeger Operator is upgraded to 1.13.0, the instance `simplest` will be upgraded to the version 1.13.0, following the official upgrade instructions from the Jaeger project.
|
||||
|
||||
The Jaeger Operator can be upgraded manually by changing the deployment (`kubectl edit deployment jaeger-operator`), or via specialized tools such as the [Operator Lifecycle Manager (OLM)](https://github.com/operator-framework/operator-lifecycle-manager).
|
||||
|
||||
# Updating a Jaeger instance (experimental)
|
||||
|
||||
A Jaeger instance can be updated by changing the `CustomResource`, either via `kubectl edit jaeger simplest`, where `simplest` is the Jaeger's instance name, or by applying the updated YAML file via `kubectl apply -f simplest.yaml`.
|
||||
|
||||
{{< danger >}}
|
||||
The name of the Jaeger instance cannot be updated, as it is part of the identifying information for the resource.
|
||||
{{< /danger >}}
|
||||
|
||||
Simpler changes such as changing the replica sizes can be applied without much concern, whereas changes to the strategy should be watched closely and might potentially cause an outage for individual components (collector/query/agent).
|
||||
|
||||
While changing the backing storage is supported, migration of the data is not.
|
||||
|
||||
# Removing a Jaeger instance
|
||||
<!-- TODO Add OKD/OpenShift commands and tabs shortcode-->
|
||||
|
||||
To remove an instance, use the `delete` command with the custom resource file used when you created the instance:
|
||||
|
||||
```bash
|
||||
kubectl delete -f simplest.yaml
|
||||
```
|
||||
|
||||
Alternatively, you can remove a Jaeger instance by running:
|
||||
|
||||
```bash
|
||||
kubectl delete jaeger simplest
|
||||
```
|
||||
|
||||
{{< info >}}
|
||||
Deleting the instance will not remove the data from any permanent storage used with this instance. Data from in-memory instances, however, will be lost.
|
||||
{{< /info >}}
|
||||
|
||||
# Monitoring the operator
|
||||
|
||||
The Jaeger Operator starts a Prometheus-compatible endpoint on `0.0.0.0:8383/metrics` with internal metrics that can be used to monitor the process.
|
||||
|
||||
{{< info >}}
|
||||
The Jaeger Operator does not yet publish its own metrics. Rather, it makes available metrics reported by the components it uses, such as the Operator SDK.
|
||||
{{< /info >}}
|
||||
|
||||
# Uninstalling the operator
|
||||
<!-- TODO Add OKD/OpenShift commands and tabs shortcode -->
|
||||
|
||||
To uninstall the operator, run the following commands:
|
||||
|
||||
```bash
|
||||
kubectl delete -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/operator.yaml
|
||||
kubectl delete -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role_binding.yaml
|
||||
kubectl delete -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role.yaml
|
||||
kubectl delete -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/service_account.yaml
|
||||
kubectl delete -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/crds/jaegertracing.io_jaegers_crd.yaml
|
||||
```
|
|
@ -1,95 +0,0 @@
|
|||
---
|
||||
title: Performance Tuning Guide
|
||||
description: Tweaking your Jaeger instance to achieve a better performance
|
||||
weight: 10
|
||||
---
|
||||
|
||||
Jaeger was built from day 1 to be able to ingest huge amounts of data in a resilient way. To better utilize resources that might cause delays, such as storage or network communications, Jaeger buffers and batches data. When more spans are generated than Jaeger is able to safely process, spans might get dropped. However, the defaults might not fit all scenarios: for instance, Agents running as a sidecar might have more memory constraints than Agents running as a daemon in bare metal.
|
||||
|
||||
## Deployment considerations
|
||||
|
||||
Although performance tuning the individual components is important, the way Jaeger is deployed can be decisive in obtaining optimal performance.
|
||||
|
||||
### Scale the Collector up and down
|
||||
|
||||
Use the auto-scaling capabilities of your platform: the Collector is nearly horizontally scalable so that more instances can be added and removed on-demand. A good way to scale up and down is by checking the `jaeger_collector_queue_length` metric: add instances when the length is higher than 50% of the maximum size for extended periods of time. Another metric that can be taken into consideration is `jaeger_collector_in_queue_latency_bucket`, which is a histogram indicating how long spans have been waiting in the queue before a worker picked it up. When the queue latency gets higher over time, it’s a good indication to increase the number of the workers, or to improve the storage performance.
|
||||
|
||||
Adding Collector instances is recommended when your platform provides auto-scaling capabilities, or when it's easier to start/stop Collector instances than changing existing, running instances. Scaling horizontally is also indicated when the CPU usage should be spread across nodes.
|
||||
|
||||
### Make sure the storage can keep up
|
||||
|
||||
Each span is written to the storage by the Collector using one worker, blocking it until the span has been stored. When the storage is too slow, the number of workers blocked by the storage might be too high, causing spans to be dropped. To help diagnose this situation, the histogram `jaeger_collector_save_latency_bucket` can be analyzed. Ideally, the latency should remain the same over time. When the histogram shows that most spans are taking longer and longer over time, it’s a good indication that your storage might need some attention.
|
||||
|
||||
### Place the Agents close to your applications
|
||||
|
||||
The Agent is meant to be placed on the same host as the instrumented application, in order to avoid UDP packet loss over the network. This is typically accomplished by having one Agent per bare metal host for traditional applications, or as a sidecar in container environments like Kubernetes, as this helps spread the load handled by Agents with the additional advantage of allowing each Agent to be tweaked individually, according to the application’s needs and importance.
|
||||
|
||||
### Consider using Apache Kafka as intermediate buffer
|
||||
|
||||
Jaeger [can use Apache Kafka](../architecture/) as a buffer between the Collector and the actual backing storage (Elasticsearch, Apache Cassandra). This is ideal for cases where the traffic spikes are relatively frequent (prime time traffic) but the storage can eventually catch up once the traffic normalizes. For that, the `SPAN_STORAGE_TYPE` environment variable should be set to `kafka` in the Collector and the Jaeger Ingester component can be used, reading data from Kafka and writing it to the storage.
|
||||
|
||||
In addition to the performance aspects, having spans written to Kafka is useful for building real time data pipeline for aggregations and feature extraction from traces.
|
||||
|
||||
## Client (Tracer) settings
|
||||
|
||||
The Jaeger Clients are built to have minimal effect to the instrumented application. As such, it has conservative defaults that might not be suitable for all cases. Note that Jaeger Clients can be configured programmatically or via [environment variables](../client-features/).
|
||||
|
||||
### Adjust the sampling configuration
|
||||
|
||||
Together, the `JAEGER_SAMPLER_TYPE` and `JAEGER_SAMPLER_PARAM` specify how often traces should be "sampled", ie, recorded and sent to the Jaeger backend. For applications generating a large number of spans, setting the sampling type to `probabilistic` and the value to `0.001` (the default) will cause traces to be reported with a 1/1000th chance. Note that the sampling decision is made at the root span and propagated down to all child spans.
|
||||
|
||||
For applications with low to medium traffic, setting the sampling type to `const` and value to `1` will cause all spans to be reported. Similarly, tracing can be disabled by setting the value to `0`, while context propagation will continue to work.
|
||||
|
||||
Some Clients support the setting `JAEGER_DISABLED` to completely disable the Jaeger Tracer. This is recommended only if the Tracer is behaving in a way that causes problems to the instrumented application, as it will not propagate the context to the downstream services.
|
||||
|
||||
{{< info >}}
|
||||
We recommend to set your clients to use the [`remote` sampling strategy](../sampling/#collector-sampling-configuration), so that admins can centrally set the concrete sampling strategy for each service.
|
||||
{{< /info >}}
|
||||
|
||||
### Increase in-memory queue size
|
||||
|
||||
Most of the Jaeger clients, such as the Java, Go, and C# clients, buffer spans in memory before sending them to the Jaeger Agent/Collector. The maximum size of this buffer is defined by the environment variable `JAEGER_REPORTER_MAX_QUEUE_SIZE` (default value: about `100` spans): the larger the size, the higher the potential memory consumption. When the instrumented application is generating a large number of spans, it’s possible that the queue will be full causing the Client to discard the new spans (metric `jaeger_tracer_reporter_spans_total{result="dropped",}`).
|
||||
|
||||
In most common scenarios, the queue will be close to empty (metric: `jaeger_tracer_reporter_queue_length`), as spans are flushed to the Agent or Collector at regular intervals or when a certain size of the batch is reached. The detailed behavior of this queue is described in this [GitHub issue](https://github.com/jaegertracing/jaeger-client-java/issues/607).
|
||||
|
||||
### Modify the batched spans flush interval
|
||||
|
||||
The Java, Go, NodeJS, Python and C# Clients allow the customization of the flush interval (default value: `1000` milliseconds, or 1 second) used by the reporters, such as the `RemoteReporter`, to trigger a `flush` operation, sending all in-memory spans to the Agent or Collector. The lower the flush interval is set to, the more frequent the flush operations happen. As most reporters will wait until enough data is in the queue, this setting will force a flush operation at periodic intervals, so that spans are sent to the backend in a timely fashion.
|
||||
|
||||
When the instrumented application is generating a large number of spans and the Agent/Collector is close to the application, the networking overhead might be low, justifying a higher number of flush operations. When the `HttpSender` is being used and the Collector is not close enough to the application, the networking overhead might be too high so that a higher value for this property makes sense.
|
||||
|
||||
## Agent settings
|
||||
|
||||
Jaeger Agents receive data from Clients, sending them in batches to the Collector. When not properly configured, it might end up discarding data even if the host machine has plenty of resources.
|
||||
|
||||
### Adjust server queue sizes
|
||||
|
||||
The set of “server queue size” properties ( `processor.jaeger-binary.server-queue-size`, `processor.jaeger-compact.server-queue-size`, `processor.zipkin-compact.server-queue-size`) indicate the maximum number of span batches that the Agent can accept and store in memory. It’s safe to assume that `jaeger-compact` is the most important processor in your Agent setup, as it’s the only one available in most Clients, such as the Java and Go Clients.
|
||||
|
||||
The default value for each queue is `1000` span batches. Given that each span batch has up to 64KiB worth of spans, each queue can hold up to 64MiB worth of spans.
|
||||
|
||||
In typical scenarios, the queue will be close to empty (metric `jaeger_agent_thrift_udp_server_queue_size`) as span batches should be quickly picked up and processed by a worker. However, sudden spikes in the number of span batches submitted by Clients might occur, causing the batches to be queued. When the queue is full, the older batches are overridden causing spans to be discarded (metric `jaeger_agent_thrift_udp_server_packets_dropped_total`).
|
||||
|
||||
### Adjust processor workers
|
||||
|
||||
The set of “processor workers” properties ( `processor.jaeger-binary.workers`, `processor.jaeger-compact.workers`, `processor.zipkin-compact.workers`) indicate the number of parallel span batch processors to start. Each worker type has a default size of `10`. In general, span batches are processed as soon as they are placed in the server queue and will block a worker until the whole packet is sent to the Collector. For Agents processing data from multiple Clients, the number of workers should be increased. Given that the cost of each worker is low, a good rule of thumb is 10 workers per Client with moderate traffic: given that each span batch might contain up to 64KiB worth of spans, it means that 10 workers are able to send about 640KiB concurrently to a Collector.
|
||||
|
||||
## Collector settings
|
||||
|
||||
The Collector receives data from Clients and Agents. When not properly configured, it might process less data than what would be possible on the same host, or it might overload the host by consuming more memory than permitted.
|
||||
|
||||
### Adjust queue size
|
||||
|
||||
Similar to the Agent, the Collector is able to receive spans and place them in an internal queue for processing. This allows the Collector to return immediately to the Client/Agent instead of waiting for the span to make its way to the storage.
|
||||
|
||||
The setting `collector.queue-size` (default: `2000`) dictates how many spans the queue should support. In the typical scenario, the queue will be close to empty, as enough workers should exist picking up spans from the queue and sending them to the storage. When the number of items in the queue (metric `jaeger_collector_queue_length`) is permanently high, it’s an indication that either the number of workers should be increased or that the storage cannot keep up with the volume of data that it’s receiving. When the queue is full, the older items in the queue are overridden, causing spans to be discarded (metric `jaeger_collector_spans_dropped_total`).
|
||||
|
||||
{{< warning >}}
|
||||
The queue size for the Agent is about _span batches_, whereas the queue size for the Collector is about _individual spans_.
|
||||
{{< /warning >}}
|
||||
|
||||
Given that the queue size should be close to empty most of the time, this setting should be as high as the available memory for the Collector, to provide maximum protection against sudden traffic spikes. However, if your storage layer is under-provisioned and cannot keep up, even a large queue will quickly fill up and start dropping data.
|
||||
|
||||
### Adjust processor workers
|
||||
|
||||
Items from the span queue in the Collector are picked up by workers. Each worker picks one span from the queue and persists it to the storage. The number of workers can be specified by the setting `collector.num-workers` (default: `50`) and should be as high as needed to keep the queue close to zero. The general rule is: the faster the backing storage, the lower the number of workers can be. Given that workers are relatively cheap, this number can be increased at will. As a general rule, one worker per 50 items in the queue should be sufficient when the storage is fast. With a `collector.queue-size` of `2000`, having about `40` workers should be sufficient. For slower storage mechanisms, this ratio should be adjusted accordingly, having more workers per queue item.
|
|
@ -1,66 +0,0 @@
|
|||
---
|
||||
title: Sampling
|
||||
hasparent: true
|
||||
---
|
||||
|
||||
Jaeger libraries implement consistent upfront (or head-based) sampling. For example, assume we have a simple call graph where service A calls service B, and B calls service C: `A -> B -> C`. When service A receives a request that contains no tracing information, Jaeger tracer will start a new {{< tip "trace" >}}, assign it a random trace ID, and make a sampling decision based on the currently installed sampling strategy. The sampling decision will be propagated with the requests to B and to C, so those services will not be making the sampling decision again but instead will respect the decision made by the top service A. This approach guarantees that if a trace is sampled, all its {{< tip "spans" "span" >}} will be recorded in the backend. If each service was making its own sampling decision we would rarely get complete traces in the backend.
|
||||
|
||||
## Client Sampling Configuration
|
||||
|
||||
When using configuration object to instantiate the tracer, the type of sampling can be selected via `sampler.type` and `sampler.param` properties. Jaeger libraries support the following samplers:
|
||||
|
||||
* **Constant** (`sampler.type=const`) sampler always makes the same decision for all traces. It either samples all traces (`sampler.param=1`) or none of them (`sampler.param=0`).
|
||||
* **Probabilistic** (`sampler.type=probabilistic`) sampler makes a random sampling decision with the probability of sampling equal to the value of `sampler.param` property. For example, with `sampler.param=0.1` approximately 1 in 10 traces will be sampled.
|
||||
* **Rate Limiting** (`sampler.type=ratelimiting`) sampler uses a leaky bucket rate limiter to ensure that traces are sampled with a certain constant rate. For example, when `sampler.param=2.0` it will sample requests with the rate of 2 traces per second.
|
||||
* **Remote** (`sampler.type=remote`, which is also the default) sampler consults Jaeger agent for the appropriate sampling strategy to use in the current service. This allows controlling the sampling strategies in the services from a [central configuration](#collector-sampling-configuration) in Jaeger backend, or even dynamically (see [Adaptive Sampling](https://github.com/jaegertracing/jaeger/issues/365)).
|
||||
|
||||
### Adaptive Sampler
|
||||
|
||||
Adaptive sampler is a composite sampler that combines two functions:
|
||||
|
||||
* It makes sampling decisions on a per-operation basis, i.e. based on {{< tip "span" >}} operation name. This is especially useful in the API services whose endpoints may have very different traffic volumes and using a single probabilistic sampler for the whole service might starve (never sample) some of the low QPS endpoints.
|
||||
* It supports a minimum guaranteed rate of sampling, such as always allowing up to N {{< tip "traces" "trace" >}} per seconds and then sampling anything above that with a certain probability (everything is per-operation, not per-service).
|
||||
|
||||
Per-operation parameters can be configured statically or pulled periodically from Jaeger backend with the help of Remote sampler. Adaptive sampler is designed to work with the upcoming [Adaptive Sampling](https://github.com/jaegertracing/jaeger/issues/365) feature of the Jaeger backend.
|
||||
|
||||
## Collector Sampling Configuration
|
||||
|
||||
Collectors can be instantiated with static sampling strategies (which are propagated to the respective service if configured with Remote sampler) via the `--sampling.strategies-file` option. This option requires a path to a json file which have the sampling strategies defined.
|
||||
|
||||
Example `strategies.json`
|
||||
```json
|
||||
{
|
||||
"service_strategies": [
|
||||
{
|
||||
"service": "foo",
|
||||
"type": "probabilistic",
|
||||
"param": 0.8,
|
||||
"operation_strategies": [
|
||||
{
|
||||
"operation": "op1",
|
||||
"type": "probabilistic",
|
||||
"param": 0.2
|
||||
},
|
||||
{
|
||||
"operation": "op2",
|
||||
"type": "probabilistic",
|
||||
"param": 0.4
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "bar",
|
||||
"type": "ratelimiting",
|
||||
"param": 5
|
||||
}
|
||||
],
|
||||
"default_strategy": {
|
||||
"type": "probabilistic",
|
||||
"param": 0.5
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`service_strategies` defines service specific sampling strategies and `operation_strategies` defines operation specific sampling strategies. There are 2 types of strategies possible: `probabilistic` and `ratelimiting` which are described [above](#client-sampling-configuration) (NOTE: `ratelimiting` is not supported for `operation_strategies`). `default_strategy` defines the catch-all sampling strategy that is propagated if the service is not included as part of `service_strategies`.
|
||||
|
||||
In the above example, all service `foo` operations are sampled probabilistically with a probability of 0.8 except `op1` and `op2` which are probabilistically sampled with a probability of 0.2 and 0.4 respectively. All operations for service `bar` are ratelimited at 5 traces per second. Any other service is probabilistically sampled with a probability of 0.5.
|
|
@ -1,70 +0,0 @@
|
|||
---
|
||||
title: Introduction
|
||||
weight: 1
|
||||
children:
|
||||
- title: Features
|
||||
url: features
|
||||
---
|
||||
|
||||
Welcome to Jaeger's documentation portal! Below, you'll find information for beginners and experienced Jaeger users.
|
||||
|
||||
If you can't find what you are looking for, or have an issue not covered here, we'd love to [hear from you](/get-in-touch).
|
||||
|
||||
## About
|
||||
|
||||
Jaeger, inspired by [Dapper][dapper] and [OpenZipkin](http://zipkin.io),
|
||||
is a distributed tracing system released as open source by [Uber Technologies][ubeross].
|
||||
It is used for monitoring and troubleshooting microservices-based distributed systems, including:
|
||||
|
||||
* Distributed context propagation
|
||||
* Distributed transaction monitoring
|
||||
* Root cause analysis
|
||||
* Service dependency analysis
|
||||
* Performance / latency optimization
|
||||
|
||||
Uber published a blog post, [Evolving Distributed Tracing at Uber](https://eng.uber.com/distributed-tracing/), where they explain the history and reasons for the architectural choices made in Jaeger. [Yuri Shkuro](https://shkuro.com), creator of Jaeger, also published a book [Mastering Distributed Tracing](https://shkuro.com/books/2019-mastering-distributed-tracing/) that covers in-depth many aspects of Jaeger design and operation, as well as distributed tracing in general.
|
||||
|
||||
## Features
|
||||
|
||||
* [OpenTracing](http://opentracing.io/) compatible data model and instrumentation libraries
|
||||
* in [Go](https://github.com/jaegertracing/jaeger-client-go), [Java](https://github.com/jaegertracing/jaeger-client-java), [Node](https://github.com/jaegertracing/jaeger-client-node), [Python](https://github.com/jaegertracing/jaeger-client-python)
|
||||
and [C++](https://github.com/jaegertracing/cpp-client)
|
||||
* Uses consistent upfront sampling with individual per service/endpoint probabilities
|
||||
* Multiple storage backends: Cassandra, Elasticsearch, memory.
|
||||
* Adaptive sampling (coming soon)
|
||||
* Post-collection data processing pipeline (coming soon)
|
||||
|
||||
See [Features](./features/) page for more details.
|
||||
|
||||
## Technical Specs
|
||||
|
||||
* Backend components implemented in Go
|
||||
* React/Javascript UI
|
||||
* Supported storage backends:
|
||||
* [Cassandra 3.4+](./deployment/#cassandra)
|
||||
* [Elasticsearch 5.x, 6.x, 7.x](./deployment/#elasticsearch)
|
||||
* [Kafka](./deployment/#kafka)
|
||||
* memory storage
|
||||
|
||||
## Quick Start
|
||||
See [running a docker all in one image](getting-started#all-in-one).
|
||||
|
||||
## Screenshots
|
||||
|
||||
### Traces View
|
||||
[](/img/traces-ss.png)
|
||||
|
||||
### Trace Detail View
|
||||
[](/img/trace-detail-ss.png)
|
||||
|
||||
## Related links
|
||||
- [Evolving Distributed tracing At Uber Engineering](https://eng.uber.com/distributed-tracing/)
|
||||
- [Mastering Distributed Tracing](https://shkuro.com/books/2019-mastering-distributed-tracing/)
|
||||
- [Tracing HTTP request latency in Go with OpenTracing](https://medium.com/opentracing/tracing-http-request-latency-in-go-with-opentracing-7cc1282a100a)
|
||||
- [Distributed Tracing with Jaeger & Prometheus on Kubernetes](https://blog.openshift.com/openshift-commons-briefing-82-distributed-tracing-with-jaeger-prometheus-on-kubernetes/)
|
||||
- [Using Jaeger with Istio](https://istio.io/docs/tasks/telemetry/distributed-tracing.html)
|
||||
- [Using Jaeger with Envoy](https://www.envoyproxy.io/docs/envoy/latest/start/sandboxes/jaeger_tracing.html)
|
||||
|
||||
[dapper]: https://research.google.com/pubs/pub36356.html
|
||||
[ubeross]: http://uber.github.io
|
||||
|
|
@ -1,87 +0,0 @@
|
|||
---
|
||||
title: APIs
|
||||
hasparent: true
|
||||
---
|
||||
|
||||
Jaeger components implement various APIs for saving or retrieving trace data.
|
||||
|
||||
The following labels are used to describe API compatibility guarantees.
|
||||
|
||||
* **stable** - the API guarantees backwards compatibility. If breaking changes are going to be made in the future, they will result in a new API version, e.g. `/api/v2` URL prefix or a different namespace in the IDL.
|
||||
* **internal** - the APIs intended for internal communications between Jaeger components and not recommended for use by external components.
|
||||
* **deprecated** - the APIs that are only maintained for legacy reasons and will be phased out in the future.
|
||||
|
||||
## Span reporting APIs
|
||||
|
||||
Agent and Collector are the two components of the Jaeger backend that can receive spans. At this time they support two sets of non-overlapping APIs.
|
||||
|
||||
### Thrift over UDP (stable)
|
||||
|
||||
The Agent can only receive spans over UDP in Thrift format. The primary API is a UDP packet that contains a Thrift-encoded `Batch` struct defined in [jaeger.thrift][jaeger.thrift] IDL file, located in the [jaeger-idl][jaeger-idl] repository. Most Jaeger Clients use Thrift's `compact` encoding, however some client libraries do not support it (notably, Node.js) and use Thrift's `binary` encoding (sent to a different UDP port). The Agent's API is defined by [agent.thrift][agent.thrift] IDL file.
|
||||
|
||||
For legacy reasons, the Agent also accepts spans in Zipkin format, however, only very old versions of Jaeger clients can send data in that format and it is officially deprecated.
|
||||
|
||||
### Protobuf via gRPC (stable)
|
||||
|
||||
In a typical Jaeger deployment, Agents receive spans from Clients and forward them to Collectors. Since Jaeger version 1.11 the official and recommended protocol between Agents and Collectors is gRPC with Protobuf.
|
||||
|
||||
The Protobuf IDL [collector.proto][collector.proto] is currently located in the main Jaeger repository, under [model/proto/api_v2][collector.proto]. In the future it will be moved to [jaeger-idl][jaeger-idl] repository ([jaeger-idl/issues/55](https://github.com/jaegertracing/jaeger-idl/issues/55)).
|
||||
|
||||
### Thrift over HTTP (stable)
|
||||
|
||||
In some cases it is not feasible to deploy Jaeger Agent next to the application, for example, when the application code is running as AWS Lambda function. In these scenarios the Jaeger Clients can be configured to submit spans directly to the Collectors over HTTP/HTTPS.
|
||||
|
||||
The same [jaeger.thrift][jaeger.thrift] payload can be submitted in HTTP POST request to `/api/traces` endpoint, for example, `https://jaeger-collector:14268/api/traces`. The `Batch` struct needs to be encoded using Thrift's `binary` encoding, and the HTTP request should specify the content type header:
|
||||
|
||||
```
|
||||
Content-Type: application/vnd.apache.thrift.binary
|
||||
```
|
||||
|
||||
### JSON over HTTP (n/a)
|
||||
|
||||
There is no official Jaeger JSON format that can be accepted by the collector. In the future the Protobuf-generated JSON may be supported.
|
||||
|
||||
### Thrift via TChannel (deprecated)
|
||||
|
||||
Agent and Collector can communicate using TChannel protocol. This protocol is generally not supported by the routing infrastructure and has been deprecated. It will be eventually removed from Jaeger.
|
||||
|
||||
### Zipkin Formats (stable)
|
||||
|
||||
Jaeger Collector can also accept spans in several Zipkin data format, namely JSON v1/v2 and Thrift. The Collector needs to be configured to enable Zipkin HTTP server, e.g. on port 9411 used by Zipkin collectors. The server enables two endpoints that expect POST requests:
|
||||
|
||||
* `/api/v1/spans` for submitting spans in Zipkin JSON v1 or Zipkin Thrift format.
|
||||
* `/api/v2/spans` for submitting spans in Zipkin JSON v2.
|
||||
|
||||
## Trace retrieval APIs
|
||||
|
||||
Traces saved in the storage can be retrieved by calling Jaeger Query Service.
|
||||
|
||||
### gRPC/Protobuf (stable)
|
||||
|
||||
The recommended way for programmatically retrieving traces and other data is via gRPC endpoint defined in [query.proto][query.proto] IDL file (located in the main Jaeger repository, similar to [collector.proto][collector.proto]).
|
||||
|
||||
### HTTP JSON (internal)
|
||||
|
||||
Jaeger UI communicates with Jaeger Query Service via JSON API. For example, a trace can be retrieved via GET request to `https://jaeger-query:16686/api/traces/{trace-id-hex-string}`. This JSON API is intentionally undocumented and subject to change.
|
||||
|
||||
## Clients configuration (internal)
|
||||
|
||||
Client libraries not only submit finished spans to Jaeger backend, but also periodically poll the Agents for various configurations, such as sampling strategies. The schema for the payload is defined by [sampling.thrift][sampling.thrift], encoded as JSON using Thrift's built-in JSON generation capabilities.
|
||||
|
||||
## Service dependencies graph (internal)
|
||||
|
||||
Can be retrieved from Query Service at `/api/dependencies` endpoint. The GET request expects two parameters:
|
||||
|
||||
* `endTs` (number of milliseconds since epoch) - the end of the time interval
|
||||
* `lookback` (in milliseconds) - the length the time interval (i.e. start-time + lookback = end-time).
|
||||
|
||||
The returned JSON is a list of edges represented as tuples `(caller, callee, count)`.
|
||||
|
||||
For programmatic access to service graph, the recommended API is gRPC/Protobuf described above.
|
||||
|
||||
[jaeger-idl]: https://github.com/jaegertracing/jaeger-idl/
|
||||
[jaeger.thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/jaeger.thrift
|
||||
[agent.thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/agent.thrift
|
||||
[sampling.thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/sampling.thrift
|
||||
[collector.proto]: https://github.com/jaegertracing/jaeger/blob/master/model/proto/api_v2/collector.proto
|
||||
[query.proto]: https://github.com/jaegertracing/jaeger/blob/master/model/proto/api_v2/query.proto
|
|
@ -1,13 +0,0 @@
|
|||
---
|
||||
title: CLI flags
|
||||
widescreen: true
|
||||
hasparent: true
|
||||
---
|
||||
|
||||
This is auto-generated documentation for CLI flags supported by Jaeger binaries.
|
||||
|
||||
* CLI flags for some binaries change depending on the `SPAN_STORAGE_TYPE` environment variable. Relevant variations are included below.
|
||||
* Some binaries support _commands_ (mostly informational), such as `env`, `docs`, and `version`. These commands are not included here.
|
||||
* All parameters can be also provided via environment variables, by changing all letters to upper-case and replacing all punctuation characters with underscore `_`. For example, the value for the flag `--cassandra.connections-per-host` can be provided via `CASSANDRA_CONNECTIONS_PER_HOST` environment variable.
|
||||
|
||||
{{< cli/tools-list >}}
|
|
@ -1,469 +0,0 @@
|
|||
---
|
||||
title: Deployment
|
||||
weight: 4
|
||||
children:
|
||||
- title: Operator for Kubernetes
|
||||
navtitle: Kubernetes
|
||||
url: operator
|
||||
- title: Frontend/UI
|
||||
url: frontend-ui
|
||||
- title: Windows
|
||||
url: windows
|
||||
- title: CLI Flags
|
||||
url: cli
|
||||
---
|
||||
|
||||
The main Jaeger backend components are released as Docker images on Docker Hub:
|
||||
|
||||
Component | Repository
|
||||
--------------------- | ---
|
||||
**jaeger-agent** | [hub.docker.com/r/jaegertracing/jaeger-agent/](https://hub.docker.com/r/jaegertracing/jaeger-agent/)
|
||||
**jaeger-collector** | [hub.docker.com/r/jaegertracing/jaeger-collector/](https://hub.docker.com/r/jaegertracing/jaeger-collector/)
|
||||
**jaeger-query** | [hub.docker.com/r/jaegertracing/jaeger-query/](https://hub.docker.com/r/jaegertracing/jaeger-query/)
|
||||
**jaeger-ingester** | [hub.docker.com/r/jaegertracing/jaeger-ingester/](https://hub.docker.com/r/jaegertracing/jaeger-ingester/)
|
||||
|
||||
There are orchestration templates for running Jaeger with:
|
||||
|
||||
* Kubernetes: [github.com/jaegertracing/jaeger-kubernetes](https://github.com/jaegertracing/jaeger-kubernetes),
|
||||
* OpenShift: [github.com/jaegertracing/jaeger-openshift](https://github.com/jaegertracing/jaeger-openshift).
|
||||
|
||||
## Configuration Options
|
||||
|
||||
Jaeger binaries can be configured in a number of ways (in the order of decreasing priority):
|
||||
|
||||
* command line arguments,
|
||||
* environment variables,
|
||||
* configuration files in JSON, TOML, YAML, HCL, or Java properties formats.
|
||||
|
||||
To see the complete list of options, run the binary with `help` command. Options that are specific to a certain storage backend are only listed if the storage type is selected. For example, to see all available options in the Collector with Cassandra storage:
|
||||
|
||||
```sh
|
||||
$ docker run --rm \
|
||||
-e SPAN_STORAGE_TYPE=cassandra \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}} \
|
||||
help
|
||||
```
|
||||
|
||||
In order to provide configuration parameters via environment variables, find the respective command line option and convert its name to UPPER_SNAKE_CASE, for example:
|
||||
|
||||
Command line option | Environment variable
|
||||
-----------------------------------|-------------------------------
|
||||
`--cassandra.connections-per-host` | `CASSANDRA_CONNECTIONS_PER_HOST`
|
||||
`--metrics-backend` | `METRICS_BACKEND`
|
||||
|
||||
## Agent
|
||||
|
||||
Jaeger client libraries expect **jaeger-agent** process to run locally on each host.
|
||||
The agent exposes the following ports:
|
||||
|
||||
Port | Protocol | Function
|
||||
----- | ------- | ---
|
||||
6831 | UDP | accept [jaeger.thrift][jaeger-thrift] in `compact` Thrift protocol used by most current Jaeger clients
|
||||
6832 | UDP | accept [jaeger.thrift][jaeger-thrift] in `binary` Thrift protocol used by Node.js Jaeger client (because [thriftrw][thriftrw] npm package does not support `compact` protocol)
|
||||
5778 | HTTP | serve configs, sampling strategies
|
||||
5775 | UDP | accept [zipkin.thrift][zipkin-thrift] in `compact` Thrift protocol (deprecated; only used by very old Jaeger clients, circa 2016)
|
||||
14271 | HTTP | Healthcheck at `/` and metrics at `/metrics`
|
||||
|
||||
It can be executed directly on the host or via Docker, as follows:
|
||||
|
||||
```sh
|
||||
## make sure to expose only the ports you use in your deployment scenario!
|
||||
docker run \
|
||||
--rm \
|
||||
-p6831:6831/udp \
|
||||
-p6832:6832/udp \
|
||||
-p5778:5778/tcp \
|
||||
-p5775:5775/udp \
|
||||
jaegertracing/jaeger-agent:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
### Discovery System Integration
|
||||
|
||||
The agents can connect point to point to a single collector address, which could be
|
||||
load balanced by another infrastructure component (e.g. DNS) across multiple collectors.
|
||||
The agent can also be configured with a static list of collector addresses.
|
||||
|
||||
On Docker, a command like the following can be used:
|
||||
|
||||
```sh
|
||||
docker run \
|
||||
--rm \
|
||||
-p5775:5775/udp \
|
||||
-p6831:6831/udp \
|
||||
-p6832:6832/udp \
|
||||
-p5778:5778/tcp \
|
||||
jaegertracing/jaeger-agent:{{< currentVersion >}} \
|
||||
--reporter.grpc.host-port=jaeger-collector.jaeger-infra.svc:14250
|
||||
```
|
||||
|
||||
Or use `--reporter.tchannel.host-port=jaeger-collector.jaeger-infra.svc:14267` to use
|
||||
legacy tchannel reporter.
|
||||
|
||||
When using gRPC, you have several options for load balancing and name resolution:
|
||||
|
||||
* Single connection and no load balancing. This is the default if you specify a single `host:port`. (example: `--reporter.grpc.host-port=jaeger-collector.jaeger-infra.svc:14250`)
|
||||
* Static list of hostnames and round-robin load balancing. This is what you get with a comma-separated list of addresses. (example: `reporter.grpc.host-port=jaeger-collector1:14250,jaeger-collector2:14250,jaeger-collector3:14250`)
|
||||
* Dynamic DNS resolution and round-robin load balancing. To get this behaviour, prefix the address with `dns:///` and gRPC will attempt to resolve the hostname using SRV records (for [external load balancing](https://github.com/grpc/grpc/blob/master/doc/load-balancing.md)), TXT records (for [service configs](https://github.com/grpc/grpc/blob/master/doc/service_config.md)), and A records. Refer to the [gRPC Name Resolution docs](https://github.com/grpc/grpc/blob/master/doc/naming.md) and the [dns_resolver.go implementation](https://github.com/grpc/grpc-go/blob/master/resolver/dns/dns_resolver.go) for more info. (example: `--reporter.grpc.host-port=dns:///jaeger-collector.jaeger-infra.svc:14250`)
|
||||
|
||||
### Agent level tags
|
||||
|
||||
Jaeger supports agent level tags, that can be added to the process tags of all spans passing through the agent. This is supported through the command line flag `--jaeger.tags=key1=value1,key2=value2,...,keyn=valuen`. Tags can also be set through an environment flag like so - `--jaeger.tags=key=${envFlag:defaultValue}` - The tag value will be set to the value of the `envFlag` environment key and `defaultValue` if not set. This feature is not supported for the tchannel reporter, enabled using the flags `--collector.host-port` or `--reporter.tchannel.host-port`.
|
||||
|
||||
## Collectors
|
||||
|
||||
The collectors are stateless and thus many instances of **jaeger-collector** can be run in parallel.
|
||||
Collectors require almost no configuration, except for the location of Cassandra cluster,
|
||||
via `--cassandra.keyspace` and `--cassandra.servers` options, or the location of Elasticsearch cluster, via
|
||||
`--es.server-urls`, depending on which storage is specified. To see all command line options run
|
||||
|
||||
```sh
|
||||
go run ./cmd/collector/main.go -h
|
||||
```
|
||||
|
||||
or, if you don't have the source code
|
||||
|
||||
```sh
|
||||
docker run -it --rm jaegertracing/jaeger-collector:{{< currentVersion >}} -h
|
||||
```
|
||||
|
||||
At default settings the collector exposes the following ports:
|
||||
|
||||
Port | Protocol | Function
|
||||
----- | ------- | ---
|
||||
14267 | TChannel | used by **jaeger-agent** to send spans in jaeger.thrift format
|
||||
14250 | gRPC | used by **jaeger-agent** to send spans in model.proto format
|
||||
14268 | HTTP | can accept spans directly from clients in jaeger.thrift format over binary thrift protocol
|
||||
9411 | HTTP | can accept Zipkin spans in Thrift, JSON and Proto (disabled by default)
|
||||
14269 | HTTP | Healthcheck at `/` and metrics at `/metrics`
|
||||
|
||||
|
||||
## Storage Backends
|
||||
|
||||
Collectors require a persistent storage backend. Cassandra and Elasticsearch are the primary supported storage backends. Additional backends are [discussed here](https://github.com/jaegertracing/jaeger/issues/638).
|
||||
|
||||
The storage type can be passed via `SPAN_STORAGE_TYPE` environment variable. Valid values are `cassandra`, `elasticsearch`, `kafka` (only as a buffer), `grpc-plugin`, `badger` (only with all-in-one) and `memory` (only with all-in-one).
|
||||
|
||||
As of version 1.6.0, it's possible to use multiple storage types at the same time by providing a comma-separated list of valid types to the `SPAN_STORAGE_TYPE` environment variable.
|
||||
It's important to note that all listed storage types are used for writing, but only the first type in the list will be used for reading and archiving.
|
||||
|
||||
### Memory
|
||||
|
||||
The in-memory storage is not intended for production workloads. It's intended as a simple solution to get started quickly and
|
||||
data will be lost once the process is gone.
|
||||
|
||||
By default, there's no limit in the amount of traces stored in memory but a limit can be established by passing an
|
||||
integer value via `--memory.max-traces`.
|
||||
|
||||
### Badger - local storage
|
||||
Experimental since Jaeger 1.9
|
||||
|
||||
[Badger](https://github.com/dgraph-io/badger) is an embedded local storage, only available
|
||||
with **all-in-one** distribution. By default it acts as an ephemeral storage using a temporary filesystem.
|
||||
This can be overridden by using the `--badger.ephemeral=false` option.
|
||||
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=badger \
|
||||
-e BADGER_EPHEMERAL=false \
|
||||
-e BADGER_DIRECTORY_VALUE=/badger/data \
|
||||
-e BADGER_DIRECTORY_KEY=/badger/key \
|
||||
-v <storage_dir_on_host>:/badger \
|
||||
-p 16686:16686 \
|
||||
jaegertracing/all-in-one:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
### Cassandra
|
||||
Supported versions: 3.4+
|
||||
|
||||
Deploying Cassandra itself is out of scope for our documentation. One good
|
||||
source of documentation is the [Apache Cassandra Docs](https://cassandra.apache.org/doc/latest/).
|
||||
|
||||
#### Configuration
|
||||
##### Minimal
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=cassandra \
|
||||
-e CASSANDRA_SERVERS=<...> \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
##### All options
|
||||
To view the full list of configuration options, you can run the following command:
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=cassandra \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}} \
|
||||
--help
|
||||
```
|
||||
|
||||
#### Schema script
|
||||
|
||||
A script is provided to initialize Cassandra keyspace and schema
|
||||
using Cassandra's interactive shell [`cqlsh`][cqlsh]:
|
||||
|
||||
```sh
|
||||
MODE=test sh ./plugin/storage/cassandra/schema/create.sh | cqlsh
|
||||
```
|
||||
|
||||
For production deployment, pass `MODE=prod DATACENTER={datacenter}` arguments to the script,
|
||||
where `{datacenter}` is the name used in the Cassandra configuration / network topology.
|
||||
|
||||
The script also allows overriding TTL, keyspace name, replication factor, etc.
|
||||
Run the script without arguments to see the full list of recognized parameters.
|
||||
|
||||
#### TLS support
|
||||
|
||||
Jaeger supports TLS client to node connections as long as you've configured
|
||||
your Cassandra cluster correctly. After verifying with e.g. `cqlsh`, you can
|
||||
configure the collector and query like so:
|
||||
|
||||
```sh
|
||||
docker run \
|
||||
-e CASSANDRA_SERVERS=<...> \
|
||||
-e CASSANDRA_TLS=true \
|
||||
-e CASSANDRA_TLS_SERVER_NAME="CN-in-certificate" \
|
||||
-e CASSANDRA_TLS_KEY=<path to client key file> \
|
||||
-e CASSANDRA_TLS_CERT=<path to client cert file> \
|
||||
-e CASSANDRA_TLS_CA=<path to your CA cert file> \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
The schema tool also supports TLS. You need to make a custom cqlshrc file like
|
||||
so:
|
||||
|
||||
```
|
||||
# Creating schema in a cassandra cluster requiring client TLS certificates.
|
||||
#
|
||||
# Create a volume for the schema docker container containing four files:
|
||||
# cqlshrc: this file
|
||||
# ca-cert: the cert authority for your keys
|
||||
# client-key: the keyfile for your client
|
||||
# client-cert: the cert file matching client-key
|
||||
#
|
||||
# if there is any sort of DNS mismatch and you want to ignore server validation
|
||||
# issues, then uncomment validate = false below.
|
||||
#
|
||||
# When running the container, map this volume to /root/.cassandra and set the
|
||||
# environment variable CQLSH_SSL=--ssl
|
||||
[ssl]
|
||||
certfile = ~/.cassandra/ca-cert
|
||||
userkey = ~/.cassandra/client-key
|
||||
usercert = ~/.cassandra/client-cert
|
||||
# validate = false
|
||||
```
|
||||
|
||||
### Elasticsearch
|
||||
Supported in Jaeger since 0.6.0
|
||||
Supported versions: 5.x, 6.x, 7.x
|
||||
|
||||
Elasticsearch version is automatically retrieved from root/ping endpoint.
|
||||
Based on this version Jaeger uses compatible index mappings and Elasticsearch REST API.
|
||||
The version can be explicitly provided via `--es.version=` flag.
|
||||
|
||||
Elasticsearch does not require initialization other than
|
||||
[installing and running Elasticsearch](https://www.elastic.co/downloads/elasticsearch).
|
||||
Once it is running, pass the correct configuration values to the Jaeger collector and query service.
|
||||
|
||||
#### Configuration
|
||||
##### Minimal
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=elasticsearch \
|
||||
-e ES_SERVER_URLS=<...> \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
##### All options
|
||||
To view the full list of configuration options, you can run the following command:
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=elasticsearch \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}} \
|
||||
--help
|
||||
```
|
||||
|
||||
#### Shards and Replicas for Elasticsearch indices
|
||||
|
||||
Shards and replicas are some configuration values to take special attention to, because this is decided upon
|
||||
index creation. [This article](https://qbox.io/blog/optimizing-elasticsearch-how-many-shards-per-index) goes into
|
||||
more information about choosing how many shards should be chosen for optimization.
|
||||
|
||||
#### Upgrade Elasticsearch version
|
||||
|
||||
Elasticsearch defines wire and index compatibility versions. The index compatibility defines
|
||||
the minimal version a node can read data from. For example Elasticsearch 7 can read indices
|
||||
created by Elasticsearch 6, however it cannot read indices created by Elasticsearch 5 even
|
||||
though they use the same index mappings. Therefore upgrade from Elasticsearch 6 to 7 does not require any
|
||||
data migration. However, upgrade from Elasticsearch 5 to 7 has to be done through Elasticsearch 6 and wait
|
||||
until indices created by ES 5.x are removed or explicitly reindexed.
|
||||
|
||||
Refer to the Elasticsearch [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current//setup-upgrade.html)
|
||||
for wire and index compatibility versions. Generally this information can be retrieved from root/ping REST endpoint.
|
||||
|
||||
##### Reindex
|
||||
|
||||
Manual reindexing can be used when upgrading from Elasticsearch 5 to 7 (through Elasticsearch 6)
|
||||
without waiting until indices created by Elasticsearch 5 are removed.
|
||||
|
||||
1. Reindex all span indices to new indices with suffix `-1`:
|
||||
|
||||
```bash
|
||||
curl -ivX POST -H "Content-Type: application/json" http://localhost:9200/_reindex -d @reindex.json
|
||||
{
|
||||
"source": {
|
||||
"index": "jaeger-span-*"
|
||||
},
|
||||
"dest": {
|
||||
"index": "jaeger-span"
|
||||
},
|
||||
"script": {
|
||||
"lang": "painless",
|
||||
"source": "ctx._index = 'jaeger-span-' + (ctx._index.substring('jaeger-span-'.length(), ctx._index.length())) + '-1'"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
2. Delete indices with old mapping:
|
||||
|
||||
```bash
|
||||
curl -ivX DELETE -H "Content-Type: application/json" http://localhost:9200/jaeger-span-\*,-\*-1
|
||||
```
|
||||
|
||||
3. Create indices without `-1` suffix:
|
||||
|
||||
```bash
|
||||
curl -ivX POST -H "Content-Type: application/json" http://localhost:9200/_reindex -d @reindex.json
|
||||
{
|
||||
"source": {
|
||||
"index": "jaeger-span-*"
|
||||
},
|
||||
"dest": {
|
||||
"index": "jaeger-span"
|
||||
},
|
||||
"script": {
|
||||
"lang": "painless",
|
||||
"source": "ctx._index = 'jaeger-span-' + (ctx._index.substring('jaeger-span-'.length(), ctx._index.length() - 2))"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
4. Remove suffixed indices:
|
||||
|
||||
```bash
|
||||
curl -ivX DELETE -H "Content-Type: application/json" http://localhost:9200/jaeger-span-\*-1
|
||||
```
|
||||
|
||||
Run the commands analogically for other Jaeger indices.
|
||||
|
||||
There might exist more effective migration procedure. Please share with the community any findings.
|
||||
|
||||
### Kafka
|
||||
Supported in Jaeger since 1.6.0
|
||||
Supported Kafka versions: 0.9+
|
||||
|
||||
Kafka can be used as an intermediary buffer between collector and an actual storage.
|
||||
The collector is configured with `SPAN_STORAGE_TYPE=kafka` that makes it write all received spans
|
||||
into a Kafka topic. A new component [Ingester](#ingester), added in version 1.7.0, is used to read from
|
||||
Kafka and store spans in another storage backend (Elasticsearch or Cassandra).
|
||||
|
||||
Writing to Kafka is particularly useful for building post-processing data pipelines.
|
||||
|
||||
#### Configuration
|
||||
##### Minimal
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=kafka \
|
||||
-e KAFKA_BROKERS=<...> \
|
||||
-e KAFKA_TOPIC=<...> \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
##### All options
|
||||
To view the full list of configuration options, you can run the following command:
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=kafka \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}} \
|
||||
--help
|
||||
```
|
||||
|
||||
#### Topic & partitions
|
||||
Unless your Kafka cluster is configured to automatically create topics, you will need to create it ahead of time. You can refer to [the Kafka quickstart documentation](https://kafka.apache.org/documentation/#quickstart_createtopic) to learn how.
|
||||
|
||||
You can find more information about topics and partitions in general in the [official documentation](https://kafka.apache.org/documentation/#intro_topics). [This article](https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/) provide more details about how to choose the number of partitions.
|
||||
|
||||
### Storage plugin
|
||||
|
||||
Jaeger supports gRPC based storage plugins. For more information refer to [jaeger/plugin/storage/grpc](https://github.com/jaegertracing/jaeger/tree/master/plugin/storage/grpc)
|
||||
|
||||
Available plugins:
|
||||
|
||||
* [InfluxDB](https://github.com/influxdata/jaeger-influxdb/)
|
||||
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=grpc-plugin \
|
||||
-e GRPC_STORAGE_PLUGIN_BINARY=<...> \
|
||||
-e GRPC_STORAGE_PLUGIN_CONFIGURATION_FILE=<...> \
|
||||
jaegertracing/all-in-one:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
## Ingester
|
||||
**jaeger-ingester** is a service which reads span data from Kafka topic and writes it to another storage backend (Elasticsearch or Cassandra).
|
||||
|
||||
Port | Protocol | Function
|
||||
----- | ------- | ---
|
||||
14270 | HTTP | Healthcheck at `/` and metrics at `/metrics`
|
||||
|
||||
To view all exposed configuration options run the following command:
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=cassandra \
|
||||
jaegertracing/jaeger-ingester:{{< currentVersion >}}
|
||||
--help
|
||||
```
|
||||
|
||||
## Query Service & UI
|
||||
|
||||
**jaeger-query** serves the API endpoints and a React/Javascript UI.
|
||||
The service is stateless and is typically run behind a load balancer, such as [**NGINX**](https://www.nginx.com/).
|
||||
|
||||
At default settings the query service exposes the following port(s):
|
||||
|
||||
Port | Protocol | Function
|
||||
----- | ------- | ---
|
||||
16686 | HTTP | `/api/*` endpoints and Jaeger UI at `/`
|
||||
16687 | HTTP | Healthcheck at `/` and metrics at `/metrics`
|
||||
|
||||
### Minimal deployment example (Elasticsearch backend):
|
||||
```sh
|
||||
docker run -d --rm \
|
||||
-p 16686:16686 \
|
||||
-p 16687:16687 \
|
||||
-e SPAN_STORAGE_TYPE=elasticsearch \
|
||||
-e ES_SERVER_URLS=http://<ES_SERVER_IP>:<ES_SERVER_PORT> \
|
||||
jaegertracing/jaeger-query:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
### UI Base Path
|
||||
|
||||
The base path for all **jaeger-query** HTTP routes can be set to a non-root value, e.g. `/jaeger` would cause all UI URLs to start with `/jaeger`. This can be useful when running **jaeger-query** behind a reverse proxy.
|
||||
|
||||
The base path can be configured via the `--query.base-path` command line parameter or the `QUERY_BASE_PATH` environment variable.
|
||||
|
||||
### UI Customization and Embedding
|
||||
|
||||
Please refer to the [dedicated Frontend/UI page](../frontend-ui/).
|
||||
|
||||
## Aggregation Jobs for Service Dependencies
|
||||
|
||||
Production deployments need an external process which aggregates data and creates dependency links between services. Project [spark-dependencies](https://github.com/jaegertracing/spark-dependencies) is a Spark job which derives dependency links and stores them directly to the storage.
|
||||
|
||||
## Configuration
|
||||
|
||||
All binaries accepts command line properties and environmental variables, power by [viper](https://github.com/spf13/viper) and [cobra](https://github.com/spf13/cobra) libraries. Please refer to the [CLI Flags](../cli/) page for more information.
|
||||
|
||||
[cqlsh]: http://cassandra.apache.org/doc/latest/tools/cqlsh.html
|
||||
[zipkin-thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/zipkincore.thrift
|
||||
[jaeger-thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/jaeger.thrift
|
||||
[thriftrw]: https://www.npmjs.com/package/thriftrw
|
|
@ -1,26 +0,0 @@
|
|||
---
|
||||
title: Frequently Asked Questions
|
||||
navtitle: FAQs
|
||||
description: Answers to some frequently asked questions about Jaeger.
|
||||
weight: 11
|
||||
---
|
||||
|
||||
## Why is the Dependencies page empty?
|
||||
|
||||
The Dependencies page shows a graph of services traced by Jaeger and connections between them. When you are using `all-in-one` binary with in-memory storage, the graph is calculated on-demand from all the traces stored in memory. However, if you are using a real distributed storage like Cassandra or Elasticsearch, it is too expensive to scan all the data in the database to build the service graph. Instead, the Jaeger project provides "big data" jobs that can be used to extract the service graph data from traces:
|
||||
|
||||
* https://github.com/jaegertracing/spark-dependencies - the older Spark job that can be run periodically
|
||||
* https://github.com/jaegertracing/jaeger-analytics - the new (experimental) streaming Flink jobs that run continuously and builds the service graph in smaller time intervals
|
||||
|
||||
## Why do I not see any spans in Jaeger?
|
||||
|
||||
Please refer to the [Troubleshooting](../troubleshooting/) guide.
|
||||
|
||||
## Do I need to run jaeger-agent?
|
||||
|
||||
`jaeger-agent` is not always necessary. Jaeger client libraries can be configured to export trace data directly to `jaeger-collector`. However, the following are the reasons why running `jaeger-agent` is recommended:
|
||||
|
||||
* If we want Jaeger client libraries to send trace data directly to collectors, we must provide them with a URL of the HTTP endpoint. It means that our applications require additional configuration containing this parameter, especially if we are running multiple Jaeger installations (e.g. in different availability zones or regions) and want the data sent to a nearby installation. In contrast, when using the agent, the libraries require no additional configuration because the agent is always accessible via `localhost`. It acts as a sidecar and proxies the requests to the appropriate collectors.
|
||||
* The agent can be configured to enrich the tracing data with infrastructure-specific metadata by adding extra tags to the spans, such as the current zone, region, etc. If the agent is running as a host daemon, it will be shared by all applications running on the same host. If the agent is running as a true sidecar, i.e. one per application, it can provide additional functionality such as strong authentication, multi-tenancy (see [this blog post](https://medium.com/jaegertracing/jaeger-and-multitenancy-99dfa1d49dc0)), pod name, etc.
|
||||
* If we want Jaeger client libraries to use sampling strategies that are centrally configured in the collectors, this is only possible by using the `/sampling` HTTP endpoint on the agent. There is no technical reason why this endpoint cannot be implemented directly in the collectors, it's just [not done yet](https://github.com/jaegertracing/jaeger/issues/1971).
|
||||
* Agents allow implementing traffic control to the collectors. If we have thousands of hosts in the data center, each running many applications, and each application sending data directly to the collectors, there may be too many open connections for each collector to handle. The agents can load balance this traffic with fewer connections.
|
|
@ -1,57 +0,0 @@
|
|||
---
|
||||
title: Features
|
||||
hasparent: true
|
||||
---
|
||||
|
||||
Jaeger is used for monitoring and troubleshooting microservices-based distributed systems, including:
|
||||
|
||||
* Distributed context propagation
|
||||
* Distributed transaction monitoring
|
||||
* Root cause analysis
|
||||
* Service dependency analysis
|
||||
* Performance / latency optimization
|
||||
|
||||
## High Scalability
|
||||
|
||||
Jaeger backend is designed to have no single points of failure and to scale with the business needs.
|
||||
For example, any given Jaeger installation at Uber is typically processing several billion {{< tip "spans" "span" >}} per day.
|
||||
|
||||
## Native support for OpenTracing
|
||||
|
||||
Jaeger backend, Web UI, and instrumentation libraries have been designed from ground up to support the OpenTracing standard.
|
||||
|
||||
* Represent {{< tip "traces" "trace" >}} as {{< tip "directed acyclic graphs" "directed acyclic graph" >}} (not just trees) via [span references](https://github.com/opentracing/specification/blob/master/specification.md#references-between-spans)
|
||||
* Support strongly typed span _tags_ and _structured logs_
|
||||
* Support general distributed context propagation mechanism via _baggage_
|
||||
|
||||
## Multiple storage backends
|
||||
|
||||
Jaeger supports two popular open source NoSQL databases as trace storage backends: Cassandra 3.4+ and Elasticsearch 5.x/6.x/7.x.
|
||||
There are ongoing community experiments using other databases, such as ScyllaDB, InfluxDB, Amazon DynamoDB. Jaeger also ships
|
||||
with a simple in-memory storage for testing setups.
|
||||
|
||||
## Modern Web UI
|
||||
|
||||
Jaeger Web UI is implemented in Javascript using popular open source frameworks like React. Several performance
|
||||
improvements have been released in v1.0 to allow the UI to efficiently deal with large volumes of data, and to display
|
||||
{{< tip "traces" "trace" >}} with tens of thousands of {{< tip "spans" "span" >}} (e.g. we tried a trace with 80,000 spans).
|
||||
|
||||
## Cloud Native Deployment
|
||||
|
||||
Jaeger backend is distributed as a collection of Docker images. The binaries support various configuration methods,
|
||||
including command line options, environment variables, and configuration files in multiple formats (yaml, toml, etc.).
|
||||
Deployment to Kubernetes clusters is assisted by a [Kubernetes operator](https://github.com/jaegertracing/jaeger-operator), [Kubernetes templates](https://github.com/jaegertracing/jaeger-kubernetes)
|
||||
and a [Helm chart](https://github.com/kubernetes/charts/tree/master/incubator/jaeger).
|
||||
|
||||
## Observability
|
||||
|
||||
All Jaeger backend components expose [Prometheus](https://prometheus.io/) metrics by default (other metrics backends are
|
||||
also supported). Logs are written to standard out using the structured logging library [zap](https://github.com/uber-go/zap).
|
||||
|
||||
## Backwards compatibility with Zipkin
|
||||
|
||||
Although we recommend instrumenting applications with OpenTracing API and binding to Jaeger client libraries to benefit
|
||||
from advanced features not available elsewhere, if your organization has already invested in the instrumentation
|
||||
using Zipkin libraries, you do not have to rewrite all that code. Jaeger provides backwards compatibility with Zipkin
|
||||
by accepting spans in Zipkin formats (Thrift, JSON v1/v2 and Protobuf) over HTTP. Switching from Zipkin backend is just a matter
|
||||
of routing the traffic from Zipkin libraries to the Jaeger backend.
|
|
@ -1,224 +0,0 @@
|
|||
---
|
||||
title: Frontend/UI Configuration
|
||||
navtitle: Frontend/UI
|
||||
hasparent: true
|
||||
weight: 7
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
Several aspects of the UI can be configured:
|
||||
|
||||
* The Dependencies section can be enabled / configured
|
||||
* Google Analytics tracking can be enabled / configured
|
||||
* Additional menu options can be added to the global nav
|
||||
|
||||
These options can be configured by a JSON configuration file. The `--query.ui-config` command line parameter of the query service must then be set to the path to the JSON file when the query service is started.
|
||||
|
||||
An example configuration file:
|
||||
|
||||
```json
|
||||
{
|
||||
"dependencies": {
|
||||
"dagMaxNumServices": 200,
|
||||
"menuEnabled": true
|
||||
},
|
||||
"archiveEnabled": true,
|
||||
"tracking": {
|
||||
"gaID": "UA-000000-2",
|
||||
"trackErrors": true
|
||||
},
|
||||
"menu": [
|
||||
{
|
||||
"label": "About Jaeger",
|
||||
"items": [
|
||||
{
|
||||
"label": "GitHub",
|
||||
"url": "https://github.com/jaegertracing/jaeger"
|
||||
},
|
||||
{
|
||||
"label": "Docs",
|
||||
"url": "http://jaeger.readthedocs.io/en/latest/"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"linkPatterns": [{
|
||||
"type": "process",
|
||||
"key": "jaeger.version",
|
||||
"url": "https://github.com/jaegertracing/jaeger-client-java/releases/tag/#{jaeger.version}",
|
||||
"text": "Information about Jaeger release #{jaeger.version}"
|
||||
}]
|
||||
}
|
||||
```
|
||||
|
||||
### Dependencies
|
||||
|
||||
`dependencies.dagMaxNumServices` defines the maximum number of services allowed before the DAG dependency view is disabled. Default: `200`.
|
||||
|
||||
`dependencies.menuEnabled` enables (`true`) or disables (`false`) the dependencies menu button. Default: `true`.
|
||||
|
||||
### Archive Support
|
||||
|
||||
`archiveEnabled` enables (`true`) or disables (`false`) the archive traces button. Default: `false`. It requires a configuration of an archive storage in Query service. Archived traces are only accessible directly by ID, they are not searchable.
|
||||
|
||||
### Google Analytics Tracking
|
||||
|
||||
`tracking.gaID` defines the Google Analytics tracking ID. This is required for Google Analytics tracking, and setting it to a non-`null` value enables Google Analytics tracking. Default: `null`.
|
||||
|
||||
`tracking.trackErrors` enables (`true`) or disables (`false`) error tracking via Google Analytics. Errors can only be tracked if a valid Google Analytics ID is provided. For additional details on error tracking via Google Analytics see the [tracking README](https://github.com/jaegertracing/jaeger-ui/blob/c622330546afc1be59a42f874bcc1c2fadf7e69a/src/utils/tracking/README.md) in the UI repo. Default: `true`.
|
||||
|
||||
### Custom Menu Items
|
||||
|
||||
`menu` allows additional links to be added to the global nav. The additional links are right-aligned.
|
||||
|
||||
In the sample JSON config above, the configured menu will have a dropdown labeled "About Jaeger" with sub-options for "GitHub" and "Docs". The format for a link in the top right menu is as follows:
|
||||
|
||||
```json
|
||||
{
|
||||
"label": "Some text here",
|
||||
"url": "https://example.com"
|
||||
}
|
||||
```
|
||||
|
||||
Links can either be members of the `menu` Array, directly, or they can be grouped into a dropdown menu option. The format for a group of links is:
|
||||
|
||||
```json
|
||||
{
|
||||
"label": "Dropdown button",
|
||||
"items": [ ]
|
||||
}
|
||||
```
|
||||
|
||||
The `items` Array should contain one or more link configurations.
|
||||
|
||||
### Link Patterns
|
||||
|
||||
The `linkPatterns` node can be used to create links from fields displayed in the Jaeger UI.
|
||||
|
||||
Field | Description
|
||||
------|------------
|
||||
type | The metadata section in which your link will be added: process, tags, logs
|
||||
key | The name of tag/process/log attribute which value will be displayed as a link
|
||||
url | The URL where the link should point to, it can be an external site or relative path in Jaeger UI
|
||||
text | The text displayed in the tooltip for the link
|
||||
|
||||
Both `url` and `text` can be defined as templates (i.e. using `#{field-name}`) where Jaeger UI will dynamically substitute values based on tags/logs data.
|
||||
|
||||
## Embedded Mode
|
||||
|
||||
Starting with version 1.9, Jaeger UI provides an "embedded" layout mode which is intended to support integrating Jaeger UI into other applications. Currently (as of `v0`), the approach taken is to remove various UI elements from the page to make the UI better suited for space-constrained layouts.
|
||||
|
||||
The embedded mode is induced and configured via URL query parameters.
|
||||
|
||||
To enter embedded mode, the `uiEmbed=v0` query parameter and value must be added to the URL. For example, the following URL will show the trace with ID `abc123` in embedded mode:
|
||||
|
||||
```
|
||||
http://localhost:16686/trace/abc123?uiEmbed=v0
|
||||
```
|
||||
|
||||
`uiEmbed=v0` is required.
|
||||
|
||||
Further, each page supported has an <img src="/img/frontend-ui/embed-open-icon.png" style="width: 20px; height:20px;display:inline;" alt="Embed open window"> button added that will open the non-embedded page in a new tab.
|
||||
|
||||
The following pages support embedded mode:
|
||||
|
||||
* Search Page
|
||||
* Trace Page
|
||||
|
||||
### Search Page
|
||||
|
||||
To integrate the Search Trace Page to our application we have to indicate to the Jaeger UI that we want to use the embed mode with `uiEmbed=v0`.
|
||||
|
||||
For example:
|
||||
|
||||
```
|
||||
http://localhost:16686/search?
|
||||
service=my-service&
|
||||
start=1543917759557000&
|
||||
end=1543921359557000&
|
||||
limit=20&
|
||||
lookback=1h&
|
||||
maxDuration&
|
||||
minDuration&
|
||||
uiEmbed=v0
|
||||
```
|
||||
|
||||

|
||||
|
||||
#### Configuration options
|
||||
|
||||
The following query parameter can be used to configure the layout of the search page :
|
||||
|
||||
* `uiSearchHideGraph=1` - disables the display of the scatter plot above the search results
|
||||
|
||||
```
|
||||
http://localhost:16686/search?
|
||||
service=my-service&
|
||||
start=1543917759557000&
|
||||
end=1543921359557000&
|
||||
limit=20&
|
||||
lookback=1h&
|
||||
maxDuration&
|
||||
minDuration&
|
||||
uiEmbed=v0&
|
||||
uiSearchHideGraph=1
|
||||
```
|
||||
|
||||

|
||||
|
||||
### Trace Page
|
||||
|
||||
|
||||
To integrate the Trace Page to our application we have to indicate to the Jaeger UI that we want to use the embed mode with `uiEmbed=v0`.
|
||||
|
||||
For example:
|
||||
|
||||
```sh
|
||||
http://localhost:16686/trace/{trace-id}?uiEmbed=v0
|
||||
```
|
||||

|
||||
|
||||
If we have navigated to this view from the search traces page we'll have a button to go back to the results page.
|
||||
|
||||

|
||||
|
||||
#### Configuration options
|
||||
|
||||
The following query parameters can be used to configure the layout of the trace page :
|
||||
|
||||
* `uiTimelineCollapseTitle=1` causes the trace header to start out collapsed, which hides the summary and the minimap.
|
||||
|
||||
```
|
||||
http://localhost:16686/trace/{trace-id}?
|
||||
uiEmbed=v0&
|
||||
uiTimelineCollapseTitle=1
|
||||
```
|
||||

|
||||
|
||||
* `uiTimelineHideMinimap=1` removes the minimap, entirely, regardless of whether the trace header is expanded or not.
|
||||
|
||||
```
|
||||
http://localhost:16686/trace/{trace-id}?
|
||||
uiEmbed=v0&
|
||||
uiTimelineHideMinimap=1
|
||||
```
|
||||

|
||||
|
||||
* `uiTimelineHideSummary=1` - removes the trace summary information (number of services, etc.) entirely, regardless of whether the trace header is expanded or not.
|
||||
|
||||
```
|
||||
http://localhost:16686/trace/{trace-id}?
|
||||
uiEmbed=v0&
|
||||
uiTimelineHideSummary=1
|
||||
```
|
||||

|
||||
|
||||
We can also combine the options:
|
||||
```
|
||||
http://localhost:16686/trace/{trace-id}?
|
||||
uiEmbed=v0&
|
||||
uiTimelineHideMinimap=1&
|
||||
uiTimelineHideSummary=1
|
||||
```
|
||||

|
|
@ -1,129 +0,0 @@
|
|||
---
|
||||
title: Getting Started
|
||||
description: Get up and running with Jaeger in your local environment
|
||||
weight: 2
|
||||
---
|
||||
|
||||
## Instrumentation
|
||||
|
||||
Your applications must be instrumented before they can send tracing data to Jaeger backend. Check the [Client Libraries](../client-libraries) section for information about how to use the OpenTracing API and how to initialize and configure Jaeger tracers.
|
||||
|
||||
## All in One
|
||||
|
||||
All-in-one is an executable designed for quick local testing, launches the Jaeger UI, collector, query, and agent, with an in memory storage component.
|
||||
|
||||
The simplest way to start the all-in-one is to use the pre-built image published to DockerHub (a single command line).
|
||||
|
||||
```bash
|
||||
$ docker run -d --name jaeger \
|
||||
-e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
|
||||
-p 5775:5775/udp \
|
||||
-p 6831:6831/udp \
|
||||
-p 6832:6832/udp \
|
||||
-p 5778:5778 \
|
||||
-p 16686:16686 \
|
||||
-p 14268:14268 \
|
||||
-p 9411:9411 \
|
||||
jaegertracing/all-in-one:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
Or run the `jaeger-all-in-one(.exe)` executable from the [binary distribution archives][download]:
|
||||
|
||||
```bash
|
||||
$ jaeger-all-in-one --collector.zipkin.http-port=9411
|
||||
```
|
||||
|
||||
You can then navigate to `http://localhost:16686` to access the Jaeger UI.
|
||||
|
||||
The container exposes the following ports:
|
||||
|
||||
Port | Protocol | Component | Function
|
||||
----- | ------- | --------- | ---
|
||||
5775 | UDP | agent | accept `zipkin.thrift` over compact thrift protocol (deprecated, used by legacy clients only)
|
||||
6831 | UDP | agent | accept `jaeger.thrift` over compact thrift protocol
|
||||
6832 | UDP | agent | accept `jaeger.thrift` over binary thrift protocol
|
||||
5778 | HTTP | agent | serve configs
|
||||
16686 | HTTP | query | serve frontend
|
||||
14268 | HTTP | collector | accept `jaeger.thrift` directly from clients
|
||||
14250 | HTTP | collector | accept `model.proto`
|
||||
9411 | HTTP | collector | Zipkin compatible endpoint (optional)
|
||||
|
||||
|
||||
## Kubernetes and OpenShift
|
||||
|
||||
* Kubernetes templates: https://github.com/jaegertracing/jaeger-kubernetes
|
||||
* Kubernetes Operator: https://github.com/jaegertracing/jaeger-operator
|
||||
* OpenShift templates: https://github.com/jaegertracing/jaeger-openshift
|
||||
|
||||
## Sample App: HotROD
|
||||
|
||||
HotROD (Rides on Demand) is a demo application that consists of several microservices and
|
||||
illustrates the use of the [OpenTracing API](http://opentracing.io).
|
||||
A tutorial / walkthrough is available in the blog post:
|
||||
[Take OpenTracing for a HotROD ride][hotrod-tutorial].
|
||||
|
||||
It can be run standalone, but requires Jaeger backend to view the traces.
|
||||
|
||||
### Features
|
||||
|
||||
- Discover architecture of the whole system via data-driven dependency
|
||||
diagram.
|
||||
- View request timeline and errors; understand how the app works.
|
||||
- Find sources of latency and lack of concurrency.
|
||||
- Highly contextualized logging.
|
||||
- Use baggage propagation to:
|
||||
|
||||
- Diagnose inter-request contention (queueing).
|
||||
- Attribute time spent in a service.
|
||||
|
||||
- Use open source libraries with OpenTracing integration to get
|
||||
vendor-neutral instrumentation for free.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- You need Go 1.11 or higher installed on your machine to run from source.
|
||||
- Requires a [running Jaeger backend](#all-in-one) to view the traces.
|
||||
|
||||
### Running
|
||||
|
||||
#### From Source
|
||||
|
||||
```bash
|
||||
mkdir -p $GOPATH/src/github.com/jaegertracing
|
||||
cd $GOPATH/src/github.com/jaegertracing
|
||||
git clone git@github.com:jaegertracing/jaeger.git jaeger
|
||||
cd jaeger
|
||||
make install
|
||||
go run ./examples/hotrod/main.go all
|
||||
```
|
||||
#### From docker
|
||||
|
||||
```bash
|
||||
$ docker run --rm -it \
|
||||
--link jaeger \
|
||||
-p8080-8083:8080-8083 \
|
||||
-e JAEGER_AGENT_HOST="jaeger" \
|
||||
jaegertracing/example-hotrod:{{< currentVersion >}} \
|
||||
all
|
||||
```
|
||||
|
||||
#### From binary distribution
|
||||
|
||||
Run `example-hotrod(.exe)` executable from the [binary distribution archives][download]:
|
||||
```bash
|
||||
$ example-hotrod all
|
||||
```
|
||||
|
||||
Then navigate to `http://localhost:8080`.
|
||||
|
||||
|
||||
## Migrating from Zipkin
|
||||
|
||||
Collector service exposes Zipkin compatible REST API `/api/v1/spans` which accepts both Thrift and JSON. Also there is `/api/v2/spans` for JSON and Proto.
|
||||
By default it's disabled. It can be enabled with `--collector.zipkin.http-port=9411`.
|
||||
|
||||
Zipkin [Thrift](https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/zipkincore.thrift) IDL and Zipkin [Proto](https://github.com/jaegertracing/jaeger-idl/blob/master/proto/zipkin.proto) IDL files can be found in [jaegertracing/jaeger-idl](https://github.com/jaegertracing/jaeger-idl) repository.
|
||||
They're compatible with [openzipkin/zipkin-api](https://github.com/openzipkin/zipkin-api) [Thrift](https://github.com/openzipkin/zipkin-api/blob/master/thrift/zipkinCore.thrift) and [Proto](https://github.com/openzipkin/zipkin-api/blob/master/zipkin.proto).
|
||||
|
||||
[hotrod-tutorial]: https://medium.com/@YuriShkuro/take-opentracing-for-a-hotrod-ride-f6e3141f7941
|
||||
[download]: ../../../download/
|
|
@ -1,903 +0,0 @@
|
|||
---
|
||||
title: Operator for Kubernetes
|
||||
hasparent: true
|
||||
---
|
||||
|
||||
# Understanding Operators
|
||||
|
||||
The Jaeger Operator is an implementation of a [Kubernetes Operator](https://coreos.com/operators/). Operators are pieces of software that ease the operational complexity of running another piece of software. More technically, _Operators_ are a method of packaging, deploying, and managing a Kubernetes application.
|
||||
|
||||
A Kubernetes application is an application that is both deployed on Kubernetes and managed using the Kubernetes APIs and `kubectl` (kubernetes) or `oc` (OKD) tooling. To be able to make the most of Kubernetes, you need a set of cohesive APIs to extend in order to service and manage your apps that run on Kubernetes. Think of Operators as the runtime that manages this type of app on Kubernetes.
|
||||
|
||||
# Installing the Operator
|
||||
|
||||
{{< info >}}
|
||||
The Jaeger Operator version tracks one version of the Jaeger components (Query, Collector, Agent). When a new version of the Jaeger components is released, a new version of the operator will be released that understands how running instances of the previous version can be upgraded to the new version.
|
||||
{{< /info >}}
|
||||
|
||||
## Installing the Operator on Kubernetes
|
||||
|
||||
The following instructions will create the `observability` namespace and install the Jaeger Operator.
|
||||
|
||||
{{< info >}}
|
||||
Make sure your `kubectl` command is properly configured to talk to a valid Kubernetes cluster. If you don't have a cluster, you can create one locally using [`minikube`](https://kubernetes.io/docs/tasks/tools/install-minikube/).
|
||||
{{< /info >}}
|
||||
|
||||
To install the operator, run:
|
||||
<!--TODO - Does Kubernetes have privileged users? Needs to be run as a system:admin on OKD/OpenShift.-->
|
||||
|
||||
```bash
|
||||
kubectl create namespace observability # <1>
|
||||
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/crds/jaegertracing.io_jaegers_crd.yaml # <2>
|
||||
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/service_account.yaml
|
||||
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role.yaml
|
||||
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role_binding.yaml
|
||||
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/operator.yaml
|
||||
```
|
||||
<1> This creates the namespace used by default in the deployment files. If you want to install the Jaeger operator in a different namespace, you must edit the deployment files to change `observability` to the desired namespace value.
|
||||
|
||||
<2> This installs the "Custom Resource Definition" for the `apiVersion: jaegertracing.io/v1`
|
||||
|
||||
At this point, there should be a `jaeger-operator` deployment available. You can view it by running the following command:
|
||||
|
||||
```bash
|
||||
$ kubectl get deployment jaeger-operator -n observability
|
||||
|
||||
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
|
||||
jaeger-operator 1 1 1 1 48s
|
||||
```
|
||||
|
||||
The operator is now ready to create Jaeger instances.
|
||||
|
||||
## Installing the Operator on OKD/OpenShift
|
||||
|
||||
<!-- TODO: Add instructions for installing via the operatorhub? -->
|
||||
|
||||
The instructions from the previous section also work for installing the operator on OKD or OpenShift. Make sure you are logged in as a privileged user, when you install the role based acces control (RBAC) rules, the custom resource definition, and the operator.
|
||||
|
||||
```bash
|
||||
oc login -u <privileged user>
|
||||
|
||||
oc new-project observability # <1>
|
||||
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/crds/jaegertracing.io_jaegers_crd.yaml # <2>
|
||||
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/service_account.yaml
|
||||
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role.yaml
|
||||
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role_binding.yaml
|
||||
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/operator.yaml
|
||||
```
|
||||
<1> This creates the namespace used by default in the deployment files. If you want to install the Jaeger operator in a different namespace, you must edit the deployment files to change `observability` to the desired namespace value.
|
||||
|
||||
<2> This installs the "Custom Resource Definition" for the `apiVersion: jaegertracing.io/v1`
|
||||
|
||||
Once the operator is installed, grant the role `jaeger-operator` to users who should be able to install individual Jaeger instances. The following example creates a role binding allowing the user `developer` to create Jaeger instances:
|
||||
|
||||
```bash
|
||||
oc create \
|
||||
rolebinding developer-jaeger-operator \
|
||||
--role=jaeger-operator \
|
||||
--user=developer
|
||||
```
|
||||
|
||||
After the role is granted, switch back to a non-privileged user.
|
||||
|
||||
# Quick Start - Deploying the AllInOne image
|
||||
|
||||
The simplest possible way to create a Jaeger instance is by creating a YAML file like the following example. This will install the default AllInOne strategy, which deploys the "all-in-one" image (agent, collector, query, ingester, Jaeger UI) in a single pod, using in-memory storage by default.
|
||||
|
||||
{{< info >}}
|
||||
This default strategy is intended for development, testing, and demo purposes, not for production.
|
||||
{{< /info >}}
|
||||
|
||||
```yaml
|
||||
apiVersion: jaegertracing.io/v1
|
||||
kind: Jaeger
|
||||
metadata:
|
||||
name: simplest
|
||||
```
|
||||
|
||||
The YAML file can then be used with `kubectl`:
|
||||
<!-- TODO - Add OKD commands and tabs shortcode. -->
|
||||
|
||||
```bash
|
||||
kubectl apply -f simplest.yaml
|
||||
```
|
||||
|
||||
In a few seconds, a new in-memory all-in-one instance of Jaeger will be available, suitable for quick demos and development purposes. To check the instances that were created, list the `jaeger` objects:
|
||||
|
||||
```bash
|
||||
$ kubectl get jaegers
|
||||
NAME CREATED AT
|
||||
simplest 28s
|
||||
```
|
||||
|
||||
To get the pod name, query for the pods belonging to the `simplest` Jaeger instance:
|
||||
|
||||
|
||||
```bash
|
||||
$ kubectl get pods -l app.kubernetes.io/instance=simplest
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
simplest-6499bb6cdd-kqx75 1/1 Running 0 2m
|
||||
```
|
||||
|
||||
Similarly, the logs can be queried either from the pod directly using the pod name obtained from the previous example, or from all pods belonging to our instance:
|
||||
|
||||
```bash
|
||||
$ kubectl logs -l app.kubernetes.io/instance=simplest
|
||||
...
|
||||
{"level":"info","ts":1535385688.0951214,"caller":"healthcheck/handler.go:133","msg":"Health Check state change","status":"ready"}
|
||||
```
|
||||
|
||||
{{< info >}}
|
||||
On OKD/OpenShift the container name must be specified.
|
||||
{{< /info >}}
|
||||
|
||||
```bash
|
||||
$ kubectl logs -l app.kubernetes.io/instance=simplest -c jaeger
|
||||
...
|
||||
{"level":"info","ts":1535385688.0951214,"caller":"healthcheck/handler.go:133","msg":"Health Check state change","status":"ready"}
|
||||
```
|
||||
|
||||
# Deployment Strategies
|
||||
|
||||
When you create a Jaeger instance, it is associated with a strategy. The strategy is defined in the custom resource file, and determines the architecture to be used for the Jaeger backend. The default strategy is `allInOne`. The other possible values are `production` and `streaming`.
|
||||
|
||||
The available strategies are described in the following sections.
|
||||
|
||||
## AllInOne (Default) strategy
|
||||
|
||||
This strategy is intended for development, testing, and demo purposes.
|
||||
|
||||
The main backend components, agent, collector and query service, are all packaged into a single executable which is configured (by default) to use in-memory storage.
|
||||
|
||||
## Production strategy
|
||||
|
||||
The `production` strategy is intended (as the name suggests) for production environments, where long term storage of trace data is important, as well as a more scalable and highly available architecture is required. Each of the backend components is therefore separately deployed.
|
||||
|
||||
The agent can be injected as a sidecar on the instrumented application or as a daemonset.
|
||||
|
||||
The query and collector services are configured with a supported storage type - currently Cassandra or Elasticsearch. Multiple instances of each of these components can be provisioned as required for performance and resilience purposes.
|
||||
|
||||
The main additional requirement is to provide the details of the storage type and options, for example:
|
||||
|
||||
```yaml
|
||||
storage:
|
||||
type: elasticsearch
|
||||
options:
|
||||
es:
|
||||
server-urls: http://elasticsearch:9200
|
||||
```
|
||||
|
||||
## Streaming strategy
|
||||
|
||||
The `streaming` strategy is designed to augment the `production` strategy by providing a streaming capability that effectively sits between the collector and the backend storage (Cassandra or Elasticsearch). This provides the benefit of reducing the pressure on the backend storage, under high load situations, and enables other trace post-processing capabilities to tap into the real time span data directly from the streaming platform (Kafka).
|
||||
|
||||
The only additional information required is to provide the details for accessing the Kafka platform, which is configured in the `collector` component (as producer) and `ingester` component (as consumer):
|
||||
|
||||
```yaml
|
||||
apiVersion: jaegertracing.io/v1
|
||||
kind: Jaeger
|
||||
metadata:
|
||||
name: simple-streaming
|
||||
spec:
|
||||
strategy: streaming
|
||||
collector:
|
||||
options:
|
||||
kafka: # <1>
|
||||
producer:
|
||||
topic: jaeger-spans
|
||||
brokers: my-cluster-kafka-brokers.kafka:9092
|
||||
ingester:
|
||||
options:
|
||||
kafka: # <1>
|
||||
consumer:
|
||||
topic: jaeger-spans
|
||||
brokers: my-cluster-kafka-brokers.kafka:9092
|
||||
ingester:
|
||||
deadlockInterval: 0 # <2>
|
||||
storage:
|
||||
type: elasticsearch
|
||||
options:
|
||||
es:
|
||||
server-urls: http://elasticsearch:9200
|
||||
```
|
||||
<1> Identifies the Kafka configuration used by the collector, to produce the messages, and the ingester to consume the messages.
|
||||
|
||||
<2> The deadlock interval can be disabled to avoid the ingester being terminated when no messages arrive within the default 1 minute period
|
||||
|
||||
{{< info >}}
|
||||
A Kafka environment can be configured using [Strimzi's Kafka operator](https://strimzi.io/).
|
||||
{{< /info >}}
|
||||
|
||||
# Understanding Custom Resource Definitions
|
||||
|
||||
In the Kubernetes API, a resource is an endpoint that stores a collection of API objects of a certain kind. For example, the built-in Pods resource contains a collection of Pod objects. A _Custom Resource Definition_ (CRD) object defines a new, unique object `Kind` in the cluster and lets the Kubernetes API server handle its entire lifecycle.
|
||||
|
||||
To create _Custom Resource_ (CR) objects, cluster administrators must first create a Custom Resource Definition (CRD). The CRDs allow cluster users to create CRs to add the new resource types into their projects. An Operator watches for custom resource objects to be created, and when it sees a custom resource being created, it creates the application based on the parameters defined in the custom resource object.
|
||||
|
||||
{{< info >}}
|
||||
While only cluster administrators can create CRDs, developers can create the CR from an existing CRD if they have read and write permission to it.
|
||||
{{< /info >}}
|
||||
|
||||
<!--
|
||||
## Jaeger Custom Resource Parameters
|
||||
|
||||
TODO Create a TABLE with all the parameters, descriptions/notes, valid values, and defaults.
|
||||
Figure out if we can generate the options? Can we filter them in any way?
|
||||
https://github.com/jaegertracing/jaeger/issues/1537
|
||||
https://github.com/jaegertracing/documentation/issues/250-->
|
||||
|
||||
For reference, here's how you can create a more complex all-in-one instance:
|
||||
|
||||
```yaml
|
||||
apiVersion: jaegertracing.io/v1
|
||||
kind: Jaeger
|
||||
metadata:
|
||||
name: my-jaeger
|
||||
spec:
|
||||
strategy: allInOne # <1>
|
||||
allInOne:
|
||||
image: jaegertracing/all-in-one:latest # <2>
|
||||
options: # <3>
|
||||
log-level: debug # <4>
|
||||
storage:
|
||||
type: memory # <5>
|
||||
options: # <6>
|
||||
memory: # <7>
|
||||
max-traces: 100000
|
||||
ingress:
|
||||
enabled: false # <8>
|
||||
agent:
|
||||
strategy: DaemonSet # <9>
|
||||
annotations:
|
||||
scheduler.alpha.kubernetes.io/critical-pod: "" # <10>
|
||||
```
|
||||
|
||||
<1> The default strategy is `allInOne`. The other possible values are `production` and `streaming`.
|
||||
|
||||
<2> The image to use, in a regular Docker syntax.
|
||||
|
||||
<3> The (non-storage related) options to be passed verbatim to the underlying binary. Refer to the Jaeger documentation and/or to the `--help` option from the related binary for all the available options.
|
||||
|
||||
<4> The option is a simple `key: value` map. In this case, we want the option `--log-level=debug` to be passed to the binary.
|
||||
|
||||
<5> The storage type to be used. By default it will be `memory`, but can be any other supported storage type (Cassandra, Elasticsearch, Kafka).
|
||||
|
||||
<6> All storage related options should be placed here, rather than under the 'allInOne' or other component options.
|
||||
|
||||
<7> Some options are namespaced and we can alternatively break them into nested objects. We could have specified `memory.max-traces: 100000`.
|
||||
|
||||
<8> By default, an ingress object is created for the query service. It can be disabled by setting its `enabled` option to `false`. If deploying on OpenShift, this will be represented by a Route object.
|
||||
|
||||
<9> By default, the operator assumes that agents are deployed as sidecars within the target pods. Specifying the strategy as "DaemonSet" changes that and makes the operator deploy the agent as DaemonSet. Note that your tracer client will probably have to override the "JAEGER_AGENT_HOST" environment variable to use the node's IP.
|
||||
|
||||
<10> Define annotations to be applied to all deployments (not services). These can be overridden by annotations defined on the individual components.
|
||||
|
||||
You can view example custom resources for different Jaeger configurations [on GitHub](https://github.com/jaegertracing/jaeger-operator/tree/master/examples).
|
||||
|
||||
# Configuring the Custom Resource
|
||||
|
||||
<!--TODO
|
||||
esIndexCleaner
|
||||
Spark dependencies
|
||||
-->
|
||||
|
||||
You can use the simplest example (shown above) and create a Jaeger instance using the defaults, or you can create your own custom resource file.
|
||||
|
||||
## Storage options
|
||||
|
||||
### Cassandra storage
|
||||
|
||||
When the storage type is set to Cassandra, the operator will automatically create a batch job that creates the required schema for Jaeger to run. This batch job will block the Jaeger installation, so that it starts only after the schema is successfuly created. The creation of this batch job can be disabled by setting the `enabled` property to `false`:
|
||||
|
||||
```yaml
|
||||
apiVersion: jaegertracing.io/v1
|
||||
kind: Jaeger
|
||||
metadata:
|
||||
name: cassandra-without-create-schema
|
||||
spec:
|
||||
strategy: allInOne
|
||||
storage:
|
||||
type: cassandra
|
||||
cassandraCreateSchema:
|
||||
enabled: false # <1>
|
||||
```
|
||||
<1> Defaults to `true`
|
||||
|
||||
Further aspects of the batch job can be configured as well. An example with all the possible options is shown below:
|
||||
|
||||
```yaml
|
||||
apiVersion: jaegertracing.io/v1
|
||||
kind: Jaeger
|
||||
metadata:
|
||||
name: cassandra-with-create-schema
|
||||
spec:
|
||||
strategy: allInOne # <1>
|
||||
storage:
|
||||
type: cassandra
|
||||
options: # <2>
|
||||
cassandra:
|
||||
servers: cassandra
|
||||
keyspace: jaeger_v1_datacenter3
|
||||
cassandraCreateSchema: # <3>
|
||||
datacenter: "datacenter3"
|
||||
mode: "test"
|
||||
```
|
||||
<1> The same works for `production` and `streaming`.
|
||||
|
||||
<2> These options are for the regular Jaeger components, like `collector` and `query`.
|
||||
|
||||
<3> The options for the `create-schema` job.
|
||||
|
||||
{{< info >}}
|
||||
The default create-schema job uses `MODE=prod`, which implies a replication factor of `2`, using `NetworkTopologyStrategy` as the class, effectively meaning that at least 3 nodes are required in the Cassandra cluster. If a `SimpleStrategy` is desired, set the mode to `test`, which then sets the replication factor of `1`. Refer to the [create-schema script](https://github.com/jaegertracing/jaeger/blob/master/plugin/storage/cassandra/schema/create.sh) for more details.
|
||||
{{< /info >}}
|
||||
|
||||
### Elasticsearch storage
|
||||
|
||||
By default Elasticsearch storage does not require any initialization job to be run. However Elasticsearch
|
||||
storage requires a cron job to be run to clean old data from the storage.
|
||||
|
||||
When rollover (`es.use-aliases`) is enabled, Jaeger operator also deploys a job to initialize Elasticsearch storage
|
||||
and another two cron jobs to perform required index management actions.
|
||||
|
||||
#### External Elasticsearch
|
||||
|
||||
Jaeger can be used with an external Elasticsearch cluster.
|
||||
The following example shows a Jaeger CR using an external Elasticsearch cluster
|
||||
with TLS CA certificate mounted from a volume and user/password stored in a secret.
|
||||
|
||||
```yaml
|
||||
apiVersion: jaegertracing.io/v1
|
||||
kind: Jaeger
|
||||
metadata:
|
||||
name: simple-prod
|
||||
spec:
|
||||
strategy: production
|
||||
storage:
|
||||
type: elasticsearch # <1>
|
||||
options:
|
||||
es:
|
||||
server-urls: https://elasticsearch.default.svc:9200 # <2>
|
||||
tls: # <3>
|
||||
ca: /es/certificates/root-ca.pem
|
||||
secretName: jaeger-secret # <4>
|
||||
volumeMounts: # <5>
|
||||
- name: certificates
|
||||
mountPath: /es/certificates/
|
||||
readOnly: true
|
||||
volumes:
|
||||
- name: certificates
|
||||
secret:
|
||||
secretName: quickstart-es-http-certs-public
|
||||
```
|
||||
|
||||
<1> Storage type Elasticsearch.
|
||||
|
||||
<2> Url to Elasticsearch service running in default namespace.
|
||||
|
||||
<3> TLS configuration. In this case only CA certificate, but it can also contain `es.tls.key` and `es.tls.cert` when using mutual TLS.
|
||||
|
||||
<4> Secret which defines environment variables `ES_PASSWORD` and `ES_USERNAME`. Created by `kubectl create secret generic jaeger-secret --from-literal=ES_PASSWORD=changeme --from-literal=ES_USERNAME=elastic`
|
||||
|
||||
<5> Volume mounts and volumes which are mounted into all storage components.
|
||||
|
||||
#### Self provisioned
|
||||
Under some circumstances, the Jaeger Operator can make use of the [Elasticsearch Operator](https://github.com/openshift/elasticsearch-operator) to provision a suitable Elasticsearch cluster.
|
||||
|
||||
{{< warning >}}
|
||||
This feature is supported only on OKD/OpenShift clusters. Spark dependencies are not supported with this feature [Issue #294](https://github.com/jaegertracing/jaeger-operator/issues/294).
|
||||
{{< /warning >}}
|
||||
|
||||
When there is no `es.server-urls` option as part of a Jaeger `production` instance and `elasticsearch` is set as the storage type, the Jaeger Operator creates an Elasticsearch cluster via the Elasticsearch Operator by creating a Custom Resource based on the configuration provided in storage section. The Elasticsearch cluster is meant to be dedicated for a single Jaeger instance.
|
||||
|
||||
The self-provision of an Elasticsearch cluster can be disabled by setting the flag `--es-provision` to `false`. The default value is `auto`, which will make the Jaeger Operator query the Kubernetes cluster for its ability to handle a `Elasticsearch` custom resource. This is usually set by the Elasticsearch Operator during its installation process, so, if the Elasticsearch Operator is expected to run *after* the Jaeger Operator, the flag can be set to `true`.
|
||||
|
||||
{{< danger >}}
|
||||
At the moment there can be only one Jaeger with self-provisioned Elasticsearch instance per namespace.
|
||||
{{< /danger >}}
|
||||
|
||||
#### Elasticsearch index cleaner job
|
||||
|
||||
When using `elasticsearch` storage by default a cron job is created to clean old traces from it, the options for it are listed below so you can configure it to your use case.
|
||||
The connection configuration is derived from the storage options.
|
||||
|
||||
```yaml
|
||||
storage:
|
||||
type: elasticsearch
|
||||
esIndexCleaner:
|
||||
enabled: true // turn the cron job deployment on and off
|
||||
numberOfDays: 7 // number of days to wait before deleting a record
|
||||
schedule: "55 23 * * *" // cron expression for it to run
|
||||
```
|
||||
|
||||
The connection configuration to storage is derived from storage options.
|
||||
|
||||
#### Elasticsearch rollover
|
||||
|
||||
This index management strategy is more complicated than using the default daily indices and
|
||||
it requires an initialisation job to prepare the storage and two cron jobs to manage indices.
|
||||
The first cron job is used for rolling-over to a new index and the second for removing
|
||||
indices from read alias. The rollover feature is used when storage option `es.use-aliases` is enabled.
|
||||
|
||||
To learn more about rollover index management in Jaeger refer to this
|
||||
[article](https://medium.com/jaegertracing/using-elasticsearch-rollover-to-manage-indices-8b3d0c77915d).
|
||||
|
||||
```yaml
|
||||
storage:
|
||||
type: elasticsearch
|
||||
options:
|
||||
es:
|
||||
use-aliases: true
|
||||
esRollover:
|
||||
enabled: true // turn the cron job deployment on and off
|
||||
conditions: "{\"max_age\": \"2d\"}" // conditions when to rollover to a new index
|
||||
readTTL: 7d // how long should be old data available for reading
|
||||
schedule: "55 23 * * *" // cron expression for it to run
|
||||
```
|
||||
|
||||
The connection configuration to storage is derived from storage options.
|
||||
|
||||
## Deriving dependencies
|
||||
|
||||
The processing to derive dependencies will collect spans from storage, analyzes links between services and store them for later presentation in the UI.
|
||||
This job can only be used with the `production` strategy and storage type `cassandra` or `elasticsearch`.
|
||||
|
||||
```yaml
|
||||
storage:
|
||||
type: elasticsearch
|
||||
dependencies:
|
||||
enabled: true # turn the job deployment on and off
|
||||
schedule: "55 23 * * *" # cron expression for it to run
|
||||
sparkMaster: # spark master connection string, when empty spark runs in embedded local mode
|
||||
```
|
||||
|
||||
The connection configuration to storage is derived from storage options.
|
||||
|
||||
## Auto-injecting Jaeger Agent Sidecars
|
||||
|
||||
The operator can inject Jaeger Agent sidecars in `Deployment` workloads, provided that the deployment has the annotation `sidecar.jaegertracing.io/inject` with a suitable value. The values can be either `"true"` (as string), or the Jaeger instance name, as returned by `kubectl get jaegers`. When `"true"` is used, there should be exactly *one* Jaeger instance for the same namespace as the deployment, otherwise, the operator can't figure out automatically which Jaeger instance to use.
|
||||
|
||||
The following snippet shows a simple application that will get a sidecar injected, with the Jaeger Agent pointing to the single Jaeger instance available in the same namespace:
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: myapp
|
||||
annotations:
|
||||
"sidecar.jaegertracing.io/inject": "true" # <1>
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app: myapp
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: myapp
|
||||
spec:
|
||||
containers:
|
||||
- name: myapp
|
||||
image: acme/myapp:myversion
|
||||
```
|
||||
<1> Either `"true"` (as string) or the Jaeger instance name.
|
||||
|
||||
A complete sample deployment is available at [`deploy/examples/business-application-injected-sidecar.yaml`](https://github.com/jaegertracing/jaeger-operator/blob/master/examples/business-application-injected-sidecar.yaml).
|
||||
|
||||
When the sidecar is injected, the Jaeger Agent can then be accessed at its default location on `localhost`.
|
||||
|
||||
## Installing the Agent as DaemonSet
|
||||
By default, the Operator expects the agents to be deployed as sidecars to the target applications. This is convenient for several purposes, like in a multi-tenant scenario or to have better load balancing, but there are scenarios where you might want to install the agent as a `DaemonSet`. In that case, specify the Agent's strategy to `DaemonSet`, as follows:
|
||||
|
||||
```yaml
|
||||
apiVersion: jaegertracing.io/v1
|
||||
kind: Jaeger
|
||||
metadata:
|
||||
name: my-jaeger
|
||||
spec:
|
||||
agent:
|
||||
strategy: DaemonSet
|
||||
```
|
||||
|
||||
{{< danger >}}
|
||||
If you attempt to install two Jaeger instances on the same cluster with `DaemonSet` as the strategy, only *one* will end up deploying a `DaemonSet`, as the agent is required to bind to well-known ports on the node. Because of that, the second daemon set will fail to bind to those ports.
|
||||
{{< /danger >}}
|
||||
|
||||
Your tracer client will then most likely need to be told where the agent is located. This is usually done by setting the environment variable `JAEGER_AGENT_HOST` to the value of the Kubernetes node's IP, for example:
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: myapp
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app: myapp
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: myapp
|
||||
spec:
|
||||
containers:
|
||||
- name: myapp
|
||||
image: acme/myapp:myversion
|
||||
env:
|
||||
- name: JAEGER_AGENT_HOST
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
fieldPath: status.hostIP
|
||||
```
|
||||
|
||||
### OpenShift
|
||||
|
||||
In OpenShift, a `HostPort` can only be set when a special security context is set. A separate service account can be used by the Jaeger Agent with the permission to bind to `HostPort`, as follows:
|
||||
|
||||
```bash
|
||||
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/examples/openshift/hostport-scc-daemonset.yaml # <1>
|
||||
oc new-project myappnamespace
|
||||
oc create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/examples/openshift/service_account_jaeger-agent-daemonset.yaml # <2>
|
||||
oc adm policy add-scc-to-user daemonset-with-hostport -z jaeger-agent-daemonset # <3>
|
||||
oc apply -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/examples/openshift/agent-as-daemonset.yaml # <4>
|
||||
```
|
||||
<1> The `SecurityContextConstraints` with the `allowHostPorts` policy
|
||||
|
||||
<2> The `ServiceAccount` to be used by the Jaeger Agent
|
||||
|
||||
<3> Adds the security policy to the service account
|
||||
|
||||
<4> Creates the Jaeger Instance using the `serviceAccount` created in the steps above
|
||||
|
||||
{{< warning >}}
|
||||
Without such a policy, errors like the following will prevent a `DaemonSet` to be created: `Warning FailedCreate 4s (x14 over 45s) daemonset-controller Error creating: pods "agent-as-daemonset-agent-daemonset-" is forbidden: unable to validate against any security context constraint: [spec.containers[0].securityContext.containers[0].hostPort: Invalid value: 5775: Host ports are not allowed to be used`
|
||||
{{< /warning >}}
|
||||
|
||||
After a few seconds, the `DaemonSet` should be up and running:
|
||||
|
||||
```bash
|
||||
$ oc get daemonset agent-as-daemonset-agent-daemonset
|
||||
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE
|
||||
agent-as-daemonset-agent-daemonset 1 1 1 1 1
|
||||
```
|
||||
|
||||
## Secrets Support
|
||||
|
||||
The Operator supports passing secrets to the Collector, Query and All-In-One deployments. This can be used for example, to pass credentials (username/password) to access the underlying storage backend (for example: Elasticsearch).
|
||||
The secrets are available as environment variables in the (Collector/Query/All-In-One) nodes.
|
||||
|
||||
```yaml
|
||||
storage:
|
||||
type: elasticsearch
|
||||
options:
|
||||
es:
|
||||
server-urls: http://elasticsearch:9200
|
||||
secretName: jaeger-secrets
|
||||
```
|
||||
|
||||
The secret itself would be managed outside of the `jaeger-operator` custom resource.
|
||||
|
||||
## Configuring the UI
|
||||
|
||||
Information on various configuration options for the UI can be found [here](../frontend-ui/#configuration), defined in json format.
|
||||
|
||||
To apply UI configuration changes within the Custom Resource, the same information can be included in yaml format as shown below:
|
||||
|
||||
```yaml
|
||||
ui:
|
||||
options:
|
||||
dependencies:
|
||||
menuEnabled: false
|
||||
tracking:
|
||||
gaID: UA-000000-2
|
||||
menu:
|
||||
- label: "About Jaeger"
|
||||
items:
|
||||
- label: "Documentation"
|
||||
url: "https://www.jaegertracing.io/docs/latest"
|
||||
linkPatterns:
|
||||
- type: "logs"
|
||||
key: "customer_id"
|
||||
url: /search?limit=20&lookback=1h&service=frontend&tags=%7B%22customer_id%22%3A%22#{customer_id}%22%7D
|
||||
text: "Search for other traces for customer_id=#{customer_id}"
|
||||
```
|
||||
|
||||
## Defining Sampling Strategies
|
||||
|
||||
{{< info >}}
|
||||
This is not relevant if a trace was started by the Istio proxy as the sampling decision is made there. And the Jaeger sampling decisions are only relevant when you are using the Jaeger tracer (client).
|
||||
{{< /info >}}
|
||||
|
||||
The operator can be used to define sampling strategies that will be supplied to tracers that have been configured to use a remote sampler:
|
||||
|
||||
```yaml
|
||||
apiVersion: jaegertracing.io/v1
|
||||
kind: Jaeger
|
||||
metadata:
|
||||
name: with-sampling
|
||||
spec:
|
||||
strategy: allInOne
|
||||
sampling:
|
||||
options:
|
||||
default_strategy:
|
||||
type: probabilistic
|
||||
param: 0.5
|
||||
```
|
||||
|
||||
This example defines a default sampling strategy that is probabilistic, with a 50% chance of the trace instances being sampled.
|
||||
|
||||
Refer to the Jaeger documentation on [Collector Sampling Configuration](https://www.jaegertracing.io/docs/latest/sampling/#collector-sampling-configuration) to see how service and endpoint sampling can be configured. The JSON representation described in that documentation can be used in the operator by converting to YAML.
|
||||
|
||||
## Finer grained configuration
|
||||
|
||||
The custom resource can be used to define finer grained Kubernetes configuration applied to all Jaeger components or at the individual component level.
|
||||
|
||||
When a common definition (for all Jaeger components) is required, it is defined under the `spec` node. When the definition relates to an individual component, it is placed under the `spec/<component>` node.
|
||||
|
||||
The types of supported configuration include:
|
||||
|
||||
* [affinity](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity) to determine which nodes a pod can be allocated to
|
||||
|
||||
* [annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/)
|
||||
|
||||
* [labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/)
|
||||
|
||||
* [resources](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container) to limit cpu and memory
|
||||
|
||||
* [tolerations](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/) in conjunction with `taints` to enable pods to avoid being repelled from a node
|
||||
|
||||
* [volumes](https://kubernetes.io/docs/concepts/storage/volumes/) and volume mounts
|
||||
|
||||
* [serviceAccount](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/) to run each component with separate identity
|
||||
|
||||
* [securityContext](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/) to define privileges of running components
|
||||
|
||||
```yaml
|
||||
apiVersion: jaegertracing.io/v1
|
||||
kind: Jaeger
|
||||
metadata:
|
||||
name: simple-prod
|
||||
spec:
|
||||
strategy: production
|
||||
storage:
|
||||
type: elasticsearch
|
||||
options:
|
||||
es:
|
||||
server-urls: http://elasticsearch:9200
|
||||
annotations:
|
||||
key1: value1
|
||||
labels:
|
||||
key2: value2
|
||||
resources:
|
||||
requests:
|
||||
memory: "64Mi"
|
||||
cpu: "250m"
|
||||
limits:
|
||||
memory: "128Mi"
|
||||
cpu: "500m"
|
||||
affinity:
|
||||
nodeAffinity:
|
||||
requiredDuringSchedulingIgnoredDuringExecution:
|
||||
nodeSelectorTerms:
|
||||
- matchExpressions:
|
||||
- key: kubernetes.io/e2e-az-name
|
||||
operator: In
|
||||
values:
|
||||
- e2e-az1
|
||||
- e2e-az2
|
||||
preferredDuringSchedulingIgnoredDuringExecution:
|
||||
- weight: 1
|
||||
preference:
|
||||
matchExpressions:
|
||||
- key: another-node-label-key
|
||||
operator: In
|
||||
values:
|
||||
- another-node-label-value
|
||||
tolerations:
|
||||
- key: "key1"
|
||||
operator: "Equal"
|
||||
value: "value1"
|
||||
effect: "NoSchedule"
|
||||
- key: "key1"
|
||||
operator: "Equal"
|
||||
value: "value1"
|
||||
effect: "NoExecute"
|
||||
serviceAccount: nameOfServiceAccount
|
||||
securityContext:
|
||||
runAsUser: 1000
|
||||
volumeMounts:
|
||||
- name: config-vol
|
||||
mountPath: /etc/config
|
||||
volumes:
|
||||
- name: config-vol
|
||||
configMap:
|
||||
name: log-config
|
||||
items:
|
||||
- key: log_level
|
||||
path: log_level
|
||||
```
|
||||
|
||||
# Accessing the Jaeger Console (UI)
|
||||
<!-- TODO Add tabs shortcode -->
|
||||
|
||||
## Kubernetes
|
||||
|
||||
The operator creates a Kubernetes [`ingress`](https://kubernetes.io/docs/concepts/services-networking/ingress/) route, which is the Kubernetes' standard for exposing a service to the outside world, but by default it does not come with Ingress providers.
|
||||
Check the [Kubernetes documentation](https://kubernetes.github.io/ingress-nginx/deploy/#verify-installation) for the most appropriate way to achieve an Ingress provider for your platform. The following command enables the Ingress provider on `minikube`:
|
||||
|
||||
```bash
|
||||
minikube addons enable ingress
|
||||
```
|
||||
|
||||
Once Ingress is enabled, the address for the Jaeger console can be found by querying the Ingress object:
|
||||
|
||||
```bash
|
||||
$ kubectl get ingress
|
||||
NAME HOSTS ADDRESS PORTS AGE
|
||||
simplest-query * 192.168.122.34 80 3m
|
||||
```
|
||||
|
||||
In this example, the Jaeger UI is available at http://192.168.122.34.
|
||||
|
||||
To enable TLS in the Ingress, pass a `secretName` with the name of a [Secret](https://kubernetes.io/docs/concepts/configuration/secret/) containing the TLS certificate:
|
||||
|
||||
```yaml
|
||||
apiVersion: jaegertracing.io/v1
|
||||
kind: Jaeger
|
||||
metadata:
|
||||
name: ingress-with-tls
|
||||
spec:
|
||||
ingress:
|
||||
secretName: my-tls-secret
|
||||
```
|
||||
|
||||
## OpenShift
|
||||
|
||||
When the Operator is running on OpenShift, the Operator will automatically create a `Route` object for the query services. Use the following command to check the hostname/port:
|
||||
|
||||
```bash
|
||||
oc get routes
|
||||
```
|
||||
|
||||
{{< info >}}
|
||||
Make sure to use `https` with the hostname/port you get from the command above, otherwise you'll see a message like: "Application is not available".
|
||||
{{< /info >}}
|
||||
|
||||
By default, the Jaeger UI is protected with OpenShift's OAuth service and any valid user is able to login. To disable this feature and leave the Jaeger UI unsecured, set the Ingress property `security` to `none` in the custom resource file:
|
||||
|
||||
```yaml
|
||||
apiVersion: jaegertracing.io/v1
|
||||
kind: Jaeger
|
||||
metadata:
|
||||
name: disable-oauth-proxy
|
||||
spec:
|
||||
ingress:
|
||||
security: none
|
||||
```
|
||||
|
||||
Custom `SAR` and `Delegate URL` values can be specified as part of the `.Spec.Ingress.OpenShift.SAR` and `.Spec.Ingress.Openshift.DelegateURLs`, as follows:
|
||||
|
||||
```yaml
|
||||
apiVersion: jaegertracing.io/v1
|
||||
kind: Jaeger
|
||||
metadata:
|
||||
name: custom-sar-oauth-proxy
|
||||
spec:
|
||||
ingress:
|
||||
openshift:
|
||||
sar: '{"namespace": "default", "resource": "pods", "verb": "get"}'
|
||||
delegateUrls: '{"/":{"namespace": "default", "resource": "pods", "verb": "get"}}'
|
||||
```
|
||||
|
||||
When the `delegateUrls` is set, the Jaeger Operator needs to create a new `ClusterRoleBinding` between the service account used by the UI Proxy (`{InstanceName}-ui-proxy`) and the role `system:auth-delegator`, as required by the OpenShift OAuth Proxy. Because of that, the service account used by the operator itself needs to have the same cluster role binding. To accomplish that, a `ClusterRoleBinding` such as the following has to be created:
|
||||
|
||||
```yaml
|
||||
kind: ClusterRoleBinding
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
metadata:
|
||||
name: jaeger-operator-with-auth-delegator
|
||||
namespace: observability
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: jaeger-operator
|
||||
namespace: observability
|
||||
roleRef:
|
||||
kind: ClusterRole
|
||||
name: system:auth-delegator
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
```
|
||||
|
||||
Cluster administrators not comfortable in letting users deploy Jaeger instances with this cluster role are free to not add this cluster role to the operator's service account. In that case, the Operator will auto-detect that the required permissions are missing and will log a message similar to: `the requested instance specifies the delegateUrls option for the OAuth Proxy, but this operator cannot assign the proper cluster role to it (system:auth-delegator). Create a cluster role binding between the operator's service account and the cluster role 'system:auth-delegator' in order to allow instances to use 'delegateUrls'`.
|
||||
|
||||
The Jaeger Operator also supports authentication using `htpasswd` files via the OpenShift OAuth Proxy. To make use of that, specify the `htpasswdFile` option within the OpenShift-specific entries, pointing to the file `htpasswd` file location in the local disk. The `htpasswd` file can be created using the `htpasswd` utility:
|
||||
|
||||
```console
|
||||
$ htpasswd -cs /tmp/htpasswd jdoe
|
||||
New password:
|
||||
Re-type new password:
|
||||
Adding password for user jdoe
|
||||
```
|
||||
|
||||
This file can then be used as the input for the `kubectl create secret` command:
|
||||
|
||||
```console
|
||||
$ kubectl create secret generic htpasswd --from-file=htpasswd=/tmp/htpasswd
|
||||
secret/htpasswd created
|
||||
```
|
||||
|
||||
Once the secret is created, it can be specified in the Jaeger CR as a volume/volume mount:
|
||||
|
||||
```yaml
|
||||
apiVersion: jaegertracing.io/v1
|
||||
kind: Jaeger
|
||||
metadata:
|
||||
name: with-htpasswd
|
||||
spec:
|
||||
ingress:
|
||||
openshift:
|
||||
sar: '{"namespace": "default", "resource": "pods", "verb": "get"}'
|
||||
htpasswdFile: /usr/local/data/htpasswd
|
||||
volumeMounts:
|
||||
- name: htpasswd-volume
|
||||
mountPath: /usr/local/data
|
||||
volumes:
|
||||
- name: htpasswd-volume
|
||||
secret:
|
||||
secretName: htpasswd
|
||||
```
|
||||
|
||||
# Upgrading the Operator and its managed instances
|
||||
|
||||
Each version of the Jaeger Operator follows one Jaeger version. Whenever a new version of the Jaeger Operator is installed, all the Jaeger instances managed by the operator will be upgraded to the Operator's supported version. For example, an instance named `simplest` that was created with Jaeger Operator 1.12.0 will be running Jaeger 1.12.0. Once the Jaeger Operator is upgraded to 1.13.0, the instance `simplest` will be upgraded to the version 1.13.0, following the official upgrade instructions from the Jaeger project.
|
||||
|
||||
The Jaeger Operator can be upgraded manually by changing the deployment (`kubectl edit deployment jaeger-operator`), or via specialized tools such as the [Operator Lifecycle Manager (OLM)](https://github.com/operator-framework/operator-lifecycle-manager).
|
||||
|
||||
# Updating a Jaeger instance (experimental)
|
||||
|
||||
A Jaeger instance can be updated by changing the `CustomResource`, either via `kubectl edit jaeger simplest`, where `simplest` is the Jaeger's instance name, or by applying the updated YAML file via `kubectl apply -f simplest.yaml`.
|
||||
|
||||
{{< danger >}}
|
||||
The name of the Jaeger instance cannot be updated, as it is part of the identifying information for the resource.
|
||||
{{< /danger >}}
|
||||
|
||||
Simpler changes such as changing the replica sizes can be applied without much concern, whereas changes to the strategy should be watched closely and might potentially cause an outage for individual components (collector/query/agent).
|
||||
|
||||
While changing the backing storage is supported, migration of the data is not.
|
||||
|
||||
# Removing a Jaeger instance
|
||||
<!-- TODO Add OKD/OpenShift commands and tabs shortcode-->
|
||||
|
||||
To remove an instance, use the `delete` command with the custom resource file used when you created the instance:
|
||||
|
||||
```bash
|
||||
kubectl delete -f simplest.yaml
|
||||
```
|
||||
|
||||
Alternatively, you can remove a Jaeger instance by running:
|
||||
|
||||
```bash
|
||||
kubectl delete jaeger simplest
|
||||
```
|
||||
|
||||
{{< info >}}
|
||||
Deleting the instance will not remove the data from any permanent storage used with this instance. Data from in-memory instances, however, will be lost.
|
||||
{{< /info >}}
|
||||
|
||||
# Monitoring the operator
|
||||
|
||||
The Jaeger Operator starts a Prometheus-compatible endpoint on `0.0.0.0:8383/metrics` with internal metrics that can be used to monitor the process.
|
||||
|
||||
{{< info >}}
|
||||
The Jaeger Operator does not yet publish its own metrics. Rather, it makes available metrics reported by the components it uses, such as the Operator SDK.
|
||||
{{< /info >}}
|
||||
|
||||
# Uninstalling the operator
|
||||
<!-- TODO Add OKD/OpenShift commands and tabs shortcode -->
|
||||
|
||||
To uninstall the operator, run the following commands:
|
||||
|
||||
```bash
|
||||
kubectl delete -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/operator.yaml
|
||||
kubectl delete -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role_binding.yaml
|
||||
kubectl delete -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role.yaml
|
||||
kubectl delete -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/service_account.yaml
|
||||
kubectl delete -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/crds/jaegertracing.io_jaegers_crd.yaml
|
||||
```
|
|
@ -1,95 +0,0 @@
|
|||
---
|
||||
title: Performance Tuning Guide
|
||||
description: Tweaking your Jaeger instance to achieve a better performance
|
||||
weight: 10
|
||||
---
|
||||
|
||||
Jaeger was built from day 1 to be able to ingest huge amounts of data in a resilient way. To better utilize resources that might cause delays, such as storage or network communications, Jaeger buffers and batches data. When more spans are generated than Jaeger is able to safely process, spans might get dropped. However, the defaults might not fit all scenarios: for instance, Agents running as a sidecar might have more memory constraints than Agents running as a daemon in bare metal.
|
||||
|
||||
## Deployment considerations
|
||||
|
||||
Although performance tuning the individual components is important, the way Jaeger is deployed can be decisive in obtaining optimal performance.
|
||||
|
||||
### Scale the Collector up and down
|
||||
|
||||
Use the auto-scaling capabilities of your platform: the Collector is nearly horizontally scalable so that more instances can be added and removed on-demand. A good way to scale up and down is by checking the `jaeger_collector_queue_length` metric: add instances when the length is higher than 50% of the maximum size for extended periods of time. Another metric that can be taken into consideration is `jaeger_collector_in_queue_latency_bucket`, which is a histogram indicating how long spans have been waiting in the queue before a worker picked it up. When the queue latency gets higher over time, it’s a good indication to increase the number of the workers, or to improve the storage performance.
|
||||
|
||||
Adding Collector instances is recommended when your platform provides auto-scaling capabilities, or when it's easier to start/stop Collector instances than changing existing, running instances. Scaling horizontally is also indicated when the CPU usage should be spread across nodes.
|
||||
|
||||
### Make sure the storage can keep up
|
||||
|
||||
Each span is written to the storage by the Collector using one worker, blocking it until the span has been stored. When the storage is too slow, the number of workers blocked by the storage might be too high, causing spans to be dropped. To help diagnose this situation, the histogram `jaeger_collector_save_latency_bucket` can be analyzed. Ideally, the latency should remain the same over time. When the histogram shows that most spans are taking longer and longer over time, it’s a good indication that your storage might need some attention.
|
||||
|
||||
### Place the Agents close to your applications
|
||||
|
||||
The Agent is meant to be placed on the same host as the instrumented application, in order to avoid UDP packet loss over the network. This is typically accomplished by having one Agent per bare metal host for traditional applications, or as a sidecar in container environments like Kubernetes, as this helps spread the load handled by Agents with the additional advantage of allowing each Agent to be tweaked individually, according to the application’s needs and importance.
|
||||
|
||||
### Consider using Apache Kafka as intermediate buffer
|
||||
|
||||
Jaeger [can use Apache Kafka](../architecture/) as a buffer between the Collector and the actual backing storage (Elasticsearch, Apache Cassandra). This is ideal for cases where the traffic spikes are relatively frequent (prime time traffic) but the storage can eventually catch up once the traffic normalizes. For that, the `SPAN_STORAGE_TYPE` environment variable should be set to `kafka` in the Collector and the Jaeger Ingester component can be used, reading data from Kafka and writing it to the storage.
|
||||
|
||||
In addition to the performance aspects, having spans written to Kafka is useful for building real time data pipeline for aggregations and feature extraction from traces.
|
||||
|
||||
## Client (Tracer) settings
|
||||
|
||||
The Jaeger Clients are built to have minimal effect to the instrumented application. As such, it has conservative defaults that might not be suitable for all cases. Note that Jaeger Clients can be configured programmatically or via [environment variables](../client-features/).
|
||||
|
||||
### Adjust the sampling configuration
|
||||
|
||||
Together, the `JAEGER_SAMPLER_TYPE` and `JAEGER_SAMPLER_PARAM` specify how often traces should be "sampled", ie, recorded and sent to the Jaeger backend. For applications generating a large number of spans, setting the sampling type to `probabilistic` and the value to `0.001` (the default) will cause traces to be reported with a 1/1000th chance. Note that the sampling decision is made at the root span and propagated down to all child spans.
|
||||
|
||||
For applications with low to medium traffic, setting the sampling type to `const` and value to `1` will cause all spans to be reported. Similarly, tracing can be disabled by setting the value to `0`, while context propagation will continue to work.
|
||||
|
||||
Some Clients support the setting `JAEGER_DISABLED` to completely disable the Jaeger Tracer. This is recommended only if the Tracer is behaving in a way that causes problems to the instrumented application, as it will not propagate the context to the downstream services.
|
||||
|
||||
{{< info >}}
|
||||
We recommend to set your clients to use the [`remote` sampling strategy](../sampling/#collector-sampling-configuration), so that admins can centrally set the concrete sampling strategy for each service.
|
||||
{{< /info >}}
|
||||
|
||||
### Increase in-memory queue size
|
||||
|
||||
Most of the Jaeger clients, such as the Java, Go, and C# clients, buffer spans in memory before sending them to the Jaeger Agent/Collector. The maximum size of this buffer is defined by the environment variable `JAEGER_REPORTER_MAX_QUEUE_SIZE` (default value: about `100` spans): the larger the size, the higher the potential memory consumption. When the instrumented application is generating a large number of spans, it’s possible that the queue will be full causing the Client to discard the new spans (metric `jaeger_tracer_reporter_spans_total{result="dropped",}`).
|
||||
|
||||
In most common scenarios, the queue will be close to empty (metric: `jaeger_tracer_reporter_queue_length`), as spans are flushed to the Agent or Collector at regular intervals or when a certain size of the batch is reached. The detailed behavior of this queue is described in this [GitHub issue](https://github.com/jaegertracing/jaeger-client-java/issues/607).
|
||||
|
||||
### Modify the batched spans flush interval
|
||||
|
||||
The Java, Go, NodeJS, Python and C# Clients allow the customization of the flush interval (default value: `1000` milliseconds, or 1 second) used by the reporters, such as the `RemoteReporter`, to trigger a `flush` operation, sending all in-memory spans to the Agent or Collector. The lower the flush interval is set to, the more frequent the flush operations happen. As most reporters will wait until enough data is in the queue, this setting will force a flush operation at periodic intervals, so that spans are sent to the backend in a timely fashion.
|
||||
|
||||
When the instrumented application is generating a large number of spans and the Agent/Collector is close to the application, the networking overhead might be low, justifying a higher number of flush operations. When the `HttpSender` is being used and the Collector is not close enough to the application, the networking overhead might be too high so that a higher value for this property makes sense.
|
||||
|
||||
## Agent settings
|
||||
|
||||
Jaeger Agents receive data from Clients, sending them in batches to the Collector. When not properly configured, it might end up discarding data even if the host machine has plenty of resources.
|
||||
|
||||
### Adjust server queue sizes
|
||||
|
||||
The set of “server queue size” properties ( `processor.jaeger-binary.server-queue-size`, `processor.jaeger-compact.server-queue-size`, `processor.zipkin-compact.server-queue-size`) indicate the maximum number of span batches that the Agent can accept and store in memory. It’s safe to assume that `jaeger-compact` is the most important processor in your Agent setup, as it’s the only one available in most Clients, such as the Java and Go Clients.
|
||||
|
||||
The default value for each queue is `1000` span batches. Given that each span batch has up to 64KiB worth of spans, each queue can hold up to 64MiB worth of spans.
|
||||
|
||||
In typical scenarios, the queue will be close to empty (metric `jaeger_agent_thrift_udp_server_queue_size`) as span batches should be quickly picked up and processed by a worker. However, sudden spikes in the number of span batches submitted by Clients might occur, causing the batches to be queued. When the queue is full, the older batches are overridden causing spans to be discarded (metric `jaeger_agent_thrift_udp_server_packets_dropped_total`).
|
||||
|
||||
### Adjust processor workers
|
||||
|
||||
The set of “processor workers” properties ( `processor.jaeger-binary.workers`, `processor.jaeger-compact.workers`, `processor.zipkin-compact.workers`) indicate the number of parallel span batch processors to start. Each worker type has a default size of `10`. In general, span batches are processed as soon as they are placed in the server queue and will block a worker until the whole packet is sent to the Collector. For Agents processing data from multiple Clients, the number of workers should be increased. Given that the cost of each worker is low, a good rule of thumb is 10 workers per Client with moderate traffic: given that each span batch might contain up to 64KiB worth of spans, it means that 10 workers are able to send about 640KiB concurrently to a Collector.
|
||||
|
||||
## Collector settings
|
||||
|
||||
The Collector receives data from Clients and Agents. When not properly configured, it might process less data than what would be possible on the same host, or it might overload the host by consuming more memory than permitted.
|
||||
|
||||
### Adjust queue size
|
||||
|
||||
Similar to the Agent, the Collector is able to receive spans and place them in an internal queue for processing. This allows the Collector to return immediately to the Client/Agent instead of waiting for the span to make its way to the storage.
|
||||
|
||||
The setting `collector.queue-size` (default: `2000`) dictates how many spans the queue should support. In the typical scenario, the queue will be close to empty, as enough workers should exist picking up spans from the queue and sending them to the storage. When the number of items in the queue (metric `jaeger_collector_queue_length`) is permanently high, it’s an indication that either the number of workers should be increased or that the storage cannot keep up with the volume of data that it’s receiving. When the queue is full, the older items in the queue are overridden, causing spans to be discarded (metric `jaeger_collector_spans_dropped_total`).
|
||||
|
||||
{{< warning >}}
|
||||
The queue size for the Agent is about _span batches_, whereas the queue size for the Collector is about _individual spans_.
|
||||
{{< /warning >}}
|
||||
|
||||
Given that the queue size should be close to empty most of the time, this setting should be as high as the available memory for the Collector, to provide maximum protection against sudden traffic spikes. However, if your storage layer is under-provisioned and cannot keep up, even a large queue will quickly fill up and start dropping data.
|
||||
|
||||
### Adjust processor workers
|
||||
|
||||
Items from the span queue in the Collector are picked up by workers. Each worker picks one span from the queue and persists it to the storage. The number of workers can be specified by the setting `collector.num-workers` (default: `50`) and should be as high as needed to keep the queue close to zero. The general rule is: the faster the backing storage, the lower the number of workers can be. Given that workers are relatively cheap, this number can be increased at will. As a general rule, one worker per 50 items in the queue should be sufficient when the storage is fast. With a `collector.queue-size` of `2000`, having about `40` workers should be sufficient. For slower storage mechanisms, this ratio should be adjusted accordingly, having more workers per queue item.
|
|
@ -1,66 +0,0 @@
|
|||
---
|
||||
title: Sampling
|
||||
hasparent: true
|
||||
---
|
||||
|
||||
Jaeger libraries implement consistent upfront (or head-based) sampling. For example, assume we have a simple call graph where service A calls service B, and B calls service C: `A -> B -> C`. When service A receives a request that contains no tracing information, Jaeger tracer will start a new {{< tip "trace" >}}, assign it a random trace ID, and make a sampling decision based on the currently installed sampling strategy. The sampling decision will be propagated with the requests to B and to C, so those services will not be making the sampling decision again but instead will respect the decision made by the top service A. This approach guarantees that if a trace is sampled, all its {{< tip "spans" "span" >}} will be recorded in the backend. If each service was making its own sampling decision we would rarely get complete traces in the backend.
|
||||
|
||||
## Client Sampling Configuration
|
||||
|
||||
When using configuration object to instantiate the tracer, the type of sampling can be selected via `sampler.type` and `sampler.param` properties. Jaeger libraries support the following samplers:
|
||||
|
||||
* **Constant** (`sampler.type=const`) sampler always makes the same decision for all traces. It either samples all traces (`sampler.param=1`) or none of them (`sampler.param=0`).
|
||||
* **Probabilistic** (`sampler.type=probabilistic`) sampler makes a random sampling decision with the probability of sampling equal to the value of `sampler.param` property. For example, with `sampler.param=0.1` approximately 1 in 10 traces will be sampled.
|
||||
* **Rate Limiting** (`sampler.type=ratelimiting`) sampler uses a leaky bucket rate limiter to ensure that traces are sampled with a certain constant rate. For example, when `sampler.param=2.0` it will sample requests with the rate of 2 traces per second.
|
||||
* **Remote** (`sampler.type=remote`, which is also the default) sampler consults Jaeger agent for the appropriate sampling strategy to use in the current service. This allows controlling the sampling strategies in the services from a [central configuration](#collector-sampling-configuration) in Jaeger backend, or even dynamically (see [Adaptive Sampling](https://github.com/jaegertracing/jaeger/issues/365)).
|
||||
|
||||
### Adaptive Sampler
|
||||
|
||||
Adaptive sampler is a composite sampler that combines two functions:
|
||||
|
||||
* It makes sampling decisions on a per-operation basis, i.e. based on {{< tip "span" >}} operation name. This is especially useful in the API services whose endpoints may have very different traffic volumes and using a single probabilistic sampler for the whole service might starve (never sample) some of the low QPS endpoints.
|
||||
* It supports a minimum guaranteed rate of sampling, such as always allowing up to N {{< tip "traces" "trace" >}} per seconds and then sampling anything above that with a certain probability (everything is per-operation, not per-service).
|
||||
|
||||
Per-operation parameters can be configured statically or pulled periodically from Jaeger backend with the help of Remote sampler. Adaptive sampler is designed to work with the upcoming [Adaptive Sampling](https://github.com/jaegertracing/jaeger/issues/365) feature of the Jaeger backend.
|
||||
|
||||
## Collector Sampling Configuration
|
||||
|
||||
Collectors can be instantiated with static sampling strategies (which are propagated to the respective service if configured with Remote sampler) via the `--sampling.strategies-file` option. This option requires a path to a json file which have the sampling strategies defined.
|
||||
|
||||
Example `strategies.json`
|
||||
```json
|
||||
{
|
||||
"service_strategies": [
|
||||
{
|
||||
"service": "foo",
|
||||
"type": "probabilistic",
|
||||
"param": 0.8,
|
||||
"operation_strategies": [
|
||||
{
|
||||
"operation": "op1",
|
||||
"type": "probabilistic",
|
||||
"param": 0.2
|
||||
},
|
||||
{
|
||||
"operation": "op2",
|
||||
"type": "probabilistic",
|
||||
"param": 0.4
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "bar",
|
||||
"type": "ratelimiting",
|
||||
"param": 5
|
||||
}
|
||||
],
|
||||
"default_strategy": {
|
||||
"type": "probabilistic",
|
||||
"param": 0.5
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`service_strategies` defines service specific sampling strategies and `operation_strategies` defines operation specific sampling strategies. There are 2 types of strategies possible: `probabilistic` and `ratelimiting` which are described [above](#client-sampling-configuration) (NOTE: `ratelimiting` is not supported for `operation_strategies`). `default_strategy` defines the catch-all sampling strategy that is propagated if the service is not included as part of `service_strategies`.
|
||||
|
||||
In the above example, all service `foo` operations are sampled probabilistically with a probability of 0.8 except `op1` and `op2` which are probabilistically sampled with a probability of 0.2 and 0.4 respectively. All operations for service `bar` are ratelimited at 5 traces per second. Any other service is probabilistically sampled with a probability of 0.5.
|
|
@ -1,43 +0,0 @@
|
|||
---
|
||||
title: Windows Service Deployment
|
||||
hasparent: true
|
||||
---
|
||||
|
||||
In Windows environments, Jaeger processes can be hosted and managed as Windows services controlled via the `sc` utility. To configure such services on Windows, download [nssm.exe](https://nssm.cc/download) for the appropriate architecture, and issue commands similar to how Jaeger is typically run. The example below showcases a basic Elasticsearch setup, configured using both environment variables and process arguments.
|
||||
|
||||
## Agent
|
||||
```bat
|
||||
nssm install JaegerAgent C:\Jaeger\jaeger-agent.exe --reporter.grpc.host-port=localhost:14250
|
||||
|
||||
nssm set JaegerAgent AppStdout C:\Jaeger\jaeger-agent.out.log
|
||||
nssm set JaegerAgent AppStderr C:\Jaeger\jaeger-agent.err.log
|
||||
nssm set JaegerAgent Description Jaeger Agent service
|
||||
|
||||
nssm start JaegerAgent
|
||||
```
|
||||
|
||||
## Collector
|
||||
```bat
|
||||
nssm install JaegerCollector C:\Jaeger\jaeger-collector.exe --es.server-urls=http://localhost:9200 --es.username=jaeger --es.password=PASSWORD
|
||||
|
||||
nssm set JaegerCollector AppStdout C:\Jaeger\jaeger-collector.out.log
|
||||
nssm set JaegerCollector AppStderr C:\Jaeger\jaeger-collector.err.log
|
||||
nssm set JaegerCollector Description Jaeger Collector service
|
||||
nssm set JaegerCollector AppEnvironmentExtra SPAN_STORAGE_TYPE=elasticsearch
|
||||
|
||||
nssm start JaegerCollector
|
||||
```
|
||||
|
||||
## Query UI
|
||||
```bat
|
||||
nssm install JaegerUI C:\Jaeger\jaeger-query.exe --es.server-urls=http://localhost:9200 --es.username=jaeger --es.password=PASSWORD
|
||||
|
||||
nssm set JaegerUI AppStdout C:\Jaeger\jaeger-ui.out.log
|
||||
nssm set JaegerUI AppStderr C:\Jaeger\jaeger-ui.err.log
|
||||
nssm set JaegerUI Description Jaeger Query service
|
||||
nssm set JaegerUI AppEnvironmentExtra SPAN_STORAGE_TYPE=elasticsearch
|
||||
|
||||
nssm start JaegerUI
|
||||
```
|
||||
|
||||
For additional information & docs, please see [the NSSM usage guide.](https://nssm.cc/usage)
|
|
@ -1,70 +0,0 @@
|
|||
---
|
||||
title: Introduction
|
||||
weight: 1
|
||||
children:
|
||||
- title: Features
|
||||
url: features
|
||||
---
|
||||
|
||||
Welcome to Jaeger's documentation portal! Below, you'll find information for beginners and experienced Jaeger users.
|
||||
|
||||
If you can't find what you are looking for, or have an issue not covered here, we'd love to [hear from you](/get-in-touch).
|
||||
|
||||
## About
|
||||
|
||||
Jaeger, inspired by [Dapper][dapper] and [OpenZipkin](http://zipkin.io),
|
||||
is a distributed tracing system released as open source by [Uber Technologies][ubeross].
|
||||
It is used for monitoring and troubleshooting microservices-based distributed systems, including:
|
||||
|
||||
* Distributed context propagation
|
||||
* Distributed transaction monitoring
|
||||
* Root cause analysis
|
||||
* Service dependency analysis
|
||||
* Performance / latency optimization
|
||||
|
||||
Uber published a blog post, [Evolving Distributed Tracing at Uber](https://eng.uber.com/distributed-tracing/), where they explain the history and reasons for the architectural choices made in Jaeger. [Yuri Shkuro](https://shkuro.com), creator of Jaeger, also published a book [Mastering Distributed Tracing](https://shkuro.com/books/2019-mastering-distributed-tracing/) that covers in-depth many aspects of Jaeger design and operation, as well as distributed tracing in general.
|
||||
|
||||
## Features
|
||||
|
||||
* [OpenTracing](http://opentracing.io/) compatible data model and instrumentation libraries
|
||||
* in [Go](https://github.com/jaegertracing/jaeger-client-go), [Java](https://github.com/jaegertracing/jaeger-client-java), [Node](https://github.com/jaegertracing/jaeger-client-node), [Python](https://github.com/jaegertracing/jaeger-client-python)
|
||||
and [C++](https://github.com/jaegertracing/cpp-client)
|
||||
* Uses consistent upfront sampling with individual per service/endpoint probabilities
|
||||
* Multiple storage backends: Cassandra, Elasticsearch, memory.
|
||||
* Adaptive sampling (coming soon)
|
||||
* Post-collection data processing pipeline (coming soon)
|
||||
|
||||
See [Features](./features/) page for more details.
|
||||
|
||||
## Technical Specs
|
||||
|
||||
* Backend components implemented in Go
|
||||
* React/Javascript UI
|
||||
* Supported storage backends:
|
||||
* [Cassandra 3.4+](./deployment/#cassandra)
|
||||
* [Elasticsearch 5.x, 6.x, 7.x](./deployment/#elasticsearch)
|
||||
* [Kafka](./deployment/#kafka)
|
||||
* memory storage
|
||||
|
||||
## Quick Start
|
||||
See [running a docker all in one image](getting-started#all-in-one).
|
||||
|
||||
## Screenshots
|
||||
|
||||
### Traces View
|
||||
[](/img/traces-ss.png)
|
||||
|
||||
### Trace Detail View
|
||||
[](/img/trace-detail-ss.png)
|
||||
|
||||
## Related links
|
||||
- [Evolving Distributed tracing At Uber Engineering](https://eng.uber.com/distributed-tracing/)
|
||||
- [Mastering Distributed Tracing](https://shkuro.com/books/2019-mastering-distributed-tracing/)
|
||||
- [Tracing HTTP request latency in Go with OpenTracing](https://medium.com/opentracing/tracing-http-request-latency-in-go-with-opentracing-7cc1282a100a)
|
||||
- [Distributed Tracing with Jaeger & Prometheus on Kubernetes](https://blog.openshift.com/openshift-commons-briefing-82-distributed-tracing-with-jaeger-prometheus-on-kubernetes/)
|
||||
- [Using Jaeger with Istio](https://istio.io/docs/tasks/telemetry/distributed-tracing.html)
|
||||
- [Using Jaeger with Envoy](https://www.envoyproxy.io/docs/envoy/latest/start/sandboxes/jaeger_tracing.html)
|
||||
|
||||
[dapper]: https://research.google.com/pubs/pub36356.html
|
||||
[ubeross]: http://uber.github.io
|
||||
|
|
@ -1,87 +0,0 @@
|
|||
---
|
||||
title: APIs
|
||||
hasparent: true
|
||||
---
|
||||
|
||||
Jaeger components implement various APIs for saving or retrieving trace data.
|
||||
|
||||
The following labels are used to describe API compatibility guarantees.
|
||||
|
||||
* **stable** - the API guarantees backwards compatibility. If breaking changes are going to be made in the future, they will result in a new API version, e.g. `/api/v2` URL prefix or a different namespace in the IDL.
|
||||
* **internal** - the APIs intended for internal communications between Jaeger components and not recommended for use by external components.
|
||||
* **deprecated** - the APIs that are only maintained for legacy reasons and will be phased out in the future.
|
||||
|
||||
## Span reporting APIs
|
||||
|
||||
Agent and Collector are the two components of the Jaeger backend that can receive spans. At this time they support two sets of non-overlapping APIs.
|
||||
|
||||
### Thrift over UDP (stable)
|
||||
|
||||
The Agent can only receive spans over UDP in Thrift format. The primary API is a UDP packet that contains a Thrift-encoded `Batch` struct defined in [jaeger.thrift][jaeger.thrift] IDL file, located in the [jaeger-idl][jaeger-idl] repository. Most Jaeger Clients use Thrift's `compact` encoding, however some client libraries do not support it (notably, Node.js) and use Thrift's `binary` encoding (sent to a different UDP port). The Agent's API is defined by [agent.thrift][agent.thrift] IDL file.
|
||||
|
||||
For legacy reasons, the Agent also accepts spans in Zipkin format, however, only very old versions of Jaeger clients can send data in that format and it is officially deprecated.
|
||||
|
||||
### Protobuf via gRPC (stable)
|
||||
|
||||
In a typical Jaeger deployment, Agents receive spans from Clients and forward them to Collectors. Since Jaeger version 1.11 the official and recommended protocol between Agents and Collectors is gRPC with Protobuf.
|
||||
|
||||
The Protobuf IDL [collector.proto][collector.proto] is currently located in the main Jaeger repository, under [model/proto/api_v2][collector.proto]. In the future it will be moved to [jaeger-idl][jaeger-idl] repository ([jaeger-idl/issues/55](https://github.com/jaegertracing/jaeger-idl/issues/55)).
|
||||
|
||||
### Thrift over HTTP (stable)
|
||||
|
||||
In some cases it is not feasible to deploy Jaeger Agent next to the application, for example, when the application code is running as AWS Lambda function. In these scenarios the Jaeger Clients can be configured to submit spans directly to the Collectors over HTTP/HTTPS.
|
||||
|
||||
The same [jaeger.thrift][jaeger.thrift] payload can be submitted in HTTP POST request to `/api/traces` endpoint, for example, `https://jaeger-collector:14268/api/traces`. The `Batch` struct needs to be encoded using Thrift's `binary` encoding, and the HTTP request should specify the content type header:
|
||||
|
||||
```
|
||||
Content-Type: application/vnd.apache.thrift.binary
|
||||
```
|
||||
|
||||
### JSON over HTTP (n/a)
|
||||
|
||||
There is no official Jaeger JSON format that can be accepted by the collector. In the future the Protobuf-generated JSON may be supported.
|
||||
|
||||
### Thrift via TChannel (deprecated)
|
||||
|
||||
Agent and Collector can communicate using TChannel protocol. This protocol is generally not supported by the routing infrastructure and has been deprecated. It will be eventually removed from Jaeger.
|
||||
|
||||
### Zipkin Formats (stable)
|
||||
|
||||
Jaeger Collector can also accept spans in several Zipkin data format, namely JSON v1/v2 and Thrift. The Collector needs to be configured to enable Zipkin HTTP server, e.g. on port 9411 used by Zipkin collectors. The server enables two endpoints that expect POST requests:
|
||||
|
||||
* `/api/v1/spans` for submitting spans in Zipkin JSON v1 or Zipkin Thrift format.
|
||||
* `/api/v2/spans` for submitting spans in Zipkin JSON v2.
|
||||
|
||||
## Trace retrieval APIs
|
||||
|
||||
Traces saved in the storage can be retrieved by calling Jaeger Query Service.
|
||||
|
||||
### gRPC/Protobuf (stable)
|
||||
|
||||
The recommended way for programmatically retrieving traces and other data is via gRPC endpoint defined in [query.proto][query.proto] IDL file (located in the main Jaeger repository, similar to [collector.proto][collector.proto]).
|
||||
|
||||
### HTTP JSON (internal)
|
||||
|
||||
Jaeger UI communicates with Jaeger Query Service via JSON API. For example, a trace can be retrieved via GET request to `https://jaeger-query:16686/api/traces/{trace-id-hex-string}`. This JSON API is intentionally undocumented and subject to change.
|
||||
|
||||
## Clients configuration (internal)
|
||||
|
||||
Client libraries not only submit finished spans to Jaeger backend, but also periodically poll the Agents for various configurations, such as sampling strategies. The schema for the payload is defined by [sampling.thrift][sampling.thrift], encoded as JSON using Thrift's built-in JSON generation capabilities.
|
||||
|
||||
## Service dependencies graph (internal)
|
||||
|
||||
Can be retrieved from Query Service at `/api/dependencies` endpoint. The GET request expects two parameters:
|
||||
|
||||
* `endTs` (number of milliseconds since epoch) - the end of the time interval
|
||||
* `lookback` (in milliseconds) - the length the time interval (i.e. start-time + lookback = end-time).
|
||||
|
||||
The returned JSON is a list of edges represented as tuples `(caller, callee, count)`.
|
||||
|
||||
For programmatic access to service graph, the recommended API is gRPC/Protobuf described above.
|
||||
|
||||
[jaeger-idl]: https://github.com/jaegertracing/jaeger-idl/
|
||||
[jaeger.thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/jaeger.thrift
|
||||
[agent.thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/agent.thrift
|
||||
[sampling.thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/sampling.thrift
|
||||
[collector.proto]: https://github.com/jaegertracing/jaeger/blob/master/model/proto/api_v2/collector.proto
|
||||
[query.proto]: https://github.com/jaegertracing/jaeger/blob/master/model/proto/api_v2/query.proto
|
|
@ -1,13 +0,0 @@
|
|||
---
|
||||
title: CLI flags
|
||||
widescreen: true
|
||||
hasparent: true
|
||||
---
|
||||
|
||||
This is auto-generated documentation for CLI flags supported by Jaeger binaries.
|
||||
|
||||
* CLI flags for some binaries change depending on the `SPAN_STORAGE_TYPE` environment variable. Relevant variations are included below.
|
||||
* Some binaries support _commands_ (mostly informational), such as `env`, `docs`, and `version`. These commands are not included here.
|
||||
* All parameters can be also provided via environment variables, by changing all letters to upper-case and replacing all punctuation characters with underscore `_`. For example, the value for the flag `--cassandra.connections-per-host` can be provided via `CASSANDRA_CONNECTIONS_PER_HOST` environment variable.
|
||||
|
||||
{{< cli/tools-list >}}
|
|
@ -1,469 +0,0 @@
|
|||
---
|
||||
title: Deployment
|
||||
weight: 4
|
||||
children:
|
||||
- title: Operator for Kubernetes
|
||||
navtitle: Kubernetes
|
||||
url: operator
|
||||
- title: Frontend/UI
|
||||
url: frontend-ui
|
||||
- title: Windows
|
||||
url: windows
|
||||
- title: CLI Flags
|
||||
url: cli
|
||||
---
|
||||
|
||||
The main Jaeger backend components are released as Docker images on Docker Hub:
|
||||
|
||||
Component | Repository
|
||||
--------------------- | ---
|
||||
**jaeger-agent** | [hub.docker.com/r/jaegertracing/jaeger-agent/](https://hub.docker.com/r/jaegertracing/jaeger-agent/)
|
||||
**jaeger-collector** | [hub.docker.com/r/jaegertracing/jaeger-collector/](https://hub.docker.com/r/jaegertracing/jaeger-collector/)
|
||||
**jaeger-query** | [hub.docker.com/r/jaegertracing/jaeger-query/](https://hub.docker.com/r/jaegertracing/jaeger-query/)
|
||||
**jaeger-ingester** | [hub.docker.com/r/jaegertracing/jaeger-ingester/](https://hub.docker.com/r/jaegertracing/jaeger-ingester/)
|
||||
|
||||
There are orchestration templates for running Jaeger with:
|
||||
|
||||
* Kubernetes: [github.com/jaegertracing/jaeger-kubernetes](https://github.com/jaegertracing/jaeger-kubernetes),
|
||||
* OpenShift: [github.com/jaegertracing/jaeger-openshift](https://github.com/jaegertracing/jaeger-openshift).
|
||||
|
||||
## Configuration Options
|
||||
|
||||
Jaeger binaries can be configured in a number of ways (in the order of decreasing priority):
|
||||
|
||||
* command line arguments,
|
||||
* environment variables,
|
||||
* configuration files in JSON, TOML, YAML, HCL, or Java properties formats.
|
||||
|
||||
To see the complete list of options, run the binary with `help` command. Options that are specific to a certain storage backend are only listed if the storage type is selected. For example, to see all available options in the Collector with Cassandra storage:
|
||||
|
||||
```sh
|
||||
$ docker run --rm \
|
||||
-e SPAN_STORAGE_TYPE=cassandra \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}} \
|
||||
help
|
||||
```
|
||||
|
||||
In order to provide configuration parameters via environment variables, find the respective command line option and convert its name to UPPER_SNAKE_CASE, for example:
|
||||
|
||||
Command line option | Environment variable
|
||||
-----------------------------------|-------------------------------
|
||||
`--cassandra.connections-per-host` | `CASSANDRA_CONNECTIONS_PER_HOST`
|
||||
`--metrics-backend` | `METRICS_BACKEND`
|
||||
|
||||
## Agent
|
||||
|
||||
Jaeger client libraries expect **jaeger-agent** process to run locally on each host.
|
||||
The agent exposes the following ports:
|
||||
|
||||
Port | Protocol | Function
|
||||
----- | ------- | ---
|
||||
6831 | UDP | accept [jaeger.thrift][jaeger-thrift] in `compact` Thrift protocol used by most current Jaeger clients
|
||||
6832 | UDP | accept [jaeger.thrift][jaeger-thrift] in `binary` Thrift protocol used by Node.js Jaeger client (because [thriftrw][thriftrw] npm package does not support `compact` protocol)
|
||||
5778 | HTTP | serve configs, sampling strategies
|
||||
5775 | UDP | accept [zipkin.thrift][zipkin-thrift] in `compact` Thrift protocol (deprecated; only used by very old Jaeger clients, circa 2016)
|
||||
14271 | HTTP | Healthcheck at `/` and metrics at `/metrics`
|
||||
|
||||
It can be executed directly on the host or via Docker, as follows:
|
||||
|
||||
```sh
|
||||
## make sure to expose only the ports you use in your deployment scenario!
|
||||
docker run \
|
||||
--rm \
|
||||
-p6831:6831/udp \
|
||||
-p6832:6832/udp \
|
||||
-p5778:5778/tcp \
|
||||
-p5775:5775/udp \
|
||||
jaegertracing/jaeger-agent:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
### Discovery System Integration
|
||||
|
||||
The agents can connect point to point to a single collector address, which could be
|
||||
load balanced by another infrastructure component (e.g. DNS) across multiple collectors.
|
||||
The agent can also be configured with a static list of collector addresses.
|
||||
|
||||
On Docker, a command like the following can be used:
|
||||
|
||||
```sh
|
||||
docker run \
|
||||
--rm \
|
||||
-p5775:5775/udp \
|
||||
-p6831:6831/udp \
|
||||
-p6832:6832/udp \
|
||||
-p5778:5778/tcp \
|
||||
jaegertracing/jaeger-agent:{{< currentVersion >}} \
|
||||
--reporter.grpc.host-port=jaeger-collector.jaeger-infra.svc:14250
|
||||
```
|
||||
|
||||
Or use `--reporter.tchannel.host-port=jaeger-collector.jaeger-infra.svc:14267` to use
|
||||
legacy tchannel reporter.
|
||||
|
||||
When using gRPC, you have several options for load balancing and name resolution:
|
||||
|
||||
* Single connection and no load balancing. This is the default if you specify a single `host:port`. (example: `--reporter.grpc.host-port=jaeger-collector.jaeger-infra.svc:14250`)
|
||||
* Static list of hostnames and round-robin load balancing. This is what you get with a comma-separated list of addresses. (example: `reporter.grpc.host-port=jaeger-collector1:14250,jaeger-collector2:14250,jaeger-collector3:14250`)
|
||||
* Dynamic DNS resolution and round-robin load balancing. To get this behaviour, prefix the address with `dns:///` and gRPC will attempt to resolve the hostname using SRV records (for [external load balancing](https://github.com/grpc/grpc/blob/master/doc/load-balancing.md)), TXT records (for [service configs](https://github.com/grpc/grpc/blob/master/doc/service_config.md)), and A records. Refer to the [gRPC Name Resolution docs](https://github.com/grpc/grpc/blob/master/doc/naming.md) and the [dns_resolver.go implementation](https://github.com/grpc/grpc-go/blob/master/resolver/dns/dns_resolver.go) for more info. (example: `--reporter.grpc.host-port=dns:///jaeger-collector.jaeger-infra.svc:14250`)
|
||||
|
||||
### Agent level tags
|
||||
|
||||
Jaeger supports agent level tags, that can be added to the process tags of all spans passing through the agent. This is supported through the command line flag `--agent.tags=key1=value1,key2=value2,...,keyn=valuen`. Tags can also be set through an environment flag like so - `--agent.tags=key=${envFlag:defaultValue}` - The tag value will be set to the value of the `envFlag` environment key and `defaultValue` if not set. This feature is not supported for the tchannel reporter, enabled using the flags `--collector.host-port` or `--reporter.tchannel.host-port`.
|
||||
|
||||
## Collectors
|
||||
|
||||
The collectors are stateless and thus many instances of **jaeger-collector** can be run in parallel.
|
||||
Collectors require almost no configuration, except for the location of Cassandra cluster,
|
||||
via `--cassandra.keyspace` and `--cassandra.servers` options, or the location of Elasticsearch cluster, via
|
||||
`--es.server-urls`, depending on which storage is specified. To see all command line options run
|
||||
|
||||
```sh
|
||||
go run ./cmd/collector/main.go -h
|
||||
```
|
||||
|
||||
or, if you don't have the source code
|
||||
|
||||
```sh
|
||||
docker run -it --rm jaegertracing/jaeger-collector:{{< currentVersion >}} -h
|
||||
```
|
||||
|
||||
At default settings the collector exposes the following ports:
|
||||
|
||||
Port | Protocol | Function
|
||||
----- | ------- | ---
|
||||
14267 | TChannel | used by **jaeger-agent** to send spans in jaeger.thrift format
|
||||
14250 | gRPC | used by **jaeger-agent** to send spans in model.proto format
|
||||
14268 | HTTP | can accept spans directly from clients in jaeger.thrift format over binary thrift protocol
|
||||
9411 | HTTP | can accept Zipkin spans in Thrift, JSON and Proto (disabled by default)
|
||||
14269 | HTTP | Healthcheck at `/` and metrics at `/metrics`
|
||||
|
||||
|
||||
## Storage Backends
|
||||
|
||||
Collectors require a persistent storage backend. Cassandra and Elasticsearch are the primary supported storage backends. Additional backends are [discussed here](https://github.com/jaegertracing/jaeger/issues/638).
|
||||
|
||||
The storage type can be passed via `SPAN_STORAGE_TYPE` environment variable. Valid values are `cassandra`, `elasticsearch`, `kafka` (only as a buffer), `grpc-plugin`, `badger` (only with all-in-one) and `memory` (only with all-in-one).
|
||||
|
||||
As of version 1.6.0, it's possible to use multiple storage types at the same time by providing a comma-separated list of valid types to the `SPAN_STORAGE_TYPE` environment variable.
|
||||
It's important to note that all listed storage types are used for writing, but only the first type in the list will be used for reading and archiving.
|
||||
|
||||
### Memory
|
||||
|
||||
The in-memory storage is not intended for production workloads. It's intended as a simple solution to get started quickly and
|
||||
data will be lost once the process is gone.
|
||||
|
||||
By default, there's no limit in the amount of traces stored in memory but a limit can be established by passing an
|
||||
integer value via `--memory.max-traces`.
|
||||
|
||||
### Badger - local storage
|
||||
Experimental since Jaeger 1.9
|
||||
|
||||
[Badger](https://github.com/dgraph-io/badger) is an embedded local storage, only available
|
||||
with **all-in-one** distribution. By default it acts as an ephemeral storage using a temporary filesystem.
|
||||
This can be overridden by using the `--badger.ephemeral=false` option.
|
||||
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=badger \
|
||||
-e BADGER_EPHEMERAL=false \
|
||||
-e BADGER_DIRECTORY_VALUE=/badger/data \
|
||||
-e BADGER_DIRECTORY_KEY=/badger/key \
|
||||
-v <storage_dir_on_host>:/badger \
|
||||
-p 16686:16686 \
|
||||
jaegertracing/all-in-one:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
### Cassandra
|
||||
Supported versions: 3.4+
|
||||
|
||||
Deploying Cassandra itself is out of scope for our documentation. One good
|
||||
source of documentation is the [Apache Cassandra Docs](https://cassandra.apache.org/doc/latest/).
|
||||
|
||||
#### Configuration
|
||||
##### Minimal
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=cassandra \
|
||||
-e CASSANDRA_SERVERS=<...> \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
##### All options
|
||||
To view the full list of configuration options, you can run the following command:
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=cassandra \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}} \
|
||||
--help
|
||||
```
|
||||
|
||||
#### Schema script
|
||||
|
||||
A script is provided to initialize Cassandra keyspace and schema
|
||||
using Cassandra's interactive shell [`cqlsh`][cqlsh]:
|
||||
|
||||
```sh
|
||||
MODE=test sh ./plugin/storage/cassandra/schema/create.sh | cqlsh
|
||||
```
|
||||
|
||||
For production deployment, pass `MODE=prod DATACENTER={datacenter}` arguments to the script,
|
||||
where `{datacenter}` is the name used in the Cassandra configuration / network topology.
|
||||
|
||||
The script also allows overriding TTL, keyspace name, replication factor, etc.
|
||||
Run the script without arguments to see the full list of recognized parameters.
|
||||
|
||||
#### TLS support
|
||||
|
||||
Jaeger supports TLS client to node connections as long as you've configured
|
||||
your Cassandra cluster correctly. After verifying with e.g. `cqlsh`, you can
|
||||
configure the collector and query like so:
|
||||
|
||||
```sh
|
||||
docker run \
|
||||
-e CASSANDRA_SERVERS=<...> \
|
||||
-e CASSANDRA_TLS=true \
|
||||
-e CASSANDRA_TLS_SERVER_NAME="CN-in-certificate" \
|
||||
-e CASSANDRA_TLS_KEY=<path to client key file> \
|
||||
-e CASSANDRA_TLS_CERT=<path to client cert file> \
|
||||
-e CASSANDRA_TLS_CA=<path to your CA cert file> \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
The schema tool also supports TLS. You need to make a custom cqlshrc file like
|
||||
so:
|
||||
|
||||
```
|
||||
# Creating schema in a cassandra cluster requiring client TLS certificates.
|
||||
#
|
||||
# Create a volume for the schema docker container containing four files:
|
||||
# cqlshrc: this file
|
||||
# ca-cert: the cert authority for your keys
|
||||
# client-key: the keyfile for your client
|
||||
# client-cert: the cert file matching client-key
|
||||
#
|
||||
# if there is any sort of DNS mismatch and you want to ignore server validation
|
||||
# issues, then uncomment validate = false below.
|
||||
#
|
||||
# When running the container, map this volume to /root/.cassandra and set the
|
||||
# environment variable CQLSH_SSL=--ssl
|
||||
[ssl]
|
||||
certfile = ~/.cassandra/ca-cert
|
||||
userkey = ~/.cassandra/client-key
|
||||
usercert = ~/.cassandra/client-cert
|
||||
# validate = false
|
||||
```
|
||||
|
||||
### Elasticsearch
|
||||
Supported in Jaeger since 0.6.0
|
||||
Supported versions: 5.x, 6.x, 7.x
|
||||
|
||||
Elasticsearch version is automatically retrieved from root/ping endpoint.
|
||||
Based on this version Jaeger uses compatible index mappings and Elasticsearch REST API.
|
||||
The version can be explicitly provided via `--es.version=` flag.
|
||||
|
||||
Elasticsearch does not require initialization other than
|
||||
[installing and running Elasticsearch](https://www.elastic.co/downloads/elasticsearch).
|
||||
Once it is running, pass the correct configuration values to the Jaeger collector and query service.
|
||||
|
||||
#### Configuration
|
||||
##### Minimal
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=elasticsearch \
|
||||
-e ES_SERVER_URLS=<...> \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
##### All options
|
||||
To view the full list of configuration options, you can run the following command:
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=elasticsearch \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}} \
|
||||
--help
|
||||
```
|
||||
|
||||
#### Shards and Replicas for Elasticsearch indices
|
||||
|
||||
Shards and replicas are some configuration values to take special attention to, because this is decided upon
|
||||
index creation. [This article](https://qbox.io/blog/optimizing-elasticsearch-how-many-shards-per-index) goes into
|
||||
more information about choosing how many shards should be chosen for optimization.
|
||||
|
||||
#### Upgrade Elasticsearch version
|
||||
|
||||
Elasticsearch defines wire and index compatibility versions. The index compatibility defines
|
||||
the minimal version a node can read data from. For example Elasticsearch 7 can read indices
|
||||
created by Elasticsearch 6, however it cannot read indices created by Elasticsearch 5 even
|
||||
though they use the same index mappings. Therefore upgrade from Elasticsearch 6 to 7 does not require any
|
||||
data migration. However, upgrade from Elasticsearch 5 to 7 has to be done through Elasticsearch 6 and wait
|
||||
until indices created by ES 5.x are removed or explicitly reindexed.
|
||||
|
||||
Refer to the Elasticsearch [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current//setup-upgrade.html)
|
||||
for wire and index compatibility versions. Generally this information can be retrieved from root/ping REST endpoint.
|
||||
|
||||
##### Reindex
|
||||
|
||||
Manual reindexing can be used when upgrading from Elasticsearch 5 to 7 (through Elasticsearch 6)
|
||||
without waiting until indices created by Elasticsearch 5 are removed.
|
||||
|
||||
1. Reindex all span indices to new indices with suffix `-1`:
|
||||
|
||||
```bash
|
||||
curl -ivX POST -H "Content-Type: application/json" http://localhost:9200/_reindex -d @reindex.json
|
||||
{
|
||||
"source": {
|
||||
"index": "jaeger-span-*"
|
||||
},
|
||||
"dest": {
|
||||
"index": "jaeger-span"
|
||||
},
|
||||
"script": {
|
||||
"lang": "painless",
|
||||
"source": "ctx._index = 'jaeger-span-' + (ctx._index.substring('jaeger-span-'.length(), ctx._index.length())) + '-1'"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
2. Delete indices with old mapping:
|
||||
|
||||
```bash
|
||||
curl -ivX DELETE -H "Content-Type: application/json" http://localhost:9200/jaeger-span-\*,-\*-1
|
||||
```
|
||||
|
||||
3. Create indices without `-1` suffix:
|
||||
|
||||
```bash
|
||||
curl -ivX POST -H "Content-Type: application/json" http://localhost:9200/_reindex -d @reindex.json
|
||||
{
|
||||
"source": {
|
||||
"index": "jaeger-span-*"
|
||||
},
|
||||
"dest": {
|
||||
"index": "jaeger-span"
|
||||
},
|
||||
"script": {
|
||||
"lang": "painless",
|
||||
"source": "ctx._index = 'jaeger-span-' + (ctx._index.substring('jaeger-span-'.length(), ctx._index.length() - 2))"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
4. Remove suffixed indices:
|
||||
|
||||
```bash
|
||||
curl -ivX DELETE -H "Content-Type: application/json" http://localhost:9200/jaeger-span-\*-1
|
||||
```
|
||||
|
||||
Run the commands analogically for other Jaeger indices.
|
||||
|
||||
There might exist more effective migration procedure. Please share with the community any findings.
|
||||
|
||||
### Kafka
|
||||
Supported in Jaeger since 1.6.0
|
||||
Supported Kafka versions: 0.9+
|
||||
|
||||
Kafka can be used as an intermediary buffer between collector and an actual storage.
|
||||
The collector is configured with `SPAN_STORAGE_TYPE=kafka` that makes it write all received spans
|
||||
into a Kafka topic. A new component [Ingester](#ingester), added in version 1.7.0, is used to read from
|
||||
Kafka and store spans in another storage backend (Elasticsearch or Cassandra).
|
||||
|
||||
Writing to Kafka is particularly useful for building post-processing data pipelines.
|
||||
|
||||
#### Configuration
|
||||
##### Minimal
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=kafka \
|
||||
-e KAFKA_PRODUCER_BROKERS=<...> \
|
||||
-e KAFKA_TOPIC=<...> \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
##### All options
|
||||
To view the full list of configuration options, you can run the following command:
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=kafka \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}} \
|
||||
--help
|
||||
```
|
||||
|
||||
#### Topic & partitions
|
||||
Unless your Kafka cluster is configured to automatically create topics, you will need to create it ahead of time. You can refer to [the Kafka quickstart documentation](https://kafka.apache.org/documentation/#quickstart_createtopic) to learn how.
|
||||
|
||||
You can find more information about topics and partitions in general in the [official documentation](https://kafka.apache.org/documentation/#intro_topics). [This article](https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/) provide more details about how to choose the number of partitions.
|
||||
|
||||
### Storage plugin
|
||||
|
||||
Jaeger supports gRPC based storage plugins. For more information refer to [jaeger/plugin/storage/grpc](https://github.com/jaegertracing/jaeger/tree/master/plugin/storage/grpc)
|
||||
|
||||
Available plugins:
|
||||
|
||||
* [InfluxDB](https://github.com/influxdata/jaeger-influxdb/)
|
||||
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=grpc-plugin \
|
||||
-e GRPC_STORAGE_PLUGIN_BINARY=<...> \
|
||||
-e GRPC_STORAGE_PLUGIN_CONFIGURATION_FILE=<...> \
|
||||
jaegertracing/all-in-one:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
## Ingester
|
||||
**jaeger-ingester** is a service which reads span data from Kafka topic and writes it to another storage backend (Elasticsearch or Cassandra).
|
||||
|
||||
Port | Protocol | Function
|
||||
----- | ------- | ---
|
||||
14270 | HTTP | Healthcheck at `/` and metrics at `/metrics`
|
||||
|
||||
To view all exposed configuration options run the following command:
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=cassandra \
|
||||
jaegertracing/jaeger-ingester:{{< currentVersion >}}
|
||||
--help
|
||||
```
|
||||
|
||||
## Query Service & UI
|
||||
|
||||
**jaeger-query** serves the API endpoints and a React/Javascript UI.
|
||||
The service is stateless and is typically run behind a load balancer, such as [**NGINX**](https://www.nginx.com/).
|
||||
|
||||
At default settings the query service exposes the following port(s):
|
||||
|
||||
Port | Protocol | Function
|
||||
----- | ------- | ---
|
||||
16686 | HTTP | `/api/*` endpoints and Jaeger UI at `/`
|
||||
16687 | HTTP | Healthcheck at `/` and metrics at `/metrics`
|
||||
|
||||
### Minimal deployment example (Elasticsearch backend):
|
||||
```sh
|
||||
docker run -d --rm \
|
||||
-p 16686:16686 \
|
||||
-p 16687:16687 \
|
||||
-e SPAN_STORAGE_TYPE=elasticsearch \
|
||||
-e ES_SERVER_URLS=http://<ES_SERVER_IP>:<ES_SERVER_PORT> \
|
||||
jaegertracing/jaeger-query:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
### UI Base Path
|
||||
|
||||
The base path for all **jaeger-query** HTTP routes can be set to a non-root value, e.g. `/jaeger` would cause all UI URLs to start with `/jaeger`. This can be useful when running **jaeger-query** behind a reverse proxy.
|
||||
|
||||
The base path can be configured via the `--query.base-path` command line parameter or the `QUERY_BASE_PATH` environment variable.
|
||||
|
||||
### UI Customization and Embedding
|
||||
|
||||
Please refer to the [dedicated Frontend/UI page](../frontend-ui/).
|
||||
|
||||
## Aggregation Jobs for Service Dependencies
|
||||
|
||||
Production deployments need an external process which aggregates data and creates dependency links between services. Project [spark-dependencies](https://github.com/jaegertracing/spark-dependencies) is a Spark job which derives dependency links and stores them directly to the storage.
|
||||
|
||||
## Configuration
|
||||
|
||||
All binaries accepts command line properties and environmental variables, power by [viper](https://github.com/spf13/viper) and [cobra](https://github.com/spf13/cobra) libraries. Please refer to the [CLI Flags](../cli/) page for more information.
|
||||
|
||||
[cqlsh]: http://cassandra.apache.org/doc/latest/tools/cqlsh.html
|
||||
[zipkin-thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/zipkincore.thrift
|
||||
[jaeger-thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/jaeger.thrift
|
||||
[thriftrw]: https://www.npmjs.com/package/thriftrw
|
|
@ -1,26 +0,0 @@
|
|||
---
|
||||
title: Frequently Asked Questions
|
||||
navtitle: FAQs
|
||||
description: Answers to some frequently asked questions about Jaeger.
|
||||
weight: 11
|
||||
---
|
||||
|
||||
## Why is the Dependencies page empty?
|
||||
|
||||
The Dependencies page shows a graph of services traced by Jaeger and connections between them. When you are using `all-in-one` binary with in-memory storage, the graph is calculated on-demand from all the traces stored in memory. However, if you are using a real distributed storage like Cassandra or Elasticsearch, it is too expensive to scan all the data in the database to build the service graph. Instead, the Jaeger project provides "big data" jobs that can be used to extract the service graph data from traces:
|
||||
|
||||
* https://github.com/jaegertracing/spark-dependencies - the older Spark job that can be run periodically
|
||||
* https://github.com/jaegertracing/jaeger-analytics - the new (experimental) streaming Flink jobs that run continuously and builds the service graph in smaller time intervals
|
||||
|
||||
## Why do I not see any spans in Jaeger?
|
||||
|
||||
Please refer to the [Troubleshooting](../troubleshooting/) guide.
|
||||
|
||||
## Do I need to run jaeger-agent?
|
||||
|
||||
`jaeger-agent` is not always necessary. Jaeger client libraries can be configured to export trace data directly to `jaeger-collector`. However, the following are the reasons why running `jaeger-agent` is recommended:
|
||||
|
||||
* If we want Jaeger client libraries to send trace data directly to collectors, we must provide them with a URL of the HTTP endpoint. It means that our applications require additional configuration containing this parameter, especially if we are running multiple Jaeger installations (e.g. in different availability zones or regions) and want the data sent to a nearby installation. In contrast, when using the agent, the libraries require no additional configuration because the agent is always accessible via `localhost`. It acts as a sidecar and proxies the requests to the appropriate collectors.
|
||||
* The agent can be configured to enrich the tracing data with infrastructure-specific metadata by adding extra tags to the spans, such as the current zone, region, etc. If the agent is running as a host daemon, it will be shared by all applications running on the same host. If the agent is running as a true sidecar, i.e. one per application, it can provide additional functionality such as strong authentication, multi-tenancy (see [this blog post](https://medium.com/jaegertracing/jaeger-and-multitenancy-99dfa1d49dc0)), pod name, etc.
|
||||
* If we want Jaeger client libraries to use sampling strategies that are centrally configured in the collectors, this is only possible by using the `/sampling` HTTP endpoint on the agent. There is no technical reason why this endpoint cannot be implemented directly in the collectors, it's just [not done yet](https://github.com/jaegertracing/jaeger/issues/1971).
|
||||
* Agents allow implementing traffic control to the collectors. If we have thousands of hosts in the data center, each running many applications, and each application sending data directly to the collectors, there may be too many open connections for each collector to handle. The agents can load balance this traffic with fewer connections.
|
|
@ -1,57 +0,0 @@
|
|||
---
|
||||
title: Features
|
||||
hasparent: true
|
||||
---
|
||||
|
||||
Jaeger is used for monitoring and troubleshooting microservices-based distributed systems, including:
|
||||
|
||||
* Distributed context propagation
|
||||
* Distributed transaction monitoring
|
||||
* Root cause analysis
|
||||
* Service dependency analysis
|
||||
* Performance / latency optimization
|
||||
|
||||
## High Scalability
|
||||
|
||||
Jaeger backend is designed to have no single points of failure and to scale with the business needs.
|
||||
For example, any given Jaeger installation at Uber is typically processing several billion {{< tip "spans" "span" >}} per day.
|
||||
|
||||
## Native support for OpenTracing
|
||||
|
||||
Jaeger backend, Web UI, and instrumentation libraries have been designed from ground up to support the OpenTracing standard.
|
||||
|
||||
* Represent {{< tip "traces" "trace" >}} as {{< tip "directed acyclic graphs" "directed acyclic graph" >}} (not just trees) via [span references](https://github.com/opentracing/specification/blob/master/specification.md#references-between-spans)
|
||||
* Support strongly typed span _tags_ and _structured logs_
|
||||
* Support general distributed context propagation mechanism via _baggage_
|
||||
|
||||
## Multiple storage backends
|
||||
|
||||
Jaeger supports two popular open source NoSQL databases as trace storage backends: Cassandra 3.4+ and Elasticsearch 5.x/6.x/7.x.
|
||||
There are ongoing community experiments using other databases, such as ScyllaDB, InfluxDB, Amazon DynamoDB. Jaeger also ships
|
||||
with a simple in-memory storage for testing setups.
|
||||
|
||||
## Modern Web UI
|
||||
|
||||
Jaeger Web UI is implemented in Javascript using popular open source frameworks like React. Several performance
|
||||
improvements have been released in v1.0 to allow the UI to efficiently deal with large volumes of data, and to display
|
||||
{{< tip "traces" "trace" >}} with tens of thousands of {{< tip "spans" "span" >}} (e.g. we tried a trace with 80,000 spans).
|
||||
|
||||
## Cloud Native Deployment
|
||||
|
||||
Jaeger backend is distributed as a collection of Docker images. The binaries support various configuration methods,
|
||||
including command line options, environment variables, and configuration files in multiple formats (yaml, toml, etc.).
|
||||
Deployment to Kubernetes clusters is assisted by a [Kubernetes operator](https://github.com/jaegertracing/jaeger-operator), [Kubernetes templates](https://github.com/jaegertracing/jaeger-kubernetes)
|
||||
and a [Helm chart](https://github.com/kubernetes/charts/tree/master/incubator/jaeger).
|
||||
|
||||
## Observability
|
||||
|
||||
All Jaeger backend components expose [Prometheus](https://prometheus.io/) metrics by default (other metrics backends are
|
||||
also supported). Logs are written to standard out using the structured logging library [zap](https://github.com/uber-go/zap).
|
||||
|
||||
## Backwards compatibility with Zipkin
|
||||
|
||||
Although we recommend instrumenting applications with OpenTracing API and binding to Jaeger client libraries to benefit
|
||||
from advanced features not available elsewhere, if your organization has already invested in the instrumentation
|
||||
using Zipkin libraries, you do not have to rewrite all that code. Jaeger provides backwards compatibility with Zipkin
|
||||
by accepting spans in Zipkin formats (Thrift, JSON v1/v2 and Protobuf) over HTTP. Switching from Zipkin backend is just a matter
|
||||
of routing the traffic from Zipkin libraries to the Jaeger backend.
|
|
@ -1,224 +0,0 @@
|
|||
---
|
||||
title: Frontend/UI Configuration
|
||||
navtitle: Frontend/UI
|
||||
hasparent: true
|
||||
weight: 7
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
Several aspects of the UI can be configured:
|
||||
|
||||
* The Dependencies section can be enabled / configured
|
||||
* Google Analytics tracking can be enabled / configured
|
||||
* Additional menu options can be added to the global nav
|
||||
|
||||
These options can be configured by a JSON configuration file. The `--query.ui-config` command line parameter of the query service must then be set to the path to the JSON file when the query service is started.
|
||||
|
||||
An example configuration file:
|
||||
|
||||
```json
|
||||
{
|
||||
"dependencies": {
|
||||
"dagMaxNumServices": 200,
|
||||
"menuEnabled": true
|
||||
},
|
||||
"archiveEnabled": true,
|
||||
"tracking": {
|
||||
"gaID": "UA-000000-2",
|
||||
"trackErrors": true
|
||||
},
|
||||
"menu": [
|
||||
{
|
||||
"label": "About Jaeger",
|
||||
"items": [
|
||||
{
|
||||
"label": "GitHub",
|
||||
"url": "https://github.com/jaegertracing/jaeger"
|
||||
},
|
||||
{
|
||||
"label": "Docs",
|
||||
"url": "http://jaeger.readthedocs.io/en/latest/"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"linkPatterns": [{
|
||||
"type": "process",
|
||||
"key": "jaeger.version",
|
||||
"url": "https://github.com/jaegertracing/jaeger-client-java/releases/tag/#{jaeger.version}",
|
||||
"text": "Information about Jaeger release #{jaeger.version}"
|
||||
}]
|
||||
}
|
||||
```
|
||||
|
||||
### Dependencies
|
||||
|
||||
`dependencies.dagMaxNumServices` defines the maximum number of services allowed before the DAG dependency view is disabled. Default: `200`.
|
||||
|
||||
`dependencies.menuEnabled` enables (`true`) or disables (`false`) the dependencies menu button. Default: `true`.
|
||||
|
||||
### Archive Support
|
||||
|
||||
`archiveEnabled` enables (`true`) or disables (`false`) the archive traces button. Default: `false`. It requires a configuration of an archive storage in Query service. Archived traces are only accessible directly by ID, they are not searchable.
|
||||
|
||||
### Google Analytics Tracking
|
||||
|
||||
`tracking.gaID` defines the Google Analytics tracking ID. This is required for Google Analytics tracking, and setting it to a non-`null` value enables Google Analytics tracking. Default: `null`.
|
||||
|
||||
`tracking.trackErrors` enables (`true`) or disables (`false`) error tracking via Google Analytics. Errors can only be tracked if a valid Google Analytics ID is provided. For additional details on error tracking via Google Analytics see the [tracking README](https://github.com/jaegertracing/jaeger-ui/blob/c622330546afc1be59a42f874bcc1c2fadf7e69a/src/utils/tracking/README.md) in the UI repo. Default: `true`.
|
||||
|
||||
### Custom Menu Items
|
||||
|
||||
`menu` allows additional links to be added to the global nav. The additional links are right-aligned.
|
||||
|
||||
In the sample JSON config above, the configured menu will have a dropdown labeled "About Jaeger" with sub-options for "GitHub" and "Docs". The format for a link in the top right menu is as follows:
|
||||
|
||||
```json
|
||||
{
|
||||
"label": "Some text here",
|
||||
"url": "https://example.com"
|
||||
}
|
||||
```
|
||||
|
||||
Links can either be members of the `menu` Array, directly, or they can be grouped into a dropdown menu option. The format for a group of links is:
|
||||
|
||||
```json
|
||||
{
|
||||
"label": "Dropdown button",
|
||||
"items": [ ]
|
||||
}
|
||||
```
|
||||
|
||||
The `items` Array should contain one or more link configurations.
|
||||
|
||||
### Link Patterns
|
||||
|
||||
The `linkPatterns` node can be used to create links from fields displayed in the Jaeger UI.
|
||||
|
||||
Field | Description
|
||||
------|------------
|
||||
type | The metadata section in which your link will be added: process, tags, logs
|
||||
key | The name of tag/process/log attribute which value will be displayed as a link
|
||||
url | The URL where the link should point to, it can be an external site or relative path in Jaeger UI
|
||||
text | The text displayed in the tooltip for the link
|
||||
|
||||
Both `url` and `text` can be defined as templates (i.e. using `#{field-name}`) where Jaeger UI will dynamically substitute values based on tags/logs data.
|
||||
|
||||
## Embedded Mode
|
||||
|
||||
Starting with version 1.9, Jaeger UI provides an "embedded" layout mode which is intended to support integrating Jaeger UI into other applications. Currently (as of `v0`), the approach taken is to remove various UI elements from the page to make the UI better suited for space-constrained layouts.
|
||||
|
||||
The embedded mode is induced and configured via URL query parameters.
|
||||
|
||||
To enter embedded mode, the `uiEmbed=v0` query parameter and value must be added to the URL. For example, the following URL will show the trace with ID `abc123` in embedded mode:
|
||||
|
||||
```
|
||||
http://localhost:16686/trace/abc123?uiEmbed=v0
|
||||
```
|
||||
|
||||
`uiEmbed=v0` is required.
|
||||
|
||||
Further, each page supported has an <img src="/img/frontend-ui/embed-open-icon.png" style="width: 20px; height:20px;display:inline;" alt="Embed open window"> button added that will open the non-embedded page in a new tab.
|
||||
|
||||
The following pages support embedded mode:
|
||||
|
||||
* Search Page
|
||||
* Trace Page
|
||||
|
||||
### Search Page
|
||||
|
||||
To integrate the Search Trace Page to our application we have to indicate to the Jaeger UI that we want to use the embed mode with `uiEmbed=v0`.
|
||||
|
||||
For example:
|
||||
|
||||
```
|
||||
http://localhost:16686/search?
|
||||
service=my-service&
|
||||
start=1543917759557000&
|
||||
end=1543921359557000&
|
||||
limit=20&
|
||||
lookback=1h&
|
||||
maxDuration&
|
||||
minDuration&
|
||||
uiEmbed=v0
|
||||
```
|
||||
|
||||

|
||||
|
||||
#### Configuration options
|
||||
|
||||
The following query parameter can be used to configure the layout of the search page :
|
||||
|
||||
* `uiSearchHideGraph=1` - disables the display of the scatter plot above the search results
|
||||
|
||||
```
|
||||
http://localhost:16686/search?
|
||||
service=my-service&
|
||||
start=1543917759557000&
|
||||
end=1543921359557000&
|
||||
limit=20&
|
||||
lookback=1h&
|
||||
maxDuration&
|
||||
minDuration&
|
||||
uiEmbed=v0&
|
||||
uiSearchHideGraph=1
|
||||
```
|
||||
|
||||

|
||||
|
||||
### Trace Page
|
||||
|
||||
|
||||
To integrate the Trace Page to our application we have to indicate to the Jaeger UI that we want to use the embed mode with `uiEmbed=v0`.
|
||||
|
||||
For example:
|
||||
|
||||
```sh
|
||||
http://localhost:16686/trace/{trace-id}?uiEmbed=v0
|
||||
```
|
||||

|
||||
|
||||
If we have navigated to this view from the search traces page we'll have a button to go back to the results page.
|
||||
|
||||

|
||||
|
||||
#### Configuration options
|
||||
|
||||
The following query parameters can be used to configure the layout of the trace page :
|
||||
|
||||
* `uiTimelineCollapseTitle=1` causes the trace header to start out collapsed, which hides the summary and the minimap.
|
||||
|
||||
```
|
||||
http://localhost:16686/trace/{trace-id}?
|
||||
uiEmbed=v0&
|
||||
uiTimelineCollapseTitle=1
|
||||
```
|
||||

|
||||
|
||||
* `uiTimelineHideMinimap=1` removes the minimap, entirely, regardless of whether the trace header is expanded or not.
|
||||
|
||||
```
|
||||
http://localhost:16686/trace/{trace-id}?
|
||||
uiEmbed=v0&
|
||||
uiTimelineHideMinimap=1
|
||||
```
|
||||

|
||||
|
||||
* `uiTimelineHideSummary=1` - removes the trace summary information (number of services, etc.) entirely, regardless of whether the trace header is expanded or not.
|
||||
|
||||
```
|
||||
http://localhost:16686/trace/{trace-id}?
|
||||
uiEmbed=v0&
|
||||
uiTimelineHideSummary=1
|
||||
```
|
||||

|
||||
|
||||
We can also combine the options:
|
||||
```
|
||||
http://localhost:16686/trace/{trace-id}?
|
||||
uiEmbed=v0&
|
||||
uiTimelineHideMinimap=1&
|
||||
uiTimelineHideSummary=1
|
||||
```
|
||||

|
|
@ -1,130 +0,0 @@
|
|||
---
|
||||
title: Getting Started
|
||||
description: Get up and running with Jaeger in your local environment
|
||||
weight: 2
|
||||
---
|
||||
|
||||
## Instrumentation
|
||||
|
||||
Your applications must be instrumented before they can send tracing data to Jaeger backend. Check the [Client Libraries](../client-libraries) section for information about how to use the OpenTracing API and how to initialize and configure Jaeger tracers.
|
||||
|
||||
## All in One
|
||||
|
||||
All-in-one is an executable designed for quick local testing, launches the Jaeger UI, collector, query, and agent, with an in memory storage component.
|
||||
|
||||
The simplest way to start the all-in-one is to use the pre-built image published to DockerHub (a single command line).
|
||||
|
||||
```bash
|
||||
$ docker run -d --name jaeger \
|
||||
-e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
|
||||
-p 5775:5775/udp \
|
||||
-p 6831:6831/udp \
|
||||
-p 6832:6832/udp \
|
||||
-p 5778:5778 \
|
||||
-p 16686:16686 \
|
||||
-p 14268:14268 \
|
||||
-p 14250:14250 \
|
||||
-p 9411:9411 \
|
||||
jaegertracing/all-in-one:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
Or run the `jaeger-all-in-one(.exe)` executable from the [binary distribution archives][download]:
|
||||
|
||||
```bash
|
||||
$ jaeger-all-in-one --collector.zipkin.http-port=9411
|
||||
```
|
||||
|
||||
You can then navigate to `http://localhost:16686` to access the Jaeger UI.
|
||||
|
||||
The container exposes the following ports:
|
||||
|
||||
Port | Protocol | Component | Function
|
||||
----- | ------- | --------- | ---
|
||||
5775 | UDP | agent | accept `zipkin.thrift` over compact thrift protocol (deprecated, used by legacy clients only)
|
||||
6831 | UDP | agent | accept `jaeger.thrift` over compact thrift protocol
|
||||
6832 | UDP | agent | accept `jaeger.thrift` over binary thrift protocol
|
||||
5778 | HTTP | agent | serve configs
|
||||
16686 | HTTP | query | serve frontend
|
||||
14268 | HTTP | collector | accept `jaeger.thrift` directly from clients
|
||||
14250 | HTTP | collector | accept `model.proto`
|
||||
9411 | HTTP | collector | Zipkin compatible endpoint (optional)
|
||||
|
||||
|
||||
## Kubernetes and OpenShift
|
||||
|
||||
* Kubernetes templates: https://github.com/jaegertracing/jaeger-kubernetes
|
||||
* Kubernetes Operator: https://github.com/jaegertracing/jaeger-operator
|
||||
* OpenShift templates: https://github.com/jaegertracing/jaeger-openshift
|
||||
|
||||
## Sample App: HotROD
|
||||
|
||||
HotROD (Rides on Demand) is a demo application that consists of several microservices and
|
||||
illustrates the use of the [OpenTracing API](http://opentracing.io).
|
||||
A tutorial / walkthrough is available in the blog post:
|
||||
[Take OpenTracing for a HotROD ride][hotrod-tutorial].
|
||||
|
||||
It can be run standalone, but requires Jaeger backend to view the traces.
|
||||
|
||||
### Features
|
||||
|
||||
- Discover architecture of the whole system via data-driven dependency
|
||||
diagram.
|
||||
- View request timeline and errors; understand how the app works.
|
||||
- Find sources of latency and lack of concurrency.
|
||||
- Highly contextualized logging.
|
||||
- Use baggage propagation to:
|
||||
|
||||
- Diagnose inter-request contention (queueing).
|
||||
- Attribute time spent in a service.
|
||||
|
||||
- Use open source libraries with OpenTracing integration to get
|
||||
vendor-neutral instrumentation for free.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- You need Go 1.11 or higher installed on your machine to run from source.
|
||||
- Requires a [running Jaeger backend](#all-in-one) to view the traces.
|
||||
|
||||
### Running
|
||||
|
||||
#### From Source
|
||||
|
||||
```bash
|
||||
mkdir -p $GOPATH/src/github.com/jaegertracing
|
||||
cd $GOPATH/src/github.com/jaegertracing
|
||||
git clone git@github.com:jaegertracing/jaeger.git jaeger
|
||||
cd jaeger
|
||||
make install
|
||||
go run ./examples/hotrod/main.go all
|
||||
```
|
||||
#### From docker
|
||||
|
||||
```bash
|
||||
$ docker run --rm -it \
|
||||
--link jaeger \
|
||||
-p8080-8083:8080-8083 \
|
||||
-e JAEGER_AGENT_HOST="jaeger" \
|
||||
jaegertracing/example-hotrod:{{< currentVersion >}} \
|
||||
all
|
||||
```
|
||||
|
||||
#### From binary distribution
|
||||
|
||||
Run `example-hotrod(.exe)` executable from the [binary distribution archives][download]:
|
||||
```bash
|
||||
$ example-hotrod all
|
||||
```
|
||||
|
||||
Then navigate to `http://localhost:8080`.
|
||||
|
||||
|
||||
## Migrating from Zipkin
|
||||
|
||||
Collector service exposes Zipkin compatible REST API `/api/v1/spans` which accepts both Thrift and JSON. Also there is `/api/v2/spans` for JSON and Proto.
|
||||
By default it's disabled. It can be enabled with `--collector.zipkin.http-port=9411`.
|
||||
|
||||
Zipkin [Thrift](https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/zipkincore.thrift) IDL and Zipkin [Proto](https://github.com/jaegertracing/jaeger-idl/blob/master/proto/zipkin.proto) IDL files can be found in [jaegertracing/jaeger-idl](https://github.com/jaegertracing/jaeger-idl) repository.
|
||||
They're compatible with [openzipkin/zipkin-api](https://github.com/openzipkin/zipkin-api) [Thrift](https://github.com/openzipkin/zipkin-api/blob/master/thrift/zipkinCore.thrift) and [Proto](https://github.com/openzipkin/zipkin-api/blob/master/zipkin.proto).
|
||||
|
||||
[hotrod-tutorial]: https://medium.com/@YuriShkuro/take-opentracing-for-a-hotrod-ride-f6e3141f7941
|
||||
[download]: ../../../download/
|
File diff suppressed because it is too large
Load Diff
|
@ -1,95 +0,0 @@
|
|||
---
|
||||
title: Performance Tuning Guide
|
||||
description: Tweaking your Jaeger instance to achieve a better performance
|
||||
weight: 10
|
||||
---
|
||||
|
||||
Jaeger was built from day 1 to be able to ingest huge amounts of data in a resilient way. To better utilize resources that might cause delays, such as storage or network communications, Jaeger buffers and batches data. When more spans are generated than Jaeger is able to safely process, spans might get dropped. However, the defaults might not fit all scenarios: for instance, Agents running as a sidecar might have more memory constraints than Agents running as a daemon in bare metal.
|
||||
|
||||
## Deployment considerations
|
||||
|
||||
Although performance tuning the individual components is important, the way Jaeger is deployed can be decisive in obtaining optimal performance.
|
||||
|
||||
### Scale the Collector up and down
|
||||
|
||||
Use the auto-scaling capabilities of your platform: the Collector is nearly horizontally scalable so that more instances can be added and removed on-demand. A good way to scale up and down is by checking the `jaeger_collector_queue_length` metric: add instances when the length is higher than 50% of the maximum size for extended periods of time. Another metric that can be taken into consideration is `jaeger_collector_in_queue_latency_bucket`, which is a histogram indicating how long spans have been waiting in the queue before a worker picked it up. When the queue latency gets higher over time, it’s a good indication to increase the number of the workers, or to improve the storage performance.
|
||||
|
||||
Adding Collector instances is recommended when your platform provides auto-scaling capabilities, or when it's easier to start/stop Collector instances than changing existing, running instances. Scaling horizontally is also indicated when the CPU usage should be spread across nodes.
|
||||
|
||||
### Make sure the storage can keep up
|
||||
|
||||
Each span is written to the storage by the Collector using one worker, blocking it until the span has been stored. When the storage is too slow, the number of workers blocked by the storage might be too high, causing spans to be dropped. To help diagnose this situation, the histogram `jaeger_collector_save_latency_bucket` can be analyzed. Ideally, the latency should remain the same over time. When the histogram shows that most spans are taking longer and longer over time, it’s a good indication that your storage might need some attention.
|
||||
|
||||
### Place the Agents close to your applications
|
||||
|
||||
The Agent is meant to be placed on the same host as the instrumented application, in order to avoid UDP packet loss over the network. This is typically accomplished by having one Agent per bare metal host for traditional applications, or as a sidecar in container environments like Kubernetes, as this helps spread the load handled by Agents with the additional advantage of allowing each Agent to be tweaked individually, according to the application’s needs and importance.
|
||||
|
||||
### Consider using Apache Kafka as intermediate buffer
|
||||
|
||||
Jaeger [can use Apache Kafka](../architecture/) as a buffer between the Collector and the actual backing storage (Elasticsearch, Apache Cassandra). This is ideal for cases where the traffic spikes are relatively frequent (prime time traffic) but the storage can eventually catch up once the traffic normalizes. For that, the `SPAN_STORAGE_TYPE` environment variable should be set to `kafka` in the Collector and the Jaeger Ingester component can be used, reading data from Kafka and writing it to the storage.
|
||||
|
||||
In addition to the performance aspects, having spans written to Kafka is useful for building real time data pipeline for aggregations and feature extraction from traces.
|
||||
|
||||
## Client (Tracer) settings
|
||||
|
||||
The Jaeger Clients are built to have minimal effect to the instrumented application. As such, it has conservative defaults that might not be suitable for all cases. Note that Jaeger Clients can be configured programmatically or via [environment variables](../client-features/).
|
||||
|
||||
### Adjust the sampling configuration
|
||||
|
||||
Together, the `JAEGER_SAMPLER_TYPE` and `JAEGER_SAMPLER_PARAM` specify how often traces should be "sampled", ie, recorded and sent to the Jaeger backend. For applications generating a large number of spans, setting the sampling type to `probabilistic` and the value to `0.001` (the default) will cause traces to be reported with a 1/1000th chance. Note that the sampling decision is made at the root span and propagated down to all child spans.
|
||||
|
||||
For applications with low to medium traffic, setting the sampling type to `const` and value to `1` will cause all spans to be reported. Similarly, tracing can be disabled by setting the value to `0`, while context propagation will continue to work.
|
||||
|
||||
Some Clients support the setting `JAEGER_DISABLED` to completely disable the Jaeger Tracer. This is recommended only if the Tracer is behaving in a way that causes problems to the instrumented application, as it will not propagate the context to the downstream services.
|
||||
|
||||
{{< info >}}
|
||||
We recommend to set your clients to use the [`remote` sampling strategy](../sampling/#collector-sampling-configuration), so that admins can centrally set the concrete sampling strategy for each service.
|
||||
{{< /info >}}
|
||||
|
||||
### Increase in-memory queue size
|
||||
|
||||
Most of the Jaeger clients, such as the Java, Go, and C# clients, buffer spans in memory before sending them to the Jaeger Agent/Collector. The maximum size of this buffer is defined by the environment variable `JAEGER_REPORTER_MAX_QUEUE_SIZE` (default value: about `100` spans): the larger the size, the higher the potential memory consumption. When the instrumented application is generating a large number of spans, it’s possible that the queue will be full causing the Client to discard the new spans (metric `jaeger_tracer_reporter_spans_total{result="dropped",}`).
|
||||
|
||||
In most common scenarios, the queue will be close to empty (metric: `jaeger_tracer_reporter_queue_length`), as spans are flushed to the Agent or Collector at regular intervals or when a certain size of the batch is reached. The detailed behavior of this queue is described in this [GitHub issue](https://github.com/jaegertracing/jaeger-client-java/issues/607).
|
||||
|
||||
### Modify the batched spans flush interval
|
||||
|
||||
The Java, Go, NodeJS, Python and C# Clients allow the customization of the flush interval (default value: `1000` milliseconds, or 1 second) used by the reporters, such as the `RemoteReporter`, to trigger a `flush` operation, sending all in-memory spans to the Agent or Collector. The lower the flush interval is set to, the more frequent the flush operations happen. As most reporters will wait until enough data is in the queue, this setting will force a flush operation at periodic intervals, so that spans are sent to the backend in a timely fashion.
|
||||
|
||||
When the instrumented application is generating a large number of spans and the Agent/Collector is close to the application, the networking overhead might be low, justifying a higher number of flush operations. When the `HttpSender` is being used and the Collector is not close enough to the application, the networking overhead might be too high so that a higher value for this property makes sense.
|
||||
|
||||
## Agent settings
|
||||
|
||||
Jaeger Agents receive data from Clients, sending them in batches to the Collector. When not properly configured, it might end up discarding data even if the host machine has plenty of resources.
|
||||
|
||||
### Adjust server queue sizes
|
||||
|
||||
The set of “server queue size” properties ( `processor.jaeger-binary.server-queue-size`, `processor.jaeger-compact.server-queue-size`, `processor.zipkin-compact.server-queue-size`) indicate the maximum number of span batches that the Agent can accept and store in memory. It’s safe to assume that `jaeger-compact` is the most important processor in your Agent setup, as it’s the only one available in most Clients, such as the Java and Go Clients.
|
||||
|
||||
The default value for each queue is `1000` span batches. Given that each span batch has up to 64KiB worth of spans, each queue can hold up to 64MiB worth of spans.
|
||||
|
||||
In typical scenarios, the queue will be close to empty (metric `jaeger_agent_thrift_udp_server_queue_size`) as span batches should be quickly picked up and processed by a worker. However, sudden spikes in the number of span batches submitted by Clients might occur, causing the batches to be queued. When the queue is full, the older batches are overridden causing spans to be discarded (metric `jaeger_agent_thrift_udp_server_packets_dropped_total`).
|
||||
|
||||
### Adjust processor workers
|
||||
|
||||
The set of “processor workers” properties ( `processor.jaeger-binary.workers`, `processor.jaeger-compact.workers`, `processor.zipkin-compact.workers`) indicate the number of parallel span batch processors to start. Each worker type has a default size of `10`. In general, span batches are processed as soon as they are placed in the server queue and will block a worker until the whole packet is sent to the Collector. For Agents processing data from multiple Clients, the number of workers should be increased. Given that the cost of each worker is low, a good rule of thumb is 10 workers per Client with moderate traffic: given that each span batch might contain up to 64KiB worth of spans, it means that 10 workers are able to send about 640KiB concurrently to a Collector.
|
||||
|
||||
## Collector settings
|
||||
|
||||
The Collector receives data from Clients and Agents. When not properly configured, it might process less data than what would be possible on the same host, or it might overload the host by consuming more memory than permitted.
|
||||
|
||||
### Adjust queue size
|
||||
|
||||
Similar to the Agent, the Collector is able to receive spans and place them in an internal queue for processing. This allows the Collector to return immediately to the Client/Agent instead of waiting for the span to make its way to the storage.
|
||||
|
||||
The setting `collector.queue-size` (default: `2000`) dictates how many spans the queue should support. In the typical scenario, the queue will be close to empty, as enough workers should exist picking up spans from the queue and sending them to the storage. When the number of items in the queue (metric `jaeger_collector_queue_length`) is permanently high, it’s an indication that either the number of workers should be increased or that the storage cannot keep up with the volume of data that it’s receiving. When the queue is full, the older items in the queue are overridden, causing spans to be discarded (metric `jaeger_collector_spans_dropped_total`).
|
||||
|
||||
{{< warning >}}
|
||||
The queue size for the Agent is about _span batches_, whereas the queue size for the Collector is about _individual spans_.
|
||||
{{< /warning >}}
|
||||
|
||||
Given that the queue size should be close to empty most of the time, this setting should be as high as the available memory for the Collector, to provide maximum protection against sudden traffic spikes. However, if your storage layer is under-provisioned and cannot keep up, even a large queue will quickly fill up and start dropping data.
|
||||
|
||||
### Adjust processor workers
|
||||
|
||||
Items from the span queue in the Collector are picked up by workers. Each worker picks one span from the queue and persists it to the storage. The number of workers can be specified by the setting `collector.num-workers` (default: `50`) and should be as high as needed to keep the queue close to zero. The general rule is: the faster the backing storage, the lower the number of workers can be. Given that workers are relatively cheap, this number can be increased at will. As a general rule, one worker per 50 items in the queue should be sufficient when the storage is fast. With a `collector.queue-size` of `2000`, having about `40` workers should be sufficient. For slower storage mechanisms, this ratio should be adjusted accordingly, having more workers per queue item.
|
|
@ -1,66 +0,0 @@
|
|||
---
|
||||
title: Sampling
|
||||
hasparent: true
|
||||
---
|
||||
|
||||
Jaeger libraries implement consistent upfront (or head-based) sampling. For example, assume we have a simple call graph where service A calls service B, and B calls service C: `A -> B -> C`. When service A receives a request that contains no tracing information, Jaeger tracer will start a new {{< tip "trace" >}}, assign it a random trace ID, and make a sampling decision based on the currently installed sampling strategy. The sampling decision will be propagated with the requests to B and to C, so those services will not be making the sampling decision again but instead will respect the decision made by the top service A. This approach guarantees that if a trace is sampled, all its {{< tip "spans" "span" >}} will be recorded in the backend. If each service was making its own sampling decision we would rarely get complete traces in the backend.
|
||||
|
||||
## Client Sampling Configuration
|
||||
|
||||
When using configuration object to instantiate the tracer, the type of sampling can be selected via `sampler.type` and `sampler.param` properties. Jaeger libraries support the following samplers:
|
||||
|
||||
* **Constant** (`sampler.type=const`) sampler always makes the same decision for all traces. It either samples all traces (`sampler.param=1`) or none of them (`sampler.param=0`).
|
||||
* **Probabilistic** (`sampler.type=probabilistic`) sampler makes a random sampling decision with the probability of sampling equal to the value of `sampler.param` property. For example, with `sampler.param=0.1` approximately 1 in 10 traces will be sampled.
|
||||
* **Rate Limiting** (`sampler.type=ratelimiting`) sampler uses a leaky bucket rate limiter to ensure that traces are sampled with a certain constant rate. For example, when `sampler.param=2.0` it will sample requests with the rate of 2 traces per second.
|
||||
* **Remote** (`sampler.type=remote`, which is also the default) sampler consults Jaeger agent for the appropriate sampling strategy to use in the current service. This allows controlling the sampling strategies in the services from a [central configuration](#collector-sampling-configuration) in Jaeger backend, or even dynamically (see [Adaptive Sampling](https://github.com/jaegertracing/jaeger/issues/365)).
|
||||
|
||||
### Adaptive Sampler
|
||||
|
||||
Adaptive sampler is a composite sampler that combines two functions:
|
||||
|
||||
* It makes sampling decisions on a per-operation basis, i.e. based on {{< tip "span" >}} operation name. This is especially useful in the API services whose endpoints may have very different traffic volumes and using a single probabilistic sampler for the whole service might starve (never sample) some of the low QPS endpoints.
|
||||
* It supports a minimum guaranteed rate of sampling, such as always allowing up to N {{< tip "traces" "trace" >}} per seconds and then sampling anything above that with a certain probability (everything is per-operation, not per-service).
|
||||
|
||||
Per-operation parameters can be configured statically or pulled periodically from Jaeger backend with the help of Remote sampler. Adaptive sampler is designed to work with the upcoming [Adaptive Sampling](https://github.com/jaegertracing/jaeger/issues/365) feature of the Jaeger backend.
|
||||
|
||||
## Collector Sampling Configuration
|
||||
|
||||
Collectors can be instantiated with static sampling strategies (which are propagated to the respective service if configured with Remote sampler) via the `--sampling.strategies-file` option. This option requires a path to a json file which have the sampling strategies defined.
|
||||
|
||||
Example `strategies.json`
|
||||
```json
|
||||
{
|
||||
"service_strategies": [
|
||||
{
|
||||
"service": "foo",
|
||||
"type": "probabilistic",
|
||||
"param": 0.8,
|
||||
"operation_strategies": [
|
||||
{
|
||||
"operation": "op1",
|
||||
"type": "probabilistic",
|
||||
"param": 0.2
|
||||
},
|
||||
{
|
||||
"operation": "op2",
|
||||
"type": "probabilistic",
|
||||
"param": 0.4
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "bar",
|
||||
"type": "ratelimiting",
|
||||
"param": 5
|
||||
}
|
||||
],
|
||||
"default_strategy": {
|
||||
"type": "probabilistic",
|
||||
"param": 0.5
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`service_strategies` defines service specific sampling strategies and `operation_strategies` defines operation specific sampling strategies. There are 2 types of strategies possible: `probabilistic` and `ratelimiting` which are described [above](#client-sampling-configuration) (NOTE: `ratelimiting` is not supported for `operation_strategies`). `default_strategy` defines the catch-all sampling strategy that is propagated if the service is not included as part of `service_strategies`.
|
||||
|
||||
In the above example, all service `foo` operations are sampled probabilistically with a probability of 0.8 except `op1` and `op2` which are probabilistically sampled with a probability of 0.2 and 0.4 respectively. All operations for service `bar` are ratelimited at 5 traces per second. Any other service is probabilistically sampled with a probability of 0.5.
|
|
@ -1,43 +0,0 @@
|
|||
---
|
||||
title: Windows Service Deployment
|
||||
hasparent: true
|
||||
---
|
||||
|
||||
In Windows environments, Jaeger processes can be hosted and managed as Windows services controlled via the `sc` utility. To configure such services on Windows, download [nssm.exe](https://nssm.cc/download) for the appropriate architecture, and issue commands similar to how Jaeger is typically run. The example below showcases a basic Elasticsearch setup, configured using both environment variables and process arguments.
|
||||
|
||||
## Agent
|
||||
```bat
|
||||
nssm install JaegerAgent C:\Jaeger\jaeger-agent.exe --reporter.grpc.host-port=localhost:14250
|
||||
|
||||
nssm set JaegerAgent AppStdout C:\Jaeger\jaeger-agent.out.log
|
||||
nssm set JaegerAgent AppStderr C:\Jaeger\jaeger-agent.err.log
|
||||
nssm set JaegerAgent Description Jaeger Agent service
|
||||
|
||||
nssm start JaegerAgent
|
||||
```
|
||||
|
||||
## Collector
|
||||
```bat
|
||||
nssm install JaegerCollector C:\Jaeger\jaeger-collector.exe --es.server-urls=http://localhost:9200 --es.username=jaeger --es.password=PASSWORD
|
||||
|
||||
nssm set JaegerCollector AppStdout C:\Jaeger\jaeger-collector.out.log
|
||||
nssm set JaegerCollector AppStderr C:\Jaeger\jaeger-collector.err.log
|
||||
nssm set JaegerCollector Description Jaeger Collector service
|
||||
nssm set JaegerCollector AppEnvironmentExtra SPAN_STORAGE_TYPE=elasticsearch
|
||||
|
||||
nssm start JaegerCollector
|
||||
```
|
||||
|
||||
## Query UI
|
||||
```bat
|
||||
nssm install JaegerUI C:\Jaeger\jaeger-query.exe --es.server-urls=http://localhost:9200 --es.username=jaeger --es.password=PASSWORD
|
||||
|
||||
nssm set JaegerUI AppStdout C:\Jaeger\jaeger-ui.out.log
|
||||
nssm set JaegerUI AppStderr C:\Jaeger\jaeger-ui.err.log
|
||||
nssm set JaegerUI Description Jaeger Query service
|
||||
nssm set JaegerUI AppEnvironmentExtra SPAN_STORAGE_TYPE=elasticsearch
|
||||
|
||||
nssm start JaegerUI
|
||||
```
|
||||
|
||||
For additional information & docs, please see [the NSSM usage guide.](https://nssm.cc/usage)
|
|
@ -1,70 +0,0 @@
|
|||
---
|
||||
title: Introduction
|
||||
weight: 1
|
||||
children:
|
||||
- title: Features
|
||||
url: features
|
||||
---
|
||||
|
||||
Welcome to Jaeger's documentation portal! Below, you'll find information for beginners and experienced Jaeger users.
|
||||
|
||||
If you can't find what you are looking for, or have an issue not covered here, we'd love to [hear from you](/get-in-touch).
|
||||
|
||||
## About
|
||||
|
||||
Jaeger, inspired by [Dapper][dapper] and [OpenZipkin](http://zipkin.io),
|
||||
is a distributed tracing system released as open source by [Uber Technologies][ubeross].
|
||||
It is used for monitoring and troubleshooting microservices-based distributed systems, including:
|
||||
|
||||
* Distributed context propagation
|
||||
* Distributed transaction monitoring
|
||||
* Root cause analysis
|
||||
* Service dependency analysis
|
||||
* Performance / latency optimization
|
||||
|
||||
Uber published a blog post, [Evolving Distributed Tracing at Uber](https://eng.uber.com/distributed-tracing/), where they explain the history and reasons for the architectural choices made in Jaeger. [Yuri Shkuro](https://shkuro.com), creator of Jaeger, also published a book [Mastering Distributed Tracing](https://shkuro.com/books/2019-mastering-distributed-tracing/) that covers in-depth many aspects of Jaeger design and operation, as well as distributed tracing in general.
|
||||
|
||||
## Features
|
||||
|
||||
* [OpenTracing](http://opentracing.io/) compatible data model and instrumentation libraries
|
||||
* in [Go](https://github.com/jaegertracing/jaeger-client-go), [Java](https://github.com/jaegertracing/jaeger-client-java), [Node](https://github.com/jaegertracing/jaeger-client-node), [Python](https://github.com/jaegertracing/jaeger-client-python),
|
||||
[C++](https://github.com/jaegertracing/cpp-client) and [C#](https://github.com/jaegertracing/jaeger-client-csharp)
|
||||
* Uses consistent upfront sampling with individual per service/endpoint probabilities
|
||||
* Multiple storage backends: Cassandra, Elasticsearch, memory.
|
||||
* Adaptive sampling (coming soon)
|
||||
* Post-collection data processing pipeline (coming soon)
|
||||
|
||||
See [Features](./features/) page for more details.
|
||||
|
||||
## Technical Specs
|
||||
|
||||
* Backend components implemented in Go
|
||||
* React/Javascript UI
|
||||
* Supported storage backends:
|
||||
* [Cassandra 3.4+](./deployment/#cassandra)
|
||||
* [Elasticsearch 5.x, 6.x, 7.x](./deployment/#elasticsearch)
|
||||
* [Kafka](./deployment/#kafka)
|
||||
* memory storage
|
||||
|
||||
## Quick Start
|
||||
See [running a docker all in one image](getting-started#all-in-one).
|
||||
|
||||
## Screenshots
|
||||
|
||||
### Traces View
|
||||
[](/img/traces-ss.png)
|
||||
|
||||
### Trace Detail View
|
||||
[](/img/trace-detail-ss.png)
|
||||
|
||||
## Related links
|
||||
- [Evolving Distributed tracing At Uber Engineering](https://eng.uber.com/distributed-tracing/)
|
||||
- [Mastering Distributed Tracing](https://shkuro.com/books/2019-mastering-distributed-tracing/)
|
||||
- [Tracing HTTP request latency in Go with OpenTracing](https://medium.com/opentracing/tracing-http-request-latency-in-go-with-opentracing-7cc1282a100a)
|
||||
- [Distributed Tracing with Jaeger & Prometheus on Kubernetes](https://blog.openshift.com/openshift-commons-briefing-82-distributed-tracing-with-jaeger-prometheus-on-kubernetes/)
|
||||
- [Using Jaeger with Istio](https://istio.io/docs/tasks/telemetry/distributed-tracing.html)
|
||||
- [Using Jaeger with Envoy](https://www.envoyproxy.io/docs/envoy/latest/start/sandboxes/jaeger_tracing.html)
|
||||
|
||||
[dapper]: https://research.google.com/pubs/pub36356.html
|
||||
[ubeross]: http://uber.github.io
|
||||
|
|
@ -1,85 +0,0 @@
|
|||
---
|
||||
title: APIs
|
||||
hasparent: true
|
||||
---
|
||||
|
||||
Jaeger components implement various APIs for saving or retrieving trace data.
|
||||
|
||||
The following labels are used to describe API compatibility guarantees.
|
||||
|
||||
* **stable** - the API guarantees backwards compatibility. If breaking changes are going to be made in the future, they will result in a new API version, e.g. `/api/v2` URL prefix or a different namespace in the IDL.
|
||||
* **internal** - the APIs intended for internal communications between Jaeger components and not recommended for use by external components.
|
||||
* **deprecated** - the APIs that are only maintained for legacy reasons and will be phased out in the future.
|
||||
|
||||
## Span reporting APIs
|
||||
|
||||
Agent and Collector are the two components of the Jaeger backend that can receive spans. At this time they support two sets of non-overlapping APIs.
|
||||
|
||||
### Thrift over UDP (stable)
|
||||
|
||||
The Agent can only receive spans over UDP in Thrift format. The primary API is a UDP packet that contains a Thrift-encoded `Batch` struct defined in [jaeger.thrift][jaeger.thrift] IDL file, located in the [jaeger-idl][jaeger-idl] repository. Most Jaeger Clients use Thrift's `compact` encoding, however some client libraries do not support it (notably, Node.js) and use Thrift's `binary` encoding (sent to a different UDP port). The Agent's API is defined by [agent.thrift][agent.thrift] IDL file.
|
||||
|
||||
For legacy reasons, the Agent also accepts spans in Zipkin format, however, only very old versions of Jaeger clients can send data in that format and it is officially deprecated.
|
||||
|
||||
### Protobuf via gRPC (stable)
|
||||
|
||||
In a typical Jaeger deployment, Agents receive spans from Clients and forward them to Collectors. Since Jaeger version 1.11 the official and recommended protocol between Agents and Collectors is gRPC with Protobuf as defined in [collector.proto][collector.proto] IDL file.
|
||||
|
||||
### Thrift over HTTP (stable)
|
||||
|
||||
In some cases it is not feasible to deploy Jaeger Agent next to the application, for example, when the application code is running as AWS Lambda function. In these scenarios the Jaeger Clients can be configured to submit spans directly to the Collectors over HTTP/HTTPS.
|
||||
|
||||
The same [jaeger.thrift][jaeger.thrift] payload can be submitted in HTTP POST request to `/api/traces` endpoint, for example, `https://jaeger-collector:14268/api/traces`. The `Batch` struct needs to be encoded using Thrift's `binary` encoding, and the HTTP request should specify the content type header:
|
||||
|
||||
```
|
||||
Content-Type: application/vnd.apache.thrift.binary
|
||||
```
|
||||
|
||||
### JSON over HTTP (n/a)
|
||||
|
||||
There is no official Jaeger JSON format that can be accepted by the collector. In the future the Protobuf-generated JSON may be supported.
|
||||
|
||||
### Thrift via TChannel (deprecated)
|
||||
|
||||
Agent and Collector can communicate using TChannel protocol. This protocol is generally not supported by the routing infrastructure and has been deprecated. It will be eventually removed from Jaeger.
|
||||
|
||||
### Zipkin Formats (stable)
|
||||
|
||||
Jaeger Collector can also accept spans in several Zipkin data format, namely JSON v1/v2 and Thrift. The Collector needs to be configured to enable Zipkin HTTP server, e.g. on port 9411 used by Zipkin collectors. The server enables two endpoints that expect POST requests:
|
||||
|
||||
* `/api/v1/spans` for submitting spans in Zipkin JSON v1 or Zipkin Thrift format.
|
||||
* `/api/v2/spans` for submitting spans in Zipkin JSON v2.
|
||||
|
||||
## Trace retrieval APIs
|
||||
|
||||
Traces saved in the storage can be retrieved by calling Jaeger Query Service.
|
||||
|
||||
### gRPC/Protobuf (stable)
|
||||
|
||||
The recommended way for programmatically retrieving traces and other data is via gRPC endpoint defined in [query.proto][query.proto] IDL file.
|
||||
|
||||
### HTTP JSON (internal)
|
||||
|
||||
Jaeger UI communicates with Jaeger Query Service via JSON API. For example, a trace can be retrieved via GET request to `https://jaeger-query:16686/api/traces/{trace-id-hex-string}`. This JSON API is intentionally undocumented and subject to change.
|
||||
|
||||
## Clients configuration (internal)
|
||||
|
||||
Client libraries not only submit finished spans to Jaeger backend, but also periodically poll the Agents for various configurations, such as sampling strategies. The schema for the payload is defined by [sampling.thrift][sampling.thrift], encoded as JSON using Thrift's built-in JSON generation capabilities.
|
||||
|
||||
## Service dependencies graph (internal)
|
||||
|
||||
Can be retrieved from Query Service at `/api/dependencies` endpoint. The GET request expects two parameters:
|
||||
|
||||
* `endTs` (number of milliseconds since epoch) - the end of the time interval
|
||||
* `lookback` (in milliseconds) - the length the time interval (i.e. start-time + lookback = end-time).
|
||||
|
||||
The returned JSON is a list of edges represented as tuples `(caller, callee, count)`.
|
||||
|
||||
For programmatic access to service graph, the recommended API is gRPC/Protobuf described above.
|
||||
|
||||
[jaeger-idl]: https://github.com/jaegertracing/jaeger-idl/
|
||||
[jaeger.thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/jaeger.thrift
|
||||
[agent.thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/agent.thrift
|
||||
[sampling.thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/sampling.thrift
|
||||
[collector.proto]: https://github.com/jaegertracing/jaeger-idl/blob/master/proto/api_v2/collector.proto
|
||||
[query.proto]: https://github.com/jaegertracing/jaeger-idl/blob/master/proto/api_v2/query.proto
|
|
@ -1,69 +0,0 @@
|
|||
---
|
||||
title: Architecture
|
||||
weight: 3
|
||||
children:
|
||||
- title: APIs
|
||||
url: apis
|
||||
- title: Sampling
|
||||
url: sampling
|
||||
---
|
||||
|
||||
Jaeger's clients adhere to the data model described in the OpenTracing standard. Reading the [specification](https://github.com/opentracing/specification/blob/master/specification.md) will help you understand this section better.
|
||||
|
||||
## Terminology
|
||||
|
||||
Let's start with a quick refresher on the terminology defined by the [OpenTracing Specification](https://github.com/opentracing/specification/blob/master/specification.md).
|
||||
|
||||
### Span
|
||||
|
||||
{{< definition "span" >}}
|
||||
|
||||

|
||||
|
||||
### Trace
|
||||
|
||||
{{< definition "trace" >}}
|
||||
|
||||
## Components
|
||||
|
||||
Jaeger can be deployed either as all-in-one binary, where all Jaeger backend components
|
||||
run in a single process, or as a scalable distributed system, discussed below.
|
||||
There are two main deployment options:
|
||||
|
||||
1. Collectors are writing directly to storage.
|
||||
2. Collectors are writing to Kafka as a preliminary buffer.
|
||||
|
||||

|
||||
*Illustration of direct-to-storage architecture*
|
||||
|
||||

|
||||
*Illustration of architecture with Kafka as intermediate buffer*
|
||||
|
||||
This section details the constituent parts of Jaeger and how they relate to each other. It is arranged by the order in which spans from your application interact with them.
|
||||
|
||||
### Jaeger client libraries
|
||||
|
||||
Jaeger clients are language specific implementations of the [OpenTracing API](https://opentracing.io). They can be used to instrument applications for distributed tracing either manually or with a variety of existing open source frameworks, such as Flask, Dropwizard, gRPC, and many more, that are already integrated with OpenTracing.
|
||||
|
||||
An instrumented service creates {{< tip "spans" "span" >}} when receiving new requests and attaches context information ({{< tip "trace" >}} id, {{< tip "span" "span" >}} id, and {{< tip "baggage" "baggage" >}}) to outgoing requests. Only the ids and baggage are propagated with requests; all other profiling data, like operation name, timing, tags and logs, is not propagated. Instead, it is transmitted out of process to the Jaeger backend asynchronously, in the background.
|
||||
|
||||
The instrumentation is designed to be always on in production. To minimize the overhead, Jaeger clients employ various sampling strategies. When a trace is sampled, the profiling span data is captured and transmitted to the Jaeger backend. When a trace is not sampled, no profiling data is collected at all, and the calls to the OpenTracing APIs are short-circuited to incur the minimal amount of overhead. By default, Jaeger clients sample 0.1% of traces (1 in 1000), and have the ability to retrieve sampling strategies from the Jaeger backend. For more information, please refer to [Sampling](../sampling/).
|
||||
|
||||

|
||||
*Illustration of context propagation*
|
||||
|
||||
### Agent
|
||||
|
||||
{{< definition "agent" >}}
|
||||
|
||||
### Collector
|
||||
|
||||
{{< definition "collector" >}}
|
||||
|
||||
### Query
|
||||
|
||||
{{< definition "query" >}}
|
||||
|
||||
### Ingester
|
||||
|
||||
{{< definition "ingester" >}}
|
|
@ -1,13 +0,0 @@
|
|||
---
|
||||
title: CLI flags
|
||||
widescreen: true
|
||||
hasparent: true
|
||||
---
|
||||
|
||||
This is auto-generated documentation for CLI flags supported by Jaeger binaries.
|
||||
|
||||
* CLI flags for some binaries change depending on the `SPAN_STORAGE_TYPE` environment variable. Relevant variations are included below.
|
||||
* Some binaries support _commands_ (mostly informational), such as `env`, `docs`, and `version`. These commands are not included here.
|
||||
* All parameters can be also provided via environment variables, by changing all letters to upper-case and replacing all punctuation characters with underscore `_`. For example, the value for the flag `--cassandra.connections-per-host` can be provided via `CASSANDRA_CONNECTIONS_PER_HOST` environment variable.
|
||||
|
||||
{{< cli/tools-list >}}
|
|
@ -1,527 +0,0 @@
|
|||
---
|
||||
title: Deployment
|
||||
weight: 4
|
||||
children:
|
||||
- title: Operator for Kubernetes
|
||||
navtitle: Kubernetes
|
||||
url: operator
|
||||
- title: Frontend/UI
|
||||
url: frontend-ui
|
||||
- title: Windows
|
||||
url: windows
|
||||
- title: CLI Flags
|
||||
url: cli
|
||||
---
|
||||
|
||||
The main Jaeger backend components are released as Docker images on Docker Hub:
|
||||
|
||||
Component | Repository
|
||||
--------------------- | ---
|
||||
**jaeger-agent** | [hub.docker.com/r/jaegertracing/jaeger-agent/](https://hub.docker.com/r/jaegertracing/jaeger-agent/)
|
||||
**jaeger-collector** | [hub.docker.com/r/jaegertracing/jaeger-collector/](https://hub.docker.com/r/jaegertracing/jaeger-collector/)
|
||||
**jaeger-query** | [hub.docker.com/r/jaegertracing/jaeger-query/](https://hub.docker.com/r/jaegertracing/jaeger-query/)
|
||||
**jaeger-ingester** | [hub.docker.com/r/jaegertracing/jaeger-ingester/](https://hub.docker.com/r/jaegertracing/jaeger-ingester/)
|
||||
|
||||
There are orchestration templates for running Jaeger with:
|
||||
|
||||
* Kubernetes: [github.com/jaegertracing/jaeger-kubernetes](https://github.com/jaegertracing/jaeger-kubernetes),
|
||||
* OpenShift: [github.com/jaegertracing/jaeger-openshift](https://github.com/jaegertracing/jaeger-openshift).
|
||||
|
||||
## Configuration Options
|
||||
|
||||
Jaeger binaries can be configured in a number of ways (in the order of decreasing priority):
|
||||
|
||||
* command line arguments,
|
||||
* environment variables,
|
||||
* configuration files in JSON, TOML, YAML, HCL, or Java properties formats.
|
||||
|
||||
To see the complete list of options, run the binary with `help` command. Options that are specific to a certain storage backend are only listed if the storage type is selected. For example, to see all available options in the Collector with Cassandra storage:
|
||||
|
||||
```sh
|
||||
$ docker run --rm \
|
||||
-e SPAN_STORAGE_TYPE=cassandra \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}} \
|
||||
help
|
||||
```
|
||||
|
||||
In order to provide configuration parameters via environment variables, find the respective command line option and convert its name to UPPER_SNAKE_CASE, for example:
|
||||
|
||||
Command line option | Environment variable
|
||||
-----------------------------------|-------------------------------
|
||||
`--cassandra.connections-per-host` | `CASSANDRA_CONNECTIONS_PER_HOST`
|
||||
`--metrics-backend` | `METRICS_BACKEND`
|
||||
|
||||
## Agent
|
||||
|
||||
Jaeger client libraries expect **jaeger-agent** process to run locally on each host.
|
||||
The agent exposes the following ports:
|
||||
|
||||
Port | Protocol | Function
|
||||
----- | ------- | ---
|
||||
6831 | UDP | accept [jaeger.thrift][jaeger-thrift] in `compact` Thrift protocol used by most current Jaeger clients
|
||||
6832 | UDP | accept [jaeger.thrift][jaeger-thrift] in `binary` Thrift protocol used by Node.js Jaeger client (because [thriftrw][thriftrw] npm package does not support `compact` protocol)
|
||||
5778 | HTTP | serve configs, sampling strategies
|
||||
5775 | UDP | accept [zipkin.thrift][zipkin-thrift] in `compact` Thrift protocol (deprecated; only used by very old Jaeger clients, circa 2016)
|
||||
14271 | HTTP | admin port: health check at `/` and metrics at `/metrics`
|
||||
|
||||
It can be executed directly on the host or via Docker, as follows:
|
||||
|
||||
```sh
|
||||
## make sure to expose only the ports you use in your deployment scenario!
|
||||
docker run \
|
||||
--rm \
|
||||
-p6831:6831/udp \
|
||||
-p6832:6832/udp \
|
||||
-p5778:5778/tcp \
|
||||
-p5775:5775/udp \
|
||||
jaegertracing/jaeger-agent:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
### Discovery System Integration
|
||||
|
||||
The agents can connect point to point to a single collector address, which could be
|
||||
load balanced by another infrastructure component (e.g. DNS) across multiple collectors.
|
||||
The agent can also be configured with a static list of collector addresses.
|
||||
|
||||
On Docker, a command like the following can be used:
|
||||
|
||||
```sh
|
||||
docker run \
|
||||
--rm \
|
||||
-p5775:5775/udp \
|
||||
-p6831:6831/udp \
|
||||
-p6832:6832/udp \
|
||||
-p5778:5778/tcp \
|
||||
jaegertracing/jaeger-agent:{{< currentVersion >}} \
|
||||
--reporter.grpc.host-port=jaeger-collector.jaeger-infra.svc:14250
|
||||
```
|
||||
|
||||
Or use `--reporter.tchannel.host-port=jaeger-collector.jaeger-infra.svc:14267` to use
|
||||
legacy tchannel reporter.
|
||||
|
||||
When using gRPC, you have several options for load balancing and name resolution:
|
||||
|
||||
* Single connection and no load balancing. This is the default if you specify a single `host:port`. (example: `--reporter.grpc.host-port=jaeger-collector.jaeger-infra.svc:14250`)
|
||||
* Static list of hostnames and round-robin load balancing. This is what you get with a comma-separated list of addresses. (example: `reporter.grpc.host-port=jaeger-collector1:14250,jaeger-collector2:14250,jaeger-collector3:14250`)
|
||||
* Dynamic DNS resolution and round-robin load balancing. To get this behaviour, prefix the address with `dns:///` and gRPC will attempt to resolve the hostname using SRV records (for [external load balancing](https://github.com/grpc/grpc/blob/master/doc/load-balancing.md)), TXT records (for [service configs](https://github.com/grpc/grpc/blob/master/doc/service_config.md)), and A records. Refer to the [gRPC Name Resolution docs](https://github.com/grpc/grpc/blob/master/doc/naming.md) and the [dns_resolver.go implementation](https://github.com/grpc/grpc-go/blob/master/resolver/dns/dns_resolver.go) for more info. (example: `--reporter.grpc.host-port=dns:///jaeger-collector.jaeger-infra.svc:14250`)
|
||||
|
||||
### Agent level tags
|
||||
|
||||
Jaeger supports agent level tags, that can be added to the process tags of all spans passing through the agent. This is supported through the command line flag `--agent.tags=key1=value1,key2=value2,...,keyn=valuen`. Tags can also be set through an environment flag like so - `--agent.tags=key=${envFlag:defaultValue}` - The tag value will be set to the value of the `envFlag` environment key and `defaultValue` if not set. This feature is not supported for the tchannel reporter, enabled using the flags `--collector.host-port` or `--reporter.tchannel.host-port`.
|
||||
|
||||
## Collectors
|
||||
|
||||
The collectors are stateless and thus many instances of **jaeger-collector** can be run in parallel.
|
||||
Collectors require almost no configuration, except for the location of Cassandra cluster,
|
||||
via `--cassandra.keyspace` and `--cassandra.servers` options, or the location of Elasticsearch cluster, via
|
||||
`--es.server-urls`, depending on which storage is specified. To see all command line options run
|
||||
|
||||
```sh
|
||||
go run ./cmd/collector/main.go -h
|
||||
```
|
||||
|
||||
or, if you don't have the source code
|
||||
|
||||
```sh
|
||||
docker run -it --rm jaegertracing/jaeger-collector:{{< currentVersion >}} -h
|
||||
```
|
||||
|
||||
At default settings the collector exposes the following ports:
|
||||
|
||||
Port | Protocol | Function
|
||||
----- | ------- | ---
|
||||
14267 | TChannel | used by **jaeger-agent** to send spans in jaeger.thrift format
|
||||
14250 | gRPC | used by **jaeger-agent** to send spans in model.proto format
|
||||
14268 | HTTP | can accept spans directly from clients in jaeger.thrift format over binary thrift protocol
|
||||
9411 | HTTP | can accept Zipkin spans in Thrift, JSON and Proto (disabled by default)
|
||||
14269 | HTTP | admin port: health check at `/` and metrics at `/metrics`
|
||||
|
||||
|
||||
## Storage Backends
|
||||
|
||||
Collectors require a persistent storage backend. Cassandra and Elasticsearch are the primary supported storage backends. Additional backends are [discussed here](https://github.com/jaegertracing/jaeger/issues/638).
|
||||
|
||||
The storage type can be passed via `SPAN_STORAGE_TYPE` environment variable. Valid values are `cassandra`, `elasticsearch`, `kafka` (only as a buffer), `grpc-plugin`, `badger` (only with all-in-one) and `memory` (only with all-in-one).
|
||||
|
||||
As of version 1.6.0, it's possible to use multiple storage types at the same time by providing a comma-separated list of valid types to the `SPAN_STORAGE_TYPE` environment variable.
|
||||
It's important to note that all listed storage types are used for writing, but only the first type in the list will be used for reading and archiving.
|
||||
|
||||
### Memory
|
||||
|
||||
The in-memory storage is not intended for production workloads. It's intended as a simple solution to get started quickly and
|
||||
data will be lost once the process is gone.
|
||||
|
||||
By default, there's no limit in the amount of traces stored in memory but a limit can be established by passing an
|
||||
integer value via `--memory.max-traces`.
|
||||
|
||||
### Badger - local storage
|
||||
Experimental since Jaeger 1.9
|
||||
|
||||
[Badger](https://github.com/dgraph-io/badger) is an embedded local storage, only available
|
||||
with **all-in-one** distribution. By default it acts as an ephemeral storage using a temporary filesystem.
|
||||
This can be overridden by using the `--badger.ephemeral=false` option.
|
||||
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=badger \
|
||||
-e BADGER_EPHEMERAL=false \
|
||||
-e BADGER_DIRECTORY_VALUE=/badger/data \
|
||||
-e BADGER_DIRECTORY_KEY=/badger/key \
|
||||
-v <storage_dir_on_host>:/badger \
|
||||
-p 16686:16686 \
|
||||
jaegertracing/all-in-one:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
### Cassandra
|
||||
Supported versions: 3.4+
|
||||
|
||||
Deploying Cassandra itself is out of scope for our documentation. One good
|
||||
source of documentation is the [Apache Cassandra Docs](https://cassandra.apache.org/doc/latest/).
|
||||
|
||||
#### Configuration
|
||||
##### Minimal
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=cassandra \
|
||||
-e CASSANDRA_SERVERS=<...> \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
##### All options
|
||||
To view the full list of configuration options, you can run the following command:
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=cassandra \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}} \
|
||||
--help
|
||||
```
|
||||
|
||||
#### Schema script
|
||||
|
||||
A script is provided to initialize Cassandra keyspace and schema
|
||||
using Cassandra's interactive shell [`cqlsh`][cqlsh]:
|
||||
|
||||
```sh
|
||||
MODE=test sh ./plugin/storage/cassandra/schema/create.sh | cqlsh
|
||||
```
|
||||
|
||||
For production deployment, pass `MODE=prod DATACENTER={datacenter}` arguments to the script,
|
||||
where `{datacenter}` is the name used in the Cassandra configuration / network topology.
|
||||
|
||||
The script also allows overriding TTL, keyspace name, replication factor, etc.
|
||||
Run the script without arguments to see the full list of recognized parameters.
|
||||
|
||||
#### TLS support
|
||||
|
||||
Jaeger supports TLS client to node connections as long as you've configured
|
||||
your Cassandra cluster correctly. After verifying with e.g. `cqlsh`, you can
|
||||
configure the collector and query like so:
|
||||
|
||||
```sh
|
||||
docker run \
|
||||
-e CASSANDRA_SERVERS=<...> \
|
||||
-e CASSANDRA_TLS=true \
|
||||
-e CASSANDRA_TLS_SERVER_NAME="CN-in-certificate" \
|
||||
-e CASSANDRA_TLS_KEY=<path to client key file> \
|
||||
-e CASSANDRA_TLS_CERT=<path to client cert file> \
|
||||
-e CASSANDRA_TLS_CA=<path to your CA cert file> \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
The schema tool also supports TLS. You need to make a custom cqlshrc file like
|
||||
so:
|
||||
|
||||
```
|
||||
# Creating schema in a cassandra cluster requiring client TLS certificates.
|
||||
#
|
||||
# Create a volume for the schema docker container containing four files:
|
||||
# cqlshrc: this file
|
||||
# ca-cert: the cert authority for your keys
|
||||
# client-key: the keyfile for your client
|
||||
# client-cert: the cert file matching client-key
|
||||
#
|
||||
# if there is any sort of DNS mismatch and you want to ignore server validation
|
||||
# issues, then uncomment validate = false below.
|
||||
#
|
||||
# When running the container, map this volume to /root/.cassandra and set the
|
||||
# environment variable CQLSH_SSL=--ssl
|
||||
[ssl]
|
||||
certfile = ~/.cassandra/ca-cert
|
||||
userkey = ~/.cassandra/client-key
|
||||
usercert = ~/.cassandra/client-cert
|
||||
# validate = false
|
||||
```
|
||||
|
||||
### Elasticsearch
|
||||
Supported in Jaeger since 0.6.0
|
||||
Supported versions: 5.x, 6.x, 7.x
|
||||
|
||||
Elasticsearch version is automatically retrieved from root/ping endpoint.
|
||||
Based on this version Jaeger uses compatible index mappings and Elasticsearch REST API.
|
||||
The version can be explicitly provided via `--es.version=` flag.
|
||||
|
||||
Elasticsearch does not require initialization other than
|
||||
[installing and running Elasticsearch](https://www.elastic.co/downloads/elasticsearch).
|
||||
Once it is running, pass the correct configuration values to the Jaeger collector and query service.
|
||||
|
||||
#### Configuration
|
||||
##### Minimal
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=elasticsearch \
|
||||
-e ES_SERVER_URLS=<...> \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
##### All options
|
||||
To view the full list of configuration options, you can run the following command:
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=elasticsearch \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}} \
|
||||
--help
|
||||
```
|
||||
|
||||
#### Shards and Replicas for Elasticsearch indices
|
||||
|
||||
Shards and replicas are some configuration values to take special attention to, because this is decided upon
|
||||
index creation. [This article](https://qbox.io/blog/optimizing-elasticsearch-how-many-shards-per-index) goes into
|
||||
more information about choosing how many shards should be chosen for optimization.
|
||||
|
||||
#### Elasticsearch Rollover
|
||||
|
||||
[Elasticsearch rollover](https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-rollover-index.html) is an index management strategy that optimizes use of resources allocated to indices.
|
||||
For example, indices that do not contain any data still allocate shards, and conversely, a single index might contain significantly more data than the others.
|
||||
Jaeger by default stores data in daily indices which might not optimally utilize resources. Rollover feature can be enabled by `--es.use-aliases=true`.
|
||||
|
||||
Rollover lets you configure when to roll over to a new index based on one or more of the following criteria:
|
||||
|
||||
* `max_age` - the maximum age of the index. It uses [time units](https://www.elastic.co/guide/en/elasticsearch/reference/master/common-options.html#time-units): `d`, `h`, `m`.
|
||||
* `max_docs` - the maximum documents in the index.
|
||||
* `max_size` - the maximum estimated size of primary shards (since Elasticsearch 6.x). It uses [byte size units](https://www.elastic.co/guide/en/elasticsearch/reference/master/common-options.html#byte-units) `tb`, `gb`, `mb`.
|
||||
|
||||
Rollover index management strategy is more complex than using the default daily indices and it requires an initialisation job to prepare the storage and two cron jobs to manage indices.
|
||||
|
||||
To learn more about rollover index management in Jaeger refer to this
|
||||
[article](https://medium.com/jaegertracing/using-elasticsearch-rollover-to-manage-indices-8b3d0c77915d).
|
||||
|
||||
##### Initialize
|
||||
|
||||
The following command prepares Elasticsearch for rollover deployment by creating index aliases, indices, and index templates:
|
||||
|
||||
```sh
|
||||
docker run -it --rm --net=host jaegertracing/jaeger-es-rollover:latest init http://localhost:9200 # <1>
|
||||
```
|
||||
|
||||
If you need to initialize archive storage, add `-e ARCHIVE=true`.
|
||||
|
||||
After the initialization Jaeger can be deployed with `--es.use-aliases=true`.
|
||||
|
||||
##### Rolling over to a new index
|
||||
|
||||
The next step is to periodically execute the rollover API which rolls the write alias to a new index based on supplied conditions. The command also adds a new index to the read alias to make new data available for search.
|
||||
|
||||
```shell
|
||||
docker run -it --rm --net=host -e CONDITIONS='{"max_age": "2d"}' jaegertracing/jaeger-es-rollover:latest rollover http://localhost:9200 # <1>
|
||||
```
|
||||
|
||||
<1> The command rolls the alias over to a new index if the age of the current write index is older than 2 days. For more conditions see [Elasticsearch docs](https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-rollover-index.html).
|
||||
|
||||
The next step is to remove old indices from read aliases. It means that old data will not be available for search. This imitates the behavior of `--es.max-span-age` flag used in the default index-per-day deployment. This step could be optional and old indices could be simply removed by index cleaner in the next step.
|
||||
|
||||
```sh
|
||||
docker run -it --rm --net=host -e UNIT=days -e UNIT_COUNT=7 jaegertracing/jaeger-es-rollover:latest lookback http://localhost:9200 # <1>
|
||||
```
|
||||
|
||||
<1> Removes indices older than 7 days from read alias.
|
||||
|
||||
##### Remove old data
|
||||
|
||||
The historical data can be removed with the `jaeger-es-index-cleaner` that is also used for daily indices.
|
||||
|
||||
```shell
|
||||
docker run -it --rm --net=host -e ROLLOVER=true jaegertracing/jaeger-es-index-cleaner:latest 14 http://localhost:9200 # <1>
|
||||
```
|
||||
|
||||
<1> Remove indices older than 14 days.
|
||||
|
||||
#### Upgrade Elasticsearch version
|
||||
|
||||
Elasticsearch defines wire and index compatibility versions. The index compatibility defines
|
||||
the minimal version a node can read data from. For example Elasticsearch 7 can read indices
|
||||
created by Elasticsearch 6, however it cannot read indices created by Elasticsearch 5 even
|
||||
though they use the same index mappings. Therefore upgrade from Elasticsearch 6 to 7 does not require any
|
||||
data migration. However, upgrade from Elasticsearch 5 to 7 has to be done through Elasticsearch 6 and wait
|
||||
until indices created by ES 5.x are removed or explicitly reindexed.
|
||||
|
||||
Refer to the Elasticsearch [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current//setup-upgrade.html)
|
||||
for wire and index compatibility versions. Generally this information can be retrieved from root/ping REST endpoint.
|
||||
|
||||
##### Reindex
|
||||
|
||||
Manual reindexing can be used when upgrading from Elasticsearch 5 to 7 (through Elasticsearch 6)
|
||||
without waiting until indices created by Elasticsearch 5 are removed.
|
||||
|
||||
1. Reindex all span indices to new indices with suffix `-1`:
|
||||
|
||||
```bash
|
||||
curl -ivX POST -H "Content-Type: application/json" http://localhost:9200/_reindex -d @reindex.json
|
||||
{
|
||||
"source": {
|
||||
"index": "jaeger-span-*"
|
||||
},
|
||||
"dest": {
|
||||
"index": "jaeger-span"
|
||||
},
|
||||
"script": {
|
||||
"lang": "painless",
|
||||
"source": "ctx._index = 'jaeger-span-' + (ctx._index.substring('jaeger-span-'.length(), ctx._index.length())) + '-1'"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
2. Delete indices with old mapping:
|
||||
|
||||
```bash
|
||||
curl -ivX DELETE -H "Content-Type: application/json" http://localhost:9200/jaeger-span-\*,-\*-1
|
||||
```
|
||||
|
||||
3. Create indices without `-1` suffix:
|
||||
|
||||
```bash
|
||||
curl -ivX POST -H "Content-Type: application/json" http://localhost:9200/_reindex -d @reindex.json
|
||||
{
|
||||
"source": {
|
||||
"index": "jaeger-span-*"
|
||||
},
|
||||
"dest": {
|
||||
"index": "jaeger-span"
|
||||
},
|
||||
"script": {
|
||||
"lang": "painless",
|
||||
"source": "ctx._index = 'jaeger-span-' + (ctx._index.substring('jaeger-span-'.length(), ctx._index.length() - 2))"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
4. Remove suffixed indices:
|
||||
|
||||
```bash
|
||||
curl -ivX DELETE -H "Content-Type: application/json" http://localhost:9200/jaeger-span-\*-1
|
||||
```
|
||||
|
||||
Run the commands analogically for other Jaeger indices.
|
||||
|
||||
There might exist more effective migration procedure. Please share with the community any findings.
|
||||
|
||||
### Kafka
|
||||
Supported in Jaeger since 1.6.0
|
||||
Supported Kafka versions: 0.9+
|
||||
|
||||
Kafka can be used as an intermediary buffer between collector and an actual storage.
|
||||
The collector is configured with `SPAN_STORAGE_TYPE=kafka` that makes it write all received spans
|
||||
into a Kafka topic. A new component [Ingester](#ingester), added in version 1.7.0, is used to read from
|
||||
Kafka and store spans in another storage backend (Elasticsearch or Cassandra).
|
||||
|
||||
Writing to Kafka is particularly useful for building post-processing data pipelines.
|
||||
|
||||
#### Configuration
|
||||
##### Minimal
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=kafka \
|
||||
-e KAFKA_PRODUCER_BROKERS=<...> \
|
||||
-e KAFKA_TOPIC=<...> \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
##### All options
|
||||
To view the full list of configuration options, you can run the following command:
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=kafka \
|
||||
jaegertracing/jaeger-collector:{{< currentVersion >}} \
|
||||
--help
|
||||
```
|
||||
|
||||
#### Topic & partitions
|
||||
Unless your Kafka cluster is configured to automatically create topics, you will need to create it ahead of time. You can refer to [the Kafka quickstart documentation](https://kafka.apache.org/documentation/#quickstart_createtopic) to learn how.
|
||||
|
||||
You can find more information about topics and partitions in general in the [official documentation](https://kafka.apache.org/documentation/#intro_topics). [This article](https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/) provide more details about how to choose the number of partitions.
|
||||
|
||||
### Storage plugin
|
||||
|
||||
Jaeger supports gRPC based storage plugins. For more information refer to [jaeger/plugin/storage/grpc](https://github.com/jaegertracing/jaeger/tree/master/plugin/storage/grpc)
|
||||
|
||||
Available plugins:
|
||||
|
||||
* [InfluxDB](https://github.com/influxdata/jaeger-influxdb/)
|
||||
* [Logz.io](https://github.com/logzio/jaeger-logzio) - secure, scalable, managed, cloud-based ELK storage.
|
||||
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=grpc-plugin \
|
||||
-e GRPC_STORAGE_PLUGIN_BINARY=<...> \
|
||||
-e GRPC_STORAGE_PLUGIN_CONFIGURATION_FILE=<...> \
|
||||
jaegertracing/all-in-one:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
## Ingester
|
||||
**jaeger-ingester** is a service which reads span data from Kafka topic and writes it to another storage backend (Elasticsearch or Cassandra).
|
||||
|
||||
Port | Protocol | Function
|
||||
----- | ------- | ---
|
||||
14270 | HTTP | admin port: health check at `/` and metrics at `/metrics`
|
||||
|
||||
To view all exposed configuration options run the following command:
|
||||
```sh
|
||||
docker run \
|
||||
-e SPAN_STORAGE_TYPE=cassandra \
|
||||
jaegertracing/jaeger-ingester:{{< currentVersion >}}
|
||||
--help
|
||||
```
|
||||
|
||||
## Query Service & UI
|
||||
|
||||
**jaeger-query** serves the API endpoints and a React/Javascript UI.
|
||||
The service is stateless and is typically run behind a load balancer, such as [**NGINX**](https://www.nginx.com/).
|
||||
|
||||
At default settings the query service exposes the following port(s):
|
||||
|
||||
Port | Protocol | Function
|
||||
----- | ------- | ---
|
||||
16686 | HTTP | `/api/*` endpoints and Jaeger UI at `/`
|
||||
16687 | HTTP | admin port: health check at `/` and metrics at `/metrics`
|
||||
|
||||
### Minimal deployment example (Elasticsearch backend):
|
||||
```sh
|
||||
docker run -d --rm \
|
||||
-p 16686:16686 \
|
||||
-p 16687:16687 \
|
||||
-e SPAN_STORAGE_TYPE=elasticsearch \
|
||||
-e ES_SERVER_URLS=http://<ES_SERVER_IP>:<ES_SERVER_PORT> \
|
||||
jaegertracing/jaeger-query:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
### UI Base Path
|
||||
|
||||
The base path for all **jaeger-query** HTTP routes can be set to a non-root value, e.g. `/jaeger` would cause all UI URLs to start with `/jaeger`. This can be useful when running **jaeger-query** behind a reverse proxy.
|
||||
|
||||
The base path can be configured via the `--query.base-path` command line parameter or the `QUERY_BASE_PATH` environment variable.
|
||||
|
||||
### UI Customization and Embedding
|
||||
|
||||
Please refer to the [dedicated Frontend/UI page](../frontend-ui/).
|
||||
|
||||
## Aggregation Jobs for Service Dependencies
|
||||
|
||||
Production deployments need an external process which aggregates data and creates dependency links between services. Project [spark-dependencies](https://github.com/jaegertracing/spark-dependencies) is a Spark job which derives dependency links and stores them directly to the storage.
|
||||
|
||||
## Configuration
|
||||
|
||||
All binaries accepts command line properties and environmental variables, power by [viper](https://github.com/spf13/viper) and [cobra](https://github.com/spf13/cobra) libraries. Please refer to the [CLI Flags](../cli/) page for more information.
|
||||
|
||||
[cqlsh]: http://cassandra.apache.org/doc/latest/tools/cqlsh.html
|
||||
[zipkin-thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/zipkincore.thrift
|
||||
[jaeger-thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/jaeger.thrift
|
||||
[thriftrw]: https://www.npmjs.com/package/thriftrw
|
|
@ -1,25 +0,0 @@
|
|||
---
|
||||
title: Frequently Asked Questions
|
||||
navtitle: FAQs
|
||||
description: Answers to some frequently asked questions about Jaeger.
|
||||
weight: 11
|
||||
---
|
||||
|
||||
## Why is the Dependencies page empty?
|
||||
|
||||
The Dependencies page shows a graph of services traced by Jaeger and connections between them. When you are using `all-in-one` binary with in-memory storage, the graph is calculated on-demand from all the traces stored in memory. However, if you are using a real distributed storage like Cassandra or Elasticsearch, it is too expensive to scan all the data in the database to build the service graph. Instead, the Jaeger project provides "big data" jobs that can be used to extract the service graph data from traces:
|
||||
|
||||
* https://github.com/jaegertracing/spark-dependencies - the older Spark job that can be run periodically
|
||||
* https://github.com/jaegertracing/jaeger-analytics - the new (experimental) streaming Flink jobs that run continuously and builds the service graph in smaller time intervals
|
||||
|
||||
## Why do I not see any spans in Jaeger?
|
||||
|
||||
Please refer to the [Troubleshooting](../troubleshooting/) guide.
|
||||
|
||||
## Do I need to run jaeger-agent?
|
||||
|
||||
`jaeger-agent` is not always necessary. Jaeger client libraries can be configured to export trace data directly to `jaeger-collector`. However, the following are the reasons why running `jaeger-agent` is recommended:
|
||||
|
||||
* If we want Jaeger client libraries to send trace data directly to collectors, we must provide them with a URL of the HTTP endpoint. It means that our applications require additional configuration containing this parameter, especially if we are running multiple Jaeger installations (e.g. in different availability zones or regions) and want the data sent to a nearby installation. In contrast, when using the agent, the libraries require no additional configuration because the agent is always accessible via `localhost`. It acts as a sidecar and proxies the requests to the appropriate collectors.
|
||||
* The agent can be configured to enrich the tracing data with infrastructure-specific metadata by adding extra tags to the spans, such as the current zone, region, etc. If the agent is running as a host daemon, it will be shared by all applications running on the same host. If the agent is running as a true sidecar, i.e. one per application, it can provide additional functionality such as strong authentication, multi-tenancy (see [this blog post](https://medium.com/jaegertracing/jaeger-and-multitenancy-99dfa1d49dc0)), pod name, etc.
|
||||
* Agents allow implementing traffic control to the collectors. If we have thousands of hosts in the data center, each running many applications, and each application sending data directly to the collectors, there may be too many open connections for each collector to handle. The agents can load balance this traffic with fewer connections.
|
|
@ -1,57 +0,0 @@
|
|||
---
|
||||
title: Features
|
||||
hasparent: true
|
||||
---
|
||||
|
||||
Jaeger is used for monitoring and troubleshooting microservices-based distributed systems, including:
|
||||
|
||||
* Distributed context propagation
|
||||
* Distributed transaction monitoring
|
||||
* Root cause analysis
|
||||
* Service dependency analysis
|
||||
* Performance / latency optimization
|
||||
|
||||
## High Scalability
|
||||
|
||||
Jaeger backend is designed to have no single points of failure and to scale with the business needs.
|
||||
For example, any given Jaeger installation at Uber is typically processing several billion {{< tip "spans" "span" >}} per day.
|
||||
|
||||
## Native support for OpenTracing
|
||||
|
||||
Jaeger backend, Web UI, and instrumentation libraries have been designed from ground up to support the OpenTracing standard.
|
||||
|
||||
* Represent {{< tip "traces" "trace" >}} as {{< tip "directed acyclic graphs" "directed acyclic graph" >}} (not just trees) via [span references](https://github.com/opentracing/specification/blob/master/specification.md#references-between-spans)
|
||||
* Support strongly typed span _tags_ and _structured logs_
|
||||
* Support general distributed context propagation mechanism via _baggage_
|
||||
|
||||
## Multiple storage backends
|
||||
|
||||
Jaeger supports two popular open source NoSQL databases as trace storage backends: Cassandra 3.4+ and Elasticsearch 5.x/6.x/7.x.
|
||||
There are ongoing community experiments using other databases, such as ScyllaDB, InfluxDB, Amazon DynamoDB, Logz.io. Jaeger also ships
|
||||
with a simple in-memory storage for testing setups.
|
||||
|
||||
## Modern Web UI
|
||||
|
||||
Jaeger Web UI is implemented in Javascript using popular open source frameworks like React. Several performance
|
||||
improvements have been released in v1.0 to allow the UI to efficiently deal with large volumes of data, and to display
|
||||
{{< tip "traces" "trace" >}} with tens of thousands of {{< tip "spans" "span" >}} (e.g. we tried a trace with 80,000 spans).
|
||||
|
||||
## Cloud Native Deployment
|
||||
|
||||
Jaeger backend is distributed as a collection of Docker images. The binaries support various configuration methods,
|
||||
including command line options, environment variables, and configuration files in multiple formats (yaml, toml, etc.).
|
||||
Deployment to Kubernetes clusters is assisted by a [Kubernetes operator](https://github.com/jaegertracing/jaeger-operator), [Kubernetes templates](https://github.com/jaegertracing/jaeger-kubernetes)
|
||||
and a [Helm chart](https://github.com/kubernetes/charts/tree/master/incubator/jaeger).
|
||||
|
||||
## Observability
|
||||
|
||||
All Jaeger backend components expose [Prometheus](https://prometheus.io/) metrics by default (other metrics backends are
|
||||
also supported). Logs are written to standard out using the structured logging library [zap](https://github.com/uber-go/zap).
|
||||
|
||||
## Backwards compatibility with Zipkin
|
||||
|
||||
Although we recommend instrumenting applications with OpenTracing API and binding to Jaeger client libraries to benefit
|
||||
from advanced features not available elsewhere, if your organization has already invested in the instrumentation
|
||||
using Zipkin libraries, you do not have to rewrite all that code. Jaeger provides backwards compatibility with Zipkin
|
||||
by accepting spans in Zipkin formats (Thrift, JSON v1/v2 and Protobuf) over HTTP. Switching from Zipkin backend is just a matter
|
||||
of routing the traffic from Zipkin libraries to the Jaeger backend.
|
|
@ -1,245 +0,0 @@
|
|||
---
|
||||
title: Frontend/UI Configuration
|
||||
navtitle: Frontend/UI
|
||||
hasparent: true
|
||||
weight: 7
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
Several aspects of the UI can be configured:
|
||||
|
||||
* The Dependencies section can be enabled / configured
|
||||
* Google Analytics tracking can be enabled / configured
|
||||
* Additional menu options can be added to the global nav
|
||||
* Search input limits can be configured
|
||||
|
||||
These options can be configured by a JSON configuration file. The `--query.ui-config` command line parameter of the query service must then be set to the path to the JSON file when the query service is started.
|
||||
|
||||
An example configuration file:
|
||||
|
||||
```json
|
||||
{
|
||||
"dependencies": {
|
||||
"dagMaxNumServices": 200,
|
||||
"menuEnabled": true
|
||||
},
|
||||
"archiveEnabled": true,
|
||||
"tracking": {
|
||||
"gaID": "UA-000000-2",
|
||||
"trackErrors": true
|
||||
},
|
||||
"menu": [
|
||||
{
|
||||
"label": "About Jaeger",
|
||||
"items": [
|
||||
{
|
||||
"label": "GitHub",
|
||||
"url": "https://github.com/jaegertracing/jaeger"
|
||||
},
|
||||
{
|
||||
"label": "Docs",
|
||||
"url": "http://jaeger.readthedocs.io/en/latest/"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"search": {
|
||||
"maxLookback": {
|
||||
"label": "2 Days",
|
||||
"value": "2d"
|
||||
},
|
||||
"maxLimit": 1500
|
||||
},
|
||||
"linkPatterns": [{
|
||||
"type": "process",
|
||||
"key": "jaeger.version",
|
||||
"url": "https://github.com/jaegertracing/jaeger-client-java/releases/tag/#{jaeger.version}",
|
||||
"text": "Information about Jaeger release #{jaeger.version}"
|
||||
}]
|
||||
}
|
||||
```
|
||||
|
||||
### Dependencies
|
||||
|
||||
`dependencies.dagMaxNumServices` defines the maximum number of services allowed before the DAG dependency view is disabled. Default: `200`.
|
||||
|
||||
`dependencies.menuEnabled` enables (`true`) or disables (`false`) the dependencies menu button. Default: `true`.
|
||||
|
||||
### Archive Support
|
||||
|
||||
`archiveEnabled` enables (`true`) or disables (`false`) the archive traces button. Default: `false`. It requires a configuration of an archive storage in Query service. Archived traces are only accessible directly by ID, they are not searchable.
|
||||
|
||||
### Google Analytics Tracking
|
||||
|
||||
`tracking.gaID` defines the Google Analytics tracking ID. This is required for Google Analytics tracking, and setting it to a non-`null` value enables Google Analytics tracking. Default: `null`.
|
||||
|
||||
`tracking.trackErrors` enables (`true`) or disables (`false`) error tracking via Google Analytics. Errors can only be tracked if a valid Google Analytics ID is provided. For additional details on error tracking via Google Analytics see the [tracking README](https://github.com/jaegertracing/jaeger-ui/blob/c622330546afc1be59a42f874bcc1c2fadf7e69a/src/utils/tracking/README.md) in the UI repo. Default: `true`.
|
||||
|
||||
### Custom Menu Items
|
||||
|
||||
`menu` allows additional links to be added to the global nav. The additional links are right-aligned.
|
||||
|
||||
In the sample JSON config above, the configured menu will have a dropdown labeled "About Jaeger" with sub-options for "GitHub" and "Docs". The format for a link in the top right menu is as follows:
|
||||
|
||||
```json
|
||||
{
|
||||
"label": "Some text here",
|
||||
"url": "https://example.com"
|
||||
}
|
||||
```
|
||||
|
||||
Links can either be members of the `menu` Array, directly, or they can be grouped into a dropdown menu option. The format for a group of links is:
|
||||
|
||||
```json
|
||||
{
|
||||
"label": "Dropdown button",
|
||||
"items": [ ]
|
||||
}
|
||||
```
|
||||
|
||||
The `items` Array should contain one or more link configurations.
|
||||
|
||||
### Search Input Limit
|
||||
|
||||
The `search.maxLimit` configures the maximum results that the input let you search.
|
||||
|
||||
The `search.maxLookback` configures the maximum time before the present users can query for traces. The options in the Lookback dropdown greater than this value will not be shown.
|
||||
|
||||
Field | Description
|
||||
------|------------
|
||||
label | The text displayed in the search form dropdown
|
||||
value | The value submitted in the search query if the label is selected
|
||||
|
||||
### Link Patterns
|
||||
|
||||
The `linkPatterns` node can be used to create links from fields displayed in the Jaeger UI.
|
||||
|
||||
Field | Description
|
||||
------|------------
|
||||
type | The metadata section in which your link will be added: process, tags, logs, traces
|
||||
key | The name of tag/process/log attribute which value will be displayed as a link, this field is not necessary for type `traces`.
|
||||
url | The URL where the link should point to, it can be an external site or relative path in Jaeger UI
|
||||
text | The text displayed in the tooltip for the link
|
||||
|
||||
Both `url` and `text` can be defined as templates (i.e. using `#{field-name}`) where Jaeger UI will dynamically substitute values based on tags/logs/traces data.
|
||||
|
||||
For traces, the supported template fields are: `duration`, `endTime`, `startTime`, `traceName` and `traceID`.
|
||||
|
||||
## Embedded Mode
|
||||
|
||||
Starting with version 1.9, Jaeger UI provides an "embedded" layout mode which is intended to support integrating Jaeger UI into other applications. Currently (as of `v0`), the approach taken is to remove various UI elements from the page to make the UI better suited for space-constrained layouts.
|
||||
|
||||
The embedded mode is induced and configured via URL query parameters.
|
||||
|
||||
To enter embedded mode, the `uiEmbed=v0` query parameter and value must be added to the URL. For example, the following URL will show the trace with ID `abc123` in embedded mode:
|
||||
|
||||
```
|
||||
http://localhost:16686/trace/abc123?uiEmbed=v0
|
||||
```
|
||||
|
||||
`uiEmbed=v0` is required.
|
||||
|
||||
Further, each page supported has an <img src="/img/frontend-ui/embed-open-icon.png" style="width: 20px; height:20px;display:inline;" alt="Embed open window"> button added that will open the non-embedded page in a new tab.
|
||||
|
||||
The following pages support embedded mode:
|
||||
|
||||
* Search Page
|
||||
* Trace Page
|
||||
|
||||
### Search Page
|
||||
|
||||
To integrate the Search Trace Page to our application we have to indicate to the Jaeger UI that we want to use the embed mode with `uiEmbed=v0`.
|
||||
|
||||
For example:
|
||||
|
||||
```
|
||||
http://localhost:16686/search?
|
||||
service=my-service&
|
||||
start=1543917759557000&
|
||||
end=1543921359557000&
|
||||
limit=20&
|
||||
lookback=1h&
|
||||
maxDuration&
|
||||
minDuration&
|
||||
uiEmbed=v0
|
||||
```
|
||||
|
||||

|
||||
|
||||
#### Configuration options
|
||||
|
||||
The following query parameter can be used to configure the layout of the search page :
|
||||
|
||||
* `uiSearchHideGraph=1` - disables the display of the scatter plot above the search results
|
||||
|
||||
```
|
||||
http://localhost:16686/search?
|
||||
service=my-service&
|
||||
start=1543917759557000&
|
||||
end=1543921359557000&
|
||||
limit=20&
|
||||
lookback=1h&
|
||||
maxDuration&
|
||||
minDuration&
|
||||
uiEmbed=v0&
|
||||
uiSearchHideGraph=1
|
||||
```
|
||||
|
||||

|
||||
|
||||
### Trace Page
|
||||
|
||||
|
||||
To integrate the Trace Page to our application we have to indicate to the Jaeger UI that we want to use the embed mode with `uiEmbed=v0`.
|
||||
|
||||
For example:
|
||||
|
||||
```sh
|
||||
http://localhost:16686/trace/{trace-id}?uiEmbed=v0
|
||||
```
|
||||

|
||||
|
||||
If we have navigated to this view from the search traces page we'll have a button to go back to the results page.
|
||||
|
||||

|
||||
|
||||
#### Configuration options
|
||||
|
||||
The following query parameters can be used to configure the layout of the trace page :
|
||||
|
||||
* `uiTimelineCollapseTitle=1` causes the trace header to start out collapsed, which hides the summary and the minimap.
|
||||
|
||||
```
|
||||
http://localhost:16686/trace/{trace-id}?
|
||||
uiEmbed=v0&
|
||||
uiTimelineCollapseTitle=1
|
||||
```
|
||||

|
||||
|
||||
* `uiTimelineHideMinimap=1` removes the minimap, entirely, regardless of whether the trace header is expanded or not.
|
||||
|
||||
```
|
||||
http://localhost:16686/trace/{trace-id}?
|
||||
uiEmbed=v0&
|
||||
uiTimelineHideMinimap=1
|
||||
```
|
||||

|
||||
|
||||
* `uiTimelineHideSummary=1` - removes the trace summary information (number of services, etc.) entirely, regardless of whether the trace header is expanded or not.
|
||||
|
||||
```
|
||||
http://localhost:16686/trace/{trace-id}?
|
||||
uiEmbed=v0&
|
||||
uiTimelineHideSummary=1
|
||||
```
|
||||

|
||||
|
||||
We can also combine the options:
|
||||
```
|
||||
http://localhost:16686/trace/{trace-id}?
|
||||
uiEmbed=v0&
|
||||
uiTimelineHideMinimap=1&
|
||||
uiTimelineHideSummary=1
|
||||
```
|
||||

|
|
@ -1,130 +0,0 @@
|
|||
---
|
||||
title: Getting Started
|
||||
description: Get up and running with Jaeger in your local environment
|
||||
weight: 2
|
||||
---
|
||||
|
||||
## Instrumentation
|
||||
|
||||
Your applications must be instrumented before they can send tracing data to Jaeger backend. Check the [Client Libraries](../client-libraries) section for information about how to use the OpenTracing API and how to initialize and configure Jaeger tracers.
|
||||
|
||||
## All in One
|
||||
|
||||
All-in-one is an executable designed for quick local testing, launches the Jaeger UI, collector, query, and agent, with an in memory storage component.
|
||||
|
||||
The simplest way to start the all-in-one is to use the pre-built image published to DockerHub (a single command line).
|
||||
|
||||
```bash
|
||||
$ docker run -d --name jaeger \
|
||||
-e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
|
||||
-p 5775:5775/udp \
|
||||
-p 6831:6831/udp \
|
||||
-p 6832:6832/udp \
|
||||
-p 5778:5778 \
|
||||
-p 16686:16686 \
|
||||
-p 14268:14268 \
|
||||
-p 14250:14250 \
|
||||
-p 9411:9411 \
|
||||
jaegertracing/all-in-one:{{< currentVersion >}}
|
||||
```
|
||||
|
||||
Or run the `jaeger-all-in-one(.exe)` executable from the [binary distribution archives][download]:
|
||||
|
||||
```bash
|
||||
$ jaeger-all-in-one --collector.zipkin.http-port=9411
|
||||
```
|
||||
|
||||
You can then navigate to `http://localhost:16686` to access the Jaeger UI.
|
||||
|
||||
The container exposes the following ports:
|
||||
|
||||
Port | Protocol | Component | Function
|
||||
----- | ------- | --------- | ---
|
||||
5775 | UDP | agent | accept `zipkin.thrift` over compact thrift protocol (deprecated, used by legacy clients only)
|
||||
6831 | UDP | agent | accept `jaeger.thrift` over compact thrift protocol
|
||||
6832 | UDP | agent | accept `jaeger.thrift` over binary thrift protocol
|
||||
5778 | HTTP | agent | serve configs
|
||||
16686 | HTTP | query | serve frontend
|
||||
14268 | HTTP | collector | accept `jaeger.thrift` directly from clients
|
||||
14250 | HTTP | collector | accept `model.proto`
|
||||
9411 | HTTP | collector | Zipkin compatible endpoint (optional)
|
||||
|
||||
|
||||
## Kubernetes and OpenShift
|
||||
|
||||
* Kubernetes templates: https://github.com/jaegertracing/jaeger-kubernetes
|
||||
* Kubernetes Operator: https://github.com/jaegertracing/jaeger-operator
|
||||
* OpenShift templates: https://github.com/jaegertracing/jaeger-openshift
|
||||
|
||||
## Sample App: HotROD
|
||||
|
||||
HotROD (Rides on Demand) is a demo application that consists of several microservices and
|
||||
illustrates the use of the [OpenTracing API](http://opentracing.io).
|
||||
A tutorial / walkthrough is available in the blog post:
|
||||
[Take OpenTracing for a HotROD ride][hotrod-tutorial].
|
||||
|
||||
It can be run standalone, but requires Jaeger backend to view the traces.
|
||||
|
||||
### Features
|
||||
|
||||
- Discover architecture of the whole system via data-driven dependency
|
||||
diagram.
|
||||
- View request timeline and errors; understand how the app works.
|
||||
- Find sources of latency and lack of concurrency.
|
||||
- Highly contextualized logging.
|
||||
- Use baggage propagation to:
|
||||
|
||||
- Diagnose inter-request contention (queueing).
|
||||
- Attribute time spent in a service.
|
||||
|
||||
- Use open source libraries with OpenTracing integration to get
|
||||
vendor-neutral instrumentation for free.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- You need Go 1.11 or higher installed on your machine to run from source.
|
||||
- Requires a [running Jaeger backend](#all-in-one) to view the traces.
|
||||
|
||||
### Running
|
||||
|
||||
#### From Source
|
||||
|
||||
```bash
|
||||
mkdir -p $GOPATH/src/github.com/jaegertracing
|
||||
cd $GOPATH/src/github.com/jaegertracing
|
||||
git clone git@github.com:jaegertracing/jaeger.git jaeger
|
||||
cd jaeger
|
||||
make install
|
||||
go run ./examples/hotrod/main.go all
|
||||
```
|
||||
#### From docker
|
||||
|
||||
```bash
|
||||
$ docker run --rm -it \
|
||||
--link jaeger \
|
||||
-p8080-8083:8080-8083 \
|
||||
-e JAEGER_AGENT_HOST="jaeger" \
|
||||
jaegertracing/example-hotrod:{{< currentVersion >}} \
|
||||
all
|
||||
```
|
||||
|
||||
#### From binary distribution
|
||||
|
||||
Run `example-hotrod(.exe)` executable from the [binary distribution archives][download]:
|
||||
```bash
|
||||
$ example-hotrod all
|
||||
```
|
||||
|
||||
Then navigate to `http://localhost:8080`.
|
||||
|
||||
|
||||
## Migrating from Zipkin
|
||||
|
||||
Collector service exposes Zipkin compatible REST API `/api/v1/spans` which accepts both Thrift and JSON. Also there is `/api/v2/spans` for JSON and Proto.
|
||||
By default it's disabled. It can be enabled with `--collector.zipkin.http-port=9411`.
|
||||
|
||||
Zipkin [Thrift](https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/zipkincore.thrift) IDL and Zipkin [Proto](https://github.com/jaegertracing/jaeger-idl/blob/master/proto/zipkin.proto) IDL files can be found in [jaegertracing/jaeger-idl](https://github.com/jaegertracing/jaeger-idl) repository.
|
||||
They're compatible with [openzipkin/zipkin-api](https://github.com/openzipkin/zipkin-api) [Thrift](https://github.com/openzipkin/zipkin-api/blob/master/thrift/zipkinCore.thrift) and [Proto](https://github.com/openzipkin/zipkin-api/blob/master/zipkin.proto).
|
||||
|
||||
[hotrod-tutorial]: https://medium.com/@YuriShkuro/take-opentracing-for-a-hotrod-ride-f6e3141f7941
|
||||
[download]: ../../../download/
|
|
@ -1,47 +0,0 @@
|
|||
---
|
||||
title: Monitoring Jaeger
|
||||
navtitle: Monitoring
|
||||
weight: 8
|
||||
---
|
||||
|
||||
Jaeger itself is a distributed, microservices based system. If you run it in production, you will likely want to setup adequate monitoring for different components, e.g. to ensure that the backend is not saturated by too much tracing data.
|
||||
|
||||
## Metrics
|
||||
|
||||
By default Jaeger microservices expose metrics in Prometheus format. It is controlled by the following command line options:
|
||||
|
||||
* `--admin-http-port` the port number where the HTTP admin server is running
|
||||
* `--metrics-backend` controls how the measurements are exposed. The default value is `prometheus`, another option is `expvar`, the Go standard mechanism for exposing process level statistics.
|
||||
* `--metrics-http-route` specifies the name of the HTTP endpoint used to scrape the metrics (`/metrics` by default).
|
||||
|
||||
Each Jaeger component exposes the metrics scraping endpoint on the admin port:
|
||||
|
||||
Component | Port
|
||||
--------------------- | ---
|
||||
**jaeger-agent** | 14271
|
||||
**jaeger-collector** | 14269
|
||||
**jaeger-query** | 16687
|
||||
**jaeger-ingester** | 14270
|
||||
**all-in-one** | 14269
|
||||
|
||||
### Prometheus monitoring mixin for Jaeger
|
||||
|
||||
The Prometheus monitoring mixin for Jaeger provides a starting point for people wanting to monitor Jaeger using Prometheus, Alertmanager, and Grafana. This includes a prebuilt [dashboard](https://github.com/jaegertracing/jaeger/blob/master/monitoring/jaeger-mixin/dashboard-for-grafana.json). For more information, see [the documentation](https://github.com/jaegertracing/jaeger/tree/master/monitoring/jaeger-mixin).
|
||||
|
||||
## Logging
|
||||
|
||||
Jaeger components only log to standard out, using structured logging library [go.uber.org/zap](https://github.com/uber-go/zap) configured to write log lines as JSON encoded strings, for example:
|
||||
|
||||
```json
|
||||
{"level":"info","ts":1517621222.261759,"caller":"healthcheck/handler.go:99","msg":"Health Check server started","http-port":14269,"status":"unavailable"}
|
||||
```
|
||||
|
||||
The log level can be adjusted via `--log-level` command line switch; default level is `info`.
|
||||
|
||||
## Traces
|
||||
|
||||
Jaeger has the ability to trace some of its own components, namely the requests to the Query service. For example, if you start `all-in-one` as described in [Getting Started](../getting-started), and refresh the UI screen a few times, you will see `jaeger-query` populated in the Services dropdown. If you prefer not to see these traces in the Jaeger UI, you can disable them by running Jaeger backend components with `JAEGER_DISABLED=true` environment variable, for example:
|
||||
|
||||
```
|
||||
docker run -e JAEGER_DISABLED=true -p 16686:16686 jaegertracing/all-in-one:{{< currentVersion >}}
|
||||
```
|
File diff suppressed because it is too large
Load Diff
|
@ -1,97 +0,0 @@
|
|||
---
|
||||
title: Performance Tuning Guide
|
||||
description: Tweaking your Jaeger instance to achieve a better performance
|
||||
weight: 10
|
||||
---
|
||||
|
||||
Jaeger was built from day 1 to be able to ingest huge amounts of data in a resilient way. To better utilize resources that might cause delays, such as storage or network communications, Jaeger buffers and batches data. When more spans are generated than Jaeger is able to safely process, spans might get dropped. However, the defaults might not fit all scenarios: for instance, Agents running as a sidecar might have more memory constraints than Agents running as a daemon in bare metal.
|
||||
|
||||
## Deployment considerations
|
||||
|
||||
Although performance tuning the individual components is important, the way Jaeger is deployed can be decisive in obtaining optimal performance.
|
||||
|
||||
### Scale the Collector up and down
|
||||
|
||||
Use the auto-scaling capabilities of your platform: the Collector is nearly horizontally scalable so that more instances can be added and removed on-demand. A good way to scale up and down is by checking the `jaeger_collector_queue_length` metric: add instances when the length is higher than 50% of the maximum size for extended periods of time. Another metric that can be taken into consideration is `jaeger_collector_in_queue_latency_bucket`, which is a histogram indicating how long spans have been waiting in the queue before a worker picked it up. When the queue latency gets higher over time, it’s a good indication to increase the number of the workers, or to improve the storage performance.
|
||||
|
||||
Adding Collector instances is recommended when your platform provides auto-scaling capabilities, or when it's easier to start/stop Collector instances than changing existing, running instances. Scaling horizontally is also indicated when the CPU usage should be spread across nodes.
|
||||
|
||||
### Make sure the storage can keep up
|
||||
|
||||
Each span is written to the storage by the Collector using one worker, blocking it until the span has been stored. When the storage is too slow, the number of workers blocked by the storage might be too high, causing spans to be dropped. To help diagnose this situation, the histogram `jaeger_collector_save_latency_bucket` can be analyzed. Ideally, the latency should remain the same over time. When the histogram shows that most spans are taking longer and longer over time, it’s a good indication that your storage might need some attention.
|
||||
|
||||
### Place the Agents close to your applications
|
||||
|
||||
The Agent is meant to be placed on the same host as the instrumented application, in order to avoid UDP packet loss over the network. This is typically accomplished by having one Agent per bare metal host for traditional applications, or as a sidecar in container environments like Kubernetes, as this helps spread the load handled by Agents with the additional advantage of allowing each Agent to be tweaked individually, according to the application’s needs and importance.
|
||||
|
||||
### Consider using Apache Kafka as intermediate buffer
|
||||
|
||||
Jaeger [can use Apache Kafka](../architecture/) as a buffer between the Collector and the actual backing storage (Elasticsearch, Apache Cassandra). This is ideal for cases where the traffic spikes are relatively frequent (prime time traffic) but the storage can eventually catch up once the traffic normalizes. For that, the `SPAN_STORAGE_TYPE` environment variable should be set to `kafka` in the Collector and the Jaeger Ingester component can be used, reading data from Kafka and writing it to the storage.
|
||||
|
||||
In addition to the performance aspects, having spans written to Kafka is useful for building real time data pipeline for aggregations and feature extraction from traces.
|
||||
|
||||
## Client (Tracer) settings
|
||||
|
||||
The Jaeger Clients are built to have minimal effect to the instrumented application. As such, it has conservative defaults that might not be suitable for all cases. Note that Jaeger Clients can be configured programmatically or via [environment variables](../client-features/).
|
||||
|
||||
### Adjust the sampling configuration
|
||||
|
||||
Together, the `JAEGER_SAMPLER_TYPE` and `JAEGER_SAMPLER_PARAM` specify how often traces should be "sampled", ie, recorded and sent to the Jaeger backend. For applications generating a large number of spans, setting the sampling type to `probabilistic` and the value to `0.001` (the default) will cause traces to be reported with a 1/1000th chance. Note that the sampling decision is made at the root span and propagated down to all child spans.
|
||||
|
||||
For applications with low to medium traffic, setting the sampling type to `const` and value to `1` will cause all spans to be reported. Similarly, tracing can be disabled by setting the value to `0`, while context propagation will continue to work.
|
||||
|
||||
Some Clients support the setting `JAEGER_DISABLED` to completely disable the Jaeger Tracer. This is recommended only if the Tracer is behaving in a way that causes problems to the instrumented application, as it will not propagate the context to the downstream services.
|
||||
|
||||
{{< info >}}
|
||||
We recommend to set your clients to use the [`remote` sampling strategy](../sampling/#collector-sampling-configuration), so that admins can centrally set the concrete sampling strategy for each service.
|
||||
{{< /info >}}
|
||||
|
||||
### Increase in-memory queue size
|
||||
|
||||
Most of the Jaeger clients, such as the Java, Go, and C# clients, buffer spans in memory before sending them to the Jaeger Agent/Collector. The maximum size of this buffer is defined by the environment variable `JAEGER_REPORTER_MAX_QUEUE_SIZE` (default value: about `100` spans): the larger the size, the higher the potential memory consumption. When the instrumented application is generating a large number of spans, it’s possible that the queue will be full causing the Client to discard the new spans (metric `jaeger_tracer_reporter_spans_total{result="dropped",}`).
|
||||
|
||||
In most common scenarios, the queue will be close to empty (metric: `jaeger_tracer_reporter_queue_length`), as spans are flushed to the Agent or Collector at regular intervals or when a certain size of the batch is reached. The detailed behavior of this queue is described in this [GitHub issue](https://github.com/jaegertracing/jaeger-client-java/issues/607).
|
||||
|
||||
### Modify the batched spans flush interval
|
||||
|
||||
The Java, Go, NodeJS, Python and C# Clients allow the customization of the flush interval (default value: `1000` milliseconds, or 1 second) used by the reporters, such as the `RemoteReporter`, to trigger a `flush` operation, sending all in-memory spans to the Agent or Collector. The lower the flush interval is set to, the more frequent the flush operations happen. As most reporters will wait until enough data is in the queue, this setting will force a flush operation at periodic intervals, so that spans are sent to the backend in a timely fashion.
|
||||
|
||||
When the instrumented application is generating a large number of spans and the Agent/Collector is close to the application, the networking overhead might be low, justifying a higher number of flush operations. When the `HttpSender` is being used and the Collector is not close enough to the application, the networking overhead might be too high so that a higher value for this property makes sense.
|
||||
|
||||
## Agent settings
|
||||
|
||||
Jaeger Agents receive data from Clients, sending them in batches to the Collector. When not properly configured, it might end up discarding data even if the host machine has plenty of resources.
|
||||
|
||||
### Adjust server queue sizes
|
||||
|
||||
The set of “server queue size” properties ( `processor.jaeger-binary.server-queue-size`, `processor.jaeger-compact.server-queue-size`, `processor.zipkin-compact.server-queue-size`) indicate the maximum number of span batches that the Agent can accept and store in memory. It’s safe to assume that `jaeger-compact` is the most important processor in your Agent setup, as it’s the only one available in most Clients, such as the Java and Go Clients.
|
||||
|
||||
The default value for each queue is `1000` span batches. Given that each span batch has up to 64KiB worth of spans, each queue can hold up to 64MiB worth of spans.
|
||||
|
||||
In typical scenarios, the queue will be close to empty (metric `jaeger_agent_thrift_udp_server_queue_size`) as span batches should be quickly picked up and processed by a worker. However, sudden spikes in the number of span batches submitted by Clients might occur, causing the batches to be queued. When the queue is full, the older batches are overridden causing spans to be discarded (metric `jaeger_agent_thrift_udp_server_packets_dropped_total`).
|
||||
|
||||
### Adjust processor workers
|
||||
|
||||
The set of “processor workers” properties ( `processor.jaeger-binary.workers`, `processor.jaeger-compact.workers`, `processor.zipkin-compact.workers`) indicate the number of parallel span batch processors to start. Each worker type has a default size of `10`. In general, span batches are processed as soon as they are placed in the server queue and will block a worker until the whole packet is sent to the Collector. For Agents processing data from multiple Clients, the number of workers should be increased. Given that the cost of each worker is low, a good rule of thumb is 10 workers per Client with moderate traffic: given that each span batch might contain up to 64KiB worth of spans, it means that 10 workers are able to send about 640KiB concurrently to a Collector.
|
||||
|
||||
## Collector settings
|
||||
|
||||
The Collector receives data from Clients and Agents. When not properly configured, it might process less data than what would be possible on the same host, or it might overload the host by consuming more memory than permitted.
|
||||
|
||||
### Adjust queue size
|
||||
|
||||
Similar to the Agent, the Collector is able to receive spans and place them in an internal queue for processing. This allows the Collector to return immediately to the Client/Agent instead of waiting for the span to make its way to the storage.
|
||||
|
||||
The setting `collector.queue-size` (default: `2000`) dictates how many spans the queue should support. In the typical scenario, the queue will be close to empty, as enough workers should exist picking up spans from the queue and sending them to the storage. When the number of items in the queue (metric `jaeger_collector_queue_length`) is permanently high, it’s an indication that either the number of workers should be increased or that the storage cannot keep up with the volume of data that it’s receiving. When the queue is full, the older items in the queue are overridden, causing spans to be discarded (metric `jaeger_collector_spans_dropped_total`).
|
||||
|
||||
{{< warning >}}
|
||||
The queue size for the Agent is about _span batches_, whereas the queue size for the Collector is about _individual spans_.
|
||||
{{< /warning >}}
|
||||
|
||||
Given that the queue size should be close to empty most of the time, this setting should be as high as the available memory for the Collector, to provide maximum protection against sudden traffic spikes. However, if your storage layer is under-provisioned and cannot keep up, even a large queue will quickly fill up and start dropping data.
|
||||
|
||||
Experimental: starting from Jaeger 1.17, the Jaeger Collector can adjust the queue size automatically based on the memory requirements and average span size. Set the flag `collector.queue-size-memory` to the maximum memory size in MiB that the collector should use, and Jaeger will periodically calculate the ideal queue size based on the average span size it has seen. For safety reasons, the maximum queue size is hard-coded to 1 million records. If you are using this feature, [give us your feedback](https://www.jaegertracing.io/get-in-touch/)!
|
||||
|
||||
### Adjust processor workers
|
||||
|
||||
Items from the span queue in the Collector are picked up by workers. Each worker picks one span from the queue and persists it to the storage. The number of workers can be specified by the setting `collector.num-workers` (default: `50`) and should be as high as needed to keep the queue close to zero. The general rule is: the faster the backing storage, the lower the number of workers can be. Given that workers are relatively cheap, this number can be increased at will. As a general rule, one worker per 50 items in the queue should be sufficient when the storage is fast. With a `collector.queue-size` of `2000`, having about `40` workers should be sufficient. For slower storage mechanisms, this ratio should be adjusted accordingly, having more workers per queue item.
|
|
@ -1,85 +0,0 @@
|
|||
---
|
||||
title: Sampling
|
||||
hasparent: true
|
||||
---
|
||||
|
||||
Jaeger libraries implement consistent upfront (or head-based) sampling. For example, assume we have a simple call graph where service A calls service B, and B calls service C: `A -> B -> C`. When service A receives a request that contains no tracing information, Jaeger tracer will start a new {{< tip "trace" >}}, assign it a random trace ID, and make a sampling decision based on the currently installed sampling strategy. The sampling decision will be propagated with the requests to B and to C, so those services will not be making the sampling decision again but instead will respect the decision made by the top service A. This approach guarantees that if a trace is sampled, all its {{< tip "spans" "span" >}} will be recorded in the backend. If each service was making its own sampling decision we would rarely get complete traces in the backend.
|
||||
|
||||
## Client Sampling Configuration
|
||||
|
||||
When using configuration object to instantiate the tracer, the type of sampling can be selected via `sampler.type` and `sampler.param` properties. Jaeger libraries support the following samplers:
|
||||
|
||||
* **Constant** (`sampler.type=const`) sampler always makes the same decision for all traces. It either samples all traces (`sampler.param=1`) or none of them (`sampler.param=0`).
|
||||
* **Probabilistic** (`sampler.type=probabilistic`) sampler makes a random sampling decision with the probability of sampling equal to the value of `sampler.param` property. For example, with `sampler.param=0.1` approximately 1 in 10 traces will be sampled.
|
||||
* **Rate Limiting** (`sampler.type=ratelimiting`) sampler uses a leaky bucket rate limiter to ensure that traces are sampled with a certain constant rate. For example, when `sampler.param=2.0` it will sample requests with the rate of 2 traces per second.
|
||||
* **Remote** (`sampler.type=remote`, which is also the default) sampler consults Jaeger agent for the appropriate sampling strategy to use in the current service. This allows controlling the sampling strategies in the services from a [central configuration](#collector-sampling-configuration) in Jaeger backend, or even dynamically (see [Adaptive Sampling](https://github.com/jaegertracing/jaeger/issues/365)).
|
||||
|
||||
### Adaptive Sampler
|
||||
|
||||
Adaptive sampler is a composite sampler that combines two functions:
|
||||
|
||||
* It makes sampling decisions on a per-operation basis, i.e. based on {{< tip "span" >}} operation name. This is especially useful in the API services whose endpoints may have very different traffic volumes and using a single probabilistic sampler for the whole service might starve (never sample) some of the low QPS endpoints.
|
||||
* It supports a minimum guaranteed rate of sampling, such as always allowing up to N {{< tip "traces" "trace" >}} per seconds and then sampling anything above that with a certain probability (everything is per-operation, not per-service).
|
||||
|
||||
Per-operation parameters can be configured statically or pulled periodically from Jaeger backend with the help of Remote sampler. Adaptive sampler is designed to work with the upcoming [Adaptive Sampling](https://github.com/jaegertracing/jaeger/issues/365) feature of the Jaeger backend.
|
||||
|
||||
## Collector Sampling Configuration
|
||||
|
||||
Collectors can be instantiated with static sampling strategies (which are propagated to the respective service if configured with Remote sampler) via the `--sampling.strategies-file` option. This option requires a path to a json file which defines the sampling strategies.
|
||||
|
||||
If no configuration is provided, the collectors will return the default probabilistic sampling policy with probability 0.001 (0.1%) for all services.
|
||||
|
||||
Example `strategies.json`:
|
||||
```json
|
||||
{
|
||||
"service_strategies": [
|
||||
{
|
||||
"service": "foo",
|
||||
"type": "probabilistic",
|
||||
"param": 0.8,
|
||||
"operation_strategies": [
|
||||
{
|
||||
"operation": "op1",
|
||||
"type": "probabilistic",
|
||||
"param": 0.2
|
||||
},
|
||||
{
|
||||
"operation": "op2",
|
||||
"type": "probabilistic",
|
||||
"param": 0.4
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "bar",
|
||||
"type": "ratelimiting",
|
||||
"param": 5
|
||||
}
|
||||
],
|
||||
"default_strategy": {
|
||||
"type": "probabilistic",
|
||||
"param": 0.5,
|
||||
"operation_strategies": [
|
||||
{
|
||||
"operation": "/health",
|
||||
"type": "probabilistic",
|
||||
"param": 0.0
|
||||
},
|
||||
{
|
||||
"operation": "/metrics",
|
||||
"type": "probabilistic",
|
||||
"param": 0.0
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`service_strategies` element defines service specific sampling strategies and `operation_strategies` defines operation specific sampling strategies. There are 2 types of strategies possible: `probabilistic` and `ratelimiting` which are described [above](#client-sampling-configuration) (NOTE: `ratelimiting` is not supported for `operation_strategies`). `default_strategy` defines the catch-all sampling strategy that is propagated if the service is not included as part of `service_strategies`.
|
||||
|
||||
In the above example:
|
||||
|
||||
* All operations of service `foo` are sampled with probability 0.8 except for operations `op1` and `op2` which are probabilistically sampled with probabilities 0.2 and 0.4 respectively.
|
||||
* All operations for service `bar` are rate-limited at 5 traces per second.
|
||||
* Any other service will be sampled with probability 0.5 defined by the `default_strategy`.
|
||||
* The `default_strategy` also includes shared per-operation strategies. In this example we disable tracing on `/health` and `/metrics` endpoints for all services by using probability 0. These per-operation strategies will apply to any new service not listed in the config, as well as to the `foo` and `bar` services unless they define their own strategies for these two operations.
|
|
@ -1,43 +0,0 @@
|
|||
---
|
||||
title: Windows Service Deployment
|
||||
hasparent: true
|
||||
---
|
||||
|
||||
In Windows environments, Jaeger processes can be hosted and managed as Windows services controlled via the `sc` utility. To configure such services on Windows, download [nssm.exe](https://nssm.cc/download) for the appropriate architecture, and issue commands similar to how Jaeger is typically run. The example below showcases a basic Elasticsearch setup, configured using both environment variables and process arguments.
|
||||
|
||||
## Agent
|
||||
```bat
|
||||
nssm install JaegerAgent C:\Jaeger\jaeger-agent.exe --reporter.grpc.host-port=localhost:14250
|
||||
|
||||
nssm set JaegerAgent AppStdout C:\Jaeger\jaeger-agent.out.log
|
||||
nssm set JaegerAgent AppStderr C:\Jaeger\jaeger-agent.err.log
|
||||
nssm set JaegerAgent Description Jaeger Agent service
|
||||
|
||||
nssm start JaegerAgent
|
||||
```
|
||||
|
||||
## Collector
|
||||
```bat
|
||||
nssm install JaegerCollector C:\Jaeger\jaeger-collector.exe --es.server-urls=http://localhost:9200 --es.username=jaeger --es.password=PASSWORD
|
||||
|
||||
nssm set JaegerCollector AppStdout C:\Jaeger\jaeger-collector.out.log
|
||||
nssm set JaegerCollector AppStderr C:\Jaeger\jaeger-collector.err.log
|
||||
nssm set JaegerCollector Description Jaeger Collector service
|
||||
nssm set JaegerCollector AppEnvironmentExtra SPAN_STORAGE_TYPE=elasticsearch
|
||||
|
||||
nssm start JaegerCollector
|
||||
```
|
||||
|
||||
## Query UI
|
||||
```bat
|
||||
nssm install JaegerUI C:\Jaeger\jaeger-query.exe --es.server-urls=http://localhost:9200 --es.username=jaeger --es.password=PASSWORD
|
||||
|
||||
nssm set JaegerUI AppStdout C:\Jaeger\jaeger-ui.out.log
|
||||
nssm set JaegerUI AppStderr C:\Jaeger\jaeger-ui.err.log
|
||||
nssm set JaegerUI Description Jaeger Query service
|
||||
nssm set JaegerUI AppEnvironmentExtra SPAN_STORAGE_TYPE=elasticsearch
|
||||
|
||||
nssm start JaegerUI
|
||||
```
|
||||
|
||||
For additional information & docs, please see [the NSSM usage guide.](https://nssm.cc/usage)
|
|
@ -1,70 +0,0 @@
|
|||
---
|
||||
title: Introduction
|
||||
weight: 1
|
||||
children:
|
||||
- title: Features
|
||||
url: features
|
||||
---
|
||||
|
||||
Welcome to Jaeger's documentation portal! Below, you'll find information for beginners and experienced Jaeger users.
|
||||
|
||||
If you can't find what you are looking for, or have an issue not covered here, we'd love to [hear from you](/get-in-touch).
|
||||
|
||||
## About
|
||||
|
||||
Jaeger, inspired by [Dapper][dapper] and [OpenZipkin](http://zipkin.io),
|
||||
is a distributed tracing system released as open source by [Uber Technologies][ubeross].
|
||||
It is used for monitoring and troubleshooting microservices-based distributed systems, including:
|
||||
|
||||
* Distributed context propagation
|
||||
* Distributed transaction monitoring
|
||||
* Root cause analysis
|
||||
* Service dependency analysis
|
||||
* Performance / latency optimization
|
||||
|
||||
Uber published a blog post, [Evolving Distributed Tracing at Uber](https://eng.uber.com/distributed-tracing/), where they explain the history and reasons for the architectural choices made in Jaeger. [Yuri Shkuro](https://shkuro.com), creator of Jaeger, also published a book [Mastering Distributed Tracing](https://shkuro.com/books/2019-mastering-distributed-tracing/) that covers in-depth many aspects of Jaeger design and operation, as well as distributed tracing in general.
|
||||
|
||||
## Features
|
||||
|
||||
* [OpenTracing](http://opentracing.io/) compatible data model and instrumentation libraries
|
||||
* in [Go](https://github.com/jaegertracing/jaeger-client-go), [Java](https://github.com/jaegertracing/jaeger-client-java), [Node](https://github.com/jaegertracing/jaeger-client-node), [Python](https://github.com/jaegertracing/jaeger-client-python),
|
||||
[C++](https://github.com/jaegertracing/cpp-client) and [C#](https://github.com/jaegertracing/jaeger-client-csharp)
|
||||
* Uses consistent upfront sampling with individual per service/endpoint probabilities
|
||||
* Multiple storage backends: Cassandra, Elasticsearch, memory.
|
||||
* Adaptive sampling (coming soon)
|
||||
* Post-collection data processing pipeline (coming soon)
|
||||
|
||||
See [Features](./features/) page for more details.
|
||||
|
||||
## Technical Specs
|
||||
|
||||
* Backend components implemented in Go
|
||||
* React/Javascript UI
|
||||
* Supported storage backends:
|
||||
* [Cassandra 3.4+](./deployment/#cassandra)
|
||||
* [Elasticsearch 5.x, 6.x, 7.x](./deployment/#elasticsearch)
|
||||
* [Kafka](./deployment/#kafka)
|
||||
* memory storage
|
||||
|
||||
## Quick Start
|
||||
See [running a docker all in one image](getting-started#all-in-one).
|
||||
|
||||
## Screenshots
|
||||
|
||||
### Traces View
|
||||
[](/img/traces-ss.png)
|
||||
|
||||
### Trace Detail View
|
||||
[](/img/trace-detail-ss.png)
|
||||
|
||||
## Related links
|
||||
- [Evolving Distributed tracing At Uber Engineering](https://eng.uber.com/distributed-tracing/)
|
||||
- [Mastering Distributed Tracing](https://shkuro.com/books/2019-mastering-distributed-tracing/)
|
||||
- [Tracing HTTP request latency in Go with OpenTracing](https://medium.com/opentracing/tracing-http-request-latency-in-go-with-opentracing-7cc1282a100a)
|
||||
- [Distributed Tracing with Jaeger & Prometheus on Kubernetes](https://blog.openshift.com/openshift-commons-briefing-82-distributed-tracing-with-jaeger-prometheus-on-kubernetes/)
|
||||
- [Using Jaeger with Istio](https://istio.io/docs/tasks/telemetry/distributed-tracing.html)
|
||||
- [Using Jaeger with Envoy](https://www.envoyproxy.io/docs/envoy/latest/start/sandboxes/jaeger_tracing.html)
|
||||
|
||||
[dapper]: https://research.google.com/pubs/pub36356.html
|
||||
[ubeross]: http://uber.github.io
|
||||
|
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue