docker
/
docs
mirror of https://github.com/docker/docs.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Review DTR job runner

This commit is contained in:
Joao Fernandes 2017-03-24 14:29:54 -07:00 committed by Joao Fernandes
parent 2fcee427e3
commit 9c46463e7c
4 changed files with 403 additions and 267 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
_data/toc.yaml
View File

@ -1336,8 +1336,6 @@ manuals:
title: Chain multiple caches
- path: /datacenter/dtr/2.2/guides/admin/configure/garbage-collection/
title: Garbage collection
- path: /datacenter/dtr/2.2/guides/admin/configure/jobrunner/
title: Jobrunner
- sectiontitle: Manage users
section:
- path: /datacenter/dtr/2.2/guides/admin/manage-users/
@ -1354,6 +1352,8 @@ manuals:
title: Monitor the cluster status
- path: /datacenter/dtr/2.2/guides/admin/monitor-and-troubleshoot/troubleshoot-with-logs/
title: Troubleshoot with logs
- path: /datacenter/dtr/2.2/guides/admin/monitor-and-troubleshoot/troubleshoot-batch-jobs/
title: Troubleshoot batch jobs
- path: /datacenter/dtr/2.2/guides/admin/backups-and-disaster-recovery/
title: Backups and disaster recovery
- sectiontitle: CLI reference

265
datacenter/dtr/2.2/guides/admin/configure/jobrunner.md
View File

@ -1,265 +0,0 @@
---
title: Jobrunner
description: Learn about the inner-workings of the jobrunner container in the DTR workflow.
keywords: docker, job, runner
---
The jobrunner container is a DTR mechanism that:
1. Consumes jobs from a cluster-wide jobs queue
2. Performs the work of the given action
There is one jobrunner container per replica.
## Jobrunner Workflow
[//]: # (uncomment once diagrams are complete) The following diagram depicts the behavior of the jobrunner:
[//]: # (Placeholder for jobrunner diagram. @sarahpark will work on the diagram)
When a job is scheduled (see [Job Actions](#job-actions) below) it is put onto the
cluster-wide jobs queue with an initial status of `waiting`. When a jobrunner worker
is available to pick up the job, it will claim the it (i.e: the workerID will be set
to the replicaID of the jobrunner container that claimed the job) and set the job
status to `running`. The worker will carry out the job and then set the appropriate
status when complete (see [Job Statuses](#job-statuses) below).
[//]: # (Placeholder for jobrunner scheduling. @sarahpark will work on the diagram)
Each jobrunner has an internal queue of `waiting` jobs sorted by their `scheduledAt`
time (from earliest to latest). When a worker is free to claim the next job, it claims
it after a delay of up to 3 seconds. This delay is imposed so that each available worker
has a chance to compete for the job. The worker that was successfully able to claim the
job will set the job's `workerID` and all other workers will drop the job from their
internal queue. If a job cannot be claimed due to capacity limits (see [Job Capacities](#job-capacities))
then it is placed into a separate queue and will go through the claiming process
when the worker has enough free capacity for it.
Jobrunners monitor each other's `heartbeatExpiration` in the workers table. When a worker
see that another worker hasn't updated its expiration in a long time, it sets the
dead worker's status to `dead` and its jobs to `worker_dead`. If the dead worker is able to
reconnect to the database and notices that it's jobs have been set to `worker_dead`,
it sets those job statuses to `worker_resurrection` and cancels them.
### Job Actions
The available job actions are:
- `gc`
: Garbage collection deletes layers associated with deleted images.
- `sleep`
: Sleep is used to test the correctness of the jobrunner. It sleeps for 60 seconds.
- `false`
: False is used to test the correctness of the jobrunner. It runs the `false` command and immediately fails.
- `tagmigration`
: Tag migration is used to sync tag and manifest information from the blobstore into the database.
This information is used to for information in the API, UI, and also for GC.
- `bloblinkmigration`
: bloblinkmigration is a 2.1 to 2.1 upgrade process that adds references for blobs to repositories in the database.
- `license_update`
: License update checks for license expiration extensions if online license updates are enabled.
- `nautilus_scan_check`
: An image security scanning job. This job does not perform the actual scanning, rather it
spawns `nautilus_scan_check_single` jobs (one for each layer in the image). Once
all of the `nautilus_scan_check_single` jobs are complete, this job will terminate.
- `nautilus_scan_check_single`
: A security scanning job for a particular layer given by the `parameter: SHA256SUM`. This job
breaks up the layer into components and checks each component for vulnerabilities
(see [Security Scanning](../../user/manage-images/scan-images-for-vulnerabilities.md)).
- `nautilus_update_db`
: A job that is created to update DTR's vulnerability database. It uses an
Internet connection to check for database updates through `https://dss-cve-updates.docker.com/` and
updates the dtr-scanningstore container if there is a new update available (see [Set up vulnerability scans](set-up-vulnerability-scans.md)).
- `webhook`
: A job that is used to dispatch a webhook payload to a single endpoint
#### Job Capacities
As mentioned above, each jobrunner container acts as one worker that can carry out these actions.
The number of a particular action a worker can carry out is defined by it's capacity which can be
seen in the `GET /api/v0/workers` endpoint. For example the workers entry may look like this:
```json
{
"workers": [
{
"id": "000000000000",
"status": "running",
"capacityMap": {
"scan": 1,
"scanCheck": 1
},
"heartbeatExpiration": "2017-02-18T00:51:02Z"
}
]
}
```
This means that the worker with the replica ID `000000000000` has a capacity of 1 `scan` and 1
`scanCheck`. A job may have a `capacityMap` field which dictates how much capacity a worker
must have available for the job to be executed.
For example, if we take the above worker's `capacityMap` and the following jobs:
```json
{
"jobs": [
{
"id": "0",
"workerID": "",
"status": "waiting",
"capacityMap": {
"scan": 1
}
},
{
"id": "1",
"workerID": "",
"status": "waiting",
"capacityMap": {
"scan": 1
}
},
{
"id": "2",
"workerID": "",
"status": "waiting",
"capacityMap": {
"scanCheck": 1
}
}
]
}
```
Our worker will be able to pick up job id `0` and `2` since it has the capacity for both,
while id `1` will have to wait until the previous scan job is complete:
```json
{
"jobs": [
{
"id": "0",
"workerID": "000000000000",
"status": "running",
"capacityMap": {
"scan": 1
}
},
{
"id": "1",
"workerID": "",
"status": "waiting",
"capacityMap": {
"scan": 1
}
},
{
"id": "2",
"workerID": "000000000000",
"status": "running",
"capacityMap": {
"scanCheck": 1
}
}
]
}
```
## Job Statuses
Jobs can have the following statuses:
- `waiting`
: the job is unclaimed and waiting to be picked up by a worker
- `running`
: the worker defined by `workerID` is currently running the job
- `done`
: the job has successfully completed
- `error`
: the job has completed with errors
- `cancel_request`
: the worker monitors the job statuses in the database. If the status for a job changes
to `cancel_request`, the worker will cancel the job
- `cancel`
: the job has been cancelled and not fully executed
- `deleted`
: the job and logs have been removed
- `worker_dead`
: the worker for this job has been declared `dead` and the job will not continue
- `worker_shutdown`
: the worker that was running this job has been gracefully stopped
- `worker_resurrection`
: the worker for this job has reconnected to the database and will cancel these jobs
## Troubleshooting a Job
An entry for a job can look like:
```json
{
"id": "1fcf4c0f-ff3b-471a-8839-5dcb631b2f7b",
"retryFromID": "1fcf4c0f-ff3b-471a-8839-5dcb631b2f7b",
"workerID": "000000000000",
"status": "done",
"scheduledAt": "2017-02-17T01:09:47.771Z",
"lastUpdated": "2017-02-17T01:10:14.117Z",
"action": "nautilus_scan_check_single",
"retriesLeft": 0,
"retriesTotal": 0,
"capacityMap": {
"scan": 1
},
"parameters": {
"SHA256SUM": "1bacd3c8ccb1f15609a10bd4a403831d0ec0b354438ddbf644c95c5d54f8eb13"
},
"deadline": "",
"stopTimeout": ""
}
```
The fields of interest here are:
- `id`: the ID of the job itself
- `workerID`: the ID of the jobrunner worker (synonymous with the DTR replica ID) that is running this job
- `status`: the current state of the job (see [Job Statuses](#job-statuses))
- `action`: what job the worker will actually perform (see [Job Actions](#job-actions))
- `capacityMap`: the available "capacity" a worker needs for this job to run (see [Job Capacities](#job-capacities))
You can view the logs of a particular job by hitting the `GET /api/v0/jobs/{jobID}/logs` endpoint
with the job's `id` as `{jobID}`.
## Cron jobs
Several of the jobs listed in [Job Actions](#job-actions) have been set to run on a
recurring schedule. You can view these jobs with the `GET /api/v0/crons` endpoint which
will return a list similar to this example:
```json
{
"crons": [
{
"id": "48875b1b-5006-48f5-9f3c-af9fbdd82255",
"action": "license_update",
"schedule": "57 54 3 * * *",
"retries": 2,
"capacityMap": null,
"parameters": null,
"deadline": "",
"stopTimeout": "",
"nextRun": "2017-02-22T03:54:57Z"
},
{
"id": "b1c1e61e-1e74-4677-8e4a-2a7dacefffdc",
"action": "nautilus_update_db",
"schedule": "0 0 3 * * *",
"retries": 0,
"capacityMap": null,
"parameters": null,
"deadline": "",
"stopTimeout": "",
"nextRun": "2017-02-22T03:00:00Z"
}
]
}
```
The `schedule` is simlar to the style of a typical Unix crontab format:
`"second minute hour day month year"`. This determines the next time the `action` will
take place.

231
datacenter/dtr/2.2/guides/admin/monitor-and-troubleshoot/troubleshoot-batch-jobs.md Normal file
View File

@ -0,0 +1,231 @@
---
title: Troubleshoot batch jobs
description: Learn how Docker Trusted Registry run batch jobs, so that you can troubleshoot when something goes wrong
keywords: docker, dtr, troubleshoot
---
DTR uses a job queue to schedule batch jobs. A job is placed on this work queue,
and a job runner component of DTR consumes work from this cluster-wide job
queue and executes it.
![batch jobs diagram](../../images/troubleshoot-batch-jobs-1.svg)
All DTR replicas have access to the job queue, and have a job runner component
that can get and execute work.
## How it works
When a job is created, it is added to a cluster-wide job queue with the
`waiting` status.
When one of the DTR replicas is ready to claim, it waits a random time of up
to 3 seconds, giving the opportunity to every replica to claim the task.
A replica gets a job by adding it's replica ID to the job. That way, other
replicas know the job has been claimed. Once a replica claims a job it adds
it to an internal queue of jobs, that is sorted by their `scheduledAt` time.
When that time happens, the replica updates the job status to `running`, and
starts executing it.
The job runner component of each DTR replica keeps an `heartbeatExpiration`
entry on the database shared by all replicas. If a replica becomes
unhealthy, other replicas notice this and update that worker status to `dead`.
Also, all the jobs that replica had claimed are updated to the status `worker_dead`,
so that other replicas can claim the job.
## Job types
DTR has several types of jobs.
| Job | Description |
|:---------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| gc | Garbage collection job that deletes layers associated with deleted images |
| sleep | Sleep is used to test the correctness of the jobrunner. It sleeps for 60 seconds |
| false | False is used to test the correctness of the jobrunner. It runs the `false` command and immediately fails |
| tagmigration | Tag migration is used to sync tag and manifest information from the blobstore into the database. This information is used to for information in the API, UI, and also for GC |
| bloblinkmigration | bloblinkmigration is a 2.1 to 2.1 upgrade process that adds references for blobs to repositories in the database |
| license_update | License update checks for license expiration extensions if online license updates are enabled |
| nautilus_scan_check | An image security scanning job. This job does not perform the actual scanning, rather it spawns `nautilus_scan_check_single` jobs (one for each layer in the image). Once all of the `nautilus_scan_check_single` jobs are complete, this job will terminate |
| nautilus_scan_check_single | A security scanning job for a particular layer given by the `parameter: SHA256SUM`. This job breaks up the layer into components and checks each component for vulnerabilities |
| nautilus_update_db | A job that is created to update DTR's vulnerability database. It uses an Internet connection to check for database updates through `https://dss-cve-updates.docker.com/` and updates the `dtr-scanningstore` container if there is a new update available |
| webhook | A job that is used to dispatch a webhook payload to a single endpoint |
## Job status
Jobs can be in one of the following status:
| Status | Description |
|:----------------|:------------------------------------------------------------------------------------------------------------------------------------------|
| waiting | The job is unclaimed and waiting to be picked up by a worker |
| running | The worker defined by `workerID` is currently running the job |
| done | The job has successfully completed |
| error | The job has completed with errors |
| cancel_request | The worker monitors the job statuses in the database. If the status for a job changes to `cancel_request`, the worker will cancel the job |
| cancel | The job has been cancelled and not fully executed |
| deleted | The job and logs have been removed |
| worker_dead | The worker for this job has been declared `dead` and the job will not continue |
| worker_shutdown | The worker that was running this job has been gracefully stopped |
| worker | resurrection| The worker for this job has reconnected to the database and will cancel these jobs |
## Job capacity
Each job runner has a limited capacity and won't claim jobs that require an
higher capacity. You can see the capacity of a job runner using the
`GET /api/v0/workers` endpoint:
```json
{
"workers": [
{
"id": "000000000000",
"status": "running",
"capacityMap": {
"scan": 1,
"scanCheck": 1
},
"heartbeatExpiration": "2017-02-18T00:51:02Z"
}
]
}
```
This means that the worker with replica ID `000000000000` has a capacity of 1
`scan` and 1 `scanCheck`. If this worker notices that the following jobs
are available:
```json
{
"jobs": [
{
"id": "0",
"workerID": "",
"status": "waiting",
"capacityMap": {
"scan": 1
}
},
{
"id": "1",
"workerID": "",
"status": "waiting",
"capacityMap": {
"scan": 1
}
},
{
"id": "2",
"workerID": "",
"status": "waiting",
"capacityMap": {
"scanCheck": 1
}
}
]
}
```
Our worker will be able to pick up job id `0` and `2` since it has the capacity
for both, while id `1` will have to wait until the previous scan job is complete:
```json
{
"jobs": [
{
"id": "0",
"workerID": "000000000000",
"status": "running",
"capacityMap": {
"scan": 1
}
},
{
"id": "1",
"workerID": "",
"status": "waiting",
"capacityMap": {
"scan": 1
}
},
{
"id": "2",
"workerID": "000000000000",
"status": "running",
"capacityMap": {
"scanCheck": 1
}
}
]
}
```
## Troubleshoot jobs
You can get the list of jobs, using the `GET /api/v0/jobs/` endpoint. Each job
looks like this:
```json
{
"id": "1fcf4c0f-ff3b-471a-8839-5dcb631b2f7b",
"retryFromID": "1fcf4c0f-ff3b-471a-8839-5dcb631b2f7b",
"workerID": "000000000000",
"status": "done",
"scheduledAt": "2017-02-17T01:09:47.771Z",
"lastUpdated": "2017-02-17T01:10:14.117Z",
"action": "nautilus_scan_check_single",
"retriesLeft": 0,
"retriesTotal": 0,
"capacityMap": {
"scan": 1
},
"parameters": {
"SHA256SUM": "1bacd3c8ccb1f15609a10bd4a403831d0ec0b354438ddbf644c95c5d54f8eb13"
},
"deadline": "",
"stopTimeout": ""
}
```
The fields of interest here are:
* `id`: the ID of the job
* `workerID`: the ID of the worker in a DTR replica that is running this job
* `status`: the current state of the job
* `action`: what job the worker will actually perform
* `capacityMap`: the available capacity a worker needs for this job to run
## Cron jobs
Several of the jobs performed by DTR are run in a recurrent schedule. You can
see those jobs using the `GET /api/v0/crons` endpoint:
```json
{
"crons": [
{
"id": "48875b1b-5006-48f5-9f3c-af9fbdd82255",
"action": "license_update",
"schedule": "57 54 3 * * *",
"retries": 2,
"capacityMap": null,
"parameters": null,
"deadline": "",
"stopTimeout": "",
"nextRun": "2017-02-22T03:54:57Z"
},
{
"id": "b1c1e61e-1e74-4677-8e4a-2a7dacefffdc",
"action": "nautilus_update_db",
"schedule": "0 0 3 * * *",
"retries": 0,
"capacityMap": null,
"parameters": null,
"deadline": "",
"stopTimeout": "",
"nextRun": "2017-02-22T03:00:00Z"
}
]
}
```
The `schedule` uses a Unix crontab syntax.

170
datacenter/dtr/2.2/guides/images/troubleshoot-batch-jobs-1.svg Normal file
View File

@ -0,0 +1,170 @@
<?xml version="1.0" encoding="UTF-8"?>
<svg width="740px" height="250px" viewBox="0 0 740 250" version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<!-- Generator: Sketch 42 (36781) - http://www.bohemiancoding.com/sketch -->
<title>troubleshoot-batch-jobs</title>
<desc>Created with Sketch.</desc>
<defs>
<circle id="path-1" cx="4" cy="4" r="4"></circle>
<mask id="mask-2" maskContentUnits="userSpaceOnUse" maskUnits="objectBoundingBox" x="-2" y="-2" width="12" height="12">
<rect x="-2" y="-2" width="12" height="12" fill="white"></rect>
<use xlink:href="#path-1" fill="black"></use>
</mask>
<circle id="path-3" cx="4" cy="4" r="4"></circle>
<mask id="mask-4" maskContentUnits="userSpaceOnUse" maskUnits="objectBoundingBox" x="-2" y="-2" width="12" height="12">
<rect x="-2" y="-2" width="12" height="12" fill="white"></rect>
<use xlink:href="#path-3" fill="black"></use>
</mask>
<circle id="path-5" cx="4" cy="4" r="4"></circle>
<mask id="mask-6" maskContentUnits="userSpaceOnUse" maskUnits="objectBoundingBox" x="-2" y="-2" width="12" height="12">
<rect x="-2" y="-2" width="12" height="12" fill="white"></rect>
<use xlink:href="#path-5" fill="black"></use>
</mask>
<rect id="path-7" x="0" y="0" width="63" height="22" rx="2"></rect>
<filter x="-50%" y="-50%" width="200%" height="200%" filterUnits="objectBoundingBox" id="filter-8">
<feOffset dx="1" dy="1" in="SourceAlpha" result="shadowOffsetOuter1"></feOffset>
<feGaussianBlur stdDeviation="2.5" in="shadowOffsetOuter1" result="shadowBlurOuter1"></feGaussianBlur>
<feColorMatrix values="0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.5 0" type="matrix" in="shadowBlurOuter1"></feColorMatrix>
</filter>
<rect id="path-9" x="0" y="0" width="63" height="22" rx="2"></rect>
<filter x="-50%" y="-50%" width="200%" height="200%" filterUnits="objectBoundingBox" id="filter-10">
<feOffset dx="1" dy="1" in="SourceAlpha" result="shadowOffsetOuter1"></feOffset>
<feGaussianBlur stdDeviation="2.5" in="shadowOffsetOuter1" result="shadowBlurOuter1"></feGaussianBlur>
<feColorMatrix values="0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.5 0" type="matrix" in="shadowBlurOuter1"></feColorMatrix>
</filter>
<rect id="path-11" x="0" y="0" width="63" height="22" rx="2"></rect>
<filter x="-50%" y="-50%" width="200%" height="200%" filterUnits="objectBoundingBox" id="filter-12">
<feOffset dx="1" dy="1" in="SourceAlpha" result="shadowOffsetOuter1"></feOffset>
<feGaussianBlur stdDeviation="2.5" in="shadowOffsetOuter1" result="shadowBlurOuter1"></feGaussianBlur>
<feColorMatrix values="0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.5 0" type="matrix" in="shadowBlurOuter1"></feColorMatrix>
</filter>
</defs>
<g id="dtr-diagrams" stroke="none" stroke-width="1" fill="none" fill-rule="evenodd">
<g id="troubleshoot-batch-jobs">
<g id="group" transform="translate(154.000000, 3.000000)">
<g id="dtr">
<text id="DTR-cluster" font-family="OpenSans-Semibold, Open Sans" font-size="10" font-weight="500" fill="#82949E">
<tspan x="7.025" y="236.009524">DTR cluster</tspan>
</text>
<g id="network" transform="translate(33.000000, 178.000000)">
<rect id="Rectangle-138" fill="#439FD1" x="0" y="0" width="366" height="22" rx="2"></rect>
<text id="job-queue" font-family="OpenSans, Open Sans" font-size="10" font-weight="normal" fill="#FFFFFF">
<tspan x="160.040527" y="15">job queue</tspan>
</text>
<g id="work" transform="translate(6.000000, 5.000000)" fill="#FFFFFF">
<rect id="Rectangle-Copy-5" fill-opacity="0.05" x="70" y="0" width="12" height="12"></rect>
<rect id="Rectangle-Copy-4" fill-opacity="0.2" x="56" y="0" width="12" height="12"></rect>
<rect id="Rectangle-Copy-3" fill-opacity="0.4" x="42" y="0" width="12" height="12"></rect>
<rect id="Rectangle-Copy-2" fill-opacity="0.6" x="28" y="0" width="12" height="12"></rect>
<rect id="Rectangle-Copy" fill-opacity="0.8" x="14" y="0" width="12" height="12"></rect>
<rect id="Rectangle" x="0" y="0" width="12" height="12"></rect>
</g>
</g>
<g id="arrows" transform="translate(104.000000, 147.000000)">
<g id="arrow-copy-2" transform="translate(218.500000, 17.000000) rotate(-90.000000) translate(-218.500000, -17.000000) translate(202.000000, 13.000000)">
<path d="M2,4 L33,4" id="Line" stroke="#439FD1" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"></path>
<g id="Oval">
<use fill="#439FD1" fill-rule="evenodd" xlink:href="#path-1"></use>
<use stroke="#F7F8F9" mask="url(#mask-2)" stroke-width="4" xlink:href="#path-1"></use>
</g>
</g>
<g id="arrow-copy-3" transform="translate(111.500000, 17.000000) rotate(-90.000000) translate(-111.500000, -17.000000) translate(95.000000, 13.000000)">
<path d="M2,4 L33,4" id="Line" stroke="#439FD1" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"></path>
<g id="Oval">
<use fill="#439FD1" fill-rule="evenodd" xlink:href="#path-3"></use>
<use stroke="#F7F8F9" mask="url(#mask-4)" stroke-width="4" xlink:href="#path-3"></use>
</g>
</g>
<g id="arrow-copy-4" transform="translate(4.500000, 17.000000) rotate(-90.000000) translate(-4.500000, -17.000000) translate(-12.000000, 13.000000)">
<path d="M2,4 L33,4" id="Line" stroke="#439FD1" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"></path>
<g id="Oval">
<use fill="#439FD1" fill-rule="evenodd" xlink:href="#path-5"></use>
<use stroke="#F7F8F9" mask="url(#mask-6)" stroke-width="4" xlink:href="#path-5"></use>
</g>
</g>
</g>
<g id="nodes" transform="translate(59.000000, 45.000000)">
<g id="node-3" transform="translate(216.000000, 0.000000)">
<g id="node" transform="translate(1.000000, 0.000000)">
<g id="node-label">
<path d="M0,2.00295631 C0,0.896754086 0.897702336,0 1.99174577,0 L71,0 L71,10.6452381 C71,16.5244408 66.2312425,21.2904762 60.3513837,21.2904762 L0,21.2904762 L0,2.00295631 Z" id="Rectangle-127" fill="#445D6E"></path>
<text id="worker-node" font-family="OpenSans, Open Sans" font-size="8" font-weight="normal" fill="#FFFFFF">
<tspan x="6" y="14">worker node</tspan>
</text>
</g>
</g>
<g id="engine" transform="translate(2.000000, 43.000000)">
<rect id="Rectangle-138" fill="#1488C6" x="0" y="0" width="95" height="58" rx="2"></rect>
<text id="DTR" font-family="OpenSans, Open Sans" font-size="10" font-weight="normal" fill="#FFFFFF">
<tspan x="38.4980469" y="51">DTR</tspan>
</text>
</g>
<g id="dtr" transform="translate(0.000000, 43.000000)">
<g id="Rectangle-138">
<use fill="black" fill-opacity="1" filter="url(#filter-8)" xlink:href="#path-7"></use>
<use fill="#439FD1" fill-rule="evenodd" xlink:href="#path-7"></use>
</g>
<text id="Job-runner" font-family="OpenSans, Open Sans" font-size="10" font-weight="normal" fill="#FFFFFF">
<tspan x="7.18798828" y="15">Job runner</tspan>
</text>
</g>
<rect id="node-border" stroke="#445D6E" stroke-width="2" x="1" y="0" width="97" height="102" rx="2"></rect>
</g>
<g id="node-2" transform="translate(108.000000, 0.000000)">
<g id="node" transform="translate(1.000000, 0.000000)">
<g id="node-label">
<path d="M0,2.00295631 C0,0.896754086 0.897702336,0 1.99174577,0 L71,0 L71,10.6452381 C71,16.5244408 66.2312425,21.2904762 60.3513837,21.2904762 L0,21.2904762 L0,2.00295631 Z" id="Rectangle-127" fill="#445D6E"></path>
<text id="worker-node" font-family="OpenSans, Open Sans" font-size="8" font-weight="normal" fill="#FFFFFF">
<tspan x="6" y="14">worker node</tspan>
</text>
</g>
</g>
<g id="engine" transform="translate(2.000000, 43.000000)">
<rect id="Rectangle-138" fill="#1488C6" x="0" y="0" width="95" height="58" rx="2"></rect>
<text id="DTR" font-family="OpenSans, Open Sans" font-size="10" font-weight="normal" fill="#FFFFFF">
<tspan x="38.4980469" y="51">DTR</tspan>
</text>
</g>
<g id="dtr" transform="translate(0.000000, 43.000000)">
<g id="Rectangle-138">
<use fill="black" fill-opacity="1" filter="url(#filter-10)" xlink:href="#path-9"></use>
<use fill="#439FD1" fill-rule="evenodd" xlink:href="#path-9"></use>
</g>
<text id="Job-runner" font-family="OpenSans, Open Sans" font-size="10" font-weight="normal" fill="#FFFFFF">
<tspan x="7.18798828" y="15">Job runner</tspan>
</text>
</g>
<rect id="node-border" stroke="#445D6E" stroke-width="2" x="1" y="0" width="97" height="102" rx="2"></rect>
</g>
<g id="node-1">
<g id="node" transform="translate(1.000000, 0.000000)">
<g id="node-label">
<path d="M0,2.00295631 C0,0.896754086 0.897702336,0 1.99174577,0 L71,0 L71,10.6452381 C71,16.5244408 66.2312425,21.2904762 60.3513837,21.2904762 L0,21.2904762 L0,2.00295631 Z" id="Rectangle-127" fill="#445D6E"></path>
<text id="worker-node" font-family="OpenSans, Open Sans" font-size="8" font-weight="normal" fill="#FFFFFF">
<tspan x="6" y="14">worker node</tspan>
</text>
</g>
</g>
<g id="engine" transform="translate(2.000000, 43.000000)">
<rect id="Rectangle-138" fill="#1488C6" x="0" y="0" width="95" height="58" rx="2"></rect>
<text id="DTR" font-family="OpenSans, Open Sans" font-size="10" font-weight="normal" fill="#FFFFFF">
<tspan x="38.4980469" y="51">DTR</tspan>
</text>
</g>
<g id="dtr" transform="translate(0.000000, 43.000000)">
<g id="Rectangle-138">
<use fill="black" fill-opacity="1" filter="url(#filter-12)" xlink:href="#path-11"></use>
<use fill="#439FD1" fill-rule="evenodd" xlink:href="#path-11"></use>
</g>
<text id="Job-runner" font-family="OpenSans, Open Sans" font-size="10" font-weight="normal" fill="#FFFFFF">
<tspan x="7.18798828" y="15">Job runner</tspan>
</text>
</g>
<rect id="node-border" stroke="#445D6E" stroke-width="2" x="1" y="0" width="97" height="102" rx="2"></rect>
</g>
</g>
<rect id="group" stroke="#82949E" stroke-width="2" stroke-dasharray="5,5,5,5" x="0" y="0" width="433" height="245" rx="2"></rect>
</g>
</g>
</g>
</g>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 13 KiB