istio.io/archive/v1.2/blog/2018/export-logs-through-stackdr.../index.html

<!doctype html><html lang=en itemscope itemtype=https://schema.org/WebPage><head><meta charset=utf-8><meta http-equiv=x-ua-compatible content="IE=edge"><meta name=viewport content="width=device-width,initial-scale=1,shrink-to-fit=no"><meta name=theme-color content=#466BB0><meta name=title content="Exporting Logs to BigQuery, GCS, Pub/Sub through Stackdriver"><meta name=description content="How to export Istio Access Logs to different sinks like BigQuery, GCS, Pub/Sub through Stackdriver."><meta name=author content="Nupur Garg and Douglas Reid"><meta name=keywords content=microservices,services,mesh><meta property=og:title content="Exporting Logs to BigQuery, GCS, Pub/Sub through Stackdriver"><meta property=og:type content=website><meta property=og:description content="How to export Istio Access Logs to different sinks like BigQuery, GCS, Pub/Sub through Stackdriver."><meta property=og:url content=/v1.2/blog/2018/export-logs-through-stackdriver/><meta property=og:image content=/v1.2/img/istio-whitelogo-bluebackground-framed.svg><meta property=og:image:alt content="Istio Logo"><meta property=og:image:width content=112><meta property=og:image:height content=150><meta property=og:site_name content=Istio><meta name=twitter:card content=summary><meta name=twitter:site content=@IstioMesh><title>Istioldie 1.2 / Exporting Logs to BigQuery, GCS, Pub/Sub through Stackdriver</title><script async src="https://www.googletagmanager.com/gtag/js?id=UA-98480406-2"></script><script>window.dataLayer=window.dataLayer||[];function gtag(){dataLayer.push(arguments);}
gtag('js',new Date());gtag('config','UA-98480406-2');</script><link rel=alternate type=application/rss+xml title="Istio Blog" href=/v1.2/feed.xml><link rel="shortcut icon" href=/v1.2/favicons/favicon.ico><link rel=apple-touch-icon href=/v1.2/favicons/apple-touch-icon-180x180.png sizes=180x180><link rel=icon type=image/png href=/v1.2/favicons/favicon-16x16.png sizes=16x16><link rel=icon type=image/png href=/v1.2/favicons/favicon-32x32.png sizes=32x32><link rel=icon type=image/png href=/v1.2/favicons/android-36x36.png sizes=36x36><link rel=icon type=image/png href=/v1.2/favicons/android-48x48.png sizes=48x48><link rel=icon type=image/png href=/v1.2/favicons/android-72x72.png sizes=72x72><link rel=icon type=image/png href=/v1.2/favicons/android-96x96.png sizes=96xW96><link rel=icon type=image/png href=/v1.2/favicons/android-144x144.png sizes=144x144><link rel=icon type=image/png href=/v1.2/favicons/android-192x192.png sizes=192x192><link rel=manifest href=/v1.2/manifest.json><meta name=apple-mobile-web-app-title content=Istio><meta name=application-name content=Istio><link rel=stylesheet href="https://fonts.googleapis.com/css?family=Work+Sans:400|Chivo:400|Work+Sans:500,300,600,300italic,400italic,500italic,600italic|Chivo:500,300,600,300italic,400italic,500italic,600italic"><link rel=stylesheet href=/v1.2/css/all.css><script src=/v1.2/js/themes_init.min.js></script></head><body class="language-unknown archive-site"><script>const branchName="release-1.2";const docTitle="Exporting Logs to BigQuery, GCS, Pub\/Sub through Stackdriver";const iconFile="\/v1.2/img/icons.svg";const buttonCopy='Copy to clipboard';const buttonPrint='Print';const buttonDownload='Download';</script><script src="https://www.google.com/cse/brand?form=search-form" defer></script><script src=/v1.2/js/all.min.js data-manual defer></script><header><nav><a id=brand href=/v1.2/><span class=logo><svg viewBox="0 0 300 300"><circle cx="150" cy="150" r="146" stroke-width="2" /><path d="M65 240H225L125 270z"/><path d="M65 230l60-10V110z"/><path d="M135 220l90 10L135 30z"/></svg></span><span class=name>Istioldie 1.2</span></a><div id=hamburger><svg class="icon"><use xlink:href="/v1.2/img/icons.svg#hamburger"/></svg></div><div id=header-links><a title="Learn how to deploy, use, and operate Istio." href=/v1.2/docs/>Docs</a>
<span title="Posts about using Istio.">Blog</span>
<a title="Frequently Asked Questions about Istio." href=/v1.2/faq/>FAQ</a>
<a title="Get a bit more in-depth info about the Istio project." href=/v1.2/about/>About</a><div class=menu><button id=gearDropdownButton class=menu-trigger title="Options and settings" aria-label="Options and Settings" aria-controls=gearDropdownContent><svg class="icon"><use xlink:href="/v1.2/img/icons.svg#gear"/></svg></button><div id=gearDropdownContent class=menu-content aria-labelledby=gearDropdownButton role=menu><a tabindex=-1 role=menuitem lang=en id=switch-lang-en class=active>English</a>
<a tabindex=-1 role=menuitem lang=zh id=switch-lang-zh>中文</a><div role=separator></div><a tabindex=-1 role=menuitem class=active id=light-theme-item>Light Theme</a>
<a tabindex=-1 role=menuitem id=dark-theme-item>Dark Theme</a><div role=separator></div><a tabindex=-1 role=menuitem id=syntax-coloring-item>Color Examples</a><div role=separator></div><h6>Other versions of this site</h6><a tabindex=-1 role=menuitem onclick="navigateToUrlOrRoot('https://istio.io/blog\/2018\/export-logs-through-stackdriver\/');return false;">Current Release</a>
<a tabindex=-1 role=menuitem onclick="navigateToUrlOrRoot('https://preliminary.istio.io/blog\/2018\/export-logs-through-stackdriver\/');return false;">Next Release</a>
<a tabindex=-1 role=menuitem href=https://archive.istio.io>Older Releases</a></div></div><button id=search-show title="Search this site" aria-label=Search><svg class="icon"><use xlink:href="/v1.2/img/icons.svg#magnifier"/></svg></button></div><form id=search-form name=cse role=search><input type=hidden name=cx value=013699703217164175118:iwwf17ikgf4>
<input type=hidden name=ie value=utf-8>
<input type=hidden name=hl value=en>
<input type=hidden id=search-page-url value=/v1.2/search.html>
<input id=search-textbox class=form-control name=q type=search aria-label="Search this site">
<button id=search-close title="Cancel search" type=reset aria-label="Cancel search"><svg class="icon"><use xlink:href="/v1.2/img/icons.svg#cancel-x"/></svg></button></form></nav></header><main class=primary><div id=sidebar-container class="sidebar-container sidebar-offcanvas"><nav id=sidebar aria-label="Section Navigation"><div class=directory><div class=card><button class="header dynamic" id=card0 title="Blog posts for 2019." aria-controls=card0-body><svg class="icon"><use xlink:href="/v1.2/img/icons.svg#blog"/></svg>2019 Posts</button><div class=body aria-labelledby=card0 role=region id=card0-body><ul role=tree aria-expanded=true class=leaf-section aria-labelledby=card0><li role=none><a role=treeitem title="Istio 1.1.14 patch release." href=/v1.2/blog/2019/announcing-1.1.14/>Announcing Istio 1.1.14</a></li><li role=none><a role=treeitem title="Istio 1.2.5 patch release." href=/v1.2/blog/2019/announcing-1.2.5/>Announcing Istio 1.2.5</a></li><li role=none><a role=treeitem title="Upcoming Istio 1.1 end of life announcement." href=/v1.2/blog/2019/announcing-1.1-eol/>Support for Istio 1.1 ends on September 19th, 2019</a></li><li role=none><a role=treeitem title="Istio 1.1.13 patch release." href=/v1.2/blog/2019/announcing-1.1.13/>Announcing Istio 1.1.13</a></li><li role=none><a role=treeitem title="Istio 1.2.4 patch release." href=/v1.2/blog/2019/announcing-1.2.4/>Announcing Istio 1.2.4</a></li><li role=none><a role=treeitem title="Security vulnerability disclosure for multiple CVEs." href=/v1.2/blog/2019/istio-security-003-004/>Security Update - ISTIO-SECURITY-003 and ISTIO-SECURITY-004</a></li><li role=none><a role=treeitem title="The design principles behind Istio's APIs and how those APIs are evolving." href=/v1.2/blog/2019/evolving-istios-apis/>The Evolution of Istio&#39;s APIs</a></li><li role=none><a role=treeitem title="Istio 1.1.12 patch release." href=/v1.2/blog/2019/announcing-1.1.12/>Announcing Istio 1.1.12</a></li><li role=none><a role=treeitem title="Istio 1.2.3 patch release." href=/v1.2/blog/2019/announcing-1.2.3/>Announcing Istio 1.2.3</a></li><li role=none><a role=treeitem title="Comparison of alternative solutions to control egress traffic including performance considerations." href=/v1.2/blog/2019/egress-traffic-control-in-istio-part-3/>Secure Control of Egress Traffic in Istio, part 3</a></li><li role=none><a role=treeitem title="Use Istio Egress Traffic Control to prevent attacks involving egress traffic." href=/v1.2/blog/2019/egress-traffic-control-in-istio-part-2/>Secure Control of Egress Traffic in Istio, part 2</a></li><li role=none><a role=treeitem title="Tools and guidance for evaluating Istio's data plane performance." href=/v1.2/blog/2019/performance-best-practices/>Best Practices: Benchmarking Service Mesh Performance</a></li><li role=none><a role=treeitem title="Istio 1.1.11 patch release." href=/v1.2/blog/2019/announcing-1.1.11/>Announcing Istio 1.1.11</a></li><li role=none><a role=treeitem title="Istio 1.0.9 patch release." href=/v1.2/blog/2019/announcing-1.0.9/>Announcing Istio 1.0.9</a></li><li role=none><a role=treeitem title="Istio 1.1.10 patch release." href=/v1.2/blog/2019/announcing-1.1.10/>Announcing Istio 1.1.10</a></li><li role=none><a role=treeitem title="Istio 1.2.2 patch release." href=/v1.2/blog/2019/announcing-1.2.2/>Announcing Istio 1.2.2</a></li><li role=none><a role=treeitem title="Security vulnerability disclosure for CVE-2019-12995." href=/v1.2/blog/2019/cve-2019-12995/>Security Update - CVE-2019-12995</a></li><li role=none><a role=treeitem title="Istio 1.2.1 patch release." href=/v1.2/blog/2019/announcing-1.2.1/>Announcing Istio 1.2.1</a></li><li role=none><a role=treeitem title="Istio 1.0 end of life announcement." href=/v1.2/blog/2019/announcing-1.0-eol-final/>Support for Istio 1.0 has ended</a></li><li role=none><a role=treeitem title="Istio 1.2 release announcement." href=/v1.2/blog/2019/announcing-1.2/>Announcing Istio 1.2</a></li><li role=none><a role=treeitem title="Istio 1.1.9 patch release." href=/v1.2/blog/2019/announcing-1.1.9/>Announcing Istio 1.1.9</a></li><li role=none><a role=treeitem title="Istio 1.0.8 patch release." href=/v1.2/blog/2019/announcing-1.0.8/>Announcing Istio 1.0.8</a></li><li role=none><a role=treeitem title="Learn how to extend the lifetime of Istio self-signed root certificate." href=/v1.2/blog/2019/root-transition/>Extending Istio Self-Signed Root Certificate Lifetime</a></li><li role=none><a role=treeitem title="Istio 1.1.8 patch release." href=/v1.2/blog/2019/announcing-1.1.8/>Announcing Istio 1.1.8</a></li><li role=none><a role=treeitem title="Security vulnerability disclosure for CVE-2019-12243." href=/v1.2/blog/2019/cve-2019-12243/>Security Update - CVE-2019-12243</a></li><li role=none><a role=treeitem title="Upcoming Istio 1.0 end of life announcement." href=/v1.2/blog/2019/announcing-1.0-eol/>Support for Istio 1.0 ends on June 19th, 2019</a></li><li role=none><a role=treeitem title="Attacks involving egress traffic and requirements for egress traffic control." href=/v1.2/blog/2019/egress-traffic-control-in-istio-part-1/>Secure Control of Egress Traffic in Istio, part 1</a></li><li role=none><a role=treeitem title="Istio 1.1.7 patch release." href=/v1.2/blog/2019/announcing-1.1.7/>Announcing Istio 1.1.7</a></li><li role=none><a role=treeitem title="Istio 1.1.6 patch release." href=/v1.2/blog/2019/announcing-1.1.6/>Announcing Istio 1.1.6</a></li><li role=none><a role=treeitem title="Istio 1.1.5 patch release." href=/v1.2/blog/2019/announcing-1.1.5/>Announcing Istio 1.1.5</a></li><li role=none><a role=treeitem title="Istio 1.1.4 patch release." href=/v1.2/blog/2019/announcing-1.1.4/>Announcing Istio 1.1.4</a></li><li role=none><a role=treeitem title="Istio 1.1.3 patch release." href=/v1.2/blog/2019/announcing-1.1.3/>Announcing Istio 1.1.3</a></li><li role=none><a role=treeitem title="Istio 1.0.7 patch releases." href=/v1.2/blog/2019/announcing-1.0.7/>Announcing Istio 1.0.7 with Important Security Update</a></li><li role=none><a role=treeitem title="Istio 1.1.2 patch release." href=/v1.2/blog/2019/announcing-1.1.2/>Announcing Istio 1.1.2 with Important Security Update</a></li><li role=none><a role=treeitem title="Istio 1.1.1 patch release." href=/v1.2/blog/2019/announcing-1.1.1/>Announcing Istio 1.1.1</a></li><li role=none><a role=treeitem title="Istio 1.1 release announcement." href=/v1.2/blog/2019/announcing-1.1/>Announcing Istio 1.1</a></li><li role=none><a role=treeitem title="An overview of Istio 1.1 performance." href=/v1.2/blog/2019/istio1.1_perf/>Architecting Istio 1.1 for Performance</a></li><li role=none><a role=treeitem title="Istio 1.0.6 patch release." href=/v1.2/blog/2019/announcing-1.0.6/>Announcing Istio 1.0.6</a></li><li role=none><a role=treeitem title="Configuring Istio route rules in a multicluster service mesh." href=/v1.2/blog/2019/multicluster-version-routing/>Version Routing in a Multicluster Service Mesh</a></li><li role=none><a role=treeitem title="Announces the new Istio blog policy." href=/v1.2/blog/2019/sail-the-blog/>Sail the Blog!</a></li><li role=none><a role=treeitem title="De-mystify how Istio manages to plugin its data-plane components into an existing deployment." href=/v1.2/blog/2019/data-plane-setup/>Demystifying Istio&#39;s Sidecar Injection Model</a></li><li role=none><a role=treeitem title="Verifies the performance impact of adding an egress gateway." href=/v1.2/blog/2019/egress-performance/>Egress Gateway Performance Investigation</a></li><li role=none><a role=treeitem title="Addressing application startup ordering and startup latency using AppSwitch." href=/v1.2/blog/2019/appswitch/>Sidestepping Dependency Ordering with AppSwitch</a></li><li role=none><a role=treeitem title="Istio has a new discussion board." href=/v1.2/blog/2019/announcing-discuss.istio.io/>Announcing discuss.istio.io</a></li><li role=none><a role=treeitem title="Describes how to deploy a custom ingress gateway using cert-manager manually." href=/v1.2/blog/2019/custom-ingress-gateway/>Deploy a Custom Ingress Gateway Using Cert-Manager</a></li></ul></div></div><div class=card><button class="header dynamic" id=card1 title="Blog posts for 2018." aria-controls=card1-body><svg class="icon"><use xlink:href="/v1.2/img/icons.svg#blog"/></svg>2018 Posts</button><div class="body default" aria-labelledby=card1 role=region id=card1-body><ul role=tree aria-expanded=true class=leaf-section aria-labelledby=card1><li role=none><a role=treeitem title="Istio 1.0.5 patch release." href=/v1.2/blog/2018/announcing-1.0.5/>Announcing Istio 1.0.5</a></li><li role=none><a role=treeitem title="Istio 1.0.4 patch release." href=/v1.2/blog/2018/announcing-1.0.4/>Announcing Istio 1.0.4</a></li><li role=none><a role=treeitem title="How to use Istio for traffic management without deploying sidecar proxies." href=/v1.2/blog/2018/incremental-traffic-management/>Incremental Istio Part 1, Traffic Management</a></li><li role=none><a role=treeitem title="Describes a simple scenario based on Istio's Bookinfo example." href=/v1.2/blog/2018/egress-mongo/>Consuming External MongoDB Services</a></li><li role=none><a role=treeitem title="Istio 1.0.3 patch release." href=/v1.2/blog/2018/announcing-1.0.3/>Announcing Istio 1.0.3</a></li><li role=none><a role=treeitem title="Istio 1.0.2 patch release." href=/v1.2/blog/2018/announcing-1.0.2/>Announcing Istio 1.0.2</a></li><li role=none><a role=treeitem title="Istio 1.0.1 patch release." href=/v1.2/blog/2018/announcing-1.0.1/>Announcing Istio 1.0.1</a></li><li role=none><a role=treeitem title="Istio hosting an all day Twitch stream to celebrate the 1.0 release." href=/v1.2/blog/2018/istio-twitch-stream/>All Day Istio Twitch Stream</a></li><li role=none><a role=treeitem title="Istio is ready for production use with its 1.0 release." href=/v1.2/blog/2018/announcing-1.0/>Announcing Istio 1.0</a></li><li role=none><a role=treeitem title="How HP is building its next-generation footwear personalization platform on Istio." href=/v1.2/blog/2018/hp/>Istio a Game Changer for HP&#39;s FitStation Platform</a></li><li role=none><a role=treeitem title="Automatic application onboarding and latency optimizations using AppSwitch." href=/v1.2/blog/2018/delayering-istio/>Delayering Istio with AppSwitch</a></li><li role=none><a role=treeitem title="Describe Istio's authorization feature and how to use it in various use cases." href=/v1.2/blog/2018/istio-authorization/>Micro-Segmentation with Istio Authorization</a></li><li role=none><span role=treeitem class=current title="How to export Istio Access Logs to different sinks like BigQuery, GCS, Pub/Sub through Stackdriver.">Exporting Logs to BigQuery, GCS, Pub/Sub through Stackdriver</span></li><li role=none><a role=treeitem title="Describes how to configure Istio for monitoring and access policies of HTTP egress traffic." href=/v1.2/blog/2018/egress-monitoring-access-control/>Monitoring and Access Policies for HTTP Egress Traffic</a></li><li role=none><a role=treeitem title="Introduction, motivation and design principles for the Istio v1alpha3 routing API." href=/v1.2/blog/2018/v1alpha3-routing/>Introducing the Istio v1alpha3 routing API</a></li><li role=none><a role=treeitem title="Describes how to configure Istio ingress with a network load balancer on AWS." href=/v1.2/blog/2018/aws-nlb/>Configuring Istio Ingress with AWS NLB</a></li><li role=none><a role=treeitem title="Using Kubernetes namespaces and RBAC to create an Istio soft multi-tenancy environment." href=/v1.2/blog/2018/soft-multitenancy/>Istio Soft Multi-Tenancy Support</a></li><li role=none><a role=treeitem title="An introduction to safer, lower-risk deployments and release to production." href=/v1.2/blog/2018/traffic-mirroring/>Traffic Mirroring with Istio for Testing in Production</a></li><li role=none><a role=treeitem title="Describes a simple scenario based on Istio's Bookinfo example." href=/v1.2/blog/2018/egress-tcp/>Consuming External TCP Services</a></li><li role=none><a role=treeitem title="Describes a simple scenario based on Istio's Bookinfo example." href=/v1.2/blog/2018/egress-https/>Consuming External Web Services</a></li></ul></div></div><div class=card><button class="header dynamic" id=card2 title="Blog posts for 2017." aria-controls=card2-body><svg class="icon"><use xlink:href="/v1.2/img/icons.svg#blog"/></svg>2017 Posts</button><div class=body aria-labelledby=card2 role=region id=card2-body><ul role=tree aria-expanded=true class=leaf-section aria-labelledby=card2><li role=none><a role=treeitem title="Improving availability and reducing latency." href=/v1.2/blog/2017/mixer-spof-myth/>Mixer and the SPOF Myth</a></li><li role=none><a role=treeitem title="Provides an overview of Mixer's plug-in architecture." href=/v1.2/blog/2017/adapter-model/>Mixer Adapter Model</a></li><li role=none><a role=treeitem title="Istio 0.2 announcement." href=/v1.2/blog/2017/0.2-announcement/>Announcing Istio 0.2</a></li><li role=none><a role=treeitem title="How Kubernetes Network Policy relates to Istio policy." href=/v1.2/blog/2017/0.1-using-network-policy/>Using Network Policy with Istio</a></li><li role=none><a role=treeitem title="Using Istio to create autoscaled canary deployments." href=/v1.2/blog/2017/0.1-canary/>Canary Deployments using Istio</a></li><li role=none><a role=treeitem title="Istio Auth 0.1 announcement." href=/v1.2/blog/2017/0.1-auth/>Using Istio to Improve End-to-End Security</a></li><li role=none><a role=treeitem title="Istio 0.1 announcement." href=/v1.2/blog/2017/0.1-announcement/>Introducing Istio</a></li></ul></div></div></div></nav></div><div class=article-container><button tabindex=-1 id=sidebar-toggler title="Toggle the navigation bar"><svg class="icon"><use xlink:href="/v1.2/img/icons.svg#pull"/></svg></button><nav aria-label=Breadcrumb><ol><li><a href=/v1.2/ title="Connect, secure, control, and observe services.">Istio</a></li><li><a href=/v1.2/blog/ title="Posts about using Istio.">Blog</a></li><li><a href=/v1.2/blog/2018/ title="Blog posts for 2018.">2018 Posts</a></li><li>Exporting Logs to BigQuery, GCS, Pub/Sub through Stackdriver</li></ol></nav><article aria-labelledby=title><div class=title-area><div><h1 id=title>Exporting Logs to BigQuery, GCS, Pub/Sub through Stackdriver</h1><p class=byline><span>By</span>
<span class=attribution>Nupur Garg and Douglas Reid</span><span> | </span><span><svg class="icon"><use xlink:href="/v1.2/img/icons.svg#calendar"/></svg><span>&nbsp;</span>July 9, 2018</span><span> | </span><span title="1007 words"><svg class="icon"><use xlink:href="/v1.2/img/icons.svg#clock"/></svg><span>&nbsp;</span>5 minute read</span></p></div></div><nav class=toc-inlined aria-label="Table of Contents"><div><hr><ol><li role=none aria-label="Before you begin"><a href=#before-you-begin>Before you begin</a><li role=none aria-label="Configuring Istio to export logs"><a href=#configuring-istio-to-export-logs>Configuring Istio to export logs</a><ol><li role=none aria-label="Setting up various log sinks"><a href=#setting-up-various-log-sinks>Setting up various log sinks</a><ol><li role=none aria-label=BigQuery><a href=#bigquery>BigQuery</a><li role=none aria-label="Google Cloud Storage (GCS)"><a href=#google-cloud-storage-gcs>Google Cloud Storage (GCS)</a><li role=none aria-label="Google Cloud Pub/Sub"><a href=#google-cloud-pub-sub>Google Cloud Pub/Sub</a></ol></li><li role=none aria-label="Setting up Stackdriver"><a href=#setting-up-stackdriver>Setting up Stackdriver</a></ol></li><li role=none aria-label="Understanding what happened"><a href=#understanding-what-happened>Understanding what happened</a><li role=none aria-label=Cleanup><a href=#cleanup>Cleanup</a><li role=none aria-label="Availability of logs in export sinks"><a href=#availability-of-logs-in-export-sinks>Availability of logs in export sinks</a></ol><hr></div></nav><p>This post shows how to direct Istio logs to <a href=https://cloud.google.com/stackdriver/>Stackdriver</a>
and export those logs to various configured sinks such as such as
<a href=https://cloud.google.com/bigquery/>BigQuery</a>, <a href=https://cloud.google.com/storage/>Google Cloud Storage</a>
or <a href=https://cloud.google.com/pubsub/>Cloud Pub/Sub</a>. At the end of this post you can perform
analytics on Istio data from your favorite places such as BigQuery, GCS or Cloud Pub/Sub.</p><p>The <a href=/v1.2/docs/examples/bookinfo/>Bookinfo</a> sample application is used as the example
application throughout this task.</p><h2 id=before-you-begin>Before you begin</h2><p><a href=/v1.2/docs/setup/>Install Istio</a> in your cluster and deploy an application.</p><h2 id=configuring-istio-to-export-logs>Configuring Istio to export logs</h2><p>Istio exports logs using the <code>logentry</code> <a href=/v1.2/docs/reference/config/policy-and-telemetry/templates/logentry>template</a>.
This specifies all the variables that are available for analysis. It
contains information like source service, destination service, auth
metrics (coming..) among others. Following is a diagram of the pipeline:</p><figure style=width:75%><div class=wrapper-with-intrinsic-ratio style=padding-bottom:75%><a data-skipendnotes=true href=/v1.2/blog/2018/export-logs-through-stackdriver/./istio-analytics-using-stackdriver.png title="Exporting logs from Istio to Stackdriver for analysis"><img class=element-to-stretch src=/v1.2/blog/2018/export-logs-through-stackdriver/./istio-analytics-using-stackdriver.png alt="Exporting logs from Istio to Stackdriver for analysis"></a></div><figcaption>Exporting logs from Istio to Stackdriver for analysis</figcaption></figure><p>Istio supports exporting logs to Stackdriver which can in turn be configured to export
logs to your favorite sink like BigQuery, Pub/Sub or GCS. Please follow the steps
below to setup your favorite sink for exporting logs first and then Stackdriver
in Istio.</p><h3 id=setting-up-various-log-sinks>Setting up various log sinks</h3><p>Common setup for all sinks:</p><ol><li>Enable <a href=https://cloud.google.com/monitoring/api/enable-api>Stackdriver Monitoring API</a> for the project.</li><li>Make sure <code>principalEmail</code> that would be setting up the sink has write access to the project and Logging Admin role permissions.</li><li>Make sure the <code>GOOGLE_APPLICATION_CREDENTIALS</code> environment variable is set. Please follow instructions <a href=https://cloud.google.com/docs/authentication/getting-started>here</a> to set it up.</li></ol><h4 id=bigquery>BigQuery</h4><ol><li><a href=https://cloud.google.com/bigquery/docs/datasets>Create a BigQuery dataset</a> as a destination for the logs export.</li><li>Record the ID of the dataset. It will be needed to configure the Stackdriver handler.
It would be of the form <code>bigquery.googleapis.com/projects/[PROJECT_ID]/datasets/[DATASET_ID]</code></li><li>Give <a href=https://cloud.google.com/logging/docs/api/tasks/exporting-logs#writing_to_the_destination>sink’s writer identity</a>: <code>cloud-logs@system.gserviceaccount.com</code> BigQuery Data Editor role in IAM.</li><li>If using <a href=/v1.2/docs/setup/kubernetes/platform-setup/gke/>Google Kubernetes Engine</a>, make sure <code>bigquery</code> <a href=https://cloud.google.com/sdk/gcloud/reference/container/clusters/create>Scope</a> is enabled on the cluster.</li></ol><h4 id=google-cloud-storage-gcs>Google Cloud Storage (GCS)</h4><ol><li><a href=https://cloud.google.com/storage/docs/creating-buckets>Create a GCS bucket</a> where you would like logs to get exported in GCS.</li><li>Recode the ID of the bucket. It will be needed to configure Stackdriver.
It would be of the form <code>storage.googleapis.com/[BUCKET_ID]</code></li><li>Give <a href=https://cloud.google.com/logging/docs/api/tasks/exporting-logs#writing_to_the_destination>sink’s writer identity</a>: <code>cloud-logs@system.gserviceaccount.com</code> Storage Object Creator role in IAM.</li></ol><h4 id=google-cloud-pub-sub>Google Cloud Pub/Sub</h4><ol><li><a href=https://cloud.google.com/pubsub/docs/admin>Create a topic</a> where you would like logs to get exported in Google Cloud Pub/Sub.</li><li>Recode the ID of the topic. It will be needed to configure Stackdriver.
It would be of the form <code>pubsub.googleapis.com/projects/[PROJECT_ID]/topics/[TOPIC_ID]</code></li><li>Give <a href=https://cloud.google.com/logging/docs/api/tasks/exporting-logs#writing_to_the_destination>sink’s writer identity</a>: <code>cloud-logs@system.gserviceaccount.com</code> Pub/Sub Publisher role in IAM.</li><li>If using <a href=/v1.2/docs/setup/kubernetes/platform-setup/gke/>Google Kubernetes Engine</a>, make sure <code>pubsub</code> <a href=https://cloud.google.com/sdk/gcloud/reference/container/clusters/create>Scope</a> is enabled on the cluster.</li></ol><h3 id=setting-up-stackdriver>Setting up Stackdriver</h3><p>A Stackdriver handler must be created to export data to Stackdriver. The configuration for
a Stackdriver handler is described <a href=/v1.2/docs/reference/config/policy-and-telemetry/adapters/stackdriver/>here</a>.</p><ol><li><p>Save the following yaml file as <code>stackdriver.yaml</code>. Replace <code>&lt;project_id&gt;,
&lt;sink_id&gt;, &lt;sink_destination&gt;, &lt;log_filter&gt;</code> with their specific values.</p><pre><code class=language-yaml data-expandlinks=true>apiVersion: &#34;config.istio.io/v1alpha2&#34;
kind: stackdriver
metadata:
  name: handler
  namespace: istio-system
spec:
  # We&#39;ll use the default value from the adapter, once per minute, so we don&#39;t need to supply a value.
  # pushInterval: 1m
  # Must be supplied for the Stackdriver adapter to work
  project_id: &#34;&lt;project_id&gt;&#34;
  # One of the following must be set; the preferred method is `appCredentials`, which corresponds to
  # Google Application Default Credentials.
  # If none is provided we default to app credentials.
  # appCredentials:
  # apiKey:
  # serviceAccountPath:
  # Describes how to map Istio logs into Stackdriver.
  logInfo:
    accesslog.logentry.istio-system:
      payloadTemplate: &#39;{{or (.sourceIp) &#34;-&#34;}} - {{or (.sourceUser) &#34;-&#34;}} [{{or (.timestamp.Format &#34;02/Jan/2006:15:04:05 -0700&#34;) &#34;-&#34;}}] &#34;{{or (.method) &#34;-&#34;}} {{or (.url) &#34;-&#34;}} {{or (.protocol) &#34;-&#34;}}&#34; {{or (.responseCode) &#34;-&#34;}} {{or (.responseSize) &#34;-&#34;}}&#39;
      httpMapping:
        url: url
        status: responseCode
        requestSize: requestSize
        responseSize: responseSize
        latency: latency
        localIp: sourceIp
        remoteIp: destinationIp
        method: method
        userAgent: userAgent
        referer: referer
      labelNames:
      - sourceIp
      - destinationIp
      - sourceService
      - sourceUser
      - sourceNamespace
      - destinationIp
      - destinationService
      - destinationNamespace
      - apiClaims
      - apiKey
      - protocol
      - method
      - url
      - responseCode
      - responseSize
      - requestSize
      - latency
      - connectionMtls
      - userAgent
      - responseTimestamp
      - receivedBytes
      - sentBytes
      - referer
      sinkInfo:
        id: &#39;&lt;sink_id&gt;&#39;
        destination: &#39;&lt;sink_destination&gt;&#39;
        filter: &#39;&lt;log_filter&gt;&#39;
---
apiVersion: &#34;config.istio.io/v1alpha2&#34;
kind: rule
metadata:
  name: stackdriver
  namespace: istio-system
spec:
  match: &#34;true&#34; # If omitted match is true.
  actions:
  - handler: handler.stackdriver
    instances:
    - accesslog.logentry
---
</code></pre></li><li><p>Push the configuration</p><pre><code class=language-bash data-expandlinks=true>$ kubectl apply -f stackdriver.yaml
stackdriver &#34;handler&#34; created
rule &#34;stackdriver&#34; created
logentry &#34;stackdriverglobalmr&#34; created
metric &#34;stackdriverrequestcount&#34; created
metric &#34;stackdriverrequestduration&#34; created
metric &#34;stackdriverrequestsize&#34; created
metric &#34;stackdriverresponsesize&#34; created
</code></pre></li><li><p>Send traffic to the sample application.</p><p>For the Bookinfo sample, visit <code>http://$GATEWAY_URL/productpage</code> in your web
browser or issue the following command:</p><pre><code class=language-bash data-expandlinks=true>$ curl http://$GATEWAY_URL/productpage
</code></pre></li><li><p>Verify that logs are flowing through Stackdriver to the configured sink.</p><ul><li>Stackdriver: Navigate to the <a href=https://pantheon.corp.google.com/logs/viewer>Stackdriver Logs
Viewer</a> for your project
and look under &ldquo;GKE Container&rdquo; -&gt; &ldquo;Cluster Name&rdquo; -&gt; &ldquo;Namespace Id&rdquo; for
Istio Access logs.</li><li>BigQuery: Navigate to the <a href=https://bigquery.cloud.google.com/>BigQuery
Interface</a> for your project and you
should find a table with prefix <code>accesslog_logentry_istio</code> in your sink
dataset.</li><li>GCS: Navigate to the <a href=https://pantheon.corp.google.com/storage/browser/>Storage
Browser</a> for your
project and you should find a bucket named
<code>accesslog.logentry.istio-system</code> in your sink bucket.</li><li>Pub/Sub: Navigate to the <a href=https://pantheon.corp.google.com/cloudpubsub/topicList>Pub/Sub
Topic List</a> for
your project and you should find a topic for <code>accesslog</code> in your sink
topic.</li></ul></li></ol><h2 id=understanding-what-happened>Understanding what happened</h2><p><code>Stackdriver.yaml</code> file above configured Istio to send access logs to
Stackdriver and then added a sink configuration where these logs could be
exported. In detail as follows:</p><ol><li><p>Added a handler of kind <code>stackdriver</code></p><pre><code class=language-yaml data-expandlinks=true>apiVersion: &#34;config.istio.io/v1alpha2&#34;
kind: stackdriver
metadata:
  name: handler
  namespace: &lt;your defined namespace&gt;
</code></pre></li><li><p>Added logInfo in spec</p><pre><code class=language-yaml data-expandlinks=true>spec:
  logInfo: accesslog.logentry.istio-system:
    labelNames:
    - sourceIp
    - destinationIp
    ...
    ...
    sinkInfo:
      id: &#39;&lt;sink_id&gt;&#39;
      destination: &#39;&lt;sink_destination&gt;&#39;
      filter: &#39;&lt;log_filter&gt;&#39;
</code></pre><p>In the above configuration sinkInfo contains information about the sink where you want
the logs to get exported to. For more information on how this gets filled for different sinks please refer
<a href=https://cloud.google.com/logging/docs/export/#sink-terms>here</a>.</p></li><li><p>Added a rule for Stackdriver</p><pre><code class=language-yaml data-expandlinks=true>apiVersion: &#34;config.istio.io/v1alpha2&#34;
kind: rule
metadata:
  name: stackdriver
  namespace: istio-system spec:
  match: &#34;true&#34; # If omitted match is true
actions:
- handler: handler.stackdriver
  instances:
  - accesslog.logentry
</code></pre></li></ol><h2 id=cleanup>Cleanup</h2><ul><li><p>Remove the new Stackdriver configuration:</p><pre><code class=language-bash data-expandlinks=true>$ kubectl delete -f stackdriver.yaml
</code></pre></li><li><p>If you are not planning to explore any follow-on tasks, refer to the
<a href=/v1.2/docs/examples/bookinfo/#cleanup>Bookinfo cleanup</a> instructions to shutdown
the application.</p></li></ul><h2 id=availability-of-logs-in-export-sinks>Availability of logs in export sinks</h2><p>Export to BigQuery is within minutes (we see it to be almost instant), GCS can
have a delay of 2 to 12 hours and Pub/Sub is almost immediately.</p></article><nav class=pagenav><div class=left><a title="Describe Istio's authorization feature and how to use it in various use cases." href=/v1.2/blog/2018/istio-authorization/><svg class="icon"><use xlink:href="/v1.2/img/icons.svg#left-arrow"/></svg>Micro-Segmentation with Istio Authorization</a></div><div class=right><a title="Describes how to configure Istio for monitoring and access policies of HTTP egress traffic." href=/v1.2/blog/2018/egress-monitoring-access-control/>Monitoring and Access Policies for HTTP Egress Traffic<svg class="icon"><use xlink:href="/v1.2/img/icons.svg#right-arrow"/></svg></a></div></nav><div id=endnotes-container aria-hidden=true><h2>Links</h2><ol id=endnotes></ol></div></div><div class=toc-container><nav class=toc aria-label="Table of Contents"><div id=toc><ol><li role=none aria-label="Before you begin"><a href=#before-you-begin>Before you begin</a><li role=none aria-label="Configuring Istio to export logs"><a href=#configuring-istio-to-export-logs>Configuring Istio to export logs</a><ol><li role=none aria-label="Setting up various log sinks"><a href=#setting-up-various-log-sinks>Setting up various log sinks</a><ol><li role=none aria-label=BigQuery><a href=#bigquery>BigQuery</a><li role=none aria-label="Google Cloud Storage (GCS)"><a href=#google-cloud-storage-gcs>Google Cloud Storage (GCS)</a><li role=none aria-label="Google Cloud Pub/Sub"><a href=#google-cloud-pub-sub>Google Cloud Pub/Sub</a></ol></li><li role=none aria-label="Setting up Stackdriver"><a href=#setting-up-stackdriver>Setting up Stackdriver</a></ol></li><li role=none aria-label="Understanding what happened"><a href=#understanding-what-happened>Understanding what happened</a><li role=none aria-label=Cleanup><a href=#cleanup>Cleanup</a><li role=none aria-label="Availability of logs in export sinks"><a href=#availability-of-logs-in-export-sinks>Availability of logs in export sinks</a></ol></div></nav></div></main><footer><div class=user-links><a class=channel title="Go download Istio 1.2.5 now" href=https://github.com/istio/istio/releases/tag/1.2.5 aria-label="Download Istio"><span>download</span><svg class="icon"><use xlink:href="/v1.2/img/icons.svg#download"/></svg>
</a><a class=channel title="Join the Istio discussion board to participate in discussions and get help troubleshooting problems" href=https://discuss.istio.io aria-label="Istio discussion board"><span>discuss</span><svg class="icon"><use xlink:href="/v1.2/img/icons.svg#discourse"/></svg></a>
<a class=channel title="Stack Overflow is where you can ask questions and find curated answers on deploying, configuring, and using Istio" href=https://stackoverflow.com/questions/tagged/istio aria-label="Stack Overflow"><span>stack overflow</span><svg class="icon"><use xlink:href="/v1.2/img/icons.svg#stackoverflow"/></svg></a>
<a class=channel title="Interactively discuss issues with the Istio community on Slack" href=https://istio.slack.com aria-label=slack><span>slack</span><svg class="icon"><use xlink:href="/v1.2/img/icons.svg#slack"/></svg></a>
<a class=channel title="Follow us on Twitter to get the latest news" href=https://twitter.com/IstioMesh aria-label=Twitter><span>twitter</span><svg class="icon"><use xlink:href="/v1.2/img/icons.svg#twitter"/></svg></a><div class=tag>for everyone</div></div><div class=info><p class=copyright>Istio Archive
1.2.5<br>&copy; 2019 Istio Authors, <a href=https://policies.google.com/privacy>Privacy Policy</a><br>Archived on September 12, 2019</p></div><div class=dev-links><a class=channel title="GitHub is where development takes place on Istio code" href=https://github.com/istio/community aria-label=GitHub><span>github</span><svg class="icon"><use xlink:href="/v1.2/img/icons.svg#github"/></svg></a>
<a class=channel title="Access our team drive if you'd like to take a look at the Istio technical design documents" href=https://groups.google.com/forum/#!forum/istio-team-drive-access aria-label="team drive"><span>drive</span><svg class="icon"><use xlink:href="/v1.2/img/icons.svg#drive"/></svg></a>
<a class=channel title="If you'd like to contribute to the Istio project, consider participating in our working groups" href=https://github.com/istio/community/blob/master/WORKING-GROUPS.md aria-label="working groups"><span>working groups</span><svg class="icon"><use xlink:href="/v1.2/img/icons.svg#working-groups"/></svg></a><div class=tag>for developers</div></div></footer><div id=scroll-to-top-container aria-hidden=true><button id=scroll-to-top title="Back to top"><svg class="icon"><use xlink:href="/v1.2/img/icons.svg#top"/></svg></button></div></body></html>