linkerd
/
linkerd2
mirror of https://github.com/linkerd/linkerd2.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

viz: add `linkerd viz allow-scrapes` command (#9182) 

Depends on #9169.

This branch adds a `linkerd viz allow-scrapes` command, which outputs
policy resources to authorize Prometheus scrapes for data plane pods
which have the `config.linkerd.io/default-inbound-policy: deny`
annotation. The output resources can be piped into `kubectl apply` to
create the policy resources to authorize scrapes.

The `viz check --proxy` check for default-deny pods added in PR #9169 is
updated so that it doesn't fail for default-deny pods if the
authorization policy generated by `allow-scrapes` exists in their
namespace.

For example, after installing emojivoto in a namespace with the
default-deny annotation, the check outputs the following:

```console
$ bin/linkerd viz check --proxy
linkerd-viz
-----------
√ linkerd-viz Namespace exists
√ linkerd-viz ClusterRoles exist
√ linkerd-viz ClusterRoleBindings exist
√ tap API server has valid cert
√ tap API server cert is valid for at least 60 days
√ tap API service is running
√ linkerd-viz pods are injected
√ viz extension pods are running
√ viz extension proxies are healthy
‼ viz extension proxies are up-to-date
    some proxies are not running the current version:
	* metrics-api-65d85d8b84-h56gk (dev-f7784aab-eliza)
	* tap-846c68c954-2gnh7 (dev-f7784aab-eliza)
	* tap-injector-98b9cd747-2gv2z (dev-f7784aab-eliza)
	* web-64dd8c9574-qwtzg (dev-f7784aab-eliza)
	* prometheus-d647db77f-lrnbb (dev-f7784aab-eliza)
    see https://linkerd.io/2/checks/#l5d-viz-proxy-cp-version for hints
‼ viz extension proxies and cli versions match
    metrics-api-65d85d8b84-h56gk running dev-f7784aab-eliza but cli running git-2b51812e
    see https://linkerd.io/2/checks/#l5d-viz-proxy-cli-version for hints
√ prometheus is installed and configured correctly
√ can initialize the client
√ viz extension self-check

linkerd-viz-data-plane
----------------------
√ data plane namespace exists
‼ prometheus is authorized to scrape data plane pods
    prometheus may not be authorized to scrape the following pods:
	* emojivoto/emoji-699d77c79-77w7f
	* emojivoto/voting-55d76f4bcb-6lsml
	* emojivoto/web-6c54d9554d-md2sd
	* emojivoto/vote-bot-b57689ffb-fq8t5
    see https://linkerd.io/2/checks/#l5d-viz-data-plane-prom-authz for hints
/ waiting for check to complete ^C
```

Running the allow-scrapes command to generate the authorization policy:
```console

$ bin/linkerd viz allow-scrapes -n emojivoto | kubectl apply -f -
server.policy.linkerd.io/proxy-admin created
httproute.policy.linkerd.io/proxy-metrics created
httproute.policy.linkerd.io/proxy-probes created
authorizationpolicy.policy.linkerd.io/prometheus-scrape created
authorizationpolicy.policy.linkerd.io/proxy-probes created
```

Now, `viz check --proxy` succeeds:

```console
$ bin/linkerd viz check --proxy
linkerd-viz
-----------
√ linkerd-viz Namespace exists
√ linkerd-viz ClusterRoles exist
√ linkerd-viz ClusterRoleBindings exist
√ tap API server has valid cert
√ tap API server cert is valid for at least 60 days
√ tap API service is running
√ linkerd-viz pods are injected
√ viz extension pods are running
√ viz extension proxies are healthy
‼ viz extension proxies are up-to-date
    some proxies are not running the current version:
	* metrics-api-65d85d8b84-h56gk (dev-f7784aab-eliza)
	* tap-846c68c954-2gnh7 (dev-f7784aab-eliza)
	* tap-injector-98b9cd747-2gv2z (dev-f7784aab-eliza)
	* web-64dd8c9574-qwtzg (dev-f7784aab-eliza)
	* prometheus-d647db77f-lrnbb (dev-f7784aab-eliza)
    see https://linkerd.io/2/checks/#l5d-viz-proxy-cp-version for hints
‼ viz extension proxies and cli versions match
    metrics-api-65d85d8b84-h56gk running dev-f7784aab-eliza but cli running git-65fc04b0
    see https://linkerd.io/2/checks/#l5d-viz-proxy-cli-version for hints
√ prometheus is installed and configured correctly
√ can initialize the client
√ viz extension self-check

linkerd-viz-data-plane
----------------------
√ data plane namespace exists
√ prometheus is authorized to scrape data plane pods
√ data plane proxy metrics are present in Prometheus

Status check results are √
```

The implementation of the `allow-scrapes` command is very simple; I
considered implementing it using Helm, but that felt like overkill.
Instead, we just embed the policy resources as a string literal, and
use Go's `text/template` package to fill in the `linkerd-viz` version
(for `created-by` annotations), the target namespace, and so on.

Fixes #9150
This commit is contained in:
Eliza Weisman 2022-08-19 11:02:54 -07:00 committed by GitHub
parent cdeca1c40d
commit 6921218256
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
4 changed files with 229 additions and 8 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

148
viz/cmd/allow-scrapes.go Normal file
View File

@ -0,0 +1,148 @@
package cmd
import (
"os"
"text/template"
pkgcmd "github.com/linkerd/linkerd2/pkg/cmd"
"github.com/linkerd/linkerd2/pkg/version"
"github.com/spf13/cobra"
)
const (
allowScrapePolicy = `---
apiVersion: policy.linkerd.io/v1beta1
kind: Server
metadata:
name: proxy-admin
namespace: {{ .TargetNs }}
annotations:
linkerd-io/created-by: {{ .ChartName }} {{ .Version }}
labels:
linkerd.io/extension: {{ .ExtensionName }}
spec:
podSelector:
matchExpressions:
- key: linkerd.io/control-plane-ns
operator: Exists
port: linkerd-admin
proxyProtocol: HTTP/1
---
apiVersion: policy.linkerd.io/v1alpha1
kind: HTTPRoute
metadata:
name: proxy-metrics
namespace: {{ .TargetNs }}
annotations:
linkerd-io/created-by: {{ .ChartName }} {{ .Version }}
labels:
linkerd.io/extension: {{ .ExtensionName }}
spec:
parentRefs:
- name: proxy-admin
kind: Server
group: policy.linkerd.io
rules:
- matches:
- path:
value: "/metrics"
---
apiVersion: policy.linkerd.io/v1alpha1
kind: HTTPRoute
metadata:
name: proxy-probes
namespace: {{ .TargetNs }}
annotations:
linkerd-io/created-by: {{ .ChartName }} {{ .Version }}
labels:
linkerd.io/extension: {{ .ExtensionName }}
spec:
parentRefs:
- name: proxy-admin
kind: Server
group: policy.linkerd.io
rules:
- matches:
- path:
value: "/live"
- path:
value: "/ready"
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata:
name: prometheus-scrape
namespace: {{ .TargetNs }}
annotations:
linkerd-io/created-by: {{ .ChartName }} {{ .Version }}
labels:
linkerd.io/extension: {{ .ExtensionName }}
spec:
targetRef:
group: policy.linkerd.io
kind: HTTPRoute
name: proxy-metrics
requiredAuthenticationRefs:
- kind: ServiceAccount
name: prometheus
namespace: {{ .VizNs }}
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata:
name: proxy-probes
namespace: {{ .TargetNs }}
annotations:
linkerd-io/created-by: {{ .ChartName }} {{ .Version }}
labels:
linkerd.io/extension: {{ .ExtensionName }}
spec:
targetRef:
group: policy.linkerd.io
kind: HTTPRoute
name: proxy-probes
requiredAuthenticationRefs:
- kind: NetworkAuthentication
group: policy.linkerd.io
name: kubelet
namespace: {{ .VizNs }}`
)
type templateOptions struct {
ChartName string
Version string
ExtensionName string
VizNs string
TargetNs string
}
// newCmdAllowScrapes creates a new cobra command `allow-scrapes`
func newCmdAllowScrapes() *cobra.Command {
options := templateOptions{
ExtensionName: ExtensionName,
ChartName: vizChartName,
Version: version.Version,
VizNs: defaultNamespace,
}
cmd := &cobra.Command{
Use: "allow-scrapes {-n | --namespace } namespace",
Short: "Output Kubernetes resources to authorize Prometheus scrapes",
Long: `Output Kubernetes resources to authorize Prometheus scrapes in a namespace or cluster with config.linkerd.io/default-inbound-policy: deny.`,
Example: `# Allow scrapes in the 'emojivoto' namespace
linkerd viz allow-scrapes --namespace emojivoto | kubectl apply -f -`,
Args: cobra.NoArgs,
PreRunE: func(cmd *cobra.Command, args []string) error {
return cmd.MarkFlagRequired("namespace")
},
RunE: func(cmd *cobra.Command, args []string) error {
t := template.Must(template.New("allow-scrapes").Parse(allowScrapePolicy))
return t.Execute(os.Stdout, options)
},
}
cmd.Flags().StringVarP(&options.TargetNs, "namespace", "n", options.TargetNs, "The namespace in which to authorize Prometheus scrapes.")
pkgcmd.ConfigureNamespaceFlagCompletion(
cmd, []string{"n", "namespace"},
kubeconfigPath, impersonate, impersonateGroup, kubeContext)
return cmd
}

1
viz/cmd/root.go
View File

@ -89,6 +89,7 @@ func NewCmdViz() *cobra.Command {
vizCmd.AddCommand(NewCmdTap())
vizCmd.AddCommand(NewCmdTop())
vizCmd.AddCommand(newCmdUninstall())
vizCmd.AddCommand(newCmdAllowScrapes())
// resource-aware completion flag configurations
pkgcmd.ConfigureNamespaceFlagCompletion(

58
viz/cmd/uninstall.go
View File

@ -7,7 +7,10 @@ import (
pkgCmd "github.com/linkerd/linkerd2/pkg/cmd"
"github.com/linkerd/linkerd2/pkg/k8s"
"github.com/linkerd/linkerd2/pkg/k8s/resource"
"github.com/spf13/cobra"
v1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
func newCmdUninstall() *cobra.Command {
@ -43,5 +46,58 @@ func uninstallRunE(ctx context.Context) error {
return err
}
return pkgCmd.Uninstall(ctx, k8sAPI, selector)
// / `Uninstall` deletes cluster-scoped resources created by the extension
// (including the extension's namespace).
if err := pkgCmd.Uninstall(ctx, k8sAPI, selector); err != nil {
return err
}
// delete any HTTPRoute, AuthorizationPolicy, and Server resources created
// by the viz extension in any namespace.
//
// note that these are not deleted by the `Uninstall` call above, because
// they are namespaced resources.
policy := k8sAPI.L5dCrdClient.PolicyV1alpha1()
authzs, err := policy.AuthorizationPolicies(v1.NamespaceAll).List(ctx, metav1.ListOptions{LabelSelector: selector})
if err != nil {
return err
}
for _, authz := range authzs.Items {
if err := deleteResource(authz.TypeMeta, authz.ObjectMeta); err != nil {
return err
}
}
rts, err := policy.HTTPRoutes(v1.NamespaceAll).List(ctx, metav1.ListOptions{LabelSelector: selector})
if err != nil {
return err
}
for _, rt := range rts.Items {
if err := deleteResource(rt.TypeMeta, rt.ObjectMeta); err != nil {
return err
}
}
srvs, err := k8sAPI.L5dCrdClient.ServerV1beta1().Servers(v1.NamespaceAll).List(ctx, metav1.ListOptions{LabelSelector: selector})
if err != nil {
return err
}
for _, srv := range srvs.Items {
if err := deleteResource(srv.TypeMeta, srv.ObjectMeta); err != nil {
return err
}
}
return nil
}
func deleteResource(ty metav1.TypeMeta, meta metav1.ObjectMeta) error {
r := resource.NewNamespaced(ty.APIVersion, ty.Kind, meta.Name, meta.Namespace)
if err := r.RenderResource(os.Stdout); err != nil {
return fmt.Errorf("error rendering Kubernetes resource: %w", err)
}
return nil
}

30
viz/pkg/healthcheck/healthcheck.go
View File

@ -390,15 +390,29 @@ func fetchTapCaBundle(ctx context.Context, kubeAPI *k8s.KubernetesAPI) ([]*x509.
}
func (hc *HealthChecker) checkPromAuthorized(ctx context.Context) error {
nses, err := hc.getDataPlaneNamespaces(ctx)
api := hc.KubeAPIClient()
nses, err := hc.getDataPlaneNamespaces(ctx, api)
if err != nil {
return err
}
unauthorizedPods := []string{}
for _, ns := range nses {
// TODO(eliza): check if this namespace has an allow-scrapes config once
// that's implemented; if it has one, skip checking its pods.
// first, let's see if this namespace has an `allow-scrapes` policy. if
// it does, skip checking its pods --- prometheus will be able to scrape
// them even if they are default-deny.
_, err := api.L5dCrdClient.PolicyV1alpha1().AuthorizationPolicies(ns.GetName()).Get(ctx, "prometheus-scrape", metav1.GetOptions{})
if kerrors.IsNotFound(err) {
// no prometheus-scrape policy exists in this namespace
} else if err != nil {
// something went wrong while talking to the kube API
return fmt.Errorf("could not get AuthorizationPolicies in the %s namespace: %w", ns.GetName(), err)
} else {
// allow-scrapes policy exists in this namespace, don't check the
// pods.
continue
}
pods, err := hc.KubeAPIClient().GetPodsByNamespace(ctx, ns.GetName())
if err != nil {
return fmt.Errorf("could not list pods in the %s namespace: %w", ns.GetName(), err)
@ -438,22 +452,24 @@ func (hc *HealthChecker) checkPromAuthorized(ctx context.Context) error {
if len(unauthorizedPods) > 0 {
podList := strings.Join(unauthorizedPods, "\n")
return fmt.Errorf("prometheus may not be authorized to scrape the following pods:\n%s", podList)
return fmt.Errorf("prometheus may not be authorized to scrape the following pods:\n%s\n"+
" consider running `linkerd viz allow-scrapes` to authorize prometheus scrapes",
podList)
}
return nil
}
func (hc *HealthChecker) getDataPlaneNamespaces(ctx context.Context) ([]corev1.Namespace, error) {
func (hc *HealthChecker) getDataPlaneNamespaces(ctx context.Context, api *k8s.KubernetesAPI) ([]corev1.Namespace, error) {
if hc.DataPlaneNamespace != "" {
ns, err := hc.KubeAPIClient().GetNamespace(ctx, hc.DataPlaneNamespace)
ns, err := api.GetNamespace(ctx, hc.DataPlaneNamespace)
if err != nil {
return nil, err
}
return []corev1.Namespace{*ns}, nil
}
nses, err := hc.KubeAPIClient().CoreV1().Namespaces().List(ctx, metav1.ListOptions{})
nses, err := api.CoreV1().Namespaces().List(ctx, metav1.ListOptions{})
if err != nil {
return nil, err
}