QPOptions (#5169)

* Adding Queue Proxy Extensions Documentations

* Create queue-extensions.md

* Update deployment.md

* Update queue-extensions.md

* Create queue-runtime-extensions.md

* Update queue-extensions.md

* Update deployment.md

* Update deployment.md

* Update queue-extensions.md

* Update queue-extensions.md

* updates after qpoptions commit merged

* lint..

* fix tab based on mkdocs output

* add nav

* fix links

* Review comments

* Review comments2

* Update docs/serving/services/using-queue-extensions.md

Co-authored-by: Samia Nneji <snneji@vmware.com>

* Update docs/serving/services/using-queue-extensions.md

Co-authored-by: Samia Nneji <snneji@vmware.com>

* Update docs/serving/services/using-queue-extensions.md

Co-authored-by: Samia Nneji <snneji@vmware.com>

* Update docs/serving/services/using-queue-extensions.md

Co-authored-by: Samia Nneji <snneji@vmware.com>

* Update docs/serving/services/using-queue-extensions.md

Co-authored-by: Samia Nneji <snneji@vmware.com>

* Update docs/serving/queue-extensions.md

Co-authored-by: Samia Nneji <snneji@vmware.com>

* Update docs/serving/queue-extensions.md

Co-authored-by: Samia Nneji <snneji@vmware.com>

* Update docs/serving/queue-extensions.md

Co-authored-by: Samia Nneji <snneji@vmware.com>

* Align QPOptions and Using extensions...

* Update docs/serving/queue-extensions.md

Co-authored-by: Samia Nneji <snneji@vmware.com>

* Fixed Prerequisites

* added description on qpoptions to using*

Co-authored-by: Samia Nneji <snneji@vmware.com>

config/nav.yml

@@ -103,6 +103,7 @@ nav:
           - Configuring certificate class: serving/services/certificate-class.md
           - Configuring custom domains: serving/services/custom-domains.md
           - Using a custom TLS certificate for DomainMapping: serving/services/byo-certificate.md
+          - Using extensions enabled by QPOptions: serving/services/using-queue-extensions.md
           # TODO: Add security section to docs?
           - Configure resource requests and limits: serving/services/configure-requests-limits-services.md
           - HTTPS redirection: serving/services/http-protocol.md
@@ -126,6 +127,7 @@ nav:
         - Configuring the ingress gateway: serving/setting-up-custom-ingress-gateway.md
         - Configuring domain names: serving/using-a-custom-domain.md
         - Converting a Kubernetes Deployment to a Knative Service: serving/convert-deployment-to-knative-service.md
+        - Extending Queue Proxy image with QPOptions: serving/queue-extensions.md
         # Serving config
         - Serving configuration:
           - Configure Deployment resources: serving/configuration/deployment.md

docs/serving/configuration/feature-flags.md

@@ -295,13 +295,13 @@ When this extension is `enabled`, the server always runs this validation.
 
 When this extension is `allowed`, the server does not run this validation by default.
 
-When this extension is `allowed`, you can run this validation for individual Services, by adding the `features.knative.dev/podspec-dryrun":"enabled"` annotation:
+When this extension is `allowed`, you can run this validation for individual Services, by adding the `features.knative.dev/podspec-dryrun: enabled` annotation:
 
 ```yaml
 apiVersion: serving.knative.dev/v1
 kind: Service
 metadata:
-  annotations: features.knative.dev/podspec-dryrun":"enabled"
+  annotations: features.knative.dev/podspec-dryrun: enabled
 ...
 ```
 
@@ -415,3 +415,24 @@ spec:
           command: ['sh', '-c', "service_setup.sh"]
 ...
 ```
+
+### Queue Proxy Pod Info
+
+* **Type**: Extension
+* **ConfigMap key:** `queueproxy.mount-podinfo`
+
+You must set this feature to either "enabled or "allowed" when using QPOptions. The flag controls whether Knative mounts the `pod-info` volume to the `queue-proxy` container.
+
+Mounting the `pod-info` volume allows extensions that use QPOptions to access the Service annotations, by reading the `/etc/podinfo/annnotations` file. See [Extending Queue Proxy image with QPOptions](../queue-extensions.md) for more details.
+
+When this feature is `enabled`, the `pod-info` volume is always mounted. This is helpful in case where all or most of the cluster Services are required to use extensions that rely on QPOptions.
+
+When this feature is `allowed`, the `pod-info` volume is not mounted by default. Instead, the volume is mounted only for Services that add the `features.knative.dev/queueproxy-podinfo: enabled` annotation as shown below:
+
+```yaml
+apiVersion: serving.knative.dev/v1
+kind: Service
+metadata:
+  annotations: features.knative.dev/queueproxy-podinfo: enabled
+...
+```

docs/serving/queue-extensions.md

@@ -0,0 +1,40 @@
+# Extending Queue Proxy image with QPOptions
+
+Knative service pods include two containers:
+
+- The user main service container, which is named `user-container`
+- The Queue Proxy - a sidecar named `queue-proxy` that serves as a reverse proxy in front of the `user-container`
+
+You can extend Queue Proxy to offer additional features. The QPOptions feature of Queue Proxy allows additional runtime packages to extend Queue Proxy capabilities.
+
+For example, the [security-guard](https://knative.dev/security-guard/README.md) repository provides an extension that uses the QPOptions feature. The [QPOption](https://knative.dev/security-guard/pkg/qpoption/README.md) package enables users to add additional security features to Queue Proxy.
+
+The runtime features available are determined when the Queue Proxy image is built. Queue Proxy defines an orderly manner to activate and to configure extensions.
+
+## Additional information
+
+- [Enabling Queue Proxy Pod Info](./configuration/feature-flags.md#queue-proxy-pod-info) - discussing a necessary step to enable the use of extensions.
+- [Using extensions enabled by QPOptions](./services/using-queue-extensions.md) - discussing how to configure a service to use features implemented in extensions.
+
+## Adding extensions
+
+You can add extensions by replacing the `cmd/queue/main.go` file before the Queue Proxy image is built. The following example shows a `cmd/queue/main.go` file that adds the `test-gate` extension:
+
+```go
+  package main
+
+  import "os"
+
+  import "knative.dev/serving/pkg/queue/sharedmain"
+  import "knative.dev/security-guard/pkg/qpoption"
+  import _ "knative.dev/security-guard/pkg/test-gate"
+
+  func main() {
+      qOpt := qpoption.NewQPSecurityPlugs()
+      defer qOpt.Shutdown()
+
+        if sharedmain.Main(qOpt.Setup) != nil {
+          os.Exit(1)
+      }
+  }
+```

docs/serving/services/using-queue-extensions.md

@@ -0,0 +1,82 @@
+# Using extensions enabled by QPOptions
+
+QPOptions is a Queue Proxy feature that enables extending Queue Proxy with additional Go packages. For example, the [security-guard](https://knative.dev/security-guard/README.md) repository extends Queue Proxy by adding runtime security features to protect user services.
+
+Once your cluster is setup with extensions enabled by QPOptions, a Service can decide which extensions it wish to use and how to configure such extensions. Activating and configuring extensions is described here.
+
+## Overview
+
+A Service can activate and configure extensions by adding `qpoption.knative.dev/*` annotations under the: `spec.template.metadata` of the Service Custom Resource Definition (CRD).
+
+Setting a value of: `qpoption.knative.dev/<ExtensionName>-activate: "enable"` activates the extension.
+
+Setting a value of: `qpoption.knative.dev/<extension-name>-config-<key>: "<value>"` adds a configuration of `key: value` to the extension.
+
+In addition, the Service must ensure that the Pod Info volume is mounted by adding the `features.knative.dev/queueproxy-podinfo: enabled` annotation under the: `spec.template.metadata` of the Service CRD.
+
+You can create a Knative Service by applying a YAML file or by using the `kn service create` CLI command.
+
+## Prerequisites
+
+Before you can use extensions enabled by QPOptions, you must:
+
+- Prepare your cluster:
+  - Make sure you are using a Queue Proxy image that was built with the extensions that you wish to use - See [Extending Queue Proxy image with QPOptions](../queue-extensions.md).
+  - Make sure that the cluster config-features is set with `queueproxy.mount-podinfo: allowed`. See [Enabling Queue Proxy Pod Info](../configuration/feature-flags.md#queue-proxy-pod-info) for more details.
+- Meet the prerequisites in [Creating a Service](./creating-services.md)
+
+## Procedure
+
+!!! tip
+
+    The following commands create a `helloworld-go` sample Service while activating and configuring the `test-gate` extension for this Service. You can modify these commands, including the extension(s) to be activated and the extension configuration.
+
+Create a sample Service:
+
+
+=== "Apply YAML"
+
+    1. Create a YAML file using the following example:
+
+        ```yaml
+        apiVersion: serving.knative.dev/v1
+        kind: Service
+        metadata:
+          name: helloworld-go
+          namespace: default
+        spec:
+          template:
+            metadata:
+                annotations:
+                  features.knative.dev/queueproxy-podinfo: enabled
+                  qpoption.knative.dev/testgate-activate: enable
+                  qpoption.knative.dev/testgate-config-response: CU
+                  qpoption.knative.dev/testgate-config-sender: Joe
+            spec:
+              containers:
+                - image: gcr.io/knative-samples/helloworld-go
+                  env:
+                    - name: TARGET
+                      value: "World"
+        ```
+
+    1. Apply the YAML file by running the command:
+
+        ```bash
+        kubectl apply -f <filename>.yaml
+        ```
+        Where `<filename>` is the name of the file you created in the previous step.
+
+=== "kn CLI"
+
+    ```
+    kn service create helloworld-go \
+        --image gcr.io/knative-samples/helloworld-go \
+        --env TARGET=World \
+        --annotation features.knative.dev/queueproxy-podinfo=enabled \
+        --annotation qpoption.knative.dev/testgate-activate=enable \
+        --annotation qpoption.knative.dev/testgate-config-response=Goodbye \
+        --annotation qpoption.knative.dev/testgate-config-sender=Joe
+    ```
+
+After the Service has been created, Knative propagates the annotations to the podSpec of the Service deployment. When a Service pod is created, the Queue Proxy sidecar will mount a volume that contains the pod annotations and activate the `testgate` extension. This occurs if the `testgate` extension is available in the Queue Proxy image. The `testgate` extension will then be configured with the configuration: `{ sender: "Joe", response: "CU"}`.