kubeflow
/
manifests
mirror of https://github.com/kubeflow/manifests.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Document secure Kserve authentication via automated tests (#3056) 

* update the kserve tests

Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com>

* Secure KServe endpoints with oauth2-proxy authentication

This change applies oauth2-proxy authentication to the cluster-local-gateway,
ensuring KServe inference endpoints require proper authentication.
Also adds a predictor-specific AuthorizationPolicy for test workflows.

Fixes #2811

Signed-off-by: madmecodes <ayushguptadev1@gmail.com>

* Fix KServe workflows: use consistent paths, namespace handling, and add wait steps

Signed-off-by: madmecodes <ayushguptadev1@gmail.com>

* Fix Fix KServe auth workflow by ordering components correctly

Signed-off-by: madmecodes <ayushguptadev1@gmail.com>

* Fix order of KF Profile creation after multi-tenancy installation

The Profile CRD needs to be installed via multi-tenancy components
before attempting to create a user profile. This ensures the
kubeflow-user-example-com namespace is properly created for tests.

Signed-off-by: madmecodes <ayushguptadev1@gmail.com>

* test: namespace manual creation Update kserve_m2m_test.yaml workflow

Signed-off-by: madmecodes <ayushguptadev1@gmail.com>

* Update kserve_m2m_test.yaml

Signed-off-by: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com>

* Update kserve_m2m_test.yaml

Signed-off-by: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com>

* Update kserve_m2m_test.yaml

Signed-off-by: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com>

* Update kserve_m2m_test.yaml

Signed-off-by: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com>

* Update kserve_m2m_test.yaml

Signed-off-by: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com>

* Update kserve_m2m_test.yaml

Signed-off-by: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com>

* Update requirements.txt

Signed-off-by: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com>

* update: attempt to Enable secure KServe inferencing with oauth2-proxy authentication

Signed-off-by: madmecodes <ayushguptadev1@gmail.com>

* enable istio-cni

Signed-off-by: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com>

* Update kserve_m2m_test.yaml

Signed-off-by: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com>

* Update dex_oauth2-proxy_test.yaml

Signed-off-by: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com>

* Delete tests/gh-actions/deploy-dex-login-environment/kustomization.yaml

Signed-off-by: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com>

* Update dex_oauth2-proxy_test.yaml

Signed-off-by: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com>

* Update dex_oauth2-proxy_test.yaml

Signed-off-by: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com>

* Update dex_oauth2-proxy_test.yaml

Signed-off-by: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com>

* Update centraldashboard_test.yaml

Signed-off-by: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com>

* Update: Istio-cni-1-24 authorizationpolicy to use custom oauth2-proxy in cluster-local-gateway

Signed-off-by: madmecodes <ayushguptadev1@gmail.com>

* Update: kserve_m2m_test.yaml attacker namespace test

Signed-off-by: madmecodes <ayushguptadev1@gmail.com>

* Update: KServe AuthorizationPolicy

Signed-off-by: madmecodes <ayushguptadev1@gmail.com>

* Update: using old KServe AuthorizationPolicy

Signed-off-by: madmecodes <ayushguptadev1@gmail.com>

* Update: Kserve Auth policy namespace access

Signed-off-by: madmecodes <ayushguptadev1@gmail.com>

* Update: fix the label in Auth Policy

Signed-off-by: madmecodes <ayushguptadev1@gmail.com>

* Update: test, kserve_m2m_test using principals not namespaces

Signed-off-by: madmecodes <ayushguptadev1@gmail.com>

* Update kserve_m2m_test.yaml

Signed-off-by: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com>

* Update kserve_m2m_test.yaml

Signed-off-by: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com>

---------

Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com>
Signed-off-by: madmecodes <ayushguptadev1@gmail.com>
Signed-off-by: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com>
Co-authored-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com>
This commit is contained in:
Ayush Gupta 2025-03-26 00:32:59 +05:30 committed by GitHub
parent 5b20e21b6d
commit 7f8837f9e9
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
8 changed files with 232 additions and 100 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

11
.github/workflows/centraldashboard_test.yaml vendored
View File

@ -5,7 +5,7 @@ on:
- tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh
- .github/workflows/centraldashboard_test.yaml
- apps/centraldashboard/upstream/**
- tests/gh-actions/install_istio.sh
- tests/gh-actions/install_istio*.sh
- common/istio*/**
jobs:
@ -21,9 +21,10 @@ jobs:
- name: Install Istio
run: ./tests/gh-actions/install_istio.sh
- name: Build & Apply manifests
- name: Create kubeflow namespace
run: kustomize build common/kubeflow-namespace/base | kubectl apply -f -
- name: Install central-dashboard
run: |
cd apps/centraldashboard/upstream
kubectl create ns kubeflow
kustomize build overlays/kserve | kubectl apply -f -
kustomize build apps/centraldashboard/upstream/overlays/kserve | kubectl apply -f -
kubectl wait --for=condition=Ready pods --all --all-namespaces --timeout 180s

19
.github/workflows/dex_oauth2-proxy_test.yaml vendored
View File

@ -9,7 +9,7 @@ on:
- common/istio*/**
- experimental/security/PSS/*
- common/dex/base/**
- tests/gh-actions/install_istio.sh
- tests/gh-actions/install_istio*.sh
jobs:
build:
@ -47,11 +47,24 @@ jobs:
echo "Waiting for pods in auth namespace to become ready..."
kubectl wait --for=condition=ready pods --all --timeout=180s -n auth
- name: Build & Apply manifests
- name: Install central-dashboard
run: |
while ! kustomize build ./tests/gh-actions/deploy-dex-login-environment | kubectl apply -f -; do echo "Retrying to apply resources"; sleep 20; done
kustomize build apps/centraldashboard/upstream/overlays/kserve | kubectl apply -f -
kubectl wait --for=condition=Ready pods --all --all-namespaces --timeout 180s
- name: Create KF Profile
run: |
kustomize build common/user-namespace/base | kubectl apply -f -
sleep 30 # for the Profile controller to create the namespace from the profile
PROFILE_CONTROLLER_POD=$(kubectl get pods -n kubeflow -o json | jq -r '.items[] | select(.metadata.name | startswith("profiles-deployment")) | .metadata.name')
if [[ -z "$PROFILE_CONTROLLER_POD" ]]; then
echo "Error: profiles-deployment pod not found in kubeflow namespace."
exit 1
fi
kubectl logs -n kubeflow "$PROFILE_CONTROLLER_POD"
KF_PROFILE=kubeflow-user-example-com
kubectl -n $KF_PROFILE get pods,configmaps,secrets
- name: port forward
run: |
ingress_gateway_service=$(kubectl get svc --namespace istio-system --selector="app=istio-ingressgateway" --output jsonpath='{.items[0].metadata.name}')

220
.github/workflows/kserve_m2m_test.yaml vendored
View File

@ -7,7 +7,7 @@ on:
- apps/kserve/**
- tests/gh-actions/install_kserve.sh
- common/istio*/**
- tests/gh-actions/install_istio.sh
- tests/gh-actions/install_istio*.sh
- common/oauth2-proxy/**
- tests/gh-actions/install_oauth2-proxy.sh
- common/cert-manager/**
@ -15,7 +15,6 @@ on:
- common/knative/**
- tests/gh-actions/install_knative.sh
jobs:
build:
runs-on: ubuntu-latest
@ -29,11 +28,8 @@ jobs:
- name: Install kubectl
run: ./tests/gh-actions/install_kubectl.sh
- name: Create kubeflow namespace
run: kustomize build common/kubeflow-namespace/base | kubectl apply -f -
- name: Install Istio
run: ./tests/gh-actions/install_istio.sh
- name: Install Istio CNI
run: ./tests/gh-actions/install_istio-cni.sh
- name: Install oauth2-proxy
run: ./tests/gh-actions/install_oauth2-proxy.sh
@ -41,14 +37,119 @@ jobs:
- name: Install cert-manager
run: ./tests/gh-actions/install_cert_manager.sh
- name: Install knative
run: ./tests/gh-actions/install_knative.sh
- name: Create kubeflow namespace
run: kustomize build common/kubeflow-namespace/base | kubectl apply -f -
- name: Install knative CNI
run: ./tests/gh-actions/install_knative-cni.sh
- name: Install KServe
run: ./tests/gh-actions/install_kserve.sh
- name: Create test namespace # TODO to be removed and instead we shall use kubeflow-user-example-com
run: kubectl create ns kserve-test
- name: Install KF Multi Tenancy
run: ./tests/gh-actions/install_multi_tenancy.sh
- name: Install kubeflow-istio-resources
run: kustomize build common/istio-1-24/kubeflow-istio-resources/base | kubectl apply -f -
- name: Create KF Profile
run: |
kustomize build common/user-namespace/base | kubectl apply -f -
sleep 30 # for the Profile controller to create the namespace from the profile
PROFILE_CONTROLLER_POD=$(kubectl get pods -n kubeflow -o json | jq -r '.items[] | select(.metadata.name | startswith("profiles-deployment")) | .metadata.name')
if [[ -z "$PROFILE_CONTROLLER_POD" ]]; then
echo "Error: profiles-deployment pod not found in kubeflow namespace."
exit 1
fi
kubectl logs -n kubeflow "$PROFILE_CONTROLLER_POD"
KF_PROFILE=kubeflow-user-example-com
kubectl -n $KF_PROFILE get pods,configmaps,secrets
- name: Diagnose KServe Service Labels
run: |
echo "=== KServe Predictor Service Labels ==="
kubectl get pods -n kubeflow-user-example-com -l serving.knative.dev/service=isvc-sklearn-predictor-default --show-labels
# TODO for follow up PR
#- name: Apply KServe predictor AuthorizationPolicy
# run: |
# cat <<EOF | kubectl apply -f -
# apiVersion: security.istio.io/v1beta1
# kind: AuthorizationPolicy
# metadata:
# name: sklearn-iris-predictor-allow
# namespace: kubeflow-user-example-com
# spec:
# selector:
# matchLabels:
# serving.knative.dev/service: isvc-sklearn-predictor
# action: ALLOW
# rules:
# - from:
# - source:
# namespaces:
# - "istio-system"
# - "knative-serving"
# - "kubeflow"
# - "kubeflow-user-example-com"
# - principals:
# - "cluster.local/ns/kubeflow-user-example-com/sa/default-editor"
# - "cluster.local/ns/kubeflow-user-example-com/sa/default"
# - "cluster.local/ns/kubeflow-user-example-com/sa/default-viewer"
# to:
# - operation:
# paths:
# - "/v1/models/*"
# - "/v2/models/*"
# EOF
- name: Apply INSECURE KServe AuthorizationPolicy
run: |
cat <<EOF | kubectl apply -f -
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: allow-in-cluster-kserve
namespace: kubeflow-user-example-com
spec:
rules:
- to:
- operation:
paths:
- /v1/models/*
- /v2/models/*
EOF
- name: Add KServe path-based routing for external access
run: |
cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: isvc-sklearn-external
namespace: kubeflow-user-example-com
spec:
gateways:
- kubeflow/kubeflow-gateway
hosts:
- '*'
http:
- match:
- uri:
prefix: /kserve/kubeflow-user-example-com/isvc-sklearn/
rewrite:
uri: /
route:
- destination:
host: knative-local-gateway.istio-system.svc.cluster.local
headers:
request:
set:
Host: isvc-sklearn-predictor-default.kubeflow-user-example-com.svc.cluster.local
weight: 100
timeout: 300s
EOF
- name: Setup python 3.12
uses: actions/setup-python@v4
@ -64,26 +165,97 @@ jobs:
nohup kubectl port-forward --namespace istio-system svc/${INGRESS_GATEWAY_SERVICE} 8080:80 &
while ! curl localhost:8080; do echo waiting for port-forwarding; sleep 1; done; echo port-forwarding ready
- name: Run kserve tests with m2m token from SA default/default # TODO Run kserve tests with m2m token from SA kubeflow-user-example-com/default-editor
- name: Verify CNI is being used instead of init containers
run: |
INIT_CONTAINERS=$(kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{range .spec.initContainers[*]}{.name}{"\n"}{end}{"\n"}{end}' | grep istio-init || echo "none")
if [[ "$INIT_CONTAINERS" != "none" ]]; then
echo "Error: Found istio-init containers, CNI is not being used properly"
echo "$INIT_CONTAINERS"
exit 1
else
echo "Success: No istio-init containers found, CNI is working correctly"
fi
- name: Run kserve tests with m2m token from SA kubeflow-user-example-com/default-editor
run: |
# TODO run the tests against kubeflow-user-example-com
export KSERVE_INGRESS_HOST_PORT=localhost:8080
export KSERVE_M2M_TOKEN="$(kubectl -n default create token default)"
# TODO export KSERVE_M2M_TOKEN="$(kubectl -n kubeflow-user-example-com create token default-editor)"
# TODO in contrib/kserve/tests/utils.py use KSERVE_TEST_NAMESPACE = "kubeflow-user-example-com"
export KSERVE_M2M_TOKEN="$(kubectl -n kubeflow-user-example-com create token default-editor)"
cd ./apps/kserve/tests && pytest . -vs --log-level info
- name: Run and fail kserve tests without kserve m2m token
- name: Detailed KServe Access Diagnostics
run: |
export KSERVE_INGRESS_HOST_PORT=localhost:8080
cd ./apps/kserve/tests
if pytest . -vs --log-level info; then
echo "This test should fail with an HTTP redirect to oauth2-proxy/dex auth."; exit 1
else
echo "Task failed successfully!"
echo "This is a provisional way of testing that m2m is enabled for kserve."
fi
export KSERVE_M2M_TOKEN="$(kubectl -n kubeflow-user-example-com create token default-editor)"
echo "=== AuthorizationPolicy Details ==="
kubectl get authorizationpolicy -n kubeflow-user-example-com -o yaml
echo "=== Detailed Curl Test ==="
curl -vv \
-H "Host: isvc-sklearn.kubeflow-user-example-com.example.com" \
-H "Authorization: Bearer ${KSERVE_M2M_TOKEN}" \
-H "Content-Type: application/json" \
"http://${KSERVE_INGRESS_HOST_PORT}/v1/models/isvc-sklearn:predict" \
-d '{"instances": [[6.8, 2.8, 4.8, 1.4], [6.0, 3.4, 4.5, 1.6]]}'
# TODO FOR FOLLOW UP PR
#- name: Run and fail kserve tests without kserve m2m token
#run: |
# export KSERVE_INGRESS_HOST_PORT=localhost:8080
# cd ./apps/kserve/tests
# if pytest . -vs --log-level info; then
# echo "This test should fail with an HTTP redirect to oauth2-proxy/dex auth."; exit 1
# else
# echo "Task failed successfully!"
# echo "This is a provisional way of testing that m2m is enabled for kserve."
# fi
# TODO FOR FOLLOW UP PR
#- name: Test that token from attacker namespace is rejected
# run: |
# export KSERVE_INGRESS_HOST_PORT=localhost:8080
# kubectl create ns kubeflow-user-example-com-attacker
# kubectl create serviceaccount attacker-sa -n kubeflow-user-example-com-attacker
# export ATTACKER_TOKEN="$(kubectl -n kubeflow-user-example-com-attacker create token attacker-sa)"
# RESPONSE=$(curl -s -o /dev/null -w "%{http_code}" -H "Host: isvc-sklearn.kubeflow-user-example-com.example.com" \
# -H "Authorization: Bearer ${ATTACKER_TOKEN}" \
# -H "Content-Type: application/json" \
# "http://${KSERVE_INGRESS_HOST_PORT}/v1/models/isvc-sklearn:predict" \
# -d '{"instances": [[6.8, 2.8, 4.8, 1.4], [6.0, 3.4, 4.5, 1.6]]}')
# if [[ "$RESPONSE" == "403" || "$RESPONSE" == "401" ]]; then
# echo "Security test passed: Request with attacker token was correctly rejected with $RESPONSE"
# else
# echo "Security test failed: Request with attacker token returned $RESPONSE instead of 403/401"
# exit 1
# fi
- name: Test path-based external access
run: |
export KSERVE_INGRESS_HOST_PORT=localhost:8080
export KSERVE_M2M_TOKEN="$(kubectl -n kubeflow-user-example-com create token default-editor)"
# Test external path-based access
curl -v -H "Host: isvc-sklearn.kubeflow-user-example-com.example.com" \
-H "Authorization: Bearer ${KSERVE_M2M_TOKEN}" \
-H "Content-Type: application/json" \
"http://${KSERVE_INGRESS_HOST_PORT}/kserve/kubeflow-user-example-com/isvc-sklearn/v1/models/isvc-sklearn:predict" \
-d '{"instances": [[6.8, 2.8, 4.8, 1.4], [6.0, 3.4, 4.5, 1.6]]}'
- name: Run kserve models webapp test
run: |
kubectl wait --for=condition=Available --timeout=300s -n kubeflow deployment/kserve-models-web-app
- name: Apply Pod Security Standards baseline levels
run: ./tests/gh-actions/enable_baseline_PSS.sh
- name: Unapply applied baseline labels
run: |
NAMESPACES=("istio-system" "auth" "cert-manager" "oauth2-proxy" "kubeflow" "knative-serving")
for NAMESPACE in "${NAMESPACES[@]}"; do
if kubectl get namespace "$NAMESPACE" >/dev/null 2>&1; then
kubectl label namespace $NAMESPACE pod-security.kubernetes.io/enforce-
fi
done
- name: Applying Pod Security Standards restricted levels
run: ./tests/gh-actions/enable_restricted_PSS.sh

2
apps/kserve/tests/requirements.txt
View File

@ -1,4 +1,4 @@
pytest>=7.0.0
kserve>=0.12.1
kserve>=0.14.1
kubernetes>=18.20.0
requests>=2.18.4

2
apps/kserve/tests/utils.py
View File

@ -26,7 +26,7 @@ from kserve import constants
logging.basicConfig(level=logging.INFO)
KSERVE_NAMESPACE = "kserve"
KSERVE_TEST_NAMESPACE = "kserve-test"
KSERVE_TEST_NAMESPACE = "kubeflow-user-example-com"
MODEL_CLASS_NAME = "modelClass"

10
common/istio-1-24/cluster-local-gateway/base/gateway-authorizationpolicy.yaml
View File

@ -1,14 +1,16 @@
# Allow all traffic to the cluster-local-gateway
# Enforce OAuth2-proxy authentication for cluster-local-gateway
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: cluster-local-gateway
name: cluster-local-gateway-oauth2-proxy
spec:
action: ALLOW
action: CUSTOM
provider:
name: oauth2-proxy
selector:
# Same as the cluster-local-gateway Service selector
matchLabels:
app: cluster-local-gateway
istio: cluster-local-gateway
rules:
- {}
- {}

10
common/istio-cni-1-24/cluster-local-gateway/base/gateway-authorizationpolicy.yaml
View File

@ -1,14 +1,16 @@
# Allow all traffic to the cluster-local-gateway
# Enforce OAuth2-proxy authentication for cluster-local-gateway
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: cluster-local-gateway
name: cluster-local-gateway-oauth2-proxy
spec:
action: ALLOW
action: CUSTOM
provider:
name: oauth2-proxy
selector:
# Same as the cluster-local-gateway Service selector
matchLabels:
app: cluster-local-gateway
istio: cluster-local-gateway
rules:
- {}
- {}

58
tests/gh-actions/deploy-dex-login-environment/kustomization.yaml
View File

@ -1,58 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
sortOptions:
order: legacy
legacySortOptions:
orderFirst:
- Namespace
- ResourceQuota
- StorageClass
- CustomResourceDefinition
- MutatingWebhookConfiguration
- ServiceAccount
- PodSecurityPolicy
- NetworkPolicy
- Role
- ClusterRole
- RoleBinding
- ClusterRoleBinding
- ConfigMap
- Secret
- Endpoints
- Service
- LimitRange
- PriorityClass
- PersistentVolume
- PersistentVolumeClaim
- Deployment
- StatefulSet
- CronJob
- PodDisruptionBudget
orderLast:
- ValidatingWebhookConfiguration
resources:
# Istio
- ../../../common/istio-1-24/istio-crds/base
- ../../../common/istio-1-24/istio-namespace/base
- ../../../common/istio-1-24/istio-install/overlays/oauth2-proxy
# oauth2-proxy
- ../../../common/oauth2-proxy/overlays/m2m-dex-and-kind
# Dex
- ../../../common/dex/overlays/oauth2-proxy
- ../../../common/istio-1-24/cluster-local-gateway/base
# Kubeflow namespace
- ../../../common/kubeflow-namespace/base
# NetworkPolicies
- ../../../common/networkpolicies/base
# Kubeflow Roles
- ../../../common/kubeflow-roles/base
# Kubeflow Istio Resources
- ../../../common/istio-1-24/kubeflow-istio-resources/base
# Central Dashboard
- ../../../apps/centraldashboard/overlays/oauth2-proxy
# Profiles + KFAM
- ../../../apps/profiles/upstream/overlays/kubeflow
# User namespace
- ../../../common/user-namespace/base