dapr
/
components-contrib
mirror of https://github.com/dapr/components-contrib.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Add automation for deploying Azure conformance test resources (#1001) 

* Add automation for deploying Azure conformance test resources

- Add .bicep definitions for Azure deployment of conformance test resources.
- Add setup-azure-conf-test.sh script for deploying .bicep definition and
  configuring other assets needed for the Azure conformance tests:
   - Creates the appropriate service principals and their credentials
     for the test.
   - Populates the key vault with the test parameters and secrets used by
     the GitHub conformance.yml test workflow.
   - Generates a .rc file that can be invoked with `source` to configure
     the local environment variables and secrets for running conformance
     tests described in tests/conformance/README.md.
   - Generates a teardown script that can be used to dispose of the
     deployed Azure resources when done.

* Move setup-azure-conf-test under azure/ subfolder

* Add documentation for Azure conformance test automation

- Add README.md to .github/infrastructure/conformance/azure describing
  use of automation script.
- Update tests/conformance/README.md with reference to Azure
  conformance test infrastructure automation.
- Fix spelling and issues in tests/conformance/README.md reported by
  markdownlint.
This commit is contained in:
Simon Leet 2021-07-28 10:06:14 -07:00 committed by GitHub
parent f58e0ca26d
commit 9c84bd202c
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
9 changed files with 834 additions and 41 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

73
.github/infrastructure/conformance/azure/README.md vendored Normal file
View File

@ -0,0 +1,73 @@
# Azure Conformance Tests Infrastructure
This folder contains helper scripts for maintainers to set up resources needed for conformance testing of Azure components.
> If you are running on Windows, all automation described in this document will need to be executed under [Git Bash](https://www.atlassian.com/git/tutorials/git-bash.)
## Prerequisites
To use the automation in this folder, you will need:
- A user account with [Owner permissions](https://docs.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#owner) to assign RBAC roles and manage all Azure resources under an [Azure Subscription](https://docs.microsoft.com/en-us/azure/cost-management-billing/manage/create-subscription#:~:text=Create%20a%20subscription%20in%20the%20Azure%20portal%201,the%20form%20for%20each%20type%20of%20billing%20account.).
- The [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli) installed.
## Setting up Azure resources for conformance tests
The `setup-azure-conf-test.sh` script will create and configure all the Azure resources necessary for the Azure conformance tests. You will need to login the Azure CLI first, and set the target account you want to run the deployment under:
```bash
az login
az account set -s "Name of Test Deployment Subscription"
./setup-azure-conf-test.sh --user="myazurealias@contoso.com" --location="EastUS"
```
> If you plan to run the `bindings.azure.eventgrid` test, you may also want to provide the `--ngrok-token` parameter at this point, or you will have to manually populate this value yourself in the test Key Vault or your local environment variable before your run the test.
For more details on the parameters that the script supports, run:
```bash
./setup-azure-conf-test.sh --help
```
By default, the script will prefix all resources it creates with your user alias, unless you provide it with an alternative `--prefix` string. It will generate several additional artifacts for your use after deployment to an output directory that defaults to `$HOME/azure-conf-test` unless otherwise specified by `--outpath` when running the script. Based on the example command for running the script, these would be:
- `myazurealias-azure-conf-test.json` is a copy of the Azure Resource Manager (ARM) template compiled from the `.bicep` files in this folder, and used to generate all the resources. It can be modified and re-deployed if desired for your own purposes.
- `myazurealias-teardown-conf-test.sh` is a script that will teardown all of the Azure resources that were created, including the test service principals and key vault.
- `myazurealias-conf-test-config.rc` contains all the environment variables needed to run the Azure conformance tests locally. **This file also contains various credentials used to access the resources.**
- `AzureKeyVaultSecretStoreCert.pfx` is a local copy of the cert for the Service Principal used in the `secretstore.azure.keyvault` conformance test. The path to this is referenced as part of the environment variables in the `*-conf-test-config.rc`.
- `AZURE_CREDENTIALS` contains the credentials for the Service Principal you can use to run the conformance test GitHub workflow against the created Azure resources.
## Running Azure conformance tests locally
1. Apply all the environment variables needed to run the Azure conformance test from your device, by sourcing the generated `*-conf-test-config.rc` file. For example:
```bash
source ~/azure-conf-test/myazurealias-conf-test-config.rc
```
2. Follow the [instructions for running individual conformance tests](../../../../tests/conformance/README.md#running-conormance-tests).
> The `bindings.azure.eventgrid` test and others may require additional setup before running the conformance test, such as setting up non-Azure resources like an Ngrok endpoint. See [conformance.yml](../../../../.github/workflows/conformance.yml) for details.
## Running Azure conformance tests via GitHub workflows
1. Fork the `dapr/components-contrib` repo.
2. In your fork on GitHub, [add the secrets](https://docs.github.com/en/actions/reference/encrypted-secrets#creating-encrypted-secrets-for-a-repository):
- `AZURE_CREDENTIALS`: copy and paste the contents of the `AZURE_CREDENTIALS` file from the output folder of the script.
- `AZURE_KEYVAULT`: copy and paste the value of the `AzureKeyVaultName` variable from the `*-conf-test-config.rc` file in the output folder of the script. This is usually of the form `<prefix>-conf-test-kv` where the _prefix_ string is the user name portion of the Azure account UPN by default, or the value specified using the `--prefix` parameter when running the script.
3. Under the Actions tab, enable the "Components Conformance Tests" workflow and [manually trigger the workflow](https://docs.github.com/en/actions/managing-workflow-runs/manually-running-a-workflow#running-a-workflow).
## Tearing down Azure resources after use
Run the teardown `<prefix>-teardown-conf-test.sh` script generated to the output path for `setup-azure-conf-test.sh`. Using the defaults from the setup example, this would be:
```bash
az login
az account set -s "Name of Test Deployment Subscription"
~/azure-conf-test/myazurealias-teardown-conf-test.sh
```
> ⚠ This will delete the resource group containing all the resources, remove the service principals that were created and purge the test Key Vault. **Purging the Key Vault is not a recoverable action, so do not run this script if you have modified it for other purposes.**

53
.github/infrastructure/conformance/azure/conf-test-azure-cosmosdb.bicep vendored Normal file
View File

@ -0,0 +1,53 @@
// ------------------------------------------------------------
// Copyright (c) Microsoft Corporation and Dapr Contributors.
// Licensed under the MIT License.
// ------------------------------------------------------------
param cosmosDbName string
param rgLocation string = resourceGroup().location
param confTestTags object = {}
var cosmosDbSqlName = '${cosmosDbName}-sqldb'
var cosmosDbSqlContainerName = '${cosmosDbSqlName}-container'
resource cosmosDb 'Microsoft.DocumentDB/databaseAccounts@2021-04-15' = {
name: cosmosDbName
location: rgLocation
tags: confTestTags
properties: {
consistencyPolicy: {
defaultConsistencyLevel: 'Session'
}
locations: [
{
locationName: rgLocation
}
]
databaseAccountOfferType: 'Standard'
}
resource cosmosDbSql 'sqlDatabases' = {
name: cosmosDbSqlName
properties: {
resource: {
id: cosmosDbSqlName
}
}
resource cosmosDbSqlContainer 'containers' = {
name: cosmosDbSqlContainerName
properties: {
resource: {
id: cosmosDbSqlContainerName
partitionKey: {
paths: [
'/partitionKey' // Defined by conformance test state.go
]
}
}
}
}
}
}
output cosmosDbSqlName string = cosmosDb::cosmosDbSql.name
output cosmosDbSqlContainerName string = cosmosDb::cosmosDbSql::cosmosDbSqlContainer.name

14
.github/infrastructure/conformance/azure/conf-test-azure-eventgrid.bicep vendored Normal file
View File

@ -0,0 +1,14 @@
// ------------------------------------------------------------
// Copyright (c) Microsoft Corporation and Dapr Contributors.
// Licensed under the MIT License.
// ------------------------------------------------------------
param eventGridTopicName string
param rgLocation string = resourceGroup().location
param confTestTags object = {}
resource eventGridTopic 'Microsoft.EventGrid/topics@2020-06-01' = {
name: eventGridTopicName
location: rgLocation
tags: confTestTags
}

97
.github/infrastructure/conformance/azure/conf-test-azure-keyvault.bicep vendored Normal file
View File

@ -0,0 +1,97 @@
// ------------------------------------------------------------
// Copyright (c) Microsoft Corporation and Dapr Contributors.
// Licensed under the MIT License.
// ------------------------------------------------------------
param adminId string
param certAuthSpId string
param sdkAuthSpId string
param keyVaultName string
param rgLocation string = resourceGroup().location
param confTestTags object = {}
param tenantId string = subscription().tenantId
resource keyVault 'Microsoft.KeyVault/vaults@2019-09-01' = {
name: keyVaultName
location: rgLocation
tags: confTestTags
properties: {
tenantId: tenantId
sku: {
family: 'A'
name: 'standard'
}
accessPolicies: [
// This access policy is to allow the user to manage the KeyVault instance.
{
tenantId: tenantId
objectId: adminId
permissions: {
keys: [
'all'
]
secrets: [
'all'
]
certificates: [
'all'
]
storage: [
'all'
]
}
}
// This access policy is used by the AKV conformance test to Get and BulkGet secrets.
{
tenantId: tenantId
objectId: certAuthSpId
permissions: {
keys: [
'get'
'list'
]
secrets: [
'get'
'list'
]
certificates: [
'get'
'list'
]
storage: [
'get'
'list'
]
}
}
// This access policy is used by GitHub workflow to retrieve the required-secrets
// and required-certs for the conformance tests.
{
tenantId: tenantId
objectId: sdkAuthSpId
permissions: {
secrets: [
'get'
]
}
}
]
}
// These test secrets are defined by the conformance tests secretstores.go
resource testsecret1 'secrets' = {
name: 'conftestsecret'
properties: {
value: 'abcd'
contentType: 'plaintext-value'
}
}
resource testsecret2 'secrets' = {
name: 'secondsecret'
properties: {
value: 'efgh'
contentType: 'plaintext-value'
}
}
}

17
.github/infrastructure/conformance/azure/conf-test-azure-servicebus.bicep vendored Normal file
View File

@ -0,0 +1,17 @@
// ------------------------------------------------------------
// Copyright (c) Microsoft Corporation and Dapr Contributors.
// Licensed under the MIT License.
// ------------------------------------------------------------
param serviceBusName string
param rgLocation string = resourceGroup().location
param confTestTags object = {}
resource serviceBus 'Microsoft.ServiceBus/namespaces@2017-04-01' = {
name: serviceBusName
location: rgLocation
tags: confTestTags
sku: {
name: 'Standard' // Need at least Standard SKU to create Topics in ServiceBus Namespace
}
}

18
.github/infrastructure/conformance/azure/conf-test-azure-storage.bicep vendored Normal file
View File

@ -0,0 +1,18 @@
// ------------------------------------------------------------
// Copyright (c) Microsoft Corporation and Dapr Contributors.
// Licensed under the MIT License.
// ------------------------------------------------------------
param storageName string
param rgLocation string = resourceGroup().location
param confTestTags object = {}
resource storageAccount 'Microsoft.Storage/storageAccounts@2021-02-01' = {
name: storageName
sku: {
name: 'Standard_RAGRS'
}
kind: 'StorageV2'
location: rgLocation
tags: confTestTags
}

104
.github/infrastructure/conformance/azure/conf-test-azure.bicep vendored Normal file
View File

@ -0,0 +1,104 @@
// ------------------------------------------------------------
// Copyright (c) Microsoft Corporation and Dapr Contributors.
// Licensed under the MIT License.
// ------------------------------------------------------------
targetScope = 'subscription'
@minLength(3)
@maxLength(15) // storageName must be < 24 characters total
@description('Provide an string prefix between 3-15 characters for all resource names deployed by this template. It should only consist of lower case alphabetical characters.')
param namePrefix string
@description('Provide a target location for the resource group and azure resources. Defaults to West US 2.')
param rgLocation string = 'West US 2'
@description('Provide tags to associate with resources created by this deployment. Defaults to Role=dapr-conf-test.')
param confTestTags object = {
Role: 'dapr-conf-test'
}
@minLength(36)
@maxLength(36)
@description('Provide the user objectId in the current tenant to set as admin for Azure Key Vault.')
param adminId string
@minLength(36)
@maxLength(36)
@description('Provide the objectId of the Service Principal using secret auth with get access to secrets in Azure Key Vault.')
param sdkAuthSpId string
@minLength(36)
@maxLength(36)
@description('Provide the objectId of the Service Principal using cert auth with get and list access to all assets in Azure Key Vault.')
param certAuthSpId string
var confTestRgName = '${toLower(namePrefix)}-conf-test-rg'
var cosmosDbName = '${toLower(namePrefix)}-conf-test-db'
var eventGridTopicName = '${toLower(namePrefix)}-conf-test-eventgrid-topic'
var keyVaultName = '${toLower(namePrefix)}-conf-test-kv'
var serviceBusName = '${toLower(namePrefix)}-conf-test-servicebus'
var storageName = '${toLower(namePrefix)}ctstorage'
resource confTestRg 'Microsoft.Resources/resourceGroups@2021-04-01' = {
name: confTestRgName
location: rgLocation
tags: confTestTags
}
module cosmosDb 'conf-test-azure-cosmosdb.bicep' = {
name: cosmosDbName
scope: resourceGroup(confTestRg.name)
params: {
confTestTags: confTestTags
cosmosDbName: cosmosDbName
}
}
module eventGridTopic 'conf-test-azure-eventGrid.bicep' = {
name: eventGridTopicName
scope: resourceGroup(confTestRg.name)
params: {
confTestTags: confTestTags
eventGridTopicName: eventGridTopicName
}
}
module keyVault 'conf-test-azure-keyVault.bicep' = {
name: keyVaultName
scope: resourceGroup(confTestRg.name)
params: {
adminId: adminId
confTestTags: confTestTags
certAuthSpId: certAuthSpId
keyVaultName: keyVaultName
sdkAuthSpId: sdkAuthSpId
}
}
module serviceBus 'conf-test-azure-servicebus.bicep' = {
name: serviceBusName
scope: resourceGroup(confTestRg.name)
params: {
confTestTags: confTestTags
serviceBusName: serviceBusName
}
}
module storage 'conf-test-azure-storage.bicep' = {
name: storageName
scope: resourceGroup(confTestRg.name)
params: {
confTestTags: confTestTags
storageName: storageName
}
}
output confTestRgName string = confTestRg.name
output cosmosDbName string = cosmosDb.name
output cosmosDbSqlName string = cosmosDb.outputs.cosmosDbSqlName
output cosmosDbSqlContainerName string = cosmosDb.outputs.cosmosDbSqlContainerName
output eventGridTopicName string = eventGridTopic.name
output keyVaultName string = keyVault.name
output serviceBusName string = serviceBus.name
output storageName string = storage.name

395
.github/infrastructure/conformance/azure/setup-azure-conf-test.sh vendored Normal file
View File

@ -0,0 +1,395 @@
#!/usr/bin/env bash
# ------------------------------------------------------------
# Copyright (c) Microsoft Corporation and Dapr Contributors.
# Licensed under the MIT License.
# ------------------------------------------------------------
set -e
##==============================================================================
##
## Process command-line options:
##
##==============================================================================
ERR=0
for opt in "$@"
do
case $opt in
-\? | -h | --help)
SHOW_USAGE=1
;;
--location=*)
DEPLOY_LOCATION="${opt#*=}"
;;
--ngrok-token=*)
NGROK_TOKEN="${opt#*=}"
;;
--outpath=*)
OUTPUT_PATH="${opt#*=}"
;;
--user=*)
ADMIN_UPN="${opt#*=}"
;;
--prefix=*)
PREFIX="${opt#*=}"
;;
*)
echo "$0: Unknown option: $opt"
ERR=1
SHOW_USAGE=1
;;
esac
done
if [[ -z ${ADMIN_UPN} && -z ${SHOW_USAGE} ]]; then
echo "$0: --user must be specified"
ERR=1
SHOW_USAGE=1
fi
##==============================================================================
##
## Display help
##
##==============================================================================
if [[ ${SHOW_USAGE} -eq 1 ]]; then
cat<<EOF
OVERVIEW:
Sets up Azure resources needed for conformance tests and populates the secrets
needed for the conformance.yml GitHub workflow to run. Also generates a .rc file
that can be used to set environment variables for the conformance test to be run
on the local device.
PREREQUISITES:
This script requires that the Azure CLI is installed, and the user is already
logged in and has already selected the subscription under which the new resources
will be deployed. For example:
$ az login
$ az account set -s "My Test Subscription"
USAGE:
$ ./setup-azure-conf-test.sh --user=<Azure user UPN> [--location="..."] \
[--prefix="..."] [--outpath="..."] [--ngrok-token="..."]
OPTIONS:
-h, --help Print this help message.
--user The UPN for the Azure user in the current subscription who
will own all created resources, e.g. "myalias@contoso.com".
--location Optional. The location for the Azure deployment. Defaults to
"WestUS2" if not specified.
--prefix Optional. 3-15 character string to prefix all created
resources. Defaults to the user name of the provided --user
UPN if not specified.
--outpath Optional. Path to write resulting config and resource files
into. Defaults to "~/azure-conf-test" if not specified.
--ngrok-token Optional. The Authtoken from your ngrok account, as found at
https://dashboard.ngrok.com/get-started/your-authtoken. Only
needed for setting AzureEventGridNgrokToken in the KeyVault,
which is used by the GitHub workflow for the EventGrid
bindings conformance test.
EOF
exit $ERR
fi
##==============================================================================
##
## Setup default parameters
##
##==============================================================================
if [[ -z ${PREFIX} ]]; then
PREFIX="$(echo "${ADMIN_UPN}" | sed -E 's/@.*|\.//g')"
echo "INFO: Using user name as resource prefix: \"${PREFIX}\""
fi
if [[ -z ${OUTPUT_PATH} ]]; then
OUTPUT_PATH="$HOME/azure-conf-test"
echo "INFO: Using default output path: \"${OUTPUT_PATH}\""
fi
if [[ -z ${DEPLOY_LOCATION} ]]; then
DEPLOY_LOCATION="WestUS2"
echo "INFO: Using default deployment location: \"${DEPLOY_LOCATION}\""
fi
if [[ -z ${NGROK_TOKEN} ]]; then
echo "WARN: --ngrok-token is not specified, will not set AzureEventGridNgrokToken used by GitHub workflow for bindings.azure.eventgrid conformance test."
fi
echo
echo "Starting setup-azure-conf-test with the following parameters:"
echo "ADMIN_UPN=${ADMIN_UPN}"
echo "PREFIX=${PREFIX}"
echo "DEPLOY_LOCATION=${DEPLOY_LOCATION}"
echo "OUTPUT_PATH=${OUTPUT_PATH}"
echo "NGROK_TOKEN=${NGROK_TOKEN}"
##==============================================================================
##
## Setup Azure environment
##
##==============================================================================
# Constant environment variable names defined by tests or GitHub workflow
COSMOS_DB_VAR_NAME="AzureCosmosDB"
COSMOS_DB_COLLECTION_VAR_NAME="AzureCosmosDBCollection"
COSMOS_DB_MASTER_KEY_VAR_NAME="AzureCosmosDBMasterKey"
COSMOS_DB_URL_VAR_NAME="AzureCosmosDBUrl"
EVENT_GRID_ACCESS_KEY_VAR_NAME="AzureEventGridAccessKey"
EVENT_GRID_CLIENT_ID_VAR_NAME="AzureEventGridClientId"
EVENT_GRID_CLIENT_SECRET_VAR_NAME="AzureEventGridClientSecret"
EVENT_GRID_NGROK_TOKEN_VAR_NAME="AzureEventGridNgrokToken"
EVENT_GRID_SCOPE_VAR_NAME="AzureEventGridScope"
EVENT_GRID_SUB_ID_VAR_NAME="AzureEventGridSubscriptionId"
EVENT_GRID_TENANT_ID_VAR_NAME="AzureEventGridTenantId"
EVENT_GRID_TOPIC_ENDPOINT_VAR_NAME="AzureEventGridTopicEndpoint"
KEYVAULT_CERT_NAME="AzureKeyVaultSecretStoreCert"
KEYVAULT_CLIENT_ID_VAR_NAME="AzureKeyVaultSecretStoreClientId"
KEYVAULT_TENANT_ID_VAR_NAME="AzureKeyVaultSecretStoreTenantId"
KEYVAULT_NAME_VAR_NAME="AzureKeyVaultName"
SERVICE_BUS_CONNECTION_STRING_VAR_NAME="AzureServiceBusConnectionString"
STORAGE_ACCESS_KEY_VAR_NAME="AzureBlobStorageAccessKey"
STORAGE_ACCOUNT_VAR_NAME="AzureBlobStorageAccount"
STORAGE_CONTAINER_VAR_NAME="AzureBlobStorageContainer"
STORAGE_QUEUE_VAR_NAME="AzureBlobStorageQueue"
# Derived variables
ADMIN_ID="$(az ad user list --upn "${ADMIN_UPN}" --query "[].objectId" | grep \" | sed -E 's/[[:space:]]|\"//g')"
SUB_ID="$(az account show --query "id" | sed -E 's/[[:space:]]|\"//g')"
TENANT_ID="$(az account show --query "tenantId" | sed -E 's/[[:space:]]|\"//g')"
DEPLOY_NAME="${PREFIX}-azure-conf-test"
# Setup output path
mkdir -p "${OUTPUT_PATH}"
# Create Service Principals for use with the conformance tests
CERT_AUTH_SP_NAME="${PREFIX}-akv-conf-test-sp"
az ad sp create-for-rbac --name "${CERT_AUTH_SP_NAME}" --skip-assignment --years 1
CERT_AUTH_SP_ID="$(az ad sp list --display-name "${CERT_AUTH_SP_NAME}" --query "[].objectId" | grep \" | sed -E 's/[[:space:]]|\"//g')"
echo "Created Service Principal for cert auth: ${CERT_AUTH_SP_NAME}"
SDK_AUTH_SP_NAME="${PREFIX}-conf-test-runner-sp"
SDK_AUTH_SP_INFO="$(az ad sp create-for-rbac --name "${SDK_AUTH_SP_NAME}" --sdk-auth --skip-assignment --years 1)"
echo "${SDK_AUTH_SP_INFO}"
echo "Created Service Principal for SDK Auth: ${SDK_AUTH_SP_NAME}"
AZURE_CREDENTIALS_FILENAME="${OUTPUT_PATH}/AZURE_CREDENTIALS"
echo "${SDK_AUTH_SP_INFO}" > "${AZURE_CREDENTIALS_FILENAME}"
SDK_AUTH_SP_CLIENT_SECRET="$(echo "${SDK_AUTH_SP_INFO}" | grep 'clientSecret' | sed -E 's/(.*clientSecret\"\: \")|\",//g')"
SDK_AUTH_SP_ID="$(az ad sp list --display-name "${SDK_AUTH_SP_NAME}" --query "[].objectId" | grep \" | sed -E 's/[[:space:]]|\"//g')"
# Build the bicep template and deploy to Azure
az bicep install
ARM_TEMPLATE_FILE="${OUTPUT_PATH}/${PREFIX}-azure-conf-test.json"
echo "Building conf-test-azure.bicep to ${ARM_TEMPLATE_FILE} ..."
az bicep build --file conf-test-azure.bicep --outfile "${ARM_TEMPLATE_FILE}"
echo "Creating azure deployment ${DEPLOY_NAME} in ${DEPLOY_LOCATION} and resource prefix ${PREFIX}-* ..."
az deployment sub create --name "${DEPLOY_NAME}" --location "${DEPLOY_LOCATION}" --template-file "${ARM_TEMPLATE_FILE}" --p namePrefix="${PREFIX}" -p adminId="${ADMIN_ID}" -p certAuthSpId="${CERT_AUTH_SP_ID}" -p sdkAuthSpId="${SDK_AUTH_SP_ID}" -p rgLocation="${DEPLOY_LOCATION}"
# Query the deployed resource names from the bicep deployment outputs
RESOURCE_GROUP_NAME="$(az deployment sub show --name "${DEPLOY_NAME}" --query "properties.outputs.confTestRgName.value" | sed -E 's/[[:space:]]|\"//g')"
echo "INFO: RESOURCE_GROUP_NAME=${RESOURCE_GROUP_NAME}"
SERVICE_BUS_NAME="$(az deployment sub show --name "${DEPLOY_NAME}" --query "properties.outputs.serviceBusName.value" | sed -E 's/[[:space:]]|\"//g')"
echo "INFO: SERVICE_BUS_NAME=${SERVICE_BUS_NAME}"
KEYVAULT_NAME="$(az deployment sub show --name "${DEPLOY_NAME}" --query "properties.outputs.keyVaultName.value" | sed -E 's/[[:space:]]|\"//g')"
echo "INFO: KEYVAULT_NAME=${KEYVAULT_NAME}"
STORAGE_NAME="$(az deployment sub show --name "${DEPLOY_NAME}" --query "properties.outputs.storageName.value" | sed -E 's/[[:space:]]|\"//g')"
echo "INFO: STORAGE_NAME=${STORAGE_NAME}"
COSMOS_DB_NAME="$(az deployment sub show --name "${DEPLOY_NAME}" --query "properties.outputs.cosmosDbName.value" | sed -E 's/[[:space:]]|\"//g')"
echo "INFO: COSMOS_DB_NAME=${COSMOS_DB_NAME}"
COSMOS_DB_SQL_NAME="$(az deployment sub show --name "${DEPLOY_NAME}" --query "properties.outputs.cosmosDbSqlName.value" | sed -E 's/[[:space:]]|\"//g')"
echo "INFO: COSMOS_DB_SQL_NAME=${COSMOS_DB_SQL_NAME}"
COSMOS_DB_CONTAINER_NAME="$(az deployment sub show --name "${DEPLOY_NAME}" --query "properties.outputs.cosmosDbSqlContainerName.value" | sed -E 's/[[:space:]]|\"//g')"
echo "INFO: COSMOS_DB_CONTAINER_NAME=${COSMOS_DB_CONTAINER_NAME}"
EVENT_GRID_TOPIC_NAME="$(az deployment sub show --name "${DEPLOY_NAME}" --query "properties.outputs.eventGridTopicName.value" | sed -E 's/[[:space:]]|\"//g')"
echo "INFO: EVENT_GRID_TOPIC_NAME=${EVENT_GRID_TOPIC_NAME}"
# Update service principal credentials and roles for created resources
echo "Creating ${CERT_AUTH_SP_NAME} certificate ..."
az ad sp credential reset --name "${CERT_AUTH_SP_NAME}" --create-cert --cert "${KEYVAULT_CERT_NAME}" --keyvault "${KEYVAULT_NAME}"
# Add an EventGrid role to the SDK auth Service Principal so that it can be reused for the EventGrid binding conformance tests.
EVENT_GRID_SCOPE="/subscriptions/${SUB_ID}/resourceGroups/${RESOURCE_GROUP_NAME}/providers/Microsoft.EventGrid/topics/${EVENT_GRID_TOPIC_NAME}"
# MSYS_NO_PATHCONV is needed to prevent MSYS in Git Bash from converting the scope string to a local filesystem path and is benign on Linux
echo "Assigning \"EventGrid EventSubscription Contributor\" role to ${SDK_AUTH_SP_NAME} in scope \"${EVENT_GRID_SCOPE}\"..."
MSYS_NO_PATHCONV=1 az role assignment create --assignee "${SDK_AUTH_SP_ID}" --role "EventGrid EventSubscription Contributor" --scope "${EVENT_GRID_SCOPE}"
##==============================================================================
##
## Create output files for environment config and teardown of conformance tests
##
##==============================================================================
# Create script for teardown of created azure resources.
TEARDOWN_SCRIPT_NAME="${OUTPUT_PATH}/${PREFIX}-teardown-conf-test.sh"
tee "${TEARDOWN_SCRIPT_NAME}" > /dev/null \
<< EOF
#!/usr/bin/env bash
# ------------------------------------------------------------
# Copyright (c) Microsoft Corporation and Dapr Contributors.
# Licensed under the MIT License.
# ------------------------------------------------------------
set +e
echo "Deleting deployment ${DEPLOY_NAME} ..."
az deployment sub delete --name "${DEPLOY_NAME}"
echo "Deleting resource group ${RESOURCE_GROUP_NAME} ..."
az group delete --name "${RESOURCE_GROUP_NAME}" --yes
echo "Purging key vault ${KEYVAULT_NAME} ..."
az keyvault purge --name "${KEYVAULT_NAME}"
echo "Deleting service principal ${CERT_AUTH_SP_NAME} ..."
az ad sp delete --id "${CERT_AUTH_SP_ID}"
echo "Deleting service principal ${SDK_AUTH_SP_NAME} ..."
az ad sp delete --id "${SDK_AUTH_SP_ID}"
echo "INFO: ${PREFIX}-teardown-conf-test completed."
EOF
chmod +x "${TEARDOWN_SCRIPT_NAME}"
echo "INFO: Created ${TEARDOWN_SCRIPT_NAME}."
# Initialize an environment variable file that can used with `source` for local execution of conformance tests.
ENV_CONFIG_FILENAME="${OUTPUT_PATH}/${PREFIX}-conf-test-config.rc"
tee "${ENV_CONFIG_FILENAME}" > /dev/null \
<< 'EOF'
#!/usr/bin/env bash
# ------------------------------------------------------------
# Copyright (c) Microsoft Corporation and Dapr Contributors.
# Licensed under the MIT License.
# ------------------------------------------------------------
if [[ -n ${NGROK_ENDPOINT} ]]; then
export AzureEventGridSubscriberEndpoint="${NGROK_ENDPOINT}/api/events"
else
echo "WARN: NGROK_ENDPOINT is not defined, AzureEventGridSubscriberEndpoint cannot be set for local testing of TestBindingsConformance/azure.eventgrid"
fi
EOF
chmod +x "${ENV_CONFIG_FILENAME}"
echo "INFO: Created ${ENV_CONFIG_FILENAME}."
##==============================================================================
##
## Populate Key Vault and config file with conformance test settings
##
##==============================================================================
# ---------------------------------
# Populate Key Vault test settings
# ---------------------------------
echo "Configuring Key Vault test settings ..."
KEYVAULT_CERT_FILE="${OUTPUT_PATH}/${KEYVAULT_CERT_NAME}.pfx"
if [ -e "${KEYVAULT_CERT_FILE}" ]; then
rm "${KEYVAULT_CERT_FILE}"
fi
az keyvault secret download --vault-name "${KEYVAULT_NAME}" --name "${KEYVAULT_CERT_NAME}" --encoding base64 --file "${KEYVAULT_CERT_FILE}"
echo export ${KEYVAULT_CERT_NAME}=\"${KEYVAULT_CERT_FILE}\" >> "${ENV_CONFIG_FILENAME}"
# Note that the credential reset of the cert auth Service Principal has already pushed its cert to the Key Vault.
echo export ${KEYVAULT_NAME_VAR_NAME}=\"${KEYVAULT_NAME}\" >> "${ENV_CONFIG_FILENAME}"
az keyvault secret set --name "${KEYVAULT_NAME_VAR_NAME}" --vault-name "${KEYVAULT_NAME}" --value "${KEYVAULT_NAME}"
KEYVAULT_TENANT_ID="$(az ad sp list --display-name "${CERT_AUTH_SP_NAME}" --query "[].appOwnerTenantId" | grep \" | sed -E 's/[[:space:]]|\"//g')"
echo export ${KEYVAULT_TENANT_ID_VAR_NAME}=\"${KEYVAULT_TENANT_ID}\" >> "${ENV_CONFIG_FILENAME}"
az keyvault secret set --name "${KEYVAULT_TENANT_ID_VAR_NAME}" --vault-name "${KEYVAULT_NAME}" --value "${KEYVAULT_TENANT_ID}"
KEYVAULT_CLIENT_ID="$(az ad sp list --display-name "${CERT_AUTH_SP_NAME}" --query "[].appId" | grep \" | sed -E 's/[[:space:]]|\"//g')"
echo export ${KEYVAULT_CLIENT_ID_VAR_NAME}=\"${KEYVAULT_CLIENT_ID}\" >> "${ENV_CONFIG_FILENAME}"
az keyvault secret set --name "${KEYVAULT_CLIENT_ID_VAR_NAME}" --vault-name "${KEYVAULT_NAME}" --value "${KEYVAULT_CLIENT_ID}"
# ------------------------------------
# Populate Blob Storage test settings
# ------------------------------------
echo "Configuring Blob Storage test settings ..."
echo export ${STORAGE_ACCOUNT_VAR_NAME}=\"${STORAGE_NAME}\" >> "${ENV_CONFIG_FILENAME}"
az keyvault secret set --name "${STORAGE_ACCOUNT_VAR_NAME}" --vault-name "${KEYVAULT_NAME}" --value "${STORAGE_NAME}"
STORAGE_CONTAINER_NAME="${PREFIX}-conf-test-container"
echo export ${STORAGE_CONTAINER_VAR_NAME}=\"${STORAGE_CONTAINER_NAME}\" >> "${ENV_CONFIG_FILENAME}"
az keyvault secret set --name "${STORAGE_CONTAINER_VAR_NAME}" --vault-name "${KEYVAULT_NAME}" --value "${STORAGE_CONTAINER_NAME}"
STORAGE_QUEUE_NAME="${PREFIX}-conf-test-queue"
echo export ${STORAGE_QUEUE_VAR_NAME}=\"${STORAGE_QUEUE_NAME}\" >> "${ENV_CONFIG_FILENAME}"
az keyvault secret set --name "${STORAGE_QUEUE_VAR_NAME}" --vault-name "${KEYVAULT_NAME}" --value "${STORAGE_QUEUE_NAME}"
STORAGE_ACCESS_KEY="$(az storage account keys list --account-name "${STORAGE_NAME}" --query "[?keyName=='key1'].value" | grep \" | sed -E 's/[[:space:]]|\"//g')"
echo export ${STORAGE_ACCESS_KEY_VAR_NAME}=\"${STORAGE_ACCESS_KEY}\" >> "${ENV_CONFIG_FILENAME}"
az keyvault secret set --name "${STORAGE_ACCESS_KEY_VAR_NAME}" --vault-name "${KEYVAULT_NAME}" --value "${STORAGE_ACCESS_KEY}"
# --------------------------------
# Populate CosmosDB test settings
# --------------------------------
echo "Configuring CosmosDB test settings ..."
echo export ${COSMOS_DB_VAR_NAME}=\"${COSMOS_DB_SQL_NAME}\" >> "${ENV_CONFIG_FILENAME}"
az keyvault secret set --name "${COSMOS_DB_VAR_NAME}" --vault-name "${KEYVAULT_NAME}" --value "${COSMOS_DB_SQL_NAME}"
# Note that CosmosDB maps SQL DB containers to collections
echo export ${COSMOS_DB_COLLECTION_VAR_NAME}=\"${COSMOS_DB_CONTAINER_NAME}\" >> "${ENV_CONFIG_FILENAME}"
az keyvault secret set --name "${COSMOS_DB_COLLECTION_VAR_NAME}" --vault-name "${KEYVAULT_NAME}" --value "${COSMOS_DB_CONTAINER_NAME}"
COSMOS_DB_URL="$(az cosmosdb list --query "[?name=='${COSMOS_DB_NAME}'].documentEndpoint" | grep \" | sed -E 's/[[:space:]]|\"//g')"
echo export ${COSMOS_DB_URL_VAR_NAME}=\"${COSMOS_DB_URL}\" >> "${ENV_CONFIG_FILENAME}"
az keyvault secret set --name "${COSMOS_DB_URL_VAR_NAME}" --vault-name "${KEYVAULT_NAME}" --value "${COSMOS_DB_URL}"
COSMOS_DB_MASTER_KEY="$(az cosmosdb keys list --name "${COSMOS_DB_NAME}" --resource-group "${RESOURCE_GROUP_NAME}" --query "primaryMasterKey" | sed -E 's/[[:space:]]|\"//g')"
echo export ${COSMOS_DB_MASTER_KEY_VAR_NAME}=\"${COSMOS_DB_MASTER_KEY}\" >> "${ENV_CONFIG_FILENAME}"
az keyvault secret set --name "${COSMOS_DB_MASTER_KEY_VAR_NAME}" --vault-name "${KEYVAULT_NAME}" --value "${COSMOS_DB_MASTER_KEY}"
# ----------------------------------
# Populate Event Grid test settings
# ----------------------------------
echo "Configuring Event Grid test settings ..."
EVENT_GRID_ACCESS_KEY="$(az eventgrid topic key list --name "${EVENT_GRID_TOPIC_NAME}" --resource-group "${RESOURCE_GROUP_NAME}" --query "key1" | sed -E 's/[[:space:]]|\"//g')"
echo export ${EVENT_GRID_ACCESS_KEY_VAR_NAME}=\"${EVENT_GRID_ACCESS_KEY}\" >> "${ENV_CONFIG_FILENAME}"
az keyvault secret set --name "${EVENT_GRID_ACCESS_KEY_VAR_NAME}" --vault-name "${KEYVAULT_NAME}" --value "${EVENT_GRID_ACCESS_KEY}"
SDK_AUTH_SP_APP_ID="$(az ad sp list --display-name "${SDK_AUTH_SP_NAME}" --query "[].appId" | grep \" | sed -E 's/[[:space:]]|\"//g')"
echo export ${EVENT_GRID_CLIENT_ID_VAR_NAME}=\"${SDK_AUTH_SP_APP_ID}\" >> "${ENV_CONFIG_FILENAME}"
az keyvault secret set --name "${EVENT_GRID_CLIENT_ID_VAR_NAME}" --vault-name "${KEYVAULT_NAME}" --value "${SDK_AUTH_SP_APP_ID}"
echo export ${EVENT_GRID_CLIENT_SECRET_VAR_NAME}=\"${SDK_AUTH_SP_CLIENT_SECRET}\" >> "${ENV_CONFIG_FILENAME}"
az keyvault secret set --name "${EVENT_GRID_CLIENT_SECRET_VAR_NAME}" --vault-name "${KEYVAULT_NAME}" --value "${SDK_AUTH_SP_CLIENT_SECRET}"
if [[ -n ${NGROK_TOKEN} ]]; then
echo export ${EVENT_GRID_NGROK_TOKEN_VAR_NAME}=\"${NGROK_TOKEN}\" >> "${ENV_CONFIG_FILENAME}"
az keyvault secret set --name "${EVENT_GRID_NGROK_TOKEN_VAR_NAME}" --vault-name "${KEYVAULT_NAME}" --value "${NGROK_TOKEN}"
else
echo "WARN: NGROK_TOKEN not specified, AzureEventGridNgrokToken secret needs to be manually added to ${KEYVAULT_NAME} before running the GitHub conformance test workflow."
fi
echo export ${EVENT_GRID_SCOPE_VAR_NAME}=\"${EVENT_GRID_SCOPE}\" >> "${ENV_CONFIG_FILENAME}"
az keyvault secret set --name "${EVENT_GRID_SCOPE_VAR_NAME}" --vault-name "${KEYVAULT_NAME}" --value "${EVENT_GRID_SCOPE}"
echo export ${EVENT_GRID_SUB_ID_VAR_NAME}=\"${SUB_ID}\" >> "${ENV_CONFIG_FILENAME}"
az keyvault secret set --name "${EVENT_GRID_SUB_ID_VAR_NAME}" --vault-name "${KEYVAULT_NAME}" --value "${SUB_ID}"
echo export ${EVENT_GRID_TENANT_ID_VAR_NAME}=\"${TENANT_ID}\" >> "${ENV_CONFIG_FILENAME}"
az keyvault secret set --name "${EVENT_GRID_TENANT_ID_VAR_NAME}" --vault-name "${KEYVAULT_NAME}" --value "${TENANT_ID}"
EVENT_GRID_TOPIC_ENDPOINT="$(az eventgrid topic list --query "[?name=='${EVENT_GRID_TOPIC_NAME}'].endpoint" | grep \" | sed -E 's/[[:space:]]|\"//g')"
echo export ${EVENT_GRID_TOPIC_ENDPOINT_VAR_NAME}=\"${EVENT_GRID_TOPIC_ENDPOINT}\" >> "${ENV_CONFIG_FILENAME}"
az keyvault secret set --name "${EVENT_GRID_TOPIC_ENDPOINT_VAR_NAME}" --vault-name "${KEYVAULT_NAME}" --value "${EVENT_GRID_TOPIC_ENDPOINT}"
# -----------------------------------
# Populate Service Bus test settings
# -----------------------------------
echo "Configuring Service Bus test settings ..."
SERVICE_BUS_CONNECTION_STRING="$(az servicebus namespace authorization-rule keys list --name RootManageSharedAccessKey --namespace-name "${SERVICE_BUS_NAME}" --resource-group "${RESOURCE_GROUP_NAME}" --query "primaryConnectionString" | sed -E 's/[[:space:]]|\"//g')"
echo export ${SERVICE_BUS_CONNECTION_STRING_VAR_NAME}=\"${SERVICE_BUS_CONNECTION_STRING}\" >> "${ENV_CONFIG_FILENAME}"
az keyvault secret set --name "${SERVICE_BUS_CONNECTION_STRING_VAR_NAME}" --vault-name "${KEYVAULT_NAME}" --value "${SERVICE_BUS_CONNECTION_STRING}"
echo "INFO: setup-azure-conf-test completed."
echo "INFO: Remember to \`source ${ENV_CONFIG_FILENAME}\` before running local conformance tests."
echo "INFO: ${AZURE_CREDENTIALS_FILENAME} contains the repository secret to set to run the GitHub conformance test workflow."
echo "INFO: To teardown the conformance test resources, run ${TEARDOWN_SCRIPT_NAME}."

104
tests/conformance/README.md
View File

@ -1,61 +1,83 @@
# Conformance Tests
## Tests structure
1. `tests/` directory contains the configuration and the test definition for conformance tests.
2. All the conformance tests are within the `tests/conformance` directory.
3. All the configurations are in the `tests/config` directory.
4. Each of the component specific `component` definition are in their specific `component type` folder in the `tests/config` folder. For eg: `redis` statestore component definition within `state` directory. The component types are `bindings`, `state`, `secretstores`, `pubsub`. Cloud specific components will be within their own `cloud` directory within the `component type` folder eg: `pubsub/azure/servicebus`.
4. Each of the component specific `component` definition are in their specific `component type` folder in the `tests/config` folder. E.g. `redis` statestore component definition within `state` directory. The component types are `bindings`, `state`, `secretstores`, `pubsub`. Cloud specific components will be within their own `cloud` directory within the `component type` folder, e.g. `pubsub/azure/servicebus`.
5. Similar to the component definitions, each component type has its own set of the conformance tests definitions.
6. Each `component type` contains a `tests.yml` definition that defines the component to be tested along with component specific test configuration. Nested folder names have their `/` in path replaced by `.` in the component name in `tests.yml` eg: `azure/servicebus` should be `azure.servicebus`
6. Each `component type` contains a `tests.yml` definition that defines the component to be tested along with component specific test configuration. Nested folder names have their `/` in path replaced by `.` in the component name in `tests.yml`, e.g. `azure/servicebus` should be `azure.servicebus`
7. All the tests configurations are defined in `common.go` file.
8. Each `component type` has its own `_test` file to trigger the conformance tests. Eg: `bindings_test.go`.
8. Each `component type` has its own `_test` file to trigger the conformance tests. E.g. `bindings_test.go`.
9. Each test added will also need to be added to the `conformance.yml` workflow file.
## Conformance test workflow
1. All components/services tested in the automated tests are currently manually setup by the Dapr team and run in an environment maintained by the Dapr team. As a contributor, follow the next two steps for integrating a new component with conformance tests.
2. If the component is tested locally within a docker container requiring no secrets, then add the component to the `pr-components` step in conformance test workflow `.github/conformance.yml`. `pr-components` step generates the component matrix for which the conformance tests will be run on each PR.
3. If the component is tested against a service and requires secrets, then add the component to the `cron-components` step in conformance test workflow `.github/conformance.yml`. `cron-components` defines the components for which the conformance test will be run against the master code in a scehduled manner.
## Integrating a new component with conformance tests
1. All components/services tested in the automated tests are currently manually setup by the Dapr team and run in an environment maintained by the Dapr team. As a contributor, follow the next two steps for integrating a new component with conformance tests.
2. If the component is tested locally within a docker container requiring no secrets, then add the component to the `pr-components` step in conformance test workflow `.github/conformance.yml`. `pr-components` step generates the component matrix for which the conformance tests will be run on each PR.
3. If the component is tested against a service and requires secrets, then add the component to the `cron-components` step in conformance test workflow `.github/conformance.yml`. `cron-components` defines the components for which the conformance test will be run against the master code in a scheduled manner.
## Integrating a new component with conformance tests
1. Add the component specific YAML to `tests/config/<COMPONENT-TYPE>/<COMPONENT>/<FILE>.yaml`.
2. All passwords will be of the form `${{PASSWORD_KEY}}` so that it is injected via environment variables.
3. Register the component `New**` function in `common.go`. Eg:
```go
...
switch tc.Component {
case "azure.servicebusqueues":
binding = b_azure_servicebusqueues.NewAzureServiceBusQueues(testLogger)
case "azure.storagequeues":
binding = b_azure_storagequeues.NewAzureStorageQueues(testLogger)
case "azure.eventgrid":
binding = b_azure_eventgrid.NewAzureEventGrid(testLogger)
case "kafka":
binding = b_kafka.NewKafka(testLogger)
case "new-component":
binding = b_new_component.NewComponent(testLogger)
default:
return nil
}
...
```
4. Add the config to `tests.yml` defined inside `tests/config/<COMPONENT-TYPE>/` folder. Eg:
```yaml
componentType: binding
components:
## All other components
- component: <COMPONENT>
allOperations: <true/false>
operations: <List of operations if needed>
```
5. Any UUID generation for keys can be specified using `$((uuid))`. Eg: See `../config/bindings/tests.yml`
3. Register the component `New**` function in `common.go`. For example:
```go
...
switch tc.Component {
case "azure.servicebusqueues":
binding = b_azure_servicebusqueues.NewAzureServiceBusQueues(testLogger)
case "azure.storagequeues":
binding = b_azure_storagequeues.NewAzureStorageQueues(testLogger)
case "azure.eventgrid":
binding = b_azure_eventgrid.NewAzureEventGrid(testLogger)
case "kafka":
binding = b_kafka.NewKafka(testLogger)
case "new-component":
binding = b_new_component.NewComponent(testLogger)
default:
return nil
}
...
```
4. Add the config to `tests.yml` defined inside `tests/config/<COMPONENT-TYPE>/` folder. For example:
```yaml
componentType: binding
components:
## All other components
- component: <COMPONENT>
allOperations: <true/false>
operations: <List of operations if needed>
```
5. Any UUID generation for keys can be specified using `$((uuid))`. E.g. see [/tests/config/bindings/tests.yml](../config/bindings/tests.yml)
6. Run the specific conformance test following the process below.
7. Follow steps 2 and 3 for changes to the [workflow](#conformance-test-workflow).
## Running conformance tests
1. Test setup is independent of the test run.
2. Run the service that needs to conformance tested locally or in your own cloud account.
- For cloud-agnostic components such as Kafka, MQTT etc., there are `docker-compose` definitions under the [/.github/infrastructure](https://github.com/dapr/components-contrib/tree/master/.github/infrastructure) folder you can use to quickly create an instance of the service. For example, to setup Kafka for conformance tests:
```bash
docker-compose -f ./.github/infrastructure/docker-compose-kafka.yml -p kafka up -d
```
- For Azure components such as Blob Storage, Key Vault etc., there is an automation script that can help you create the resources under your subscription, and extract the environment variables needed to run the conformance tests. See [/.github/infrastructure/conformance/azure/README.md](../../.github/infrastructure/conformance/azure/README.md) for more details.
> Given the variability in components and how they need to be set up for the conformance tests, you may need to refer to the [GitHub workflow for conformance tests](../../.github/workflows/conformance.yml) for any extra setup required by some components. E.g. Azure Event Grid bindings require setting up an Ngrok instance or similar endpoint for the test.
3. Some conformance tests require credentials in the form of environment variables. For examples Azure CosmosDB conformance tests will need to have Azure CosmosDB credentials. You will need to supply them to make these tests pass.
5. To run specific tests, run:
```bash
# TEST_NAME can be TestPubsubConformance, TestStateConformance, TestSecretStoreConformance or TestBindingsConformance
# COMPONENT_NAME is the component name from the tests.yml file eg: azure.servicebus, redis, mongodb etc.
go test -v -tags=conftests -count=1 ./tests/conformance -run="${TEST_NAME}/${COMPONENT_NAME}"
4. To run specific tests, run:
```bash
# TEST_NAME can be TestPubsubConformance, TestStateConformance, TestSecretStoreConformance or TestBindingsConformance
# COMPONENT_NAME is the component name from the tests.yml file, e.g. azure.servicebus, redis, mongodb etc.
go test -v -tags=conftests -count=1 ./tests/conformance -run="${TEST_NAME}/${COMPONENT_NAME}"
```