RANCHER 🔹 TFP-AUTOMATION

Airgap	Post Release	Proxy	Recurring	Registry	Sanity	KDM

tfp-automation is a Github Actions based testing framework designed to handle the following tasks:

• Conduct daily regression testing amongst supported Rancher release lines
• Automate release testing across different permutations of a Rancher HA environment (e.g. normal, airgap, proxy)
• Support infrastructure creation for various node providers

The above points are done with an emphasis on testing the Rancher2 Terraform provider. This framework utilizes Terratest alongside Go to accomplish these goals.

---

📜 Table of contents

• Configurations
  • Infrastructure
  • Rancher
  • Terraform
    • AKS
    • EKS
    • GKE
    • AZURE_RKE1
    • EC2_RKE1
    • HARVESTER_RKE1
    • LINODE_RKE1
    • VSPHERE_RKE1
    • AZURE_RKE2 + AZURE_K3S
    • EC2_RKE2 + EC2_K3S
    • HARVESTER_RKE2 + HARVESTER_K3S
    • LINODE_RKE2 + LINODE_K3S
    • VSPHERE_RKE2 + VSPHERE_K3S
  • Terratest
    • Nodepools
      • AKS Nodepools
      • EKS Nodepools
      • GKE Nodepools
      • RKE1, RKE2, K3S Nodepools
    • Provision
    • Kubernetes Upgrade
    • Snapshots
    • Build Module
    • Cleanup

---

Configurations

When testing locally, the following environment variables should be exported:

export RANCHER2_PROVIDER_VERSION=""                                     # Required
export CLOUD_PROVIDER_VERSION=""                                        # Required for custom cluster / infrastructure building
export KUBERNETES_PROVIDER_VERSION=""                                   # Required for infrastructure building using Harvester
export LOCALS_PROVIDER_VERSION=""                                       # Required for custom cluster / infrastructure building
export QASE_AUTOMATION_TOKEN=""                                         # Required for local Qase reporting
export QASE_TEST_RUN_ID=""                                              # Required for local Qase reporting

These tests require an accurately configured cattle-config.yaml to successfully run.

Each cattle-config.yaml must include the following configurations:

rancher:
  # define rancher specific configs here

terraform:
  # define module specific configs here
  
terratest:
  # define test specific configs here

---

🔺 Back to top

As part of this framework, you have the ability to spin up a Rancher HA environment using various node providers. The infrasturcture folder hosts a series of different test files that explain further. As far as different configurations needed to make this happen, see below:

Infrastructure

terraform:
  cni: "calico"
  defaultClusterRoleForProjectMembers: "true"
  enableNetworkPolicy: false
  provider: ""                              # The following providers are supported: aws | linode | harvester
  privateKeyPath: ""
  resourcePrefix: ""
  windowsPrivateKeyPath: ""

  # Fill out the AWS section if provider is set to aws.
  awsCredentials:
    awsAccessKey: ""
    awsSecretKey: ""
  awsConfig:
    ami: ""
    awsKeyName: ""
    awsInstanceType: ""
    region: ""
    awsSecurityGroups: [""]
    awsSecurityGroupNames: [""]
    awsSubnetID: ""
    awsVpcID: ""
    awsZoneLetter: "a"
    awsRootSize: 100
    awsRoute53Zone: ""
    awsUser: ""
    sshConnectionType: "ssh"
    timeout: "5m"
    windowsAMI: ""
    windowsAwsUser: ""
    windowsInstanceType: ""
    windowsKeyName: ""

  # Fill out the Linode section if provider is set to linode.
  linodeCredentials:
    linodeToken: ""  
  linodeConfig:
    clientConnThrottle: 20
    domain: ""
    linodeImage: ""
    linodeRootPass: ""
    privateIP: true
    region: ""
    soaEmail: ""
    swapSize: 256
    tags: [""]
    timeout: "5m"
    type: ""
  
  # Fill out this Harvester section if provider is set to harvester.
  harvesterCredentials:
    clusterId: ""
    clusterType: "imported"
    kubeconfigContent: ""
  harvesterConfig:
    diskSize: "30"
    cpuCount: "4"
    memorySize: "8"
    networkNames: [""]
    imageName: ""
    vmNamespace: "default"
    sshUser: ""

  # Fill out this vSphere section if provider is set to vsphere.
  vsphereCredentials:
    password: ""
    username: ""
    vcenter: ""
  vsphereConfig:  
    cloneFrom: ""
    cpuCount: ""
    datacenter: ""
    datastore: ""
    datastoreCluster: ""
    diskSize: ""
    guestID: "ubuntu64Guest"      # This will change depending on the OS you're using
    folder: ""
    hostSystem: ""
    memorySize: ""
    standaloneNetwork: ""
    vsphereUser: ""

  standalone:
    bootstrapPassword: ""
    certManagerVersion: "v1.15.3"
    osUser: ""
    osGroup: ""
    rancherChartRepository: "https://releases.rancher.com/server-charts/"
    rancherHostname: ""
    rancherImage: "rancher/rancher"
    rancherTagVersion: "v2.11.0"
    repo: "latest"
    rke2Version: "v1.30.9+rke2r1"

---

🔺 Back to top

The rancher configurations in the cattle-config.yaml will remain consistent across all modules and tests.

Rancher

rancher:
  host: url-to-rancher-server.com
  adminToken: token-XXXXX:XXXXXXXXXXXXXXX
  insecure: true
  cleanup: true

---

🔺 Back to top

The terraform configurations in the cattle-config.yaml are module specific. Fields to configure vary per module. Below are generic fields that are applicable regardless of module. See them below:

Terraform

terraform:
  etcd:                                       # This is an optional block.
    disableSnapshot: false
    snapshotCron: "0 */5 * * *"
    snapshotRetention: 6
    s3:
      bucket: ""
      cloudCredentialName: ""
      endpoint: ""
      endpointCA: ""
      folder: ""
      region: ""
      skipSSLVerify: true
  etcdRKE1:                                   # This is an optional block
    backupConfig:
      enabled: true
      intervalHours: 12
      safeTimestamp: true
      timeout: 120
      s3BackupConfig:
        accessKey: ""
        bucketName: ""
        endpoint: ""
        folder: ""
        region: ""
        secretKey: ""
    retention: "72h"
    snapshot: false
  cloudCredentialName: ""
  defaultClusterRoleForProjectMembers: "true" # Can be "true" or "false"
  enableNetworkPolicy: false                  # Can be true or false
  hostnamePrefix: ""   
  machineConfigName: ""                       # RKE2/K3S specific
  networkPlugin: ""                           # RKE1 specific
  nodeTemplateName: ""                        # RKE1 specific
  privateRegistries:                          # This is an optional block. You must already have a private registry stood up
    engineInsecureRegistry: ""                # RKE1 specific
    url: ""
    systemDefaultRegistry: ""                 # RKE2/K3S specific, can be left blank
    username: ""                              # RKE1 specific
    password: ""                              # RKE1 specific
    insecure: true
    authConfigSecretName: ""                  # RKE2/K3S specific
    mirrorHostname: ""
    mirrorEndpoint: ""
    mirrorRewrite: ""
  chartValues: |-			      # Provided as a multiline string
    rke2-cilium:                              # RKE2/Cilium specific example of how to do a Kube-proxy Replacement deployment
      k8sServiceHost: 127.0.0.1               
      k8sServicePort: 6443
      kubeProxyReplacement: true
  cni: cilium				      # RKE2 specific
  disable-kube-proxy: true		      # Can be "true" or "false"

Note: At this time, private registries for RKE2/K3s MUST be used with provider version 3.1.1. This is due to issue https://github.com/rancher/terraform-provider-rancher2/issues/1305.

🔺 Back to top

AKS

terraform:
  module: aks
  cloudCredentialName: tf-aks
  azureCredentials:
    clientId: ""
    clientSecret: ""
    environment: "AzurePublicCloud"
    subscriptionId: ""
    tenantId: ""
  azureConfig:
    availabilityZones:
      - '1'
      - '2'
      - '3'
    image: ""
    location: ""
    managedDisks: false
    mode: "System"
    name: "agentpool"
    networkDNSServiceIP: ""
    networkDockerBridgeCIDR: ""
    networkPlugin: ""
    networkServiceCIDR: ""
    noPublicIp: false
    osDiskSizeGB: 128
    outboundType: "loadBalancer"
    resourceGroup: ""
    resourceLocation: ""
    subnet: ""
    taints: ["none:PreferNoSchedule"]
    vmSize: Standard_DS2_v2
    vnet: ""

---

🔺 Back to top

EKS

terraform:
  module: eks
  cloudCredentialName: tf-eks
  hostnamePrefix: tfp
  awsCredentials:
    awsAccessKey: ""
    awsSecretKey: ""
  awsConfig:
    awsInstanceType: t3.medium
    region: us-east-2
    awsSubnets:
      - ""
      - ""
    awsSecurityGroups:
      - ""
    publicAccess: true
    privateAccess: true

---

🔺 Back to top

GKE

terraform:
  module: gke
  cloudCredentialName: tf-creds-gke
  hostnamePrefix: tfp
  googleCredentials:
    authEncodedJson: |-
      {
        "type": "service_account",
        "project_id": "",
        "private_key_id": "",
        "private_key": "",
        "client_email": "",
        "client_id": "",
        "auth_uri": "https://accounts.google.com/o/oauth2/auth",
        "token_uri": "https://oauth2.googleapis.com/token",
        "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
        "client_x509_cert_url": ""
      }      
  googleConfig:
    region: us-central1-c
    projectID: ""
    network: default
    subnetwork: default

---

🔺 Back to top

AZURE_RKE1

terraform:
  module: azure_rke1
  networkPlugin: canal
  nodeTemplateName: tf-rke1-template
  hostnamePrefix: tfp
  azureCredentials:
    clientId: ""
    clientSecret: ""
    environment: "AzurePublicCloud"
    subscriptionId: ""
    tenantId: ""
  azureConfig:
    availabilitySet: "docker-machine"
    subscriptionId: ""
    customData: ""
    diskSize: "100"
    dockerPort: "2376"
    faultDomainCount: "3"
    image: "Canonical:0001-com-ubuntu-server-jammy:22_04-lts:latest"
    location: "westus2"
    managedDisks: false
    noPublicIp: false
    openPort: ["6443/tcp","2379/tcp","2380/tcp","8472/udp","4789/udp","9796/tcp","10256/tcp","10250/tcp","10251/tcp","10252/tcp"]
    privateIpAddress: ""
    resourceGroup: ""
    size: "Standard_D2_v2"
    sshUser: "azureuser"
    staticPublicIp: false
    storageType: "Standard_LRS"
    updateDomainCount: "5"

---

🔺 Back to top

EC2_RKE1

terraform:
  module: ec2_rke1
  networkPlugin: canal
  nodeTemplateName: tf-rke1-template
  hostnamePrefix: tfp
  awsCredentials:
    awsAccessKey: ""
    awsSecretKey: ""
  awsConfig:
    ami:
    awsInstanceType: t3.medium
    region: us-east-2
    awsSecurityGroupNames:
      - security-group-name
    awsSubnetID: subnet-xxxxxxxx
    awsVpcID: vpc-xxxxxxxx
    awsZoneLetter: a
    awsRootSize: 80

---

🔺 Back to top

HARVESTER_RKE1

terraform:
  module: harvester_rke1
  networkPlugin: canal
  nodeTemplateName: tf-rke1-template
  hostnamePrefix: tfp
  harvesterCredentials:
    clusterId: "c-m-clusterID"
    clusterType: "imported"
    kubeconfigContent: |
      kubeconfig-content      
  harvesterConfig:
    diskSize: "30"
    cpuCount: "4"
    memorySize: "8"
    networkNames: ["default/net-name"]
    imageName: "default/image-name"
    vmNamespace: "default"
    sshUser: "ubuntu"

---

🔺 Back to top

LINODE_RKE1

terraform:
  module: linode_rke1
  networkPlugin: canal
  nodeTemplateName: tf-rke1-template
  hostnamePrefix: tfp
  linodeCredentials:
    linodeToken: ""
  linodeConfig:
    region: us-east
    linodeRootPass: ""

---

🔺 Back to top

VSPHERE_RKE1

terraform:
  module: vsphere_rke1
  networkPlugin: canal
  nodeTemplateName: tf-rke1-template
  hostnamePrefix: tfp
  vsphereCredentials:
    password: ""
    username: ""
    vcenter: ""
    vcenterPort: "443"
  vsphereConfig:  
    cfgparam: ["disk.enableUUID=TRUE"]
    cloneFrom: ""
    cloudConfig: ""
    contentLibrary: ""
    cpuCount: "4"
    creationType: "template"
    datacenter: ""
    datastore: ""
    datastoreCluster: ""
    diskSize: "40000"
    folder: ""
    hostsystem: ""
    memorySize: "8192"
    network: [""]
    pool: ""
    sshPassword: "tcuser"
    sshPort: "22"
    sshUser: "docker"
    sshUserGroup: "staff"

---

🔺 Back to top

AZURE_RKE2 + AZURE_K3S

terraform:
  module: azure_k3s
  networkPlugin: canal
  nodeTemplateName: tf-rke1-template
  hostnamePrefix: tfp
  azureCredentials:
    clientId: ""
    clientSecret: ""
    environment: "AzurePublicCloud"
    subscriptionId: ""
    tenantId: ""
  azureConfig:
    availabilitySet: "docker-machine"
    customData: ""
    diskSize: "100"
    dockerPort: "2376"
    faultDomainCount: "3"
    image: "Canonical:0001-com-ubuntu-server-jammy:22_04-lts:latest"
    location: "westus2"
    managedDisks: false
    noPublicIp: false
    openPort: ["6443/tcp","2379/tcp","2380/tcp","8472/udp","4789/udp","9796/tcp","10256/tcp","10250/tcp","10251/tcp","10252/tcp"]
    privateIpAddress: ""
    resourceGroup: ""
    size: "Standard_D2_v2"
    sshUser: ""
    staticPublicIp: false
    storageType: "Standard_LRS"
    updateDomainCount: "5"

---

🔺 Back to top

EC2_RKE2 + EC2_K3S

terraform:
  module: ec2_rke2
  cloudCredentialName: tf-creds-rke2
  machineConfigName: tf-rke2
  enableNetworkPolicy: false
  defaultClusterRoleForProjectMembers: user
  awsCredentials:
    awsAccessKey: ""
    awsSecretKey: ""
  awsConfig:
    ami:
    region: us-east-2
    awsSecurityGroupNames:
      - my-security-group
    awsSubnetID: subnet-xxxxxxxx
    awsVpcID: vpc-xxxxxxxx
    awsZoneLetter: a

---

🔺 Back to top

HARVESTER_RKE2 + HARVESTER_K3S

terraform:
  module: harvester_rke2
  hostnamePrefix: tfp
  machineConfigName: tf-hvst
  harvesterCredentials:
    clusterId: "c-m-clusterID"
    clusterType: "imported"
    kubeconfigContent: |
      kubeconfig-content      
  harvesterConfig:
    diskSize: "30"
    cpuCount: "4"
    memorySize: "8"
    networkNames: ["default/net-name"]
    imageName: "default/image-name"
    vmNamespace: "default"
    sshUser: "ubuntu"

---

🔺 Back to top

LINODE_RKE2 + LINODE_K3S

terraform:
  module: linode_k3s
  cloudCredentialName: tf-linode-creds
  machineConfigName: tf-k3s
  enableNetworkPolicy: false
  defaultClusterRoleForProjectMembers: user
  linodeCredentials:
    linodeToken: ""
  linodeConfig:
    linodeImage: linode/ubuntu20.04
    region: us-east
    linodeRootPass: xxxxxxxxxxxx

---

🔺 Back to top

VSPHERE_RKE2 + VSPHERE_K3S

terraform:
  module: vsphere_k3s
  networkPlugin: canal
  nodeTemplateName: tf-rke1-template
  hostnamePrefix: tfp
  vsphereCredentials:
    password: ""
    username: ""
    vcenter: ""
    vcenterPort: ""
  vsphereConfig:  
    cfgparam: ["disk.enableUUID=TRUE"]
    cloneFrom: ""
    cloudConfig: ""
    contentLibrary: ""
    cpuCount: "4"
    creationType: "template"
    datacenter: ""
    datastore: ""
    datastoreCluster: ""
    diskSize: "40000"
    folder: ""
    hostsystem: ""
    memorySize: "8192"
    network: [""]
    pool: ""
    sshPassword: "tcuser"
    sshPort: "22"
    sshUser: "docker"
    sshUserGroup: "staff"

---

🔺 Back to top

The terratest configurations in the cattle-config.yaml are test specific. Fields to configure vary per test. The nodepools field in the below configurations will vary depending on the module. I will outline what each module expects first, then proceed to show the whole test specific configurations.

🔺 Back to top

Nodepools

type: []Nodepool

🔺 Back to top

AKS Nodepool

AKS nodepools only need the quantity of nodes per pool to be provided, of type int64. The below example will create a cluster with three node pools, each with a single node.

Example:

nodepools:
  - quantity: 1
  - quantity: 1
  - quantity: 1

🔺 Back to top

EKS Nodepool

EKS nodepools require the instanceType, as type string, the desiredSize of the nodepool, as type int64, the maxSize of the node pool, as type int64, and the minSize of the node pool, as type int64. The minimum requirement for an EKS nodepool's desiredSize is 2. This must be respected or the cluster will fail to provision.

Example:

nodepools:
  - instanceType: t3.medium
    desiredSize: 3
    maxSize: 3
    minSize: 0

🔺 Back to top

GKE Nodepool

GKE nodepools require the quantity of the node pool, as type int64, and the maxPodsContraint, as type int64.

Example:

nodepools:
  - quantity: 2
    maxPodsContraint: 110

🔺 Back to top

RKE1, RKE2, and K3S - all share the same nodepool configurations

For these modules, the required nodepool fields are the quantity, as type int64, as well as the roles to be assigned, each to be set or toggled via boolean - [etcd, controlplane, worker]. The following example will create three node pools, each with individual roles, and one node per pool.

Example:

nodepools:
  - quantity: 1
    etcd: true
    controlplane: false
    worker: false
  - quantity: 1
    etcd: false
    controlplane: true
    worker: false
  - quantity: 1
    etcd: false
    controlplane: false
    worker: true

That wraps up the sub-section on nodepools, circling back to the test specific configs now...

Test specific fields to configure in this section are as follows:

🔺 Back to top

Provision

terratest:
  pathToRepo: # REQUIRED - path to repo from user's go directory i.e. ../go/<path/to/repo/tfp-automation>
  nodepools:
    - quantity: 1
      etcd: true
      controlplane: false
      worker: false
    - quantity: 1
      etcd: false
      controlplane: true
      worker: false
    - quantity: 1
      etcd: false
      controlplane: false
      worker: true
  aksKubernetesVersion: ""
  eksKubernetesVersion: ""
  gkeKubernetesVersion: ""
  kubernetesVersion: ""
  nodeCount: 3

  # Below are the expected formats for all module kubernetes versions
  
  # AKS without leading v
  # e.g. '1.28.5'
  
  # EKS without leading v or any tail ending
  # e.g. '1.28'
  
  # GKE without leading v but with tail ending included
  # e.g. 1.28.7-gke.1026000
  
  # RKE1 with leading v and -rancher1-1 tail
  # e.g. v1.28.7-rancher1-1

  # RKE2 with leading v and +rke2r# tail
  # e.g. v1.28.7+rke2r1

  # K3S with leading v and +k3s# tail
  # e.g. v1.28.7+k3s1

Note: In this test suite, Terraform explicitly cleans up resources after each test case is performed. This is because Terraform will experience caching issues, causing tests to fail.

---

🔺 Back to top

Kubernetes Upgrade

terratest:
  pathToRepo: # REQUIRED - path to repo from user's go directory i.e. ../go/<path/to/repo/tfp-automation>
  nodepools:
    - quantity: 1
      etcd: true
      controlplane: false
      worker: false
    - quantity: 1
      etcd: false
      controlplane: true
      worker: false
    - quantity: 1
      etcd: false
      controlplane: false
      worker: true
  nodeCount: 3
  aksKubernetesVersion: ""
  eksKubernetesVersion: ""
  gkeKubernetesVersion: ""
  kubernetesVersion: ""
  upgradedAKSKubernetesVersion: ""
  upgradedEKSKubernetesVersion: ""
  upgradedGKEKubernetesVersion: ""
  upgradedKubernetesVersion: ""

Note: In this test suite, Terraform explicitly cleans up resources after each test case is performed. This is because Terraform will experience caching issues, causing tests to fail.

---

🔺 Back to top

ETCD Snapshots

terratest:
  pathToRepo: # REQUIRED - path to repo from user's go directory i.e. ../go/<path/to/repo/tfp-automation>
  snapshotInput: {}

Note: In this test suite, Terraform explicitly cleans up resources after each test case is performed. This is because Terraform will experience caching issues, causing tests to fail.

---

🔺 Back to top

Build Module

Build module test may be used and ran to create a main.tf terraform configuration file for the desired module. This is logged to the output for future reference and use.

Testing configurations for this are the same as outlined in provisioning test above. Please review provisioning test configurations for more details.

---

🔺 Back to top

Cleanup

Cleanup test may be used to clean up resources in situations where rancher config has cleanup set to false. This may be helpful in debugging. This test expects the same configurations used to initially create this environment, to properly clean them up.