Debugging an e2e test with delve
This commit is contained in:
Mauricio Poppe
parent
08b22fe560
commit
97f9d03363
|
|
@ -10,6 +10,7 @@
|
|||
- [Extracting a specific version of Kubernetes](#extracting-a-specific-version-of-kubernetes)
|
||||
- [Bringing up a cluster for testing](#bringing-up-a-cluster-for-testing)
|
||||
- [Debugging clusters](#debugging-clusters)
|
||||
- [Debugging an E2E test with a debugger (delve)](#debugging-an-e2e-test-with-a-debugger-delve)
|
||||
- [Local clusters](#local-clusters)
|
||||
- [Testing against local clusters](#testing-against-local-clusters)
|
||||
- [Version-skewed and upgrade testing](#version-skewed-and-upgrade-testing)
|
||||
|
|
@ -229,6 +230,7 @@ stale permissions can cause problems.
|
|||
- `sudo iptables -F`, clear ip tables rules left by the kube-proxy.
|
||||
|
||||
### Reproducing failures in flaky tests
|
||||
|
||||
You can run a test repeatedly until it fails. This is useful when debugging
|
||||
flaky tests. In order to do so, you need to set the following environment
|
||||
variable:
|
||||
|
|
@ -259,6 +261,121 @@ the provided directory (which should already exist).
|
|||
The Google-run Jenkins builds automatically collected these logs for every
|
||||
build, saving them in the `artifacts` directory uploaded to GCS.
|
||||
|
||||
### Debugging an E2E test with a debugger (delve)
|
||||
|
||||
When debugging E2E tests it's sometimes useful to pause in the middle of an E2E test
|
||||
to check the value of a variable or to check something in the cluster, instead of adding
|
||||
`time.Sleep(...)` we can run the E2E test with `delve`
|
||||
|
||||
Requirements:
|
||||
|
||||
- delve (https://github.com/go-delve/delve/tree/master/Documentation/installation)
|
||||
|
||||
For this example we'll debug a [sig-storage test that will provision storage from a snapshot](https://github.com/kubernetes/kubernetes/blob/3ed71cf190a3d6a6dcb965cf73224538059e8e5e/test/e2e/storage/testsuites/provisioning.go#L200-L236)
|
||||
|
||||
First, compile the E2E test suite with additional compiler flags
|
||||
|
||||
```sh
|
||||
# -N Disable optimizations.
|
||||
# -l Disable inlining.
|
||||
make WHAT=test/e2e/e2e.test GOGCFLAGS="all=-N -l" GOLDFLAGS=""
|
||||
```
|
||||
|
||||
Then set the env var `E2E_TEST_DEBUG_TOOL=delve` and then run the test with `./hack/gingko.sh` instead of `kubetest`, you should see the delve command line prompt
|
||||
|
||||
```sh
|
||||
E2E_TEST_DEBUG_TOOL=delve ./hack/ginkgo-e2e.sh --ginkgo.focus="sig-storage.*csi-hostpath.*Dynamic.PV.*default.fs.*provisioning.should.provision.storage.with.snapshot.data.source" --allowed-not-ready-nodes=10
|
||||
---
|
||||
Setting up for KUBERNETES_PROVIDER="gce".
|
||||
Project: ...
|
||||
Network Project: ...
|
||||
Zone: ...
|
||||
Trying to find master named '...'
|
||||
Looking for address '...'
|
||||
Using master: ... (external IP: XX.XXX.XXX.XX; internal IP: (not set))
|
||||
Type 'help' for list of commands.
|
||||
(dlv)
|
||||
```
|
||||
|
||||
Use the commands described in the [delve command lists](https://github.com/go-delve/delve/blob/master/Documentation/cli/README.md), for our example we'll set a breakpoint at the start of the method
|
||||
|
||||
```sh
|
||||
(dlv) break test/e2e/storage/testsuites/provisioning.go:201
|
||||
Breakpoint 1 set at 0x72856f2 for k8s.io/kubernetes/test/e2e/storage/testsuites.(*provisioningTestSuite).DefineTests.func4() _output/local/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:201
|
||||
```
|
||||
|
||||
When you're done setting breakpoints execute `continue` to continue the test, once the breakpoint hits you have the chance to explore variables in the test
|
||||
|
||||
```sh
|
||||
(dlv) continue
|
||||
Apr 16 20:29:18.724: INFO: Fetching cloud provider for "gce"
|
||||
I0416 20:29:18.725327 3669683 gce.go:909] Using DefaultTokenSource &oauth2.reuseTokenSource{new:(*oauth2.tokenRefresher)(0xc002b65d10), mu:sync.Mutex{state:0, sema:0x0}, t:(*oauth2.Token)(0xc0028e43c0)}
|
||||
W0416 20:29:18.891866 3669683 gce.go:477] No network name or URL specified.
|
||||
I0416 20:29:18.892058 3669683 e2e.go:129] Starting e2e run "ae1b58af-9e9e-4745-b1f4-27d763451f8e" on Ginkgo node 1
|
||||
{"msg":"Test Suite starting","total":1,"completed":0,"skipped":0,"failed":0}
|
||||
Running Suite: Kubernetes e2e suite
|
||||
===================================
|
||||
Random Seed: 1618604956 - Will randomize all specs
|
||||
Will run 1 of 5745 specs
|
||||
...
|
||||
------------------------------
|
||||
[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] provisioning
|
||||
should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource]
|
||||
_output/local/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:200
|
||||
[BeforeEach] [Testpattern: Dynamic PV (default fs)] provisioning
|
||||
_output/local/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51
|
||||
[BeforeEach] [Testpattern: Dynamic PV (default fs)] provisioning
|
||||
_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185
|
||||
STEP: Creating a kubernetes client
|
||||
Apr 16 20:29:24.747: INFO: >>> kubeConfig: ...
|
||||
STEP: Building a namespace api object, basename provisioning
|
||||
W0416 20:29:24.901750 3669683 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
|
||||
Apr 16 20:29:24.901: INFO: No PodSecurityPolicies found; assuming PodSecurityPolicy is disabled.
|
||||
STEP: Waiting for a default service account to be provisioned in namespace
|
||||
[It] should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource]
|
||||
_output/local/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:200
|
||||
> k8s.io/kubernetes/test/e2e/storage/testsuites.(*provisioningTestSuite).DefineTests.func4() _output/local/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:201 (hits goroutine(165):1 total:1) (PC: 0x72856f2)
|
||||
Warning: listing may not match stale executable
|
||||
196:
|
||||
197: l.testCase.TestDynamicProvisioning()
|
||||
198: })
|
||||
199:
|
||||
200: ginkgo.It("should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource]", func() {
|
||||
=> 201: if !dInfo.Capabilities[storageframework.CapSnapshotDataSource] {
|
||||
202: e2eskipper.Skipf("Driver %q does not support populate data from snapshot - skipping", dInfo.Name)
|
||||
203: }
|
||||
204: if !dInfo.SupportedFsType.Has(pattern.FsType) {
|
||||
205: e2eskipper.Skipf("Driver %q does not support %q fs type - skipping", dInfo.Name, pattern.FsType)
|
||||
206: }
|
||||
(dlv) print dInfo
|
||||
*k8s.io/kubernetes/test/e2e/storage/framework.DriverInfo {
|
||||
Name: "csi-hostpath",
|
||||
InTreePluginName: "",
|
||||
FeatureTag: "",
|
||||
MaxFileSize: 104857600,
|
||||
SupportedSizeRange: k8s.io/kubernetes/test/e2e/framework/volume.SizeRange {Max: "", Min: "1Mi"},
|
||||
SupportedFsType: k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/sets.String [
|
||||
"": {},
|
||||
],
|
||||
SupportedMountOption: k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/sets.String nil,
|
||||
RequiredMountOption: k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/sets.String nil,
|
||||
Capabilities: map[k8s.io/kubernetes/test/e2e/storage/framework.Capability]bool [
|
||||
"persistence": true,
|
||||
"snapshotDataSource": true,
|
||||
"multipods": true,
|
||||
"block": true,
|
||||
"pvcDataSource": true,
|
||||
"controllerExpansion": true,
|
||||
"singleNodeVolume": true,
|
||||
"volumeLimits": true,
|
||||
],
|
||||
RequiredAccessModes: []k8s.io/kubernetes/vendor/k8s.io/api/core/v1.PersistentVolumeAccessMode len: 0, cap: 0, nil,
|
||||
TopologyKeys: []string len: 0, cap: 0, nil,
|
||||
NumAllowedTopologies: 0,
|
||||
StressTestOptions: *k8s.io/kubernetes/test/e2e/storage/framework.StressTestOptions {NumPods: 10, NumRestarts: 10},
|
||||
VolumeSnapshotStressTestOptions: *k8s.io/kubernetes/test/e2e/storage/framework.VolumeSnapshotStressTestOptions {NumPods: 10, NumSnapshots: 10},}
|
||||
```
|
||||
|
||||
### Local clusters
|
||||
|
||||
It can be much faster to iterate on a local cluster instead of a cloud-based
|
||||
|
|
|
|||
Loading…
Reference in New Issue