Commit Graph

571 Commits

Author SHA1 Message Date
Madhav Jivrajani db00da87d8 cacher: Fix watch behaviour for unset RV
The original design was to honour strong consistency
semantics for when the RV is unset, i.e. serve the
watch by doing a quorum read.

However, the implementation did not match the intent,
in that, the Cacher did not distinguish between set
and unset RV. This commit rectifies that behaviour by
serving the watch from the underlying storage if the
RV is unset.

This commit subsequently also adds a test for the same.

Signed-off-by: Madhav Jivrajani <madhav.jiv@gmail.com>

Kubernetes-commit: 610b67031c79c6c38964631d27dd59df357c6d2e
2023-01-16 11:10:05 +05:30
Madhav Jivrajani fd369afe45 cacher: Allow injecting errors for Watch()
This commit allows injecting errors for the
Watch() method of the dummy storage impl.

As a consequence of this, a race is introduced
between when the injected error is written and
read whenever a Watch() is invoked using the
dummy storage. This commit adds locking in order
to mitigate this.

Signed-off-by: Madhav Jivrajani <madhav.jiv@gmail.com>

Kubernetes-commit: 2593671337ad449f51b9dcc0b63aa190dd07ab68
2023-01-16 12:48:43 +05:30
ZhangYu 7e4c7c03fd change unnecessary switch to if in metrics.go
Kubernetes-commit: 48b9f7bb6c8298a00cdc04e394af15afe5804a3e
2022-11-30 11:04:03 +08:00
Rita Zhang f471919cab kmsv2: add metrics
Signed-off-by: Rita Zhang <rita.z.zhang@gmail.com>

Kubernetes-commit: bd0f7f8ee8f7f1c7809e17fa60804bb37f65c495
2023-01-29 22:40:18 -08:00
Anish Ramasekar 0304cd9f08 logcheck.conf: ensure that kms and value/encrypt uses structured logging
Signed-off-by: Anish Ramasekar <anish.ramasekar@gmail.com>

Kubernetes-commit: b6b00e65bcfe40fdaca9ce8b214baf5015ad7c5c
2023-02-10 00:17:54 +00:00
Anish Ramasekar c9656aa265 [KMS] Use structured logging in value/encrypt
Signed-off-by: Anish Ramasekar <anish.ramasekar@gmail.com>

Kubernetes-commit: 6db7496db60816c36b519a940fa1b0af1c0898da
2023-02-09 19:54:37 +00:00
Anish Ramasekar 63ca46e40a [KMSv2] Add metrics for grpc service
Signed-off-by: Anish Ramasekar <anish.ramasekar@gmail.com>

Kubernetes-commit: de3b2d525b42d6eedeaa0b157f0f5ab98a4821f4
2023-02-09 07:46:29 +00:00
Anish Ramasekar 421ef770de [KMSv2] store hash of encrypted DEK as key in cache
Signed-off-by: Anish Ramasekar <anish.ramasekar@gmail.com>

Kubernetes-commit: f72cf5c510cf2cf7b8ee375f5c2ec835e3ed225a
2023-01-27 00:19:32 +00:00
baomingwang 3f56cdd970 Added serialization from etcd error metric
Kubernetes-commit: 92c490f0bca03b6caea896ea10415a0d77c9efdc
2022-12-08 19:34:09 -08:00
Anish Ramasekar 9fb6b944f0 kmsv2: implement expire cache with clock
Signed-off-by: Anish Ramasekar <anish.ramasekar@gmail.com>

Kubernetes-commit: 4804baa01187b4251bd632e07721d875f567d6f1
2022-09-14 20:01:45 +00:00
Tim Hockin 979698ec2b Set proto go_package: kms API
This creates some diff to the *.pb.go files to note that
in the "options".

You can dump the gzipped blob with the following program (thanks
StackOverflow!):

```go
package main

import (
	"bytes"
	"compress/gzip"
	"encoding/json"
	"fmt"
	"os"

	"io/ioutil"

	proto "github.com/golang/protobuf/proto"
	dpb "github.com/golang/protobuf/protoc-gen-go/descriptor"
)

func main() {
	m := map[string][]byte{
		"before": blobv1,
		"after": blobv2,
	}
	arg := os.Args[1]
	dump(m[arg])
}

func dump(bytes []byte) {
	fd, err := decodeFileDesc(bytes)
	if err != nil {
		panic(err)
	}
	b, err := json.MarshalIndent(fd, "", "  ")
	if err != nil {
		panic(err)
	}
	fmt.Println(string(b))
}

// decompress does gzip decompression.
func decompress(b []byte) ([]byte, error) {
	r, err := gzip.NewReader(bytes.NewReader(b))
	if err != nil {
		return nil, fmt.Errorf("bad gzipped descriptor: %v", err)
	}
	out, err := ioutil.ReadAll(r)
	if err != nil {
		return nil, fmt.Errorf("bad gzipped descriptor: %v", err)
	}
	return out, nil
}

func decodeFileDesc(enc []byte) (*dpb.FileDescriptorProto, error) {
	raw, err := decompress(enc)
	if err != nil {
		return nil, fmt.Errorf("failed to decompress enc: %v", err)
	}

	fd := new(dpb.FileDescriptorProto)
	if err := proto.Unmarshal(raw, fd); err != nil {
		return nil, fmt.Errorf("bad descriptor: %v", err)
	}
	return fd, nil
}
```

Kubernetes-commit: ab11d8a4495aa0ff03b94c8d1a5345abfcca2c59
2023-01-14 10:21:53 -08:00
Patrick Ohly 8f8c30ff8f logging: fix names of keys
The stricter checking with the upcoming logcheck v0.4.1 pointed out these names
which don't comply with our recommendations in
https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/migration-to-structured-logging.md#name-arguments.

Kubernetes-commit: bc6c7fa91201348d010b638fbadf32007c0ac546
2023-01-16 15:04:01 +01:00
Tim Hockin 72929fc438 Call update-proto-bindings from update-codegen
One script to bring them all ...

Kubernetes-commit: 4dae505d531e149881788dc36148602967419c75
2023-01-05 15:41:51 -08:00
Tim Hockin 05e000c1bd Merge 5 fragile proto-bindings scripts into 1
Each of these scripts is basically identical, and all were too brittle.
Now they should be more resilient and easier to manage.  The script
still needs to be updated if we add new ones, which I do not love.

More cleanup to follow.

Kubernetes-commit: e0ecccff3f5148cc167117ac73233b4edc1640d8
2023-01-05 13:53:59 -08:00
Rita Zhang 19c40b040a kmsv2: use status key ID to update staleness of encrypted data
Signed-off-by: Rita Zhang <rita.z.zhang@gmail.com>

Kubernetes-commit: 510ac9b3919b62cb52e166192a7d0d09a9b2dfb5
2022-12-02 13:32:34 -08:00
Krzysztof Ostrowski a85078bf03 apiserver/kmsv2: mv Service interface into kmsv2
Signed-off-by: Krzysztof Ostrowski <kostrows@redhat.com>

Kubernetes-commit: b7701b00eaa8cdc2103beb8ab78f625cc3b62d90
2023-01-09 14:36:06 +01:00
Wojciech Tyczyński 8808b718c9 Explicit sorting in TestList storage test
Kubernetes-commit: 34de5fa73dabd9dcbd6f535cf6a853371ec653f1
2022-11-21 12:03:54 +01:00
Wojciech Tyczyński 299c158ba3 Reuse generic List test for watchcache
Kubernetes-commit: ef8c4fbca8e5bed1e7edc162b95c412a7f1a758e
2022-11-04 22:01:37 +01:00
Wojciech Tyczyński 012ace9f52 Fix lack of sorting of list results in watchcache
Kubernetes-commit: aaaee155dd3c9402e674d066f4d6ca30cc971c8e
2022-11-08 13:06:56 +01:00
Wojciech Tyczyński 8cd19efa73 Fix inconsistency in returning list results both in etcd3 and watchcache
Kubernetes-commit: da8d197a7205c01fdf376246995b0a1b3ab53859
2022-11-08 11:47:28 +01:00
Jordan Liggitt c6b4e07ee4 Fix indentation/spacing in comments to render correctly in godoc
Kubernetes-commit: 78cb3862f11225135afdf76f3424e2d7b33104c7
2022-12-17 17:31:05 -05:00
Wojciech Tyczyński a29f52cb30 Standardize naming in generic storage watcher tests
Kubernetes-commit: 593742f092e1ae81988c12e6d833acb29c02009e
2022-11-08 09:52:59 +01:00
sanwishe cb73680ff8 fix etcd compaction doc
Kubernetes-commit: 8a013b36f3af870aad85654cb317366dc7831cb0
2022-08-17 02:35:16 +00:00
Tim Allclair 3268b0561b Validate etcd paths
Kubernetes-commit: f1693a02c46f79f8c90a4ef17c4a750034f21484
2022-10-10 18:15:22 -07:00
Wojciech Tyczyński 0ed87195b0 Standardize naming in generic storage tests
Kubernetes-commit: e49c225eb9e7c428aca97e8031089e7ad91ca7c4
2022-11-05 12:14:56 +01:00
Wojciech Tyczyński 82397f8421 Fix naming in storage test setup
Kubernetes-commit: 1bc95675b19c68807ed3ae45a24701fc5c2fb935
2022-11-04 15:37:47 +01:00
Wojciech Tyczyński c51bbc3183 Parallelize tests cases in some storage tests
Kubernetes-commit: 6d85f947bf86d12e4459d31029c41d6049d40c0e
2022-11-04 10:23:40 +01:00
Wojciech Tyczyński 44c802979e Reuse generic ListNonRecurisve test for cacher
Kubernetes-commit: e301306d89a69ff1a328a4ae3fa39a9e3deb022e
2022-11-03 14:58:34 +01:00
Wojciech Tyczyński b312c7eb5b Stop exporting storage testing utility functions
Kubernetes-commit: 37db332298fc6d14a798f610ce6049792299f0b1
2022-11-03 08:47:38 +01:00
Wojciech Tyczyński a47bbc6347 Minor cleanup in etcd3 tests
Kubernetes-commit: 83399ccfb8822cc31a6a3a66e8352591d88feaa8
2022-10-31 18:02:40 +01:00
Wojciech Tyczyński a24d8963d1 Refactor remaining etcd3 tests to make them generic
Kubernetes-commit: e6ddb69e6fd06bf9a02fa731f12472f3c375762d
2022-10-31 18:02:19 +01:00
Wojciech Tyczyński c38aa93533 Reuse generic TestGet in cache tests.
Kubernetes-commit: 75a1ef87b32213a7d3d1c2027dd515e4f74c1777
2022-10-27 16:00:30 +02:00
Wojciech Tyczyński 07db92af49 Fix setting resource version on deletion
Kubernetes-commit: bbcf5e38776f2b18026539a0fbcf3aa505386c1f
2022-10-26 21:37:19 +02:00
Wojciech Tyczyński 499bbb88dc Refactor WatchError test to make it generic
Kubernetes-commit: b3f9272d57a71a9c50b38754b7309ddf68adb93f
2022-10-28 11:35:58 +02:00
Wojciech Tyczyński 3f5a3e0ae0 Refactor etcd3 list consistency test
Kubernetes-commit: cd5da36c92f3ed52debc22c39a7cd9d369b0eecd
2022-10-27 10:19:09 +02:00
Wojciech Tyczyński c527f6432a Minor cleanup of etcd3 tests
Kubernetes-commit: bbe1ebc82aa019118c91e8447517f91b9d036c1e
2022-10-27 09:48:06 +02:00
Wojciech Tyczyński f6802e074e Refactor storage tests using compaction
Kubernetes-commit: b02f172cbdf7e824d13a6a9c3a9b9fe4f3f92afb
2022-10-26 20:47:19 +02:00
Wojciech Tyczyński 834cf0fc14 Refactor compaction in etcd3 tests
Kubernetes-commit: 7da7ddd779f9ea835f0c57deae05e050c543066b
2022-10-26 16:34:28 +02:00
David Ashpole aa161f2fc0 migrate apiserver utiltrace usage to component-base/tracing
Kubernetes-commit: de26b9023f2872c5cd7e15fad5dd5ab649222c13
2022-10-20 18:15:38 +00:00
Tim Allclair bd7c7f52c2 Consolidate AuditContext
Kubernetes-commit: f1d684b7b60b39b7dc1eb4156307c593f0ba74e1
2022-07-12 11:53:57 -07:00
Wojciech Tyczyński 3afe8ebf5f Minor cleanup of etcd3 tests
Kubernetes-commit: e04fe81dfcff189ef5162b9b26b55760f613a010
2022-10-26 13:05:49 +02:00
Wojciech Tyczyński 73a664638b Refactor pagination tests
Kubernetes-commit: 6c8ce894e18a2e2cae20edd2f6e79db9407ebce3
2022-10-26 12:18:21 +02:00
Wojciech Tyczyński dec6f492e8 Refactor GetReads
Kubernetes-commit: 8472e1bc13b0fa8aa3a67e5fcf2f13bfd7974cd3
2022-10-26 11:30:32 +02:00
Wojciech Tyczyński 5a8fdf8480 Move GuaranteedUpdateChecksData test to generic package
Kubernetes-commit: afc5ded83931cdb8a2643486d6834eb8d9b04588
2022-10-25 14:31:20 +02:00
Wojciech Tyczyński d9a3685d6f Minor cleanup of storage tests
Kubernetes-commit: 5344bc5e1b7e50d5ff359c714d522b421b28bd12
2022-10-25 14:09:36 +02:00
Wojciech Tyczyński 1960fa837f Move GuaranteedUpdate test to generic test package
Kubernetes-commit: 012676acc3dd2b235bd601e4105294d2715f155b
2022-10-25 14:04:00 +02:00
Han Kang e9f139f163 swap name for storage metric for underlying storage db
Change-Id: Ic7cd9bf5ef29d9c51a728ef37c94d77172fbf678

Kubernetes-commit: 3a2d1bab26bfb9e7d887207dec0c51645c7017fe
2022-10-24 15:27:05 -04:00
Oscar Utbult 9d3d7b483a grammar: replace all occurrences of "the the" with "the"
Kubernetes-commit: e4f776f23098ecf942cafa898777195adbc800f1
2022-09-17 22:58:01 +02:00
lixiaobing1 c144979a82 replace WithInsecure() with WithTransportCredentials()
Kubernetes-commit: 7892175acdb329d44cf1f34230f78e608b3cb736
2022-10-15 16:41:53 +08:00
Monis Khan f4ac8fc8bb prefixTransformers: allocate after transformation length is known
Signed-off-by: Monis Khan <mok@microsoft.com>

Kubernetes-commit: 59e1a32fc8ed35e328a3971d3a1d640ffc28ff55
2022-10-17 16:56:40 -04:00