# Aggregated Kubernetes API Endpoint

The newly introduced [karmada-aggregated-apiserver](https://github.com/karmada-io/karmada/blob/master/cmd/aggregated-apiserver/main.go) component aggregates all registered clusters and allows users to access member clusters through Karmada by the proxy endpoint,

For detailed discussion topic, see [here](https://github.com/karmada-io/karmada/discussions/1077).

Here's a quick start.

## Quick start

To quickly experience this feature, we experimented with karmada-apiserver certificate.

### Step1: Obtain the karmada-apiserver Certificate

For karmada deployed using `hack/local-up-karmada.sh`, you can directly copy it from the `/root/.kube/` directory.

```
cp /root/.kube/karmada.config karmada-apiserver.config
```

### Step2: Grant permission to user `system:admin`

`system:admin` is the user for karmada-apiserver certificate. We need to grant the `clusters/proxy` permission to it explicitly.

Apply the following yaml file:

cluster-proxy-rbac.yaml:

<details>

<summary>unfold me to see the yaml</summary>

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: cluster-proxy-clusterrole
rules:
- apiGroups:
  - 'cluster.karmada.io'
  resources:
  - clusters/proxy
  resourceNames:
  - member1
  - member2
  - member3
  verbs:
  - '*'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cluster-proxy-clusterrolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-proxy-clusterrole
subjects:
  - kind: User
    name: "system:admin"
```

</details>

```
kubectl --kubeconfig /root/.kube/karmada.config --context karmada-apiserver apply -f cluster-proxy-rbac.yaml
```

### Step3: Access member clusters

Run the below command (replace `{clustername}` with your actual cluster name):

```
kubectl --kubeconfig karmada-apiserver.config get --raw /apis/cluster.karmada.io/v1alpha1/clusters/{clustername}/proxy/api/v1/nodes
```

Or append `/apis/cluster.karmada.io/v1alpha1/clusters/{clustername}/proxy ` to the server address of karmada-apiserver.config, and then you can directly use:

```
kubectl --kubeconfig karmada-apiserver.config get node
```

> Note: For a member cluster that joins karmada in pull mode, we can [deploy apiserver-network-proxy (ANP)](../working-with-anp.md) to access it.

## Unified authentication

For one or a group of user subjects (users, groups, or service accounts) in a member cluster, we can import them into karmada control plane and grant them the `clusters/proxy` permission, so that we can access the member cluster with permission of the user subject through karmada.

In this section, we use a serviceaccount named `tom` for the test.

### Step1: Create ServiceAccount in member1 cluster (optional)

If the serviceaccount has been created in your environment, you can skip this step.

Create a serviceaccount that does not have any permission:

```
kubectl --kubeconfig /root/.kube/members.config --context member1 create serviceaccount tom
```

### Step2: Create ServiceAccount in karmada control plane

```
kubectl --kubeconfig /root/.kube/karmada.config --context karmada-apiserver create serviceaccount tom
```

In order to grant serviceaccount the `clusters/proxy` permission, apply the following rbac yaml file:

cluster-proxy-rbac.yaml:

<details>

<summary>unfold me to see the yaml</summary>

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: cluster-proxy-clusterrole
rules:
- apiGroups:
  - 'cluster.karmada.io'
  resources:
  - clusters/proxy
  resourceNames:
  - member1
  verbs:
  - '*'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cluster-proxy-clusterrolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-proxy-clusterrole
subjects:
  - kind: ServiceAccount
    name: tom
    namespace: default
  # The token generated by the serviceaccount can parse the group information. Therefore, you need to specify the group information below.
  - kind: Group
    name: "system:serviceaccounts"
  - kind: Group
    name: "system:serviceaccounts:default"
```

</details>

```
kubectl --kubeconfig /root/.kube/karmada.config --context karmada-apiserver apply -f cluster-proxy-rbac.yaml
```

### Step3: Access member1 cluster

Obtain token of serviceaccount `tom`:

```
kubectl get secret `kubectl get sa tom -oyaml | grep token | awk '{print $3}'` -oyaml | grep token: | awk '{print $2}' | base64 -d
```

Then construct a kubeconfig file `tom.config` for `tom` serviceaccount:

```yaml
apiVersion: v1
clusters:
- cluster:
    insecure-skip-tls-verify: true
    server: {karmada-apiserver-address} # Replace {karmada-apiserver-address} with karmada-apiserver-address. You can find it in /root/.kube/karmada.config file.
  name: tom
contexts:
- context:
    cluster: tom
    user: tom
  name: tom
current-context: tom
kind: Config
users:
- name: tom
  user:
    token: {token} # Replace {token} with the token obtain above.
```

Run the command below to access member1 cluster:

```
kubectl --kubeconfig tom.config get --raw /apis/cluster.karmada.io/v1alpha1/clusters/member1/proxy/apis
```

We can found that we were able to access, but run the command below:

```
kubectl --kubeconfig tom.config get --raw /apis/cluster.karmada.io/v1alpha1/clusters/member1/proxy/api/v1/nodes
```

It will fail because serviceaccount `tom` does not have any permissions in the member1 cluster.

### Step4: Grant permission to Serviceaccount in member1 cluster

Apply the following YAML file:

member1-rbac.yaml

<details>

<summary>unfold me to see the yaml</summary>

```
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: tom
rules:
- apiGroups:
  - '*'
  resources:
  - '*'
  verbs:
  - '*'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tom
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: tom
subjects:
  - kind: ServiceAccount
    name: tom
    namespace: default
```

</details>

```
kubectl --kubeconfig /root/.kube/members.config --context member1 apply -f member1-rbac.yaml
```

Run the command that failed in the previous step again:

```
kubectl --kubeconfig tom.config get --raw /apis/cluster.karmada.io/v1alpha1/clusters/member1/proxy/api/v1/nodes
```

The access will be successful.

Or we can append `/apis/cluster.karmada.io/v1alpha1/clusters/member1/proxy ` to the server address of tom.config , and then you can directly use:

```
kubectl --kubeconfig tom.config get node
```

> Note: For a member cluster that joins karmada in pull mode,  we can [deploy apiserver-network-proxy (ANP)](../working-with-anp.md) to access it.