diff --git a/hazelcast/README.md b/hazelcast/README.md index 2c841744..56a835f4 100644 --- a/hazelcast/README.md +++ b/hazelcast/README.md @@ -7,7 +7,7 @@ Any topology changes are communicated and handled by Hazelcast nodes themselves. This document also attempts to describe the core components of Kubernetes, _Pods_, _Services_ and _Replication Controllers_. ### Prerequisites -This example assumes that you have a Kubernetes cluster installed and running, and that you have installed the ```kubectl``` command line tool somewhere in your path. Please see the [getting started](https://github.com/GoogleCloudPlatform/kubernetes/tree/master/docs/getting-started-guides) for installation instructions for your platform. +This example assumes that you have a Kubernetes cluster installed and running, and that you have installed the `kubectl` command line tool somewhere in your path. Please see the [getting started](https://github.com/GoogleCloudPlatform/kubernetes/tree/master/docs/getting-started-guides) for installation instructions for your platform. ### A note for the impatient This is a somewhat long tutorial. If you want to jump straight to the "do it now" commands, please see the [tl; dr](#tl-dr) at the end. @@ -15,52 +15,14 @@ This is a somewhat long tutorial. If you want to jump straight to the "do it no ### Sources Source is freely available at: -* Docker image - https://github.com/pires/hazelcast-kubernetes * Hazelcast Discovery - https://github.com/pires/hazelcast-kubernetes-bootstrapper +* Dockerfile - https://github.com/pires/hazelcast-kubernetes * Docker Trusted Build - https://registry.hub.docker.com/u/pires/hazelcast-k8s ### Simple Single Pod Hazelcast Node -In Kubernetes, the atomic unit of an application is a [_Pod_](http://docs.k8s.io/pods.md). A Pod is one or more containers that _must_ be scheduled onto the same host. All containers in a pod share a network namespace, and may optionally share mounted volumes. In this simple case, we define a single container running Hazelcast for our pod: +In Kubernetes, the atomic unit of an application is a [_Pod_](http://docs.k8s.io/pods.md). A Pod is one or more containers that _must_ be scheduled onto the same host. All containers in a pod share a network namespace, and may optionally share mounted volumes. -```yaml -apiVersion: v1beta3 -kind: Pod -metadata: - labels: - name: hazelcast - name: hazelcast -spec: - containers: - - image: pires/hazelcast-k8s - name: hazelcast - ports: - - containerPort: 5701 - name: hazelcast - protocol: TCP - resources: - limits: - cpu: "1" -``` - -There are a few things to note in this description. First is that we are running the ```pires/hazelcast-k8s``` image. This is a standard Ubuntu 14.04 installation with Java 8. However it also adds a custom [```application ```](https://github.com/pires/hazelcast-kubernetes-bootstrapper) that finds any Hazelcast nodes in the cluster and bootstraps an Hazelcast instance. The ```HazelcastDiscoveryController``` discovers the Kubernetes API Server using the built in Kubernetes discovery service, and then uses the Kubernetes API to find new nodes (more on this later). - -You may also note that we tell Kubernetes that the container exposes the ```hazelcast``` port. Finally, we tell the cluster manager that we need 1 cpu core. - -Given this configuration, we can create the pod as follows: - -```sh -$ kubectl create -f hazelcast.yaml -``` - -After a few moments, you should be able to see the pod running: - -```sh -$ kubectl get pods hazelcast - -POD IP CONTAINER(S) IMAGE(S) HOST LABELS STATUS CREATED MESSAGE -hazelcast 10.245.2.68 e2e-test-minion-vj7k/104.197.8.214 name=hazelcast Running 11 seconds - hazelcast pires/hazelcast-k8s Running 1 seconds -``` +In this case, we shall not run a single Hazelcast pod, because the discovery mechanism now relies on a service definition. ### Adding a Hazelcast Service @@ -82,14 +44,60 @@ spec: name: hazelcast ``` -The important thing to note here is the ```selector```. It is a query over labels, that identifies the set of _Pods_ contained by the _Service_. In this case the selector is ```name: hazelcast```. If you look back at the Pod specification above, you'll see that the pod has the corresponding label, so it will be selected for membership in this Service. +The important thing to note here is the `selector`. It is a query over labels, that identifies the set of _Pods_ contained by the _Service_. In this case the selector is `name: hazelcast`. If you look at the Replication Controller specification below, you'll see that the pod has the corresponding label, so it will be selected for membership in this Service. Create this service as follows: ```sh $ kubectl create -f hazelcast-service.yaml ``` -Once the service is created, you can query it's endpoints: +### Adding replicated nodes +The real power of Kubernetes and Hazelcast lies in easily building a replicated, resizable Hazelcast cluster. + +In Kubernetes a _Replication Controller_ is responsible for replicating sets of identical pods. Like a _Service_ it has a selector query which identifies the members of it's set. Unlike a _Service_ it also has a desired number of replicas, and it will create or delete _Pods_ to ensure that the number of _Pods_ matches up with it's desired state. + +Replication Controllers will "adopt" existing pods that match their selector query, so let's create a Replication Controller with a single replica to adopt our existing Hazelcast Pod. + +```yaml +apiVersion: v1beta3 +kind: ReplicationController +metadata: + labels: + name: hazelcast + name: hazelcast +spec: + replicas: 1 + selector: + name: hazelcast + template: + metadata: + labels: + name: hazelcast + spec: + containers: + - resources: + limits: + cpu: 1 + image: pires/hazelcast-k8s:0.2 + name: hazelcast + ports: + - containerPort: 5701 + name: hazelcast +``` + +There are a few things to note in this description. First is that we are running the `pires/hazelcast-k8s` image, tag `0.2`. This is a `busybox` installation with JRE 8. However it also adds a custom [`application`](https://github.com/pires/hazelcast-kubernetes-bootstrapper) that finds any Hazelcast nodes in the cluster and bootstraps an Hazelcast instance accordingle. The `HazelcastDiscoveryController` discovers the Kubernetes API Server using the built in Kubernetes discovery service, and then uses the Kubernetes API to find new nodes (more on this later). + +You may also note that we tell Kubernetes that the container exposes the `hazelcast` port. Finally, we tell the cluster manager that we need 1 cpu core. + +The bulk of the replication controller config is actually identical to the Hazelcast pod declaration above, it simply gives the controller a recipe to use when creating new pods. The other parts are the ```selector``` which contains the controller's selector query, and the ```replicas``` parameter which specifies the desired number of replicas, in this case 1. + +Create this controller: + +```sh +$ kubectl create -f hazelcast-controller.yaml +``` + +After the controller provisions successfully the pod, you can query the service endpoints: ```sh $ kubectl get endpoints hazelcast -o yaml apiVersion: v1beta3 @@ -117,53 +125,11 @@ subsets: protocol: TCP ``` -You can see that the _Service_ has found the pod we created in step one. +You can see that the _Service_ has found the pod created by the replication controller. -### Adding replicated nodes -Of course, a single node cluster isn't particularly interesting. The real power of Kubernetes and Hazelcast lies in easily building a replicated, resizable Hazelcast cluster. +Now it gets even more interesting. -In Kubernetes a _Replication Controller_ is responsible for replicating sets of identical pods. Like a _Service_ it has a selector query which identifies the members of it's set. Unlike a _Service_ it also has a desired number of replicas, and it will create or delete _Pods_ to ensure that the number of _Pods_ matches up with it's desired state. - -Replication Controllers will "adopt" existing pods that match their selector query, so let's create a Replication Controller with a single replica to adopt our existing Hazelcast Pod. - -```yaml -apiVersion: v1beta3 -kind: ReplicationController -metadata: - labels: - name: hazelcast - name: hazelcast -spec: - replicas: 1 - selector: - name: hazelcast - template: - metadata: - labels: - name: hazelcast - spec: - containers: - - resources: - limits: - cpu: 1 - image: pires/hazelcast-k8s - name: hazelcast - ports: - - containerPort: 5701 - name: hazelcast -``` - -The bulk of the replication controller config is actually identical to the Hazelcast pod declaration above, it simply gives the controller a recipe to use when creating new pods. The other parts are the ```selector``` which contains the controller's selector query, and the ```replicas``` parameter which specifies the desired number of replicas, in this case 1. - -Create this controller: - -```sh -$ kubectl create -f hazelcast-controller.yaml -``` - -Now this is actually not that interesting, since we haven't actually done anything new. Now it will get interesting. - -Let's resize our cluster to 2: +Let's resize our cluster to 2 pods: ```sh $ kubectl resize rc hazelcast --replicas=2 ``` @@ -173,34 +139,40 @@ Now if you list the pods in your cluster, you should see two hazelcast pods: ```sh $ kubectl get pods POD IP CONTAINER(S) IMAGE(S) HOST LABELS STATUS CREATED MESSAGE -hazelcast 10.245.2.68 e2e-test-minion-vj7k/104.197.8.214 name=hazelcast Running 14 seconds - hazelcast pires/hazelcast-k8s Running 2 seconds -hazelcast-ulkws 10.245.1.80 e2e-test-minion-2x1f/146.148.62.37 name=hazelcast Running 7 seconds - hazelcast pires/hazelcast-k8s Running 6 seconds +hazelcast-pkyzd 10.244.90.3 e2e-test-minion-vj7k/104.197.8.214 name=hazelcast Running 14 seconds + hazelcast pires/hazelcast-k8s:0.2 Running 2 seconds +hazelcast-ulkws 10.244.66.2 e2e-test-minion-2x1f/146.148.62.37 name=hazelcast Running 7 seconds + hazelcast pires/hazelcast-k8s:0.2 Running 6 seconds ``` -Notice that one of the pods has the human readable name ```hazelcast``` that you specified in your config before, and one has a random string, since it was named by the replication controller. - -To prove that this all works, you can use the ```log``` command to examine the logs of one pod, for example: +To prove that this all works, you can use the `log` command to examine the logs of one pod, for example: ```sh -$ kubectl log 16b2beab-94a1-11e4-8a8b-42010af0e23e hazelcast -2014-12-24T01:21:09.731468790Z 2014-12-24 01:21:09.701 INFO 10 --- [ main] c.g.p.h.HazelcastDiscoveryController : Asking k8s registry at http://10.160.211.80:80.. -2014-12-24T01:21:13.686978543Z 2014-12-24 01:21:13.686 INFO 10 --- [ main] c.g.p.h.HazelcastDiscoveryController : Found 3 pods running Hazelcast. -2014-12-24T01:21:13.772599736Z 2014-12-24 01:21:13.772 INFO 10 --- [ main] c.g.p.h.HazelcastDiscoveryController : Added member 10.160.2.3 -2014-12-24T01:21:13.783689690Z 2014-12-24 01:21:13.783 INFO 10 --- [ main] c.g.p.h.HazelcastDiscoveryController : Added member 10.160.2.4 +$ kubectl log hazelcast-ulkws hazelcast +2015-05-09 22:06:20.016 INFO 5 --- [ main] com.github.pires.hazelcast.Application : Starting Application v0.2-SNAPSHOT on hazelcast-enyli with PID 5 (/bootstrapper.jar started by root in /) +2015-05-09 22:06:20.071 INFO 5 --- [ main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@5424f110: startup date [Sat May 09 22:06:20 GMT 2015]; root of context hierarchy +2015-05-09 22:06:21.511 INFO 5 --- [ main] o.s.j.e.a.AnnotationMBeanExporter : Registering beans for JMX exposure on startup +2015-05-09 22:06:21.549 INFO 5 --- [ main] c.g.p.h.HazelcastDiscoveryController : Asking k8s registry at http://10.100.0.1:80.. +2015-05-09 22:06:22.031 INFO 5 --- [ main] c.g.p.h.HazelcastDiscoveryController : Found 2 pods running Hazelcast. +2015-05-09 22:06:22.176 INFO 5 --- [ main] c.h.instance.DefaultAddressPicker : [LOCAL] [someGroup] [3.4.2] Interfaces is disabled, trying to pick one address from TCP-IP config addresses: [10.244.90.3, 10.244.66.2] +2015-05-09 22:06:22.177 INFO 5 --- [ main] c.h.instance.DefaultAddressPicker : [LOCAL] [someGroup] [3.4.2] Prefer IPv4 stack is true. +2015-05-09 22:06:22.189 INFO 5 --- [ main] c.h.instance.DefaultAddressPicker : [LOCAL] [someGroup] [3.4.2] Picked Address[10.244.66.2]:5701, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=5701], bind any local is true +2015-05-09 22:06:22.642 INFO 5 --- [ main] com.hazelcast.spi.OperationService : [10.244.66.2]:5701 [someGroup] [3.4.2] Backpressure is disabled +2015-05-09 22:06:22.647 INFO 5 --- [ main] c.h.spi.impl.BasicOperationScheduler : [10.244.66.2]:5701 [someGroup] [3.4.2] Starting with 2 generic operation threads and 2 partition operation threads. +2015-05-09 22:06:22.796 INFO 5 --- [ main] com.hazelcast.system : [10.244.66.2]:5701 [someGroup] [3.4.2] Hazelcast 3.4.2 (20150326 - f6349a4) starting at Address[10.244.66.2]:5701 +2015-05-09 22:06:22.798 INFO 5 --- [ main] com.hazelcast.system : [10.244.66.2]:5701 [someGroup] [3.4.2] Copyright (C) 2008-2014 Hazelcast.com +2015-05-09 22:06:22.800 INFO 5 --- [ main] com.hazelcast.instance.Node : [10.244.66.2]:5701 [someGroup] [3.4.2] Creating TcpIpJoiner +2015-05-09 22:06:22.801 INFO 5 --- [ main] com.hazelcast.core.LifecycleService : [10.244.66.2]:5701 [someGroup] [3.4.2] Address[10.244.66.2]:5701 is STARTING +2015-05-09 22:06:23.108 INFO 5 --- [cached.thread-2] com.hazelcast.nio.tcp.SocketConnector : [10.244.66.2]:5701 [someGroup] [3.4.2] Connecting to /10.244.90.3:5701, timeout: 0, bind-any: true +2015-05-09 22:06:23.182 INFO 5 --- [cached.thread-2] c.h.nio.tcp.TcpIpConnectionManager : [10.244.66.2]:5701 [someGroup] [3.4.2] Established socket connection between /10.244.66.2:48051 and 10.244.90.3/10.244.90.3:5701 +2015-05-09 22:06:29.158 INFO 5 --- [ration.thread-1] com.hazelcast.cluster.ClusterService : [10.244.66.2]:5701 [someGroup] [3.4.2] -(...) +Members [2] { + Member [10.244.90.3]:5701 + Member [10.244.66.2]:5701 this +} -2014-12-24T01:21:16.007729519Z 2014-12-24 01:21:16.000 INFO 10 --- [cached.thread-3] c.h.nio.tcp.TcpIpConnectionManager : [10.160.2.4]:5701 [someGroup] [3.3.3] Established socket connection between /10.160.2.4:54931 and /10.160.2.3:5701 -2014-12-24T01:21:16.427289059Z 2014-12-24 01:21:16.427 INFO 10 --- [thread-Acceptor] com.hazelcast.nio.tcp.SocketAcceptor : [10.160.2.4]:5701 [someGroup] [3.3.3] Accepting socket connection from /10.160.2.3:50660 -2014-12-24T01:21:16.433763738Z 2014-12-24 01:21:16.433 INFO 10 --- [cached.thread-3] c.h.nio.tcp.TcpIpConnectionManager : [10.160.2.4]:5701 [someGroup] [3.3.3] Established socket connection between /10.160.2.4:5701 and /10.160.2.3:50660 -2014-12-24T01:21:23.036227250Z 2014-12-24 01:21:23.035 INFO 10 --- [ration.thread-1] com.hazelcast.cluster.ClusterService : [10.160.2.4]:5701 [someGroup] [3.3.3] -2014-12-24T01:21:23.036227250Z -2014-12-24T01:21:23.036227250Z Members [3] { -2014-12-24T01:21:23.036227250Z Member [10.160.2.4]:5701 this -2014-12-24T01:21:23.036227250Z Member [10.160.2.3]:5701 -2014-12-24T01:21:23.036227250Z } +2015-05-09 22:06:31.177 INFO 5 --- [ main] com.hazelcast.core.LifecycleService : [10.244.66.2]:5701 [someGroup] [3.4.2] Address[10.244.66.2]:5701 is STARTED ``` Now let's resize our cluster to 4 nodes: @@ -214,9 +186,6 @@ Examine the status again by checking a node’s log and you should see the 4 mem For those of you who are impatient, here is the summary of the commands we ran in this tutorial. ```sh -# create a single hazelcast node -kubectl create -f hazelcast.yaml - # create a service to track all hazelcast nodes kubectl create -f hazelcast-service.yaml @@ -226,9 +195,6 @@ kubectl create -f hazelcast-controller.yaml # scale up to 2 nodes kubectl resize rc hazelcast --replicas=2 -# validate the cluster -docker exec nodetool status - # scale up to 4 nodes kubectl resize rc hazelcast --replicas=4 ``` @@ -236,6 +202,10 @@ kubectl resize rc hazelcast --replicas=4 ### Hazelcast Discovery Source ```java +package com.github.pires.hazelcast; + +import com.fasterxml.jackson.annotation.JsonIgnoreProperties; +import com.fasterxml.jackson.databind.ObjectMapper; import com.hazelcast.config.Config; import com.hazelcast.config.GroupConfig; import com.hazelcast.config.JoinConfig; @@ -244,9 +214,8 @@ import com.hazelcast.config.NetworkConfig; import com.hazelcast.config.SSLConfig; import com.hazelcast.config.TcpIpConfig; import com.hazelcast.core.Hazelcast; -import io.fabric8.kubernetes.api.KubernetesClient; -import io.fabric8.kubernetes.api.KubernetesFactory; -import io.fabric8.kubernetes.api.model.Pod; +import java.io.IOException; +import java.net.URL; import java.util.List; import java.util.UUID; import java.util.concurrent.CopyOnWriteArrayList; @@ -256,7 +225,7 @@ import org.springframework.boot.CommandLineRunner; import org.springframework.stereotype.Controller; /** - * Read from Kubernetes API all labeled Hazelcast pods, get their IP and connect to them. + * Read from Kubernetes API all Hazelcast service bound pods, get their IP and connect to them. */ @Controller public class HazelcastDiscoveryController implements CommandLineRunner { @@ -264,8 +233,23 @@ public class HazelcastDiscoveryController implements CommandLineRunner { private static final Logger log = LoggerFactory.getLogger( HazelcastDiscoveryController.class); - private static final String HAZELCAST_LABEL_NAME = "name"; - private static final String HAZELCAST_LABEL_VALUE = "hazelcast"; + @JsonIgnoreProperties(ignoreUnknown = true) + static class Address { + + public String IP; + } + + @JsonIgnoreProperties(ignoreUnknown = true) + static class Subset { + + public List
addresses; + } + + @JsonIgnoreProperties(ignoreUnknown = true) + static class Endpoints { + + public List subsets; + } private static String getEnvOrDefault(String var, String def) { final String val = System.getenv(var); @@ -276,25 +260,39 @@ public class HazelcastDiscoveryController implements CommandLineRunner { @Override public void run(String... args) { - final String kubeApiHost = getEnvOrDefault("KUBERNETES_RO_SERVICE_HOST", + final String hostName = getEnvOrDefault("KUBERNETES_RO_SERVICE_HOST", "localhost"); - final String kubeApiPort = getEnvOrDefault("KUBERNETES_RO_SERVICE_PORT", + final String hostPort = getEnvOrDefault("KUBERNETES_RO_SERVICE_PORT", "8080"); - final String kubeUrl = "http://" + kubeApiHost + ":" + kubeApiPort; - log.info("Asking k8s registry at {}..", kubeUrl); - final KubernetesClient kube = new KubernetesClient(new KubernetesFactory( - kubeUrl)); - final List hazelcastPods = new CopyOnWriteArrayList<>(); - kube.getPods().getItems().parallelStream().filter(pod - -> pod.getLabels().get(HAZELCAST_LABEL_NAME).equals( - HAZELCAST_LABEL_VALUE)).forEach(hazelcastPods::add); - log.info("Found {} pods running Hazelcast.", hazelcastPods.size()); - if (!hazelcastPods.isEmpty()) { - runHazelcast(hazelcastPods); + String serviceName = getEnvOrDefault("HAZELCAST_SERVICE", "hazelcast"); + String path = "/api/v1beta3/namespaces/default/endpoints/"; + final String host = "http://" + hostName + ":" + hostPort; + log.info("Asking k8s registry at {}..", host); + + final List hazelcastEndpoints = new CopyOnWriteArrayList<>(); + + try { + URL url = new URL(host + path + serviceName); + ObjectMapper mapper = new ObjectMapper(); + Endpoints endpoints = mapper.readValue(url, Endpoints.class); + if (endpoints != null) { + if (endpoints.subsets != null && !endpoints.subsets.isEmpty()) { + endpoints.subsets.parallelStream().forEach(subset -> { + subset.addresses.parallelStream().forEach( + addr -> hazelcastEndpoints.add(addr.IP)); + }); + } + } + } catch (IOException ex) { + log.warn("Request to Kubernetes API failed", ex); } + + log.info("Found {} pods running Hazelcast.", hazelcastEndpoints.size()); + + runHazelcast(hazelcastEndpoints); } - private void runHazelcast(final List hazelcastPods) { + private void runHazelcast(final List nodes) { // configure Hazelcast instance final Config cfg = new Config(); cfg.setInstanceName(UUID.randomUUID().toString()); @@ -313,9 +311,7 @@ public class HazelcastDiscoveryController implements CommandLineRunner { mcCfg.setEnabled(false); // tcp final TcpIpConfig tcpCfg = new TcpIpConfig(); - hazelcastPods.parallelStream().forEach(pod -> { - tcpCfg.addMember(pod.getCurrentState().getPodIP()); - }); + nodes.parallelStream().forEach(tcpCfg::addMember); tcpCfg.setEnabled(true); // network join configuration final JoinConfig joinCfg = new JoinConfig(); @@ -331,4 +327,5 @@ public class HazelcastDiscoveryController implements CommandLineRunner { } } + ``` diff --git a/hazelcast/hazelcast-controller.yaml b/hazelcast/hazelcast-controller.yaml index f9d87e3e..cde79303 100644 --- a/hazelcast/hazelcast-controller.yaml +++ b/hazelcast/hazelcast-controller.yaml @@ -17,7 +17,7 @@ spec: - resources: limits: cpu: 1 - image: pires/hazelcast-k8s + image: pires/hazelcast-k8s:0.2 name: hazelcast ports: - containerPort: 5701 diff --git a/hazelcast/hazelcast.yaml b/hazelcast/hazelcast.yaml deleted file mode 100644 index 97fdb86f..00000000 --- a/hazelcast/hazelcast.yaml +++ /dev/null @@ -1,17 +0,0 @@ -apiVersion: v1beta3 -kind: Pod -metadata: - labels: - name: hazelcast - name: hazelcast -spec: - containers: - - image: pires/hazelcast-k8s - name: hazelcast - ports: - - containerPort: 5701 - name: hazelcast - protocol: TCP - resources: - limits: - cpu: "1"