{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Agents on Kubeflow 🤓"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial we will be training a reinforcement learning agent from the [tensorflow/agents](https://github.com/tensorflow/agents) project on Kubernetes using [Kubeflow](https://github.com/google/kubeflow). The agent will be learning to operate a simulated robotic arm."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This demo assumes you've followed the setup steps provided in the README file.\n",
    "\n",
    "Firstly we need to provide the path on our filesystem where the examples can be found, the namespace in which to run jobs (and in which NFS is deployed, and the ID and mount path for our NFS volume."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "EXAMPLES_ROOT=\"/home/jovyan/kubeflow-examples\" # Your examples root here if different\n",
    "NAMESPACE=\"kubeflow\"\n",
    "APP_NAME=\"agents\"\n",
    "PROJECT=\"kubeflow-rl\" # Your gcloud project here!\n",
    "NFS_CLAIM_ID=\"nfs-1\"\n",
    "NFS_MOUNT_PATH=\"/mnt/nfs-1\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "APP_ROOT=os.path.join(EXAMPLES_ROOT, \"agents\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Configure ksonnet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ksonnet is a tool to simplify configuration management for Kubernetes deployments which extends to making it easier to specify and re-configure distributed TensorFlow training jobs on Kubeflow. This example ships with a Ksonnet workspace in the app subdirectory. In order to use it first we'll need to register our Kubernetes cluster by running `ks env set default` from the root of the Ksonnet app."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.chdir(os.path.join(APP_ROOT, \"app\"))\n",
    "!ks env add default\n",
    "# Did the above fail? That means you already have a default environment.\n",
    "# Try running `ks env set default` instead!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more information on Ksonnet check out their documentation [here](ksonnet.io)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building training image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Currently in order to train our model on Kubeflow we'll need to bundle our local workspace into a Docker container. We can perform such a build using Google Container Builder. Running this command requires that the current workspace has been authenticated with the Google Cloud Platform. In the future this will not be required."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import datetime\n",
    "import uuid\n",
    "\n",
    "def gen_trainer_tag(project, app_name, registry_target=\"gcr.io\"):\n",
    "    now=datetime.datetime.now()\n",
    "    build_id=now.strftime(\"%m%d-%H%M\") + \"-\" + uuid.uuid4().hex[0:4]\n",
    "    return \"%s/%s/%s:%s\" % (registry_target, project, app_name, build_id)\n",
    "\n",
    "TRAINER_TAG = gen_trainer_tag(PROJECT, APP_NAME)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [],
   "source": [
    "LAST_CACHE_FROM=\"tensorflow/tensorflow:1.4.1\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/tmp/tmp_dzp1ks3/build.yaml'"
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import yaml\n",
    "import logging\n",
    "import tempfile\n",
    "\n",
    "def _generate_gcb_config(trainer_tag, build_dir, cache_from=LAST_CACHE_FROM):\n",
    "    if not os.path.exists(build_dir):\n",
    "        raise ValueError(\"Can't find provided build directory: %s\" % build_dir)\n",
    "    os.chdir(build_dir)\n",
    "        \n",
    "    build_config = {\n",
    "        \"steps\": [\n",
    "            {\n",
    "                \"name\": \"gcr.io/cloud-builders/docker\",\n",
    "                \"args\": [\n",
    "                    \"pull\", cache_from\n",
    "                ]\n",
    "            },\n",
    "            {\n",
    "                \"name\": \"gcr.io/cloud-builders/docker\",\n",
    "                \"args\": [\n",
    "                    \"build\",\n",
    "                    \"--cache-from\",\n",
    "                    cache_from,\n",
    "                    \"-t\", trainer_tag,\n",
    "                    \".\"\n",
    "                ]\n",
    "            }\n",
    "        ],\n",
    "        \"images\": [trainer_tag]\n",
    "    }\n",
    "    \n",
    "    d = tempfile.mkdtemp()\n",
    "    \n",
    "    output_config_path = os.path.join(d, \"build.yaml\")\n",
    "    if build_dir is not None:\n",
    "        output_config_path = os.path.join(build_dir, output_config_path)\n",
    "    with open(output_config_path, \"w\") as f:\n",
    "        f.write(yaml.dump(build_config))\n",
    "\n",
    "    logging.info(\"Generated build config: %s\" % output_config_path)\n",
    "    return output_config_path\n",
    "        \n",
    "BUILD_CONFIG_PATH = _generate_gcb_config(TRAINER_TAG, APP_ROOT, LAST_CACHE_FROM)\n",
    "BUILD_CONFIG_PATH"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.chdir(APP_ROOT)\n",
    "!gcloud container builds submit --config {BUILD_CONFIG_PATH} .\n",
    "# This is still slow because GCB doesn't cache images, has to pull multi-GB image every time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Only run this cell if the build above was successful!\n",
    "LAST_CACHE_FROM=TRAINER_TAG"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The time to build this image is rather long on account of the time it takes to pip install pybullet and the lack of caching from previous builds. Hang tight as we're working on a much faster method to ship workspaces to speed up this process."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The objective of the training phase is to learn the parameterization of our model that confers a high level of performance on the robot arm control task (picking up blocks). Here we'll launch and monitor such a job."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Launching the TFJob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In local jargon a Study is simply collection of (probably related) Experiments which are themselves simply sets of parameters used for separate training runs. Here we'll run a study of the repeatability of training runs performed with identical parameter sets. As you can imaging there is a lot of room for expressing and carrying out more interesting structured studies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/mnt/nfs-1/train_dirs/studies/replicated-kuka-demo\n"
     ]
    }
   ],
   "source": [
    "study = {\"name\": \"replicated-kuka-demo\",\n",
    "         \"experiments\": [{\"name\": \"kuka\",\n",
    "                          \"image\": TRAINER_TAG,\n",
    "                          \"env\": \"KukaBulletEnv-v0\",\n",
    "                          \"num_replicas\": 4}]\n",
    "        }\n",
    "\n",
    "STUDY_LOGS_ROOT = \"/mnt/nfs-1/train_dirs/studies/{0}\".format(study[\"name\"])\n",
    "os.makedirs(STUDY_LOGS_ROOT, exist_ok=True)\n",
    "\n",
    "print(STUDY_LOGS_ROOT)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll use [ksonnet](https://ksonnet.io/) to run a TFJob for each experiment in our study. Here when `ks param set` is called a parameter is written to the corresponding section of app/components/params.libsonnet. Note that parameters set from previous runs do carry over from previous usages of components.\n",
    "\n",
    "A YAML form of the configuration for any component can be displayed using `ks show {environment e.g. default} -c {component name e.g. train}`. Jobs are submitted via `ks apply {environment} -c {component name}`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Preparing study: replicated-kuka-demo...\n",
      "Preparing experiment: kuka\n",
      "\u001b[34mINFO  \u001b[0mParameter 'image' successfully set to '\"gcr.io/kubeflow-rl/agents:0405-1658-39bf\"' for component 'train'\n",
      "\u001b[34mINFO  \u001b[0mParameter 'env' successfully set to '\"KukaBulletEnv-v0\"' for component 'train'\n",
      "\u001b[34mINFO  \u001b[0mParameter 'namespace' successfully set to '\"kubeflow\"' for component 'train'\n",
      "\u001b[34mINFO  \u001b[0mParameter 'algorithm' successfully set to '\"agents.ppo.PPOAlgorithm\"' for component 'train'\n",
      "\u001b[34mINFO  \u001b[0mParameter 'network' successfully set to '\"agents.scripts.networks.feed_forward_gaussian\"' for component 'train'\n",
      "\u001b[34mINFO  \u001b[0mParameter 'run_mode' successfully set to '\"train\"' for component 'train'\n",
      "\u001b[34mINFO  \u001b[0mParameter 'num_cpu' successfully set to '30' for component 'train'\n",
      "\u001b[34mINFO  \u001b[0mParameter 'num_agents' successfully set to '30' for component 'train'\n",
      "\u001b[34mINFO  \u001b[0mParameter 'steps' successfully set to '15e6' for component 'train'\n",
      "\u001b[34mINFO  \u001b[0mParameter 'update_every' successfully set to '60' for component 'train'\n",
      "\u001b[34mINFO  \u001b[0mParameter 'eval_episodes' successfully set to '25' for component 'train'\n",
      "\u001b[34mINFO  \u001b[0mParameter 'name' successfully set to '\"kuka-0405-1707-5ed3\"' for component 'train'\n",
      "\u001b[34mINFO  \u001b[0mParameter 'log_dir' successfully set to '\"/mnt/nfs-1/train_dirs/studies/replicated-kuka-demo/kuka-0405-1707-5ed3\"' for component 'train'\n",
      "Preparing replica 1 of [{'log_dir': '/mnt/nfs-1/train_dirs/studies/replicated-kuka-demo/kuka-0405-1707-5ed3'}] for experiment kuka\n",
      "\u001b[34mINFO  \u001b[0mUpdating tfjobs kubeflow.kuka-0405-1707-5ed3\n",
      "\u001b[34mINFO  \u001b[0mCreating non-existent tfjobs kubeflow.kuka-0405-1707-5ed3\n",
      "\u001b[34mINFO  \u001b[0mParameter 'name' successfully set to '\"kuka-0405-1707-ae54\"' for component 'train'\n",
      "\u001b[34mINFO  \u001b[0mParameter 'log_dir' successfully set to '\"/mnt/nfs-1/train_dirs/studies/replicated-kuka-demo/kuka-0405-1707-ae54\"' for component 'train'\n",
      "Preparing replica 2 of [{'log_dir': '/mnt/nfs-1/train_dirs/studies/replicated-kuka-demo/kuka-0405-1707-5ed3'}, {'log_dir': '/mnt/nfs-1/train_dirs/studies/replicated-kuka-demo/kuka-0405-1707-ae54'}] for experiment kuka\n",
      "\u001b[34mINFO  \u001b[0mUpdating tfjobs kubeflow.kuka-0405-1707-ae54\n",
      "\u001b[34mINFO  \u001b[0mCreating non-existent tfjobs kubeflow.kuka-0405-1707-ae54\n",
      "\u001b[34mINFO  \u001b[0mParameter 'name' successfully set to '\"kuka-0405-1707-e6f1\"' for component 'train'\n",
      "\u001b[34mINFO  \u001b[0mParameter 'log_dir' successfully set to '\"/mnt/nfs-1/train_dirs/studies/replicated-kuka-demo/kuka-0405-1707-e6f1\"' for component 'train'\n",
      "Preparing replica 3 of [{'log_dir': '/mnt/nfs-1/train_dirs/studies/replicated-kuka-demo/kuka-0405-1707-5ed3'}, {'log_dir': '/mnt/nfs-1/train_dirs/studies/replicated-kuka-demo/kuka-0405-1707-ae54'}, {'log_dir': '/mnt/nfs-1/train_dirs/studies/replicated-kuka-demo/kuka-0405-1707-e6f1'}] for experiment kuka\n",
      "\u001b[34mINFO  \u001b[0mUpdating tfjobs kubeflow.kuka-0405-1707-e6f1\n",
      "\u001b[34mINFO  \u001b[0mCreating non-existent tfjobs kubeflow.kuka-0405-1707-e6f1\n",
      "\u001b[34mINFO  \u001b[0mParameter 'name' successfully set to '\"kuka-0405-1707-545d\"' for component 'train'\n",
      "\u001b[34mINFO  \u001b[0mParameter 'log_dir' successfully set to '\"/mnt/nfs-1/train_dirs/studies/replicated-kuka-demo/kuka-0405-1707-545d\"' for component 'train'\n",
      "Preparing replica 4 of [{'log_dir': '/mnt/nfs-1/train_dirs/studies/replicated-kuka-demo/kuka-0405-1707-5ed3'}, {'log_dir': '/mnt/nfs-1/train_dirs/studies/replicated-kuka-demo/kuka-0405-1707-ae54'}, {'log_dir': '/mnt/nfs-1/train_dirs/studies/replicated-kuka-demo/kuka-0405-1707-e6f1'}, {'log_dir': '/mnt/nfs-1/train_dirs/studies/replicated-kuka-demo/kuka-0405-1707-545d'}] for experiment kuka\n",
      "\u001b[34mINFO  \u001b[0mUpdating tfjobs kubeflow.kuka-0405-1707-545d\n",
      "\u001b[34mINFO  \u001b[0mCreating non-existent tfjobs kubeflow.kuka-0405-1707-545d\n"
     ]
    }
   ],
   "source": [
    "import datetime\n",
    "import uuid\n",
    "import pprint\n",
    "\n",
    "os.chdir(os.path.join(APP_ROOT, \"app\"))\n",
    "\n",
    "print(\"Preparing study: %s...\" % study[\"name\"])\n",
    "\n",
    "for experiment in study[\"experiments\"]:\n",
    "    \n",
    "    print(\"Preparing experiment: %s\" % experiment[\"name\"])\n",
    "    \n",
    "    # Get and set the job container image for this experiment\n",
    "    IMAGE = experiment[\"image\"]\n",
    "    !ks param set train image {IMAGE}\n",
    "\n",
    "    # Set the gym learning environment on which to train\n",
    "    ENVIRONMENT = experiment[\"env\"]\n",
    "    !ks param set train env {ENVIRONMENT}\n",
    "    \n",
    "    !ks param set train namespace {NAMESPACE}\n",
    "\n",
    "    # Set the algorithm and network part to use for policy and value networks\n",
    "    !ks param set train algorithm \"agents.ppo.PPOAlgorithm\"\n",
    "    !ks param set train network \"agents.scripts.networks.feed_forward_gaussian\"\n",
    "\n",
    "    # Run in training mode with 30 CPU and 30 agents for 20M steps\n",
    "    !ks param set train run_mode train\n",
    "    !ks param set train num_cpu 30\n",
    "    !ks param set train num_agents 30\n",
    "    !ks param set train steps 15e6\n",
    "\n",
    "    !ks param set train update_every 60\n",
    "    !ks param set train eval_episodes 25\n",
    "\n",
    "    for replica_id in range(experiment[\"num_replicas\"]):\n",
    "\n",
    "        # Construct a unique name for the training job based on experiment[\"name\"]\n",
    "        now=datetime.datetime.now()\n",
    "        JOB_SALT=now.strftime(\"%m%d-%H%M\") + \"-\" + uuid.uuid4().hex[0:4]\n",
    "        BASE_NAME = experiment[\"name\"]\n",
    "        TRAIN_JOB_NAME=BASE_NAME + \"-\" + JOB_SALT\n",
    "        !ks param set train name {TRAIN_JOB_NAME}\n",
    "\n",
    "        # Construct a log dir path for this experiment\n",
    "        LOG_DIR=\"{0}/{1}\".format(STUDY_LOGS_ROOT, TRAIN_JOB_NAME)\n",
    "        !ks param set train log_dir {LOG_DIR}\n",
    "\n",
    "        if \"replicas\" not in experiment:\n",
    "            experiment[\"replicas\"] = []\n",
    "        experiment[\"replicas\"].append({\"log_dir\": LOG_DIR})\n",
    "        \n",
    "        print(\"Preparing replica %s of %s for experiment %s\" % (replica_id + 1, experiment[\"replicas\"], experiment[\"name\"]))\n",
    "        !ks apply default -c train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following shows our updated study config with the log directories to which each of our experiments will be writing TensorBoard logs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'experiments': [{'env': 'KukaBulletEnv-v0',\n",
       "   'image': 'gcr.io/kubeflow-rl/agents:0405-1658-39bf',\n",
       "   'name': 'kuka',\n",
       "   'num_replicas': 4,\n",
       "   'replicas': [{'log_dir': '/mnt/nfs-1/train_dirs/studies/replicated-kuka-demo/kuka-0405-1707-5ed3'},\n",
       "    {'log_dir': '/mnt/nfs-1/train_dirs/studies/replicated-kuka-demo/kuka-0405-1707-ae54'},\n",
       "    {'log_dir': '/mnt/nfs-1/train_dirs/studies/replicated-kuka-demo/kuka-0405-1707-e6f1'},\n",
       "    {'log_dir': '/mnt/nfs-1/train_dirs/studies/replicated-kuka-demo/kuka-0405-1707-545d'}]}],\n",
       " 'name': 'replicated-kuka-demo'}"
      ]
     },
     "execution_count": 137,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "study"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Monitoring training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the following we can list the TFJobs that are currently running in {NAMESPACE} and verify that the above has created the jobs we expected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME                  AGE\n",
      "kuka-0405-1707-545d   4s\n",
      "kuka-0405-1707-5ed3   12s\n",
      "kuka-0405-1707-ae54   10s\n",
      "kuka-0405-1707-e6f1   6s\n"
     ]
    }
   ],
   "source": [
    "!kubectl get tfjobs -n {NAMESPACE}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can display the IDs and status of all pods in the namespace of our training jobs via the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME                                      READY     STATUS    RESTARTS   AGE\n",
      "ambassador-56c8966c67-9pvbm               2/2       Running   0          36d\n",
      "ambassador-56c8966c67-mvqjc               2/2       Running   0          31d\n",
      "ambassador-56c8966c67-sch5s               2/2       Running   0          36d\n",
      "argo-ui-5d7fbb58d4-m6lmm                  1/1       Running   0          31d\n",
      "jupyter-cwbeitel                          1/1       Running   0          16d\n",
      "kuka-0405-1707-545d-master-b1pj-0-26qj4   1/1       Running   0          1m\n",
      "kuka-0405-1707-5ed3-master-oa84-0-td6bq   1/1       Running   0          1m\n",
      "kuka-0405-1707-ae54-master-h5wu-0-j8zv2   1/1       Running   0          1m\n",
      "kuka-0405-1707-e6f1-master-wg99-0-9jjz4   1/1       Running   0          1m\n",
      "nfs-1-provisioner-7497d85d76-j9492        1/1       Running   0          31d\n",
      "nfs-2-provisioner-59bf96c5d4-g22wx        1/1       Running   0          31d\n",
      "tf-hub-0                                  1/1       Running   0          36d\n",
      "tf-job-operator-5c497d644d-rd5j9          1/1       Running   6          31d\n",
      "workflow-controller-7d8f4bc5df-xjtw8      1/1       Running   0          36d\n"
     ]
    }
   ],
   "source": [
    "!kubectl get pods -n {NAMESPACE} --show-all"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following can be used to print logs from the last TFJob submitted above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'kuka-0405-1707-545d'"
      ]
     },
     "execution_count": 149,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TRAIN_JOB_NAME"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'kuka-0405-1707-545d-master-b1pj-0-26qj4'\n",
      "/usr/local/lib/python2.7/dist-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n",
      "INFO:tensorflow:Start a new run and write summaries and checkpoints to /mnt/nfs-1/train_dirs/studies/replicated-kuka-demo/kuka-0405-1707-545d.\n",
      "{'algorithm': <class 'agents.ppo.algorithm.PPOAlgorithm'>,\n",
      " 'debug': True,\n",
      " 'discount': 0.995,\n",
      " 'env': 'KukaBulletEnv-v0',\n",
      " 'env_processes': True,\n",
      " 'eval_episodes': 25,\n",
      " 'hparam_set_id': 'pybullet_kuka_ff',\n",
      " 'init_logstd': -1,\n",
      " 'init_mean_factor': 0.1,\n",
      " 'init_output_factor': 0.1,\n",
      " 'init_std': 0.35,\n",
      " 'kl_cutoff_coef': 1000,\n",
      " 'kl_cutoff_factor': 2,\n",
      " 'kl_init_penalty': 1,\n",
      " 'kl_target': 0.01,\n",
      " 'learning_rate': 0.0001,\n",
      " 'log_device_placement': False,\n",
      " 'logdir': '/mnt/nfs-1/train_dirs/studies/replicated-kuka-demo/kuka-0405-1707-545d',\n",
      " 'max_length': 1000,\n",
      " 'network': <function feed_forward_gaussian at 0x7f543588f488>,\n",
      " 'normalize_ranges': True,\n",
      " 'num_agents': 30,\n",
      " 'num_gpus': 0,\n",
      " 'optimizer': <class 'tensorflow.python.training.adam.AdamOptimizer'>,\n",
      " 'policy_layers': (200, 100),\n",
      " 'render_out_dir': None,\n",
      " 'render_secs': 600,\n",
      " 'run_base_tag': '0e90193e',\n",
      " 'run_mode': 'train',\n",
      " 'save_checkpoint_secs': 600,\n",
      " 'steps': 15000000,\n",
      " 'update_epochs': 25,\n",
      " 'update_every': 60,\n",
      " 'use_gpu': False,\n",
      " 'value_layers': (200, 100),\n",
      " 'weight_summaries': {'all': '.*',\n",
      "                      'policy': '.*/policy/.*',\n",
      "                      'value': '.*/value/.*'}}\n",
      "INFO:tensorflow:Graph contains 44607 trainable variables.\n",
      "2018-04-05 17:07:48.368737: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA\n",
      "INFO:tensorflow:\n",
      "--------------------------------------------------\n",
      "Phase train (phase step 0, global step 0).\n",
      "2018-04-05 17:07:52.639292: I tensorflow/core/kernels/logging_ops.cc:79] return and value: [-0.322072715][0.0247094277]\n",
      "2018-04-05 17:07:52.639757: I tensorflow/core/kernels/logging_ops.cc:79] normalized advantage: [4.529953e-06]\n",
      "2018-04-05 17:07:56.642998: I tensorflow/core/kernels/logging_ops.cc:79] value loss: [0.598414481]\n",
      "2018-04-05 17:07:56.642998: I tensorflow/core/kernels/logging_ops.cc:79] policy loss: [0.245057404]\n",
      "2018-04-05 17:07:56.643401: I tensorflow/core/kernels/logging_ops.cc:79] current penalty: [1]\n",
      "2018-04-05 17:07:56.667916: I tensorflow/core/kernels/logging_ops.cc:79] kl change: [0.000576514052]\n",
      "2018-04-05 17:07:56.668041: I tensorflow/core/kernels/logging_ops.cc:79] decrease penalty [0]\n",
      "2018-04-05 17:08:01.188702: I tensorflow/core/kernels/logging_ops.cc:79] return and value: [-0.112732142][-0.122791454]\n",
      "2018-04-05 17:08:01.188830: I tensorflow/core/kernels/logging_ops.cc:79] normalized advantage: [-4.44787338e-06]\n",
      "2018-04-05 17:08:05.169252: I tensorflow/core/kernels/logging_ops.cc:79] policy loss: [0.0643984899]\n",
      "2018-04-05 17:08:05.169253: I tensorflow/core/kernels/logging_ops.cc:79] value loss: [0.631897569]\n",
      "2018-04-05 17:08:05.169620: I tensorflow/core/kernels/logging_ops.cc:79] current penalty: [0.666666687]\n",
      "2018-04-05 17:08:05.196198: I tensorflow/core/kernels/logging_ops.cc:79] kl change: [0.00174093829]\n",
      "2018-04-05 17:08:05.196311: I tensorflow/core/kernels/logging_ops.cc:79] decrease penalty [0]\n",
      "2018-04-05 17:08:10.431437: I tensorflow/core/kernels/logging_ops.cc:79] return and value: [-0.0636681393][-0.17371732]\n",
      "2018-04-05 17:08:10.431553: I tensorflow/core/kernels/logging_ops.cc:79] normalized advantage: [2.87914918e-05]\n",
      "2018-04-05 17:08:14.490344: I tensorflow/core/kernels/logging_ops.cc:79] policy loss: [0.018688973]\n",
      "2018-04-05 17:08:14.490361: I tensorflow/core/kernels/logging_ops.cc:79] value loss: [0.676567554]\n",
      "2018-04-05 17:08:14.493676: I tensorflow/core/kernels/logging_ops.cc:79] current penalty: [0.444444448]\n",
      "2018-04-05 17:08:14.518774: I tensorflow/core/kernels/logging_ops.cc:79] kl change: [0.00242574792]\n",
      "2018-04-05 17:08:14.518928: I tensorflow/core/kernels/logging_ops.cc:79] decrease penalty [0]\n",
      "2018-04-05 17:08:20.096456: I tensorflow/core/kernels/logging_ops.cc:79] return and value: [-0.132051274][-0.186460227]\n",
      "2018-04-05 17:08:20.096586: I tensorflow/core/kernels/logging_ops.cc:79] normalized advantage: [-3.82823964e-06]\n",
      "2018-04-05 17:08:24.128271: I tensorflow/core/kernels/logging_ops.cc:79] value loss: [0.489235461]\n",
      "2018-04-05 17:08:24.128271: I tensorflow/core/kernels/logging_ops.cc:79] policy loss: [0.0739461407]\n",
      "2018-04-05 17:08:24.133808: I tensorflow/core/kernels/logging_ops.cc:79] current penalty: [0.296296299]\n",
      "2018-04-05 17:08:24.157936: I tensorflow/core/kernels/logging_ops.cc:79] kl change: [0.00183598453]\n",
      "2018-04-05 17:08:24.158047: I tensorflow/core/kernels/logging_ops.cc:79] decrease penalty [0]\n"
     ]
    }
   ],
   "source": [
    "import subprocess\n",
    "master_pod = subprocess.check_output([\"kubectl\", \"-n\", NAMESPACE, \"get\", \"pods\", \"--selector=tf_job_name=\" + TRAIN_JOB_NAME,\n",
    "                                      \"-o\", \"jsonpath='{.items[*].metadata.name}'\"]).decode(\"utf-8\")\n",
    "print(master_pod)\n",
    "!kubectl logs -n {NAMESPACE} {master_pod}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Awesome! It looks like our training job started up just fine. Next we'll want to be able to look at things with TensorBoard."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Launching tensorboard"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you're running this demo in `gcr.io/kubeflow/tensorflow-notebook-cpu` or any other environment with the TensorHub extension installed then you're only a few clicks away from monitoring your job with TensorBoard.\n",
    "\n",
    "But first we need to know that this extension is configured to run the command `tensorboard --logdir /home/jovyan` so in order for our logs on NFS to occur (at least symbollically) with in this tree we'll need to create a symbolic link as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ln -s {NFS_MOUNT_PATH}/train_dirs /home/jovyan/logs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we're ready to open TensorBoard. This can be done from the \"New\" dropdown on `http://localhost:8000/user/{your-username}/tree` by selecting \"Tensorboard\" which will open TensorBoard in a new tab. Note that you may need to grant permission for this through your pop-up blocker if you have one."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For reference below we have a plot of the mean_score variable with respect to number of training steps for a study parameterized as above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](tboard_mean_score.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see there is variability both between and within replicas but some still seem to reach stably high performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Rendering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look and see what our agent looks like when it's performing the robotic-arm-control-picking-up-blocks task. This will involve running render jobs in batch that will use parameters from the most recent checkpoint to restore a version of the model and then capture video sequences as the task is performed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Initiating render jobs\n",
    "\n",
    "Here we'll create a render job for each of the experiments in our study. This will generate multiple MP4 videos of the agent performing the task for each replica run with renders landing in ${LOG_DIR}/render/[some_unique_subdir_name]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[34mINFO  \u001b[0mParameter 'name' successfully set to '\"render-0319-2043-cccf\"' for component 'render'\n",
      "\u001b[34mINFO  \u001b[0mParameter 'log_dir' successfully set to '\"/mnt/nfs-1/train_dirs/kubeflow-rl/studies/replicated-kuka-demo-1/kuka-0319-1734-9cff\"' for component 'render'\n",
      "\u001b[34mINFO  \u001b[0mParameter 'image' successfully set to '\"gcr.io/kubeflow-rl/agents:0319-1806-6614\"' for component 'render'\n",
      "\u001b[34mINFO  \u001b[0mUpdating tfjobs kubeflow.render-0319-2043-cccf\n",
      "\u001b[34mINFO  \u001b[0mCreating non-existent tfjobs kubeflow.render-0319-2043-cccf\n",
      "\u001b[34mINFO  \u001b[0mParameter 'name' successfully set to '\"render-0319-2043-a832\"' for component 'render'\n",
      "\u001b[34mINFO  \u001b[0mParameter 'log_dir' successfully set to '\"/mnt/nfs-1/train_dirs/kubeflow-rl/studies/replicated-kuka-demo-1/kuka-0319-1734-6554\"' for component 'render'\n",
      "\u001b[34mINFO  \u001b[0mParameter 'image' successfully set to '\"gcr.io/kubeflow-rl/agents:0319-1806-6614\"' for component 'render'\n",
      "\u001b[34mINFO  \u001b[0mUpdating tfjobs kubeflow.render-0319-2043-a832\n",
      "\u001b[34mINFO  \u001b[0mCreating non-existent tfjobs kubeflow.render-0319-2043-a832\n",
      "\u001b[34mINFO  \u001b[0mParameter 'name' successfully set to '\"render-0319-2043-49f7\"' for component 'render'\n",
      "\u001b[34mINFO  \u001b[0mParameter 'log_dir' successfully set to '\"/mnt/nfs-1/train_dirs/kubeflow-rl/studies/replicated-kuka-demo-1/kuka-0319-1735-f24e\"' for component 'render'\n",
      "\u001b[34mINFO  \u001b[0mParameter 'image' successfully set to '\"gcr.io/kubeflow-rl/agents:0319-1806-6614\"' for component 'render'\n",
      "\u001b[34mINFO  \u001b[0mUpdating tfjobs kubeflow.render-0319-2043-49f7\n",
      "\u001b[34mINFO  \u001b[0mCreating non-existent tfjobs kubeflow.render-0319-2043-49f7\n",
      "\u001b[34mINFO  \u001b[0mParameter 'name' successfully set to '\"render-0319-2043-47e6\"' for component 'render'\n",
      "\u001b[34mINFO  \u001b[0mParameter 'log_dir' successfully set to '\"/mnt/nfs-1/train_dirs/kubeflow-rl/studies/replicated-kuka-demo-1/kuka-0319-1735-222e\"' for component 'render'\n",
      "\u001b[34mINFO  \u001b[0mParameter 'image' successfully set to '\"gcr.io/kubeflow-rl/agents:0319-1806-6614\"' for component 'render'\n",
      "\u001b[34mINFO  \u001b[0mUpdating tfjobs kubeflow.render-0319-2043-47e6\n",
      "\u001b[34mINFO  \u001b[0mCreating non-existent tfjobs kubeflow.render-0319-2043-47e6\n"
     ]
    }
   ],
   "source": [
    "import datetime\n",
    "import uuid\n",
    "import os\n",
    "\n",
    "os.chdir(os.path.join(APP_ROOT, \"app\"))\n",
    "\n",
    "for experiment in study[\"experiments\"]:\n",
    "    for replica in experiment[\"replicas\"]:\n",
    "        \n",
    "        LOG_DIR = replica[\"log_dir\"]\n",
    "        IMAGE = experiment[\"image\"]\n",
    "        \n",
    "        now=datetime.datetime.now()\n",
    "\n",
    "        JOB_SALT=now.strftime(\"%m%d-%H%M\") + \"-\" + uuid.uuid4().hex[0:4]\n",
    "        RENDER_JOB_NAME=\"render-\" + JOB_SALT\n",
    "        !ks param set render name {RENDER_JOB_NAME}\n",
    "\n",
    "        !ks param set render log_dir {LOG_DIR}\n",
    "        !ks param set render image {IMAGE}\n",
    "        \n",
    "        !ks apply default -c render"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It will take about 15min for each render to run. We can see whether the renders are complete by checking the third column in the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME                                  DESIRED   SUCCESSFUL   AGE\n",
      "kuka-0319-1734-6554-master-a43s-0     1         0            14m\n",
      "kuka-0319-1734-9cff-master-dbpf-0     1         0            14m\n",
      "kuka-0319-1735-222e-master-raq6-0     1         0            13m\n",
      "kuka-0319-1735-f24e-master-gx11-0     1         0            14m\n",
      "render-0319-1748-3227-master-rncq-0   1         0            41s\n",
      "render-0319-1748-7e6f-master-kv1m-0   1         0            34s\n",
      "render-0319-1748-9983-master-bdrg-0   1         0            36s\n",
      "render-0319-1748-d6d4-master-r8g6-0   1         0            39s\n"
     ]
    }
   ],
   "source": [
    "!kubectl get jobs -n {NAMESPACE} --show-all"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First let's get the GCS path of the directory containing renders for an experiment of interest. For a reminder, here's the structure of our study:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'experiments': [{'env': 'KukaBulletEnv-v0',\n",
       "   'image': 'gcr.io/kubeflow-rl/agents:0319-1806-6614',\n",
       "   'name': 'kuka',\n",
       "   'num_replicas': 4,\n",
       "   'replicas': [{'log_dir': '/mnt/nfs-1/train_dirs/kubeflow-rl/studies/replicated-kuka-demo-1/kuka-0319-1734-9cff'},\n",
       "    {'log_dir': '/mnt/nfs-1/train_dirs/kubeflow-rl/studies/replicated-kuka-demo-1/kuka-0319-1734-6554'},\n",
       "    {'log_dir': '/mnt/nfs-1/train_dirs/kubeflow-rl/studies/replicated-kuka-demo-1/kuka-0319-1735-f24e'},\n",
       "    {'log_dir': '/mnt/nfs-1/train_dirs/kubeflow-rl/studies/replicated-kuka-demo-1/kuka-0319-1735-222e'}]}],\n",
       " 'name': 'replicated-kuka-demo-1'}"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "study"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the renders for the first replica of the first experiment. If we've run multiple renders each render job will be stored in a separate subdirectory of render/ named according to the render ID, e.g.:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 188,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0319-1815-77cc\t0319-2043-c21d\n"
     ]
    }
   ],
   "source": [
    "kuka_experiment_logdir = study[\"experiments\"][0][\"replicas\"][0][\"log_dir\"]\n",
    "!ls {kuka_experiment_logdir}/render"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When OpenAI Gym writes renders to disk it includes a file manifest and some basic statistics. We'll use the following functions to load and display these for all of the renders that have been performed for the log dir we have chosen."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pprint\n",
    "def load_gym_metadata_for_render_path(path):\n",
    "    \"\"\"Load render manifest and stats from render output path.\n",
    "    \n",
    "    path: a path to which renders and metadata was written using an OpenAI Gym monitor call.\n",
    "    \n",
    "    \"\"\"\n",
    "    manifest = None\n",
    "    stats = None\n",
    "    counter = 0\n",
    "    files = os.listdir(path)\n",
    "    for filename in files:\n",
    "        file_path = os.path.join(path, filename)\n",
    "        if \"manifest\" in filename:\n",
    "            with open(file_path, \"r\") as f:\n",
    "                manifest = json.loads(f.readline())\n",
    "        elif \"episode_batch\" in filename:\n",
    "            with open(file_path, \"r\") as f:\n",
    "                stats = json.loads(f.readline())\n",
    "                stats[\"readable_timestamp\"] = datetime.datetime.fromtimestamp(\n",
    "                    float(stats[\"initial_reset_timestamp\"])).strftime('%Y-%m-%d %H:%M:%S')\n",
    "                    \n",
    "    return manifest, stats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'0319-1815-77cc': {'manifest': {'env_info': {'env_id': 'KukaBulletEnv-v0',\n",
      "                                              'gym_version': '0.9.4'},\n",
      "                                 'stats': 'openaigym.episode_batch.0.45.stats.json',\n",
      "                                 'videos': [['openaigym.video.0.45.video000000.mp4',\n",
      "                                             'openaigym.video.0.45.video000000.meta.json'],\n",
      "                                            ['openaigym.video.0.45.video000001.mp4',\n",
      "                                             'openaigym.video.0.45.video000001.meta.json'],\n",
      "                                            ['openaigym.video.0.45.video000002.mp4',\n",
      "                                             'openaigym.video.0.45.video000002.meta.json'],\n",
      "                                            ['openaigym.video.0.45.video000003.mp4',\n",
      "                                             'openaigym.video.0.45.video000003.meta.json'],\n",
      "                                            ['openaigym.video.0.45.video000004.mp4',\n",
      "                                             'openaigym.video.0.45.video000004.meta.json'],\n",
      "                                            ['openaigym.video.0.45.video000005.mp4',\n",
      "                                             'openaigym.video.0.45.video000005.meta.json']]},\n",
      "                    'stats': {'episode_lengths': [198, 199, 199, 199, 200],\n",
      "                              'episode_rewards': [-941.2689749679664,\n",
      "                                                  -939.333099645894,\n",
      "                                                  -925.3936405428902,\n",
      "                                                  -920.6183247825004,\n",
      "                                                  -940.2925663953266],\n",
      "                              'episode_types': ['t', 't', 't', 't', 't', 't'],\n",
      "                              'initial_reset_timestamp': 1521483337.954766,\n",
      "                              'readable_timestamp': '2018-03-19 18:15:37',\n",
      "                              'timestamps': [1521483434.4448,\n",
      "                                             1521483532.085795,\n",
      "                                             1521483628.837687,\n",
      "                                             1521483726.584196,\n",
      "                                             1521483825.303928]}},\n",
      " '0319-2043-c21d': {'manifest': {'env_info': {'env_id': 'KukaBulletEnv-v0',\n",
      "                                              'gym_version': '0.9.4'},\n",
      "                                 'stats': 'openaigym.episode_batch.0.44.stats.json',\n",
      "                                 'videos': [['openaigym.video.0.44.video000000.mp4',\n",
      "                                             'openaigym.video.0.44.video000000.meta.json'],\n",
      "                                            ['openaigym.video.0.44.video000001.mp4',\n",
      "                                             'openaigym.video.0.44.video000001.meta.json'],\n",
      "                                            ['openaigym.video.0.44.video000002.mp4',\n",
      "                                             'openaigym.video.0.44.video000002.meta.json'],\n",
      "                                            ['openaigym.video.0.44.video000003.mp4',\n",
      "                                             'openaigym.video.0.44.video000003.meta.json'],\n",
      "                                            ['openaigym.video.0.44.video000004.mp4',\n",
      "                                             'openaigym.video.0.44.video000004.meta.json']]},\n",
      "                    'stats': {'episode_lengths': [202, 201, 198, 202],\n",
      "                              'episode_rewards': [-898.8354846838023,\n",
      "                                                  -884.6022196284799,\n",
      "                                                  -946.5015700164948,\n",
      "                                                  -874.5368365146066],\n",
      "                              'episode_types': ['t', 't', 't', 't', 't'],\n",
      "                              'initial_reset_timestamp': 1521492234.784211,\n",
      "                              'readable_timestamp': '2018-03-19 20:43:54',\n",
      "                              'timestamps': [1521492325.675906,\n",
      "                                             1521492416.316332,\n",
      "                                             1521492505.737932,\n",
      "                                             1521492597.085779]}}}\n"
     ]
    }
   ],
   "source": [
    "def list_renders(logdir):\n",
    "    renders = {}\n",
    "    for _, render_ids, _ in os.walk(os.path.join(renders_root)):\n",
    "        for render_id in render_ids:\n",
    "            render_dir = os.path.join(renders_root, render_id)\n",
    "            manifest, stats = load_gym_metadata_for_render_path(render_dir)\n",
    "            renders[render_id] = {\n",
    "                \"manifest\": manifest,\n",
    "                \"stats\": stats\n",
    "            }\n",
    "\n",
    "    return renders\n",
    "\n",
    "renders = list_renders(kuka_experiment_logdir)\n",
    "pprint.pprint(renders)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Inspecting the result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly we'll pick one of the renders above to display inline in the notebook by providing a render ID and filename (ending in mp4). This file path will then be passed to a snippet that will load and display the content of the video as an HTML widget."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_render_path(log_dir, render_id, filename):\n",
    "    return os.path.join(log_dir, \"render\", render_id, filename)\n",
    "\n",
    "render_path = get_render_path(kuka_experiment_logdir, \"0319-2043-c21d\", \"openaigym.video.0.44.video000000.mp4\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<video alt=\"test\" controls>\n",
       "                <source src=\"data:video/mp4;base64,[encoded data, 284872 characters]\" type=\"video/mp4\" />\n",
       "             </video>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 191,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import io\n",
    "import base64\n",
    "from IPython.display import HTML\n",
    "\n",
    "video = io.open(render_path, 'rb').read()\n",
    "encoded = base64.b64encode(video)\n",
    "HTML(data='''<video alt=\"test\" controls>\n",
    "                <source src=\"data:video/mp4;base64,{0}\" type=\"video/mp4\" />\n",
    "             </video>'''.format(encoded.decode('ascii')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Great job! 🎉🎉🎉"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well that concludes this demonstration. If you have suggestions on how to make it better please feel free to [open an issue](https://github.com/kubeflow/examples) and let us know your feedback.\n",
    "\n",
    "If this is your first time working with these technologies you might be interested in some suggestions of good next steps. Here are some ideas:\n",
    "- Try training with some other learning environments (from the ID fields [here](https://github.com/bulletphysics/bullet3/blob/master/examples/pybullet/gym/pybullet_envs/__init__.py)) and tweet your results! E.g.\n",
    "    - RacecarBulletEnv-v0\n",
    "    - MinitaurBulletDuckEnv-v0\n",
    "    - HalfCheetahBulletEnv-v0\n",
    "- Take a shot at implementing your own gym learning environment and repeat the above."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
