chore: PyTorch Samples - README.md update and notebook cleanup (#5833)
* Cleaning up the notebooks Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com> * Clean up Bert Notebook Signed-off-by: ankan94 <ankan@ideas2it.com> * Clean up Bert Notebooks Signed-off-by: ankan94 <ankan@ideas2it.com> * Clean up Bert Notebooks Signed-off-by: ankan94 <ankan@ideas2it.com> * Fix: Readme changes - adding prerequisites readme * Clean up Bert Notebooks Signed-off-by: ankan94 <ankan@ideas2it.com> * Clean up Bert Notebooks Signed-off-by: ankan94 <ankan@ideas2it.com> * Clean up Bert Notebooks Signed-off-by: ankan94 <ankan@ideas2it.com> * Clean up Bert Notebooks Signed-off-by: ankan94 <ankan@ideas2it.com> * clean up cifar10 notebook and new notebook for captum insights Signed-off-by: Arvind-Ideas2IT <arvindkumarsingh.gautam@ideas2it.com> * Adding desc to BERT notebook Signed-off-by: ankan94 <ankan@ideas2it.com> * Adding desc to BERT notebook Signed-off-by: ankan94 <ankan@ideas2it.com> * Fix: Add install dependencies script * Updating README files Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com> * Updating google vertex example link Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com> * Adding link to kfp python sdk Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com> * update dependencies for captum insights in cifar10 notebook Signed-off-by: Arvind-Ideas2IT <arvindkumarsingh.gautam@ideas2it.com> Co-authored-by: ankan94 <ankan@ideas2it.com> Co-authored-by: jagadeesh <jagadeeshj@ideas2it.com> Co-authored-by: Arvind-Ideas2IT <arvindkumarsingh.gautam@ideas2it.com>
This commit is contained in:
shrinath-suresh
GitHub
parent
b3c2351a02
commit
ea5c6977bf
|
|
@ -21,3 +21,9 @@ Use the following commands to install PyTorch Kubeflow Pipeline Components from
|
|||
git clone https://github.com/kubeflow/pipelines.git
|
||||
pip install pipelines/components/PyTorch/pytorch_kfp_components/.
|
||||
```
|
||||
|
||||
### Samples
|
||||
|
||||
For running the samples follow the instruction mentioned as below
|
||||
|
||||
[Samples README.md](../../../samples/contrib/pytorch-samples/README.md)
|
||||
|
|
@ -1,253 +1,59 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 25,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Copyright (c) Facebook, Inc. and its affiliates.\n",
|
||||
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
||||
"# you may not use this file except in compliance with the License.\n",
|
||||
"# You may obtain a copy of the License at\n",
|
||||
"#\n",
|
||||
"# http://www.apache.org/licenses/LICENSE-2.0\n",
|
||||
"#\n",
|
||||
"# Unless required by applicable law or agreed to in writing, software\n",
|
||||
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
||||
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
||||
"# See the License for the specific language governing permissions and\n",
|
||||
"# limitations under the License."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Bert Pipeline"
|
||||
"# Bert Pipeline : PyTorch BERT News Classfication\n",
|
||||
"\n",
|
||||
"This notebook shows PyTorch BERT end-to-end news classification example using Kubeflow Pipelines.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"An example notebook that demonstrates how to:\n",
|
||||
"\n",
|
||||
"* Get different tasks needed for the pipeline\n",
|
||||
"* Create a Kubeflow pipeline\n",
|
||||
"* Include Pytorch KFP components to preprocess, train, visualize and deploy the model in the pipeline\n",
|
||||
"* Submit a job for execution\n",
|
||||
"* Query(prediction and explain) the final deployed model\n",
|
||||
"* Interpretation of the model using the Captum Insights\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"execution_count": 1,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Found existing installation: kfp 1.6.3\n",
|
||||
"Uninstalling kfp-1.6.3:\n",
|
||||
" Successfully uninstalled kfp-1.6.3\n",
|
||||
"Collecting kfp\n",
|
||||
" Using cached kfp-1.6.3-py3-none-any.whl\n",
|
||||
"Requirement already satisfied: captum in /opt/conda/lib/python3.8/site-packages (0.3.1)\n",
|
||||
"Requirement already satisfied: torchvision in /opt/conda/lib/python3.8/site-packages (0.9.1)\n",
|
||||
"Requirement already satisfied: matplotlib in /opt/conda/lib/python3.8/site-packages (3.4.1)\n",
|
||||
"Requirement already satisfied: pillow in /opt/conda/lib/python3.8/site-packages (8.2.0)\n",
|
||||
"Requirement already satisfied: pytorch-lightning in /opt/conda/lib/python3.8/site-packages (1.3.3)\n",
|
||||
"Requirement already satisfied: flask in /opt/conda/lib/python3.8/site-packages (2.0.1)\n",
|
||||
"Requirement already satisfied: flask-compress in /opt/conda/lib/python3.8/site-packages (1.9.0)\n",
|
||||
"Requirement already satisfied: ipywidgets in /opt/conda/lib/python3.8/site-packages (7.6.3)\n",
|
||||
"Requirement already satisfied: minio in /opt/conda/lib/python3.8/site-packages (7.0.3)\n",
|
||||
"Requirement already satisfied: torch>=1.2 in /opt/conda/lib/python3.8/site-packages (from captum) (1.8.1)\n",
|
||||
"Requirement already satisfied: numpy in /opt/conda/lib/python3.8/site-packages (from captum) (1.20.2)\n",
|
||||
"Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.8/site-packages (from torch>=1.2->captum) (3.7.4.3)\n",
|
||||
"Requirement already satisfied: click>=7.1.2 in /opt/conda/lib/python3.8/site-packages (from flask) (7.1.2)\n",
|
||||
"Requirement already satisfied: Jinja2>=3.0 in /opt/conda/lib/python3.8/site-packages (from flask) (3.0.1)\n",
|
||||
"Requirement already satisfied: Werkzeug>=2.0 in /opt/conda/lib/python3.8/site-packages (from flask) (2.0.1)\n",
|
||||
"Requirement already satisfied: itsdangerous>=2.0 in /opt/conda/lib/python3.8/site-packages (from flask) (2.0.1)\n",
|
||||
"Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.8/site-packages (from Jinja2>=3.0->flask) (2.0.1)\n",
|
||||
"Requirement already satisfied: brotli in /opt/conda/lib/python3.8/site-packages (from flask-compress) (1.0.9)\n",
|
||||
"Requirement already satisfied: traitlets>=4.3.1 in /opt/conda/lib/python3.8/site-packages (from ipywidgets) (5.0.5)\n",
|
||||
"Requirement already satisfied: nbformat>=4.2.0 in /opt/conda/lib/python3.8/site-packages (from ipywidgets) (5.1.3)\n",
|
||||
"Requirement already satisfied: widgetsnbextension~=3.5.0 in /opt/conda/lib/python3.8/site-packages (from ipywidgets) (3.5.1)\n",
|
||||
"Requirement already satisfied: jupyterlab-widgets>=1.0.0 in /opt/conda/lib/python3.8/site-packages (from ipywidgets) (1.0.0)\n",
|
||||
"Requirement already satisfied: ipython>=4.0.0 in /opt/conda/lib/python3.8/site-packages (from ipywidgets) (7.22.0)\n",
|
||||
"Requirement already satisfied: ipykernel>=4.5.1 in /opt/conda/lib/python3.8/site-packages (from ipywidgets) (5.5.3)\n",
|
||||
"Requirement already satisfied: tornado>=4.2 in /opt/conda/lib/python3.8/site-packages (from ipykernel>=4.5.1->ipywidgets) (6.1)\n",
|
||||
"Requirement already satisfied: jupyter-client in /opt/conda/lib/python3.8/site-packages (from ipykernel>=4.5.1->ipywidgets) (6.1.12)\n",
|
||||
"Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /opt/conda/lib/python3.8/site-packages (from ipython>=4.0.0->ipywidgets) (3.0.18)\n",
|
||||
"Requirement already satisfied: setuptools>=18.5 in /opt/conda/lib/python3.8/site-packages (from ipython>=4.0.0->ipywidgets) (49.6.0.post20210108)\n",
|
||||
"Requirement already satisfied: pygments in /opt/conda/lib/python3.8/site-packages (from ipython>=4.0.0->ipywidgets) (2.8.1)\n",
|
||||
"Requirement already satisfied: decorator in /opt/conda/lib/python3.8/site-packages (from ipython>=4.0.0->ipywidgets) (4.4.2)\n",
|
||||
"Requirement already satisfied: jedi>=0.16 in /opt/conda/lib/python3.8/site-packages (from ipython>=4.0.0->ipywidgets) (0.18.0)\n",
|
||||
"Requirement already satisfied: pickleshare in /opt/conda/lib/python3.8/site-packages (from ipython>=4.0.0->ipywidgets) (0.7.5)\n",
|
||||
"Requirement already satisfied: backcall in /opt/conda/lib/python3.8/site-packages (from ipython>=4.0.0->ipywidgets) (0.2.0)\n",
|
||||
"Requirement already satisfied: pexpect>4.3 in /opt/conda/lib/python3.8/site-packages (from ipython>=4.0.0->ipywidgets) (4.8.0)\n",
|
||||
"Requirement already satisfied: parso<0.9.0,>=0.8.0 in /opt/conda/lib/python3.8/site-packages (from jedi>=0.16->ipython>=4.0.0->ipywidgets) (0.8.2)\n",
|
||||
"Requirement already satisfied: ipython-genutils in /opt/conda/lib/python3.8/site-packages (from nbformat>=4.2.0->ipywidgets) (0.2.0)\n",
|
||||
"Requirement already satisfied: jupyter-core in /opt/conda/lib/python3.8/site-packages (from nbformat>=4.2.0->ipywidgets) (4.7.1)\n",
|
||||
"Requirement already satisfied: jsonschema!=2.5.0,>=2.4 in /opt/conda/lib/python3.8/site-packages (from nbformat>=4.2.0->ipywidgets) (3.2.0)\n",
|
||||
"Requirement already satisfied: six>=1.11.0 in /opt/conda/lib/python3.8/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets) (1.15.0)\n",
|
||||
"Requirement already satisfied: pyrsistent>=0.14.0 in /opt/conda/lib/python3.8/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets) (0.17.3)\n",
|
||||
"Requirement already satisfied: attrs>=17.4.0 in /opt/conda/lib/python3.8/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets) (20.3.0)\n",
|
||||
"Requirement already satisfied: ptyprocess>=0.5 in /opt/conda/lib/python3.8/site-packages (from pexpect>4.3->ipython>=4.0.0->ipywidgets) (0.7.0)\n",
|
||||
"Requirement already satisfied: wcwidth in /opt/conda/lib/python3.8/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=4.0.0->ipywidgets) (0.2.5)\n",
|
||||
"Requirement already satisfied: notebook>=4.4.1 in /opt/conda/lib/python3.8/site-packages (from widgetsnbextension~=3.5.0->ipywidgets) (6.3.0)\n",
|
||||
"Requirement already satisfied: pyzmq>=17 in /opt/conda/lib/python3.8/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (22.0.3)\n",
|
||||
"Requirement already satisfied: terminado>=0.8.3 in /opt/conda/lib/python3.8/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (0.9.4)\n",
|
||||
"Requirement already satisfied: argon2-cffi in /opt/conda/lib/python3.8/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (20.1.0)\n",
|
||||
"Requirement already satisfied: nbconvert in /opt/conda/lib/python3.8/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (6.0.7)\n",
|
||||
"Requirement already satisfied: Send2Trash>=1.5.0 in /opt/conda/lib/python3.8/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (1.5.0)\n",
|
||||
"Requirement already satisfied: prometheus-client in /opt/conda/lib/python3.8/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (0.10.0)\n",
|
||||
"Requirement already satisfied: python-dateutil>=2.1 in /opt/conda/lib/python3.8/site-packages (from jupyter-client->ipykernel>=4.5.1->ipywidgets) (2.8.1)\n",
|
||||
"Requirement already satisfied: requests-toolbelt<1,>=0.8.0 in /opt/conda/lib/python3.8/site-packages (from kfp) (0.9.1)\n",
|
||||
"Requirement already satisfied: google-api-python-client<2,>=1.7.8 in /opt/conda/lib/python3.8/site-packages (from kfp) (1.12.8)\n",
|
||||
"Requirement already satisfied: docstring-parser<1,>=0.7.3 in /opt/conda/lib/python3.8/site-packages (from kfp) (0.8.1)\n",
|
||||
"Requirement already satisfied: cloudpickle<2,>=1.3.0 in /opt/conda/lib/python3.8/site-packages (from kfp) (1.6.0)\n",
|
||||
"Requirement already satisfied: tabulate<1,>=0.8.6 in /opt/conda/lib/python3.8/site-packages (from kfp) (0.8.9)\n",
|
||||
"Requirement already satisfied: absl-py<=0.11,>=0.9 in /opt/conda/lib/python3.8/site-packages (from kfp) (0.11.0)\n",
|
||||
"Requirement already satisfied: strip-hints<1,>=0.1.8 in /opt/conda/lib/python3.8/site-packages (from kfp) (0.1.9)\n",
|
||||
"Requirement already satisfied: protobuf<4,>=3.13.0 in /opt/conda/lib/python3.8/site-packages (from kfp) (3.15.7)\n",
|
||||
"Requirement already satisfied: kubernetes<13,>=8.0.0 in /opt/conda/lib/python3.8/site-packages (from kfp) (10.0.1)\n",
|
||||
"Requirement already satisfied: google-cloud-storage<2,>=1.20.0 in /opt/conda/lib/python3.8/site-packages (from kfp) (1.37.1)\n",
|
||||
"Requirement already satisfied: kfp-pipeline-spec<0.2.0,>=0.1.8 in /opt/conda/lib/python3.8/site-packages (from kfp) (0.1.8)\n",
|
||||
"Requirement already satisfied: PyYAML<6,>=5.3 in /opt/conda/lib/python3.8/site-packages (from kfp) (5.4.1)\n",
|
||||
"Requirement already satisfied: kfp-server-api<2.0.0,>=1.1.2 in /opt/conda/lib/python3.8/site-packages (from kfp) (1.6.0)\n",
|
||||
"Requirement already satisfied: google-auth<2,>=1.6.1 in /opt/conda/lib/python3.8/site-packages (from kfp) (1.28.1)\n",
|
||||
"Requirement already satisfied: fire<1,>=0.3.1 in /opt/conda/lib/python3.8/site-packages (from kfp) (0.4.0)\n",
|
||||
"Requirement already satisfied: Deprecated<2,>=1.2.7 in /opt/conda/lib/python3.8/site-packages (from kfp) (1.2.12)\n",
|
||||
"Requirement already satisfied: wrapt<2,>=1.10 in /opt/conda/lib/python3.8/site-packages (from Deprecated<2,>=1.2.7->kfp) (1.12.1)\n",
|
||||
"Requirement already satisfied: termcolor in /opt/conda/lib/python3.8/site-packages (from fire<1,>=0.3.1->kfp) (1.1.0)\n",
|
||||
"Requirement already satisfied: google-auth-httplib2>=0.0.3 in /opt/conda/lib/python3.8/site-packages (from google-api-python-client<2,>=1.7.8->kfp) (0.1.0)\n",
|
||||
"Requirement already satisfied: uritemplate<4dev,>=3.0.0 in /opt/conda/lib/python3.8/site-packages (from google-api-python-client<2,>=1.7.8->kfp) (3.0.1)\n",
|
||||
"Requirement already satisfied: httplib2<1dev,>=0.15.0 in /opt/conda/lib/python3.8/site-packages (from google-api-python-client<2,>=1.7.8->kfp) (0.19.1)\n",
|
||||
"Requirement already satisfied: google-api-core<2dev,>=1.21.0 in /opt/conda/lib/python3.8/site-packages (from google-api-python-client<2,>=1.7.8->kfp) (1.26.3)\n",
|
||||
"Requirement already satisfied: packaging>=14.3 in /opt/conda/lib/python3.8/site-packages (from google-api-core<2dev,>=1.21.0->google-api-python-client<2,>=1.7.8->kfp) (20.9)\n",
|
||||
"Requirement already satisfied: requests<3.0.0dev,>=2.18.0 in /opt/conda/lib/python3.8/site-packages (from google-api-core<2dev,>=1.21.0->google-api-python-client<2,>=1.7.8->kfp) (2.25.1)\n",
|
||||
"Requirement already satisfied: pytz in /opt/conda/lib/python3.8/site-packages (from google-api-core<2dev,>=1.21.0->google-api-python-client<2,>=1.7.8->kfp) (2021.1)\n",
|
||||
"Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.6.0 in /opt/conda/lib/python3.8/site-packages (from google-api-core<2dev,>=1.21.0->google-api-python-client<2,>=1.7.8->kfp) (1.53.0)\n",
|
||||
"Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/conda/lib/python3.8/site-packages (from google-auth<2,>=1.6.1->kfp) (0.2.8)\n",
|
||||
"Requirement already satisfied: rsa<5,>=3.1.4 in /opt/conda/lib/python3.8/site-packages (from google-auth<2,>=1.6.1->kfp) (4.7.2)\n",
|
||||
"Requirement already satisfied: cachetools<5.0,>=2.0.0 in /opt/conda/lib/python3.8/site-packages (from google-auth<2,>=1.6.1->kfp) (4.2.1)\n",
|
||||
"Requirement already satisfied: google-cloud-core<2.0dev,>=1.4.1 in /opt/conda/lib/python3.8/site-packages (from google-cloud-storage<2,>=1.20.0->kfp) (1.6.0)\n",
|
||||
"Requirement already satisfied: google-resumable-media<2.0dev,>=1.2.0 in /opt/conda/lib/python3.8/site-packages (from google-cloud-storage<2,>=1.20.0->kfp) (1.2.0)\n",
|
||||
"Requirement already satisfied: google-crc32c<2.0dev,>=1.0 in /opt/conda/lib/python3.8/site-packages (from google-resumable-media<2.0dev,>=1.2.0->google-cloud-storage<2,>=1.20.0->kfp) (1.1.2)\n",
|
||||
"Requirement already satisfied: cffi>=1.0.0 in /opt/conda/lib/python3.8/site-packages (from google-crc32c<2.0dev,>=1.0->google-resumable-media<2.0dev,>=1.2.0->google-cloud-storage<2,>=1.20.0->kfp) (1.14.5)\n",
|
||||
"Requirement already satisfied: pycparser in /opt/conda/lib/python3.8/site-packages (from cffi>=1.0.0->google-crc32c<2.0dev,>=1.0->google-resumable-media<2.0dev,>=1.2.0->google-cloud-storage<2,>=1.20.0->kfp) (2.20)\n",
|
||||
"Requirement already satisfied: pyparsing<3,>=2.4.2 in /opt/conda/lib/python3.8/site-packages (from httplib2<1dev,>=0.15.0->google-api-python-client<2,>=1.7.8->kfp) (2.4.7)\n",
|
||||
"Requirement already satisfied: urllib3>=1.15 in /opt/conda/lib/python3.8/site-packages (from kfp-server-api<2.0.0,>=1.1.2->kfp) (1.26.4)\n",
|
||||
"Requirement already satisfied: certifi in /opt/conda/lib/python3.8/site-packages (from kfp-server-api<2.0.0,>=1.1.2->kfp) (2020.12.5)\n",
|
||||
"Requirement already satisfied: requests-oauthlib in /opt/conda/lib/python3.8/site-packages (from kubernetes<13,>=8.0.0->kfp) (1.3.0)\n",
|
||||
"Requirement already satisfied: websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0 in /opt/conda/lib/python3.8/site-packages (from kubernetes<13,>=8.0.0->kfp) (0.58.0)\n",
|
||||
"Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /opt/conda/lib/python3.8/site-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.1->kfp) (0.4.8)\n",
|
||||
"Requirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.8/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core<2dev,>=1.21.0->google-api-python-client<2,>=1.7.8->kfp) (2.10)\n",
|
||||
"Requirement already satisfied: chardet<5,>=3.0.2 in /opt/conda/lib/python3.8/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core<2dev,>=1.21.0->google-api-python-client<2,>=1.7.8->kfp) (4.0.0)\n",
|
||||
"Requirement already satisfied: wheel in /opt/conda/lib/python3.8/site-packages (from strip-hints<1,>=0.1.8->kfp) (0.36.2)\n",
|
||||
"Requirement already satisfied: kiwisolver>=1.0.1 in /opt/conda/lib/python3.8/site-packages (from matplotlib) (1.3.1)\n",
|
||||
"Requirement already satisfied: cycler>=0.10 in /opt/conda/lib/python3.8/site-packages (from matplotlib) (0.10.0)\n",
|
||||
"Requirement already satisfied: tensorboard!=2.5.0,>=2.2.0 in /opt/conda/lib/python3.8/site-packages (from pytorch-lightning) (2.4.1)\n",
|
||||
"Requirement already satisfied: pyDeprecate==0.3.0 in /opt/conda/lib/python3.8/site-packages (from pytorch-lightning) (0.3.0)\n",
|
||||
"Requirement already satisfied: tqdm>=4.41.0 in /opt/conda/lib/python3.8/site-packages (from pytorch-lightning) (4.60.0)\n",
|
||||
"Requirement already satisfied: torchmetrics>=0.2.0 in /opt/conda/lib/python3.8/site-packages (from pytorch-lightning) (0.3.2)\n",
|
||||
"Requirement already satisfied: future>=0.17.1 in /opt/conda/lib/python3.8/site-packages (from pytorch-lightning) (0.18.2)\n",
|
||||
"Requirement already satisfied: fsspec[http]>=2021.4.0 in /opt/conda/lib/python3.8/site-packages (from pytorch-lightning) (2021.5.0)\n",
|
||||
"Requirement already satisfied: aiohttp in /opt/conda/lib/python3.8/site-packages (from fsspec[http]>=2021.4.0->pytorch-lightning) (3.7.4.post0)\n",
|
||||
"Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /opt/conda/lib/python3.8/site-packages (from tensorboard!=2.5.0,>=2.2.0->pytorch-lightning) (0.4.4)\n",
|
||||
"Requirement already satisfied: markdown>=2.6.8 in /opt/conda/lib/python3.8/site-packages (from tensorboard!=2.5.0,>=2.2.0->pytorch-lightning) (3.3.4)\n",
|
||||
"Requirement already satisfied: grpcio>=1.24.3 in /opt/conda/lib/python3.8/site-packages (from tensorboard!=2.5.0,>=2.2.0->pytorch-lightning) (1.38.0)\n",
|
||||
"Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /opt/conda/lib/python3.8/site-packages (from tensorboard!=2.5.0,>=2.2.0->pytorch-lightning) (1.8.0)\n",
|
||||
"Requirement already satisfied: oauthlib>=3.0.0 in /opt/conda/lib/python3.8/site-packages (from requests-oauthlib->kubernetes<13,>=8.0.0->kfp) (3.1.0)\n",
|
||||
"Requirement already satisfied: async-timeout<4.0,>=3.0 in /opt/conda/lib/python3.8/site-packages (from aiohttp->fsspec[http]>=2021.4.0->pytorch-lightning) (3.0.1)\n",
|
||||
"Requirement already satisfied: yarl<2.0,>=1.0 in /opt/conda/lib/python3.8/site-packages (from aiohttp->fsspec[http]>=2021.4.0->pytorch-lightning) (1.6.3)\n",
|
||||
"Requirement already satisfied: multidict<7.0,>=4.5 in /opt/conda/lib/python3.8/site-packages (from aiohttp->fsspec[http]>=2021.4.0->pytorch-lightning) (5.1.0)\n",
|
||||
"Requirement already satisfied: testpath in /opt/conda/lib/python3.8/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (0.4.4)\n",
|
||||
"Requirement already satisfied: entrypoints>=0.2.2 in /opt/conda/lib/python3.8/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (0.3)\n",
|
||||
"Requirement already satisfied: defusedxml in /opt/conda/lib/python3.8/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (0.7.1)\n",
|
||||
"Requirement already satisfied: bleach in /opt/conda/lib/python3.8/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (3.3.0)\n",
|
||||
"Requirement already satisfied: mistune<2,>=0.8.1 in /opt/conda/lib/python3.8/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (0.8.4)\n",
|
||||
"Requirement already satisfied: pandocfilters>=1.4.1 in /opt/conda/lib/python3.8/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (1.4.3)\n",
|
||||
"Requirement already satisfied: nbclient<0.6.0,>=0.5.0 in /opt/conda/lib/python3.8/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (0.5.3)\n",
|
||||
"Requirement already satisfied: jupyterlab-pygments in /opt/conda/lib/python3.8/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (0.1.2)\n",
|
||||
"Requirement already satisfied: nest-asyncio in /opt/conda/lib/python3.8/site-packages (from nbclient<0.6.0,>=0.5.0->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (1.5.1)\n",
|
||||
"Requirement already satisfied: async-generator in /opt/conda/lib/python3.8/site-packages (from nbclient<0.6.0,>=0.5.0->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (1.10)\n",
|
||||
"Requirement already satisfied: webencodings in /opt/conda/lib/python3.8/site-packages (from bleach->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (0.5.1)\n",
|
||||
"Installing collected packages: kfp\n",
|
||||
"Successfully installed kfp-1.6.3\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! pip uninstall -y kfp\n",
|
||||
"! pip install kfp captum torchvision matplotlib pillow pytorch-lightning flask flask-compress ipywidgets minio"
|
||||
"! pip install --no-cache-dir kfp"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 46,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"v14.16.0\n",
|
||||
"\u001b[K\u001b[?25h \u001b[27m] - reify:npm: \u001b[32;40mtiming\u001b[0m \u001b[35mreify:createSparse\u001b[0m Completed in 0ms\u001b[0m\u001b[Kms\u001b[0m\u001b[K\n",
|
||||
"changed 14 packages, and audited 255 packages in 2s\n",
|
||||
"\n",
|
||||
"11 packages are looking for funding\n",
|
||||
" run `npm fund` for details\n",
|
||||
"\n",
|
||||
"found \u001b[32m\u001b[1m0\u001b[22m\u001b[39m vulnerabilities\n",
|
||||
"\u001b[K\u001b[?25h \u001b[27m] \\ reify:yarn: \u001b[7msill\u001b[0m \u001b[35maudit\u001b[0m bulk request { yarn: [ '1.22.10'\u001b[0m\u001b[K0'\u001b[0m\u001b[K\n",
|
||||
"changed 1 package, and audited 2 packages in 766ms\n",
|
||||
"\n",
|
||||
"found \u001b[32m\u001b[1m0\u001b[22m\u001b[39m vulnerabilities\n",
|
||||
"1.22.10\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# Node Version\n",
|
||||
"! node --version\n",
|
||||
"\n",
|
||||
"# Install yarn\n",
|
||||
"! npm install -g npm\n",
|
||||
"! npm install --global yarn\n",
|
||||
"! yarn --version"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 47,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Installing /opt/conda/lib/python3.8/site-packages/captum/insights/attr_vis/widget/static -> jupyter-captum-insights\n",
|
||||
"- Validating: \u001b[32mOK\u001b[0m\n",
|
||||
"\n",
|
||||
" To initialize this nbextension in the browser every time the notebook (or other app) loads:\n",
|
||||
" \n",
|
||||
" jupyter nbextension enable captum.insights.attr_vis.widget --py --sys-prefix\n",
|
||||
" \n",
|
||||
"Traceback (most recent call last):\n",
|
||||
" File \"/opt/conda/bin/jupyter\", line 8, in <module>\n",
|
||||
" sys.exit(main())\n",
|
||||
" File \"/opt/conda/lib/python3.8/site-packages/jupyter_core/command.py\", line 285, in main\n",
|
||||
" command = _jupyter_abspath(subcommand)\n",
|
||||
" File \"/opt/conda/lib/python3.8/site-packages/jupyter_core/command.py\", line 124, in _jupyter_abspath\n",
|
||||
" raise Exception(\n",
|
||||
"Exception: Jupyter command `jupyter-nbbextension` not found.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# Install Jupyter Notebook Widgets\n",
|
||||
"! jupyter nbextension install --py --symlink --sys-prefix captum.insights.attr_vis.widget\n",
|
||||
"! jupyter nbextension install jupyter-matplotlib"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 48,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Enabling notebook extension jupyter-js-widgets/extension...\n",
|
||||
" - Validating: \u001b[32mOK\u001b[0m\n",
|
||||
"Enabling notebook extension jupyter-captum-insights/extension...\n",
|
||||
" - Validating: \u001b[32mOK\u001b[0m\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# Enable Jupyter Notebook Extensions\n",
|
||||
"! jupyter nbextension enable --py widgetsnbextension\n",
|
||||
"! jupyter nbextension enable captum.insights.attr_vis.widget --py --sys-prefix"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 20,
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
|
|
@ -256,7 +62,7 @@
|
|||
"'1.6.3'"
|
||||
]
|
||||
},
|
||||
"execution_count": 20,
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
|
|
@ -271,22 +77,6 @@
|
|||
"from kfp import dsl\n",
|
||||
"from kfp import compiler\n",
|
||||
"\n",
|
||||
"import numpy as np\n",
|
||||
"import logging\n",
|
||||
"\n",
|
||||
"from PIL import Image\n",
|
||||
"import torchvision.transforms as transforms\n",
|
||||
"\n",
|
||||
"from matplotlib.colors import LinearSegmentedColormap\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"\n",
|
||||
"import torch\n",
|
||||
"import torch.nn.functional as F\n",
|
||||
"\n",
|
||||
"import captum\n",
|
||||
"from captum.attr import IntegratedGradients, Occlusion, LayerGradCam, LayerAttribution\n",
|
||||
"from captum.attr import visualization as viz\n",
|
||||
"\n",
|
||||
"kfp.__version__"
|
||||
]
|
||||
},
|
||||
|
|
@ -309,12 +99,12 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"INGRESS_GATEWAY='http://istio-ingressgateway.istio-system.svc.cluster.local'\n",
|
||||
"AUTH=\"MTYyMzEzOTA4M3xOd3dBTkVNeU4xQTBVek5WUVVoYVVra3pNalEzU2paYVEwSTBRVU5FUkZSUFJFWkNRVTVMTTFreVZFVkNSRWhTTnpKR1UwNVdWRkU9fGKXic0yyJFpWhuQhq9LjvsTTQaBS_TnwSJfF4kSKCgE\"\n",
|
||||
"AUTH=\"<enter your token here>\"\n",
|
||||
"NAMESPACE=\"kubeflow-user-example-com\"\n",
|
||||
"COOKIE=\"authservice_session=\"+AUTH\n",
|
||||
"EXPERIMENT=\"Default\""
|
||||
|
|
@ -329,7 +119,7 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
|
|
@ -340,7 +130,7 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 17,
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
|
|
@ -349,7 +139,7 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 18,
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
|
|
@ -378,7 +168,7 @@
|
|||
" 'storage_state': 'STORAGESTATE_AVAILABLE'}"
|
||||
]
|
||||
},
|
||||
"execution_count": 18,
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
|
|
@ -399,7 +189,7 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
|
|
@ -409,7 +199,7 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 20,
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
|
|
@ -435,7 +225,7 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 21,
|
||||
"execution_count": 9,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
|
|
@ -603,7 +393,7 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 22,
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
|
|
@ -613,13 +403,13 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 23,
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<a href=\"http://istio-ingressgateway.istio-system.svc.cluster.local/pipeline/#/runs/details/72058c34-a86c-4087-8142-f7652b9a538a\" target=\"_blank\" >Run details</a>."
|
||||
"<a href=\"http://istio-ingressgateway.istio-system.svc.cluster.local/pipeline/#/runs/details/308b618c-38d0-43d6-ae6d-3b0d11418f07\" target=\"_blank\" >Run details</a>."
|
||||
],
|
||||
"text/plain": [
|
||||
"<IPython.core.display.HTML object>"
|
||||
|
|
@ -643,7 +433,7 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"execution_count": 13,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
|
|
@ -652,9 +442,9 @@
|
|||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"NAME URL READY PREV LATEST PREVROLLEDOUTREVISION LATESTREADYREVISION AGE\n",
|
||||
"bertserve http://bertserve.kubeflow-user-example-com.example.com True 100 bertserve-predictor-default-f5s8c 6m26s\n",
|
||||
"torchserve False 173m\n"
|
||||
"NAME URL READY PREV LATEST PREVROLLEDOUTREVISION LATESTREADYREVISION AGE\n",
|
||||
"bertserve http://bertserve.kubeflow-user-example-com.example.com True 100 bertserve-predictor-default-zckhh 44h\n",
|
||||
"torchserve http://torchserve.kubeflow-user-example-com.example.com True 100 torchserve-predictor-default-tdknk 47h\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
|
@ -671,16 +461,16 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"execution_count": 14,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"'/bin/bash: jq: command not found'"
|
||||
"'bertserve.kubeflow-user-example-com.example.com'"
|
||||
]
|
||||
},
|
||||
"execution_count": 8,
|
||||
"execution_count": 14,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
|
|
@ -700,7 +490,7 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 24,
|
||||
"execution_count": 27,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
|
|
@ -716,7 +506,7 @@
|
|||
"> Host: bertserve.kubeflow-user-example-com.example.com\n",
|
||||
"> User-Agent: curl/7.68.0\n",
|
||||
"> Accept: */*\n",
|
||||
"> Cookie: authservice_session=MTYyMzEzOTA4M3xOd3dBTkVNeU4xQTBVek5WUVVoYVVra3pNalEzU2paYVEwSTBRVU5FUkZSUFJFWkNRVTVMTTFreVZFVkNSRWhTTnpKR1UwNVdWRkU9fGKXic0yyJFpWhuQhq9LjvsTTQaBS_TnwSJfF4kSKCgE\n",
|
||||
"> Cookie: authservice_session=MTYyMzI1NDI0NHxOd3dBTkVGU1UxaElXRXN5VUVKTVJrZFVUMWhDU1VoVlNVMUhSRFZaVVRWQlNrVkhORTAzUTFWTFVqZExSa0pHVmpWU016SmFOa0U9fO86sBQIDoqYUxX9ffUnG7xS8xyysaWppWJa0c3QBRJd\n",
|
||||
"> Content-Length: 84\n",
|
||||
"> Content-Type: application/x-www-form-urlencoded\n",
|
||||
"> \n",
|
||||
|
|
@ -726,12 +516,12 @@
|
|||
"< HTTP/1.1 200 OK\n",
|
||||
"< content-length: 33\n",
|
||||
"< content-type: application/json; charset=UTF-8\n",
|
||||
"< date: Tue, 08 Jun 2021 11:54:16 GMT\n",
|
||||
"< date: Thu, 10 Jun 2021 08:51:03 GMT\n",
|
||||
"< server: istio-envoy\n",
|
||||
"< x-envoy-upstream-service-time: 499\n",
|
||||
"< x-envoy-upstream-service-time: 535\n",
|
||||
"< \n",
|
||||
"{ [33 bytes data]\n",
|
||||
"100 117 100 33 100 84 63 162 --:--:-- --:--:-- --:--:-- 225\n",
|
||||
"100 117 100 33 100 84 52 132 --:--:-- --:--:-- --:--:-- 214\n",
|
||||
"* Connection #0 to host istio-ingressgateway.istio-system.svc.cluster.local left intact\n"
|
||||
]
|
||||
}
|
||||
|
|
@ -742,7 +532,7 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 25,
|
||||
"execution_count": 16,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
|
|
@ -784,22 +574,22 @@
|
|||
"> Host: bertserve.kubeflow-user-example-com.example.com\n",
|
||||
"> User-Agent: curl/7.68.0\n",
|
||||
"> Accept: */*\n",
|
||||
"> Cookie: authservice_session=MTYyMzEzOTA4M3xOd3dBTkVNeU4xQTBVek5WUVVoYVVra3pNalEzU2paYVEwSTBRVU5FUkZSUFJFWkNRVTVMTTFreVZFVkNSRWhTTnpKR1UwNVdWRkU9fGKXic0yyJFpWhuQhq9LjvsTTQaBS_TnwSJfF4kSKCgE\n",
|
||||
"> Cookie: authservice_session=MTYyMzI1NDI0NHxOd3dBTkVGU1UxaElXRXN5VUVKTVJrZFVUMWhDU1VoVlNVMUhSRFZaVVRWQlNrVkhORTAzUTFWTFVqZExSa0pHVmpWU016SmFOa0U9fO86sBQIDoqYUxX9ffUnG7xS8xyysaWppWJa0c3QBRJd\n",
|
||||
"> Content-Length: 84\n",
|
||||
"> Content-Type: application/x-www-form-urlencoded\n",
|
||||
"> \n",
|
||||
"} [84 bytes data]\n",
|
||||
"* upload completely sent off: 84 out of 84 bytes\n",
|
||||
"100 84 0 0 100 84 0 1 0:01:24 0:00:44 0:00:40 0* Mark bundle as not supporting multiuse\n",
|
||||
"100 84 0 0 100 84 0 1 0:01:24 0:00:52 0:00:32 000:02 00:30 0:00:12 08 0:00:04 0 0 0 1 0:01:24 0:00:45 0:00:39 0* Mark bundle as not supporting multiuse\n",
|
||||
"< HTTP/1.1 200 OK\n",
|
||||
"< content-length: 320\n",
|
||||
"< content-type: application/json; charset=UTF-8\n",
|
||||
"< date: Tue, 08 Jun 2021 11:51:36 GMT\n",
|
||||
"< date: Thu, 10 Jun 2021 08:23:36 GMT\n",
|
||||
"< server: istio-envoy\n",
|
||||
"< x-envoy-upstream-service-time: 44680\n",
|
||||
"< x-envoy-upstream-service-time: 52441\n",
|
||||
"< \n",
|
||||
"{ [320 bytes data]\n",
|
||||
"100 404 100 320 100 84 7 1 0:01:24 0:00:44 0:00:40 93\n",
|
||||
"100 404 100 320 100 84 6 1 0:01:24 0:00:52 0:00:32 77\n",
|
||||
"* Connection #0 to host istio-ingressgateway.istio-system.svc.cluster.local left intact\n"
|
||||
]
|
||||
}
|
||||
|
|
@ -827,8 +617,11 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 21,
|
||||
"metadata": {},
|
||||
"execution_count": 19,
|
||||
"metadata": {
|
||||
"scrolled": true,
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
|
|
@ -852,7 +645,7 @@
|
|||
" 'delta': 0.12081503337965546}]}"
|
||||
]
|
||||
},
|
||||
"execution_count": 21,
|
||||
"execution_count": 19,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
|
|
@ -864,8 +657,10 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 26,
|
||||
"metadata": {},
|
||||
"execution_count": 20,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"prediction_json = json.loads(open(\"./bert_prediction_output.json\", \"r\").read())"
|
||||
|
|
@ -873,8 +668,10 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 91,
|
||||
"metadata": {},
|
||||
"execution_count": 21,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import torch\n",
|
||||
|
|
@ -890,9 +687,18 @@
|
|||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 92,
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Visualization of Predictions"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 23,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from captum.attr import visualization\n",
|
||||
|
|
@ -910,37 +716,19 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 93,
|
||||
"execution_count": 24,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<table width: 100%><div style=\"border-top: 1px solid; margin-top: 5px; padding-top: 5px; display: inline-block\"><b>Legend: </b><span style=\"display: inline-block; width: 10px; height: 10px; border: 1px solid; background-color: hsl(0, 75%, 60%)\"></span> Negative <span style=\"display: inline-block; width: 10px; height: 10px; border: 1px solid; background-color: hsl(0, 75%, 100%)\"></span> Neutral <span style=\"display: inline-block; width: 10px; height: 10px; border: 1px solid; background-color: hsl(120, 75%, 50%)\"></span> Positive </div><tr><th>True Label</th><th>Predicted Label</th><th>Attribution Label</th><th>Attribution Score</th><th>Word Importance</th><tr><td><text style=\"padding-right:2em\"><b>Business</b></text></td><td><text style=\"padding-right:2em\"><b>\"Sci/Tech\" (0.75)</b></text></td><td><text style=\"padding-right:2em\"><b>world</b></text></td><td><text style=\"padding-right:2em\"><b>0.58</b></text></td><td><mark style=\"background-color: hsl(120, 75%, 76%); opacity:1.0; line-height:1.75\"><font color=\"black\"> [CLS] </font></mark><mark style=\"background-color: hsl(0, 75%, 99%); opacity:1.0; line-height:1.75\"><font color=\"black\"> bloomberg </font></mark><mark style=\"background-color: hsl(0, 75%, 91%); opacity:1.0; line-height:1.75\"><font color=\"black\"> has </font></mark><mark style=\"background-color: hsl(120, 75%, 93%); opacity:1.0; line-height:1.75\"><font color=\"black\"> reported </font></mark><mark style=\"background-color: hsl(120, 75%, 96%); opacity:1.0; line-height:1.75\"><font color=\"black\"> on </font></mark><mark style=\"background-color: hsl(120, 75%, 92%); opacity:1.0; line-height:1.75\"><font color=\"black\"> the </font></mark><mark style=\"background-color: hsl(120, 75%, 74%); opacity:1.0; line-height:1.75\"><font color=\"black\"> economy </font></mark><mark style=\"background-color: hsl(0, 75%, 77%); opacity:1.0; line-height:1.75\"><font color=\"black\"> [SEP] </font></mark></td><tr></table>"
|
||||
],
|
||||
"text/plain": [
|
||||
"<IPython.core.display.HTML object>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<table width: 100%><div style=\"border-top: 1px solid; margin-top: 5px; padding-top: 5px; display: inline-block\"><b>Legend: </b><span style=\"display: inline-block; width: 10px; height: 10px; border: 1px solid; background-color: hsl(0, 75%, 60%)\"></span> Negative <span style=\"display: inline-block; width: 10px; height: 10px; border: 1px solid; background-color: hsl(0, 75%, 100%)\"></span> Neutral <span style=\"display: inline-block; width: 10px; height: 10px; border: 1px solid; background-color: hsl(120, 75%, 50%)\"></span> Positive </div><tr><th>True Label</th><th>Predicted Label</th><th>Attribution Label</th><th>Attribution Score</th><th>Word Importance</th><tr><td><text style=\"padding-right:2em\"><b>Business</b></text></td><td><text style=\"padding-right:2em\"><b>\"Sci/Tech\" (0.75)</b></text></td><td><text style=\"padding-right:2em\"><b>world</b></text></td><td><text style=\"padding-right:2em\"><b>0.58</b></text></td><td><mark style=\"background-color: hsl(120, 75%, 76%); opacity:1.0; line-height:1.75\"><font color=\"black\"> [CLS] </font></mark><mark style=\"background-color: hsl(0, 75%, 99%); opacity:1.0; line-height:1.75\"><font color=\"black\"> bloomberg </font></mark><mark style=\"background-color: hsl(0, 75%, 91%); opacity:1.0; line-height:1.75\"><font color=\"black\"> has </font></mark><mark style=\"background-color: hsl(120, 75%, 93%); opacity:1.0; line-height:1.75\"><font color=\"black\"> reported </font></mark><mark style=\"background-color: hsl(120, 75%, 96%); opacity:1.0; line-height:1.75\"><font color=\"black\"> on </font></mark><mark style=\"background-color: hsl(120, 75%, 92%); opacity:1.0; line-height:1.75\"><font color=\"black\"> the </font></mark><mark style=\"background-color: hsl(120, 75%, 74%); opacity:1.0; line-height:1.75\"><font color=\"black\"> economy </font></mark><mark style=\"background-color: hsl(0, 75%, 77%); opacity:1.0; line-height:1.75\"><font color=\"black\"> [SEP] </font></mark></td><tr></table>"
|
||||
],
|
||||
"text/plain": [
|
||||
"<IPython.core.display.HTML object>"
|
||||
]
|
||||
},
|
||||
"execution_count": 93,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"visualization.visualize_text(vis_data_records)"
|
||||
"vis = visualization.visualize_text(vis_data_records)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### visualization appreas as below\n",
|
||||
""
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -952,7 +740,7 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 56,
|
||||
"execution_count": 10,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
|
|
@ -963,12 +751,19 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! kubectl delete pod --field-selector=status.phase==Succeeded -n $NAMESPACE"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
|
|
|
|||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
|
|
@ -5,6 +5,19 @@ This folder contains different PyTorch Kubeflow pipeline examples using the PyTo
|
|||
1. Cifar10 example for Computer Vision
|
||||
2. BERT example for NLP
|
||||
|
||||
Please navigate to the following link for running the examples with Google Vertex AI pipeline
|
||||
|
||||
https://github.com/amygdala/code-snippets/tree/master/ml/vertex_pipelines/pytorch/cifar
|
||||
|
||||
Use the following link for installing KFP python sdk
|
||||
|
||||
https://github.com/kubeflow/pipelines/tree/master/sdk/python
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Check the following prerequisites before running the examples
|
||||
|
||||
[Prequisites](prerequisites.md)
|
||||
|
||||
## Steps to Run the examples in Cluster Environment
|
||||
|
||||
|
|
@ -25,10 +38,37 @@ Ex: `pipelines/samples/contrib/pytorch-samples/iris`
|
|||
### Build and push the docker image
|
||||
```
|
||||
docker build -t image_name:tag -f Dockerfile-cpu .
|
||||
```
|
||||
|
||||
to run the example in gpu, run the following commands for building docker image
|
||||
|
||||
```
|
||||
docker build -t image_name:tag -f Dockerfile-gpu .
|
||||
```
|
||||
|
||||
push the docker image
|
||||
|
||||
```
|
||||
docker tag image_name:tag username/image_name:tag
|
||||
docker push username/image_name
|
||||
```
|
||||
|
||||
Note for gpu testing:
|
||||
|
||||
Following changes needs to be done in the examples notebook
|
||||
|
||||
1. Make sure to set `node selectors`, `gpus`, `accelerator` variables under the train task
|
||||
|
||||
push the docker image.
|
||||
|
||||
2. Use `isvc_gpu_yaml` for GPU inference.
|
||||
|
||||
### Tensorboard Image Update
|
||||
|
||||
A custom tensorboard image is used for viewing pytorch profiler statistics
|
||||
|
||||
Update tensorboard image name in the notebook (variable_name: `TENSORBOARD_IMAGE`) for using any other custom tensorboard image.
|
||||
|
||||
### Update component.yaml files
|
||||
|
||||
To pick the latest changes, component.yaml files needs to be updated.
|
||||
|
|
@ -46,6 +86,34 @@ Open the example notebook and run the cells to deploy the example in KFP.
|
|||
Once the deployment is done, run the prediction and explanations.
|
||||
|
||||
|
||||
### Captum Insights Visualization
|
||||
|
||||
Run the following command to port forward kubeflow dashboard
|
||||
|
||||
```
|
||||
kubectl port-forward svc/istio-ingressgateway -n istio-system 8080:80
|
||||
```
|
||||
|
||||
To view the captum insights UI in the local environment, run the following port forwarding command
|
||||
|
||||
```
|
||||
kubectl port-forward pod/root-0 -n kubeflow-user-example-com <port>:6080
|
||||
```
|
||||
|
||||
For example:
|
||||
|
||||
```
|
||||
kubectl port-forward pod/root-0 -n kubeflow-user-example-com 8999:6080
|
||||
```
|
||||
|
||||
The captum insights UI can be accessed via
|
||||
|
||||
```
|
||||
http://localhost:8999
|
||||
```
|
||||
|
||||
|
||||
|
||||
## Steps to run the examples in local environment
|
||||
|
||||
Use the following notebook files for running the existing Cifar 10 and Bert examples
|
||||
|
|
|
|||
|
|
@ -0,0 +1,19 @@
|
|||
#!/bin/bash
|
||||
|
||||
# Install jq
|
||||
apt-get update
|
||||
apt-get install -y jq
|
||||
|
||||
# Install Pip Packages
|
||||
pip install captum torchvision matplotlib pillow pytorch-lightning flask flask-compress ipywidgets minio
|
||||
|
||||
# Install Yarn
|
||||
npm install npm
|
||||
npm install yarn
|
||||
|
||||
# Install Jupyter Notebook Widgets
|
||||
jupyter nbextension install --py --symlink --sys-prefix captum.insights.attr_vis.widget
|
||||
|
||||
# Enable Jupyter Notebook Extensions
|
||||
jupyter nbextension enable --py widgetsnbextension
|
||||
jupyter nbextension enable captum.insights.attr_vis.widget --py --sys-prefix
|
||||
|
|
@ -0,0 +1,94 @@
|
|||
# Prerequisites
|
||||
|
||||
Following prerequisites needs to be performed for kubeflow 1.3 or lower.
|
||||
|
||||
### Modify KFServing torchserve image
|
||||
|
||||
Edit inferenceservice-config configmap
|
||||
|
||||
```kubectl edit cm inferenceservice-config -n kubeflow```
|
||||
|
||||
Update the following keys under `predictors->pytorch->v2` block
|
||||
|
||||
```
|
||||
"image": "public.ecr.aws/y1x1p2u5/torchserve-kfs",
|
||||
"defaultImageVersion": "0.4.0",
|
||||
"defaultGpuImageVersion": "0.4.0-gpu",
|
||||
```
|
||||
|
||||
### Modify ml-pipeline-ui image
|
||||
|
||||
Edit deployment ml-pipeline-ui
|
||||
|
||||
```
|
||||
kubectl edit deploy ml-pipeline-ui -n kubeflow
|
||||
```
|
||||
|
||||
Edit the image field with below image:
|
||||
|
||||
```
|
||||
image:gcr.io/ml-pipeline-test/bab6577be93b15db1da3ba8d85d3bced08d111da/frontend@sha256:3c8ff77766c08da5a6f38445284aa585dfb74b3930717271f093a621886602d4
|
||||
```
|
||||
|
||||
Save and exit the edit.
|
||||
|
||||
### Add Minio secret for KFServing
|
||||
|
||||
Apply below secret and sa for KFServing to access minio server
|
||||
|
||||
mino-secret.yaml
|
||||
|
||||
```
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: mysecret
|
||||
annotations:
|
||||
serving.kubeflow.org/s3-endpoint: minio-service.kubeflow:9000 # replace with your s3 endpoint
|
||||
serving.kubeflow.org/s3-usehttps: "0" # by default 1, for testing with minio you need to set to 0
|
||||
serving.kubeflow.org/s3-region: "minio" # replace with the region the bucket is created in
|
||||
serving.kubeflow.org/s3-useanoncredential: "false" # omitting this is the same as false, if true will ignore credential provided and use anonymous credentials
|
||||
type: Opaque
|
||||
data:
|
||||
AWS_ACCESS_KEY_ID: <base-64-minio-access-key> # replace with your base64 encoded minio credential
|
||||
AWS_SECRET_ACCESS_KEY: <base-64-minio-secret-key> # replace with your base64 encoded minio credential
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: sa
|
||||
secrets:
|
||||
- name: mysecret
|
||||
```
|
||||
|
||||
Run the following command to set the secrets
|
||||
|
||||
```
|
||||
Kubectl apply -f minio-secret.yaml -n kubeflow-user-example-com
|
||||
```
|
||||
|
||||
### Disable sidecar injection
|
||||
|
||||
Run the following command to disable sidecar injection
|
||||
|
||||
```kubectl label namespace kubeflow-user-example-com istio-injection=disabled --overwrite```
|
||||
|
||||
|
||||
### Creating custom notebook server
|
||||
|
||||
For installing packages via Jupyter Notebook , root permissions are needed.
|
||||
|
||||
Custom jupyter docker image can be created using the docker file mentioned below
|
||||
|
||||
https://github.com/kubeflow/kubeflow/blob/master/components/example-notebook-servers/jupyter/Dockerfile
|
||||
|
||||
In the kubeflow dashboard, create a new jupyter notebook server using following custom image
|
||||
|
||||
a. Open kubeflow dashboard
|
||||
b. Select Notebooks and Click on New servers
|
||||
c. Add new notebook server by choosing custom image
|
||||
d. Use the below custom image
|
||||
|
||||
```public.ecr.aws/y1x1p2u5/jupyter:latest-cpu```
|
||||
|
||||
Click launch on completion
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 12 KiB |
Loading…
Reference in New Issue