Commit Graph

6 Commits

Author SHA1 Message Date
cliveseldon 8d728f0b06 GitHub Summarization Seldon Update (#472)
* Update model inference wrapping to use S2I and update docs

* Add s2i reference in docs

* Fix typo highlighted in review

* Add pyLint annotation to allow protected-access on keras make predict function method
2019-01-17 16:07:34 -08:00
Michelle Casbon 70a22d6d7b [GH Issue Summarization] Upgrade to kf v0.4.0-rc.2 (#450)
* Update tfjob components to v1beta1

Remove old version of tensor2tensor component

* Combine UI into a single jsonnet file

* Upgrade GH issue summarization to kf v0.4.0-rc.2

Use latest ksonnet v0.13.1
Use latest seldon v1alpha2
Remove ksonnet app with full kubeflow platform & replace with components specific to this example.
Remove outdated scripts
Add cluster creation links to Click-to-deploy & kfctl
Add warning not to use the Training with an Estimator guide
Replace commandline with bash for better syntax highlighting
Replace messy port-forwarding commands with svc/ambassador
Add modelUrl param to ui component
Modify teardown instructions to remove the deployment
Fix grammatical mistakes

* Rearrange tfjob instructions
2018-12-30 20:05:29 -08:00
Jeremy Lewi 1043bc0c26 A bunch of changes to support distributed training using tf.estimator (#265)
* Unify the code for training with Keras and TF.Estimator

Create a single train.py and trainer.py which uses Keras inside TensorFlow
Provide options to either train with Keras or TF.TensorFlow
The code to train with TF.estimator doesn't worki

See #196
The original PR (#203) worked around a blocking issue with Keras and TF.Estimator by commenting
certain layers in the model architecture leading to a model that wouldn't generate meaningful
predictions
We weren't able to get TF.Estimator working but this PR should make it easier to troubleshoot further

We've unified the existing code so that we don't duplicate the code just to train with TF.estimator
We've added unitttests that can be used to verify training with TF.estimator works. This test
can also be used to reproduce the current errors with TF.estimator.
Add a Makefile to build the Docker image

Add a NFS PVC to our Kubeflow demo deployment.

Create a tfjob-estimator component in our ksonnet component.

changes to distributed/train.py as part of merging with notebooks/train.py
* Add command line arguments to specify paths rather than hard coding them.
* Remove the code at the start of train.py to wait until the input data
becomes available.
* I think the original intent was to allow the TFJob to be started simultaneously with the preprocessing
job and just block until the data is available
* That should be unnecessary since we can just run the preprocessing job as a separate job.

Fix notebooks/train.py (#186)

The code wasn't actually calling Model Fit
Add a unittest to verify we can invoke fit and evaluate without throwing exceptions.

* Address comments.
2018-11-07 16:23:59 -08:00
Michelle Casbon 836ad70421 Fix model file upload (#160)
* Add component parameters

Add model_url & port arguments to flask app
Add service_type, image, and model_url parameters to ui component
Fix problem argument in tensor2tensor component

* Fix broken UI component

Fix broken UI component structure by adding all, service, & deployment parts
Add parameter defaults for tfjob to resolve failures deploying other components

* Add missing imports in flask app

Fix syntax error in argument parsing
Remove underscores from parameter names to workaround ksonnet bug #554: https://github.com/ksonnet/ksonnet/issues/554

* Fix syntax errors in t2t instructions

Add CPU image build arg to docker build command for t2t-training
Fix link to ksonnet app dir
Correct param names for tensor2tensor component
Add missing params for tensor2tensor component
Fix apply command syntax
Swap out log view pod for t2t-master instead of tf-operator
Fix link to training with tfjob

* Fix model file upload

Update default params for tfjob-v1alpha2
Fix build directory path in Makefile

* Resolve lint issues

Lines too long

* Add specific image tag to tfjob-v1alpha2 default

* Fix defaults for training output files

Update image tag
Add UI image tag

* Revert service account secret details

Update associated readme
2018-06-29 18:41:20 -07:00
Jeremy Lewi e12231bae3 Make it easier to demo serving and run in Katacoda (#107)
* Make it easier to demo serving and run in Katacoda

* Allow the model path to be specified via environment variables so that
  we could potentially load the model from PVC.

* Continue to bake the model into the image so that we don't need to train
  in order to serve.

* Parameterize download_data.sh so we could potentially fetch different sources.

* Update the Makefile so that we can build and set the image for the serving
  component.

* Fix lint.

* Update the serving docs.
2018-04-28 08:11:18 -07:00
Jeremy Lewi 4b33d44af6 Support training using a PVC for the data. (#98)
* Support training using a PVC for the data.

* This will make it easier to run the example on Katacoda and non-GCP platforms.

* Modify train.py so we can use a GCS location or local file paths.

* Update the Dockerfile. The jupyter Docker images and had a bunch of
  dependencies removed and the latest images don't have the dependencies
  needed to run the examples.

* Creat a tfjob-pvc component that trains reading/writing using PVC
  and not GCP.

* * Address reviewer comments

* Ignore changes to the ksonnet parameters when determining whether to include
  dirty and sha of the diff in the image. This way we can update the
  ksonnet app with the newly built image without it leading to subsequent
  images being marked dirty.

* Fix lint issues.

* Fix lint import issue.
2018-04-27 04:08:19 -07:00