kubeflow
/
pipelines
mirror of https://github.com/kubeflow/pipelines.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Refactor to match new samples folder structure (#1741)

This commit is contained in:
carolynwang 2019-08-06 01:23:56 -07:00 committed by Kubernetes Prow Robot
parent dd59bc2597
commit 351f4562a4
7 changed files with 15 additions and 15 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
components/aws/sagemaker/ground_truth/README.md
View File

@ -53,7 +53,7 @@ active_learning_model_arn | ARN of the resulting active learning model
# Samples # Samples
## Used in a pipeline with workteam creation and training ## Used in a pipeline with workteam creation and training
Mini image classification demo: [Demo](https://github.com/kubeflow/pipelines/blob/master/samples/aws-samples/ground_truth_pipeline_demo/) Mini image classification demo: [Demo](https://github.com/kubeflow/pipelines/blob/master/samples/contrib/aws-samples/ground_truth_pipeline_demo/)
# References # References
* [Ground Truth documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/sms.html) * [Ground Truth documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/sms.html)

6
components/aws/sagemaker/hyperparameter_tuning/README.md
View File

@ -64,9 +64,9 @@ training_image | The registry path of the Docker image that contains the trainin
# Samples # Samples
## On its own ## On its own
K-Means algorithm tuning on MNIST dataset: [pipeline](https://github.com/kubeflow/pipelines/blob/master/samples/aws-samples/mnist-kmeans-sagemaker/kmeans-hpo-pipeline.py) K-Means algorithm tuning on MNIST dataset: [pipeline](https://github.com/kubeflow/pipelines/blob/master/samples/contrib/aws-samples/mnist-kmeans-sagemaker/kmeans-hpo-pipeline.py)
Follow the steps as in the [README](https://github.com/kubeflow/pipelines/blob/master/samples/aws-samples/mnist-kmeans-sagemaker/README.md) with some modification: Follow the steps as in the [README](https://github.com/kubeflow/pipelines/blob/master/samples/contrib/aws-samples/mnist-kmeans-sagemaker/README.md) with some modification:
1. Get and store data in S3 buckets 1. Get and store data in S3 buckets
2. Prepare an IAM roles with permissions to run SageMaker jobs 2. Prepare an IAM roles with permissions to run SageMaker jobs
3. Add 'aws-secret' to your kubeflow namespace 3. Add 'aws-secret' to your kubeflow namespace
@ -78,7 +78,7 @@ dsl-compile --py kmeans-hpo-pipeline.py --output kmeans-hpo-pipeline.tar.gz
6. Once the pipeline completes, you can see the outputs under 'Output parameters' in the HPO component's Input/Output section. 6. Once the pipeline completes, you can see the outputs under 'Output parameters' in the HPO component's Input/Output section.
## Integrated into a pipeline ## Integrated into a pipeline
MNIST Classification using K-Means pipeline: [Pipeline](https://github.com/kubeflow/pipelines/blob/master/samples/aws-samples/mnist-kmeans-sagemaker/mnist-classification-pipeline.py) | [Steps](https://github.com/kubeflow/pipelines/blob/master/samples/aws-samples/mnist-kmeans-sagemaker/README.md) MNIST Classification using K-Means pipeline: [Pipeline](https://github.com/kubeflow/pipelines/blob/master/samples/contrib/aws-samples/mnist-kmeans-sagemaker/mnist-classification-pipeline.py) | [Steps](https://github.com/kubeflow/pipelines/blob/master/samples/contrib/aws-samples/mnist-kmeans-sagemaker/README.md)
# Resources # Resources
* [Using Amazon built-in algorithms](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-algo-docker-registry-paths.html) * [Using Amazon built-in algorithms](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-algo-docker-registry-paths.html)

2
components/aws/sagemaker/workteam/README.md
View File

@ -38,7 +38,7 @@ workteam_arn | ARN of the workteam
# Samples # Samples
## In a pipeline with Ground Truth and training ## In a pipeline with Ground Truth and training
Mini image classification: [Demo](https://github.com/kubeflow/pipelines/blob/master/samples/aws-samples/ground_truth_pipeline_demo/) Mini image classification: [Demo](https://github.com/kubeflow/pipelines/tree/master/samples/contrib/aws-samples/ground_truth_pipeline_demo)
# References # References
* [Managing a private workforce](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-workforce-management-private.html) * [Managing a private workforce](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-workforce-management-private.html)

2
samples/contrib/aws-samples/ground_truth_pipeline_demo/README.md
View File

@ -13,7 +13,7 @@ Run the following to download `openimgs-annotations.csv`:
```bash ```bash
wget https://storage.googleapis.com/openimages/2018_04/test/test-annotations-human-imagelabels-boxable.csv -O openimgs-annotations.csv wget https://storage.googleapis.com/openimages/2018_04/test/test-annotations-human-imagelabels-boxable.csv -O openimgs-annotations.csv
``` ```
Create a s3 bucket and run [this python script](https://github.com/kubeflow/pipelines/tree/master/samples/aws-samples/ground_truth_pipeline_demo/prep_inputs.py) to get the images and generate `train.manifest`, `validation.manifest`, `class_labels.json`, and `instuctions.template`. Create a s3 bucket and run [this python script](https://github.com/kubeflow/pipelines/tree/master/samples/contrib/aws-samples/ground_truth_pipeline_demo/prep_inputs.py) to get the images and generate `train.manifest`, `validation.manifest`, `class_labels.json`, and `instuctions.template`.
## Amazon Cognito user groups ## Amazon Cognito user groups

6
samples/contrib/aws-samples/ground_truth_pipeline_demo/mini-image-classification-pipeline.py
View File

@ -5,9 +5,9 @@ from kfp import components
from kfp import dsl from kfp import dsl
from kfp.aws import use_aws_secret from kfp.aws import use_aws_secret
sagemaker_workteam_op = components.load_component_from_file('../../../components/aws/sagemaker/workteam/component.yaml') sagemaker_workteam_op = components.load_component_from_file('../../../../components/aws/sagemaker/workteam/component.yaml')
sagemaker_gt_op = components.load_component_from_file('../../../components/aws/sagemaker/ground_truth/component.yaml') sagemaker_gt_op = components.load_component_from_file('../../../../components/aws/sagemaker/ground_truth/component.yaml')
sagemaker_train_op = components.load_component_from_file('../../../components/aws/sagemaker/train/component.yaml') sagemaker_train_op = components.load_component_from_file('../../../../components/aws/sagemaker/train/component.yaml')
@dsl.pipeline( @dsl.pipeline(
name='Ground Truth image classification test pipeline', name='Ground Truth image classification test pipeline',

2
samples/contrib/aws-samples/mnist-kmeans-sagemaker/kmeans-hpo-pipeline.py
View File

@ -5,7 +5,7 @@ from kfp import components
from kfp import dsl from kfp import dsl
from kfp.aws import use_aws_secret from kfp.aws import use_aws_secret
sagemaker_hpo_op = components.load_component_from_file('../../../components/aws/sagemaker/hyperparameter_tuning/component.yaml') sagemaker_hpo_op = components.load_component_from_file('../../../../components/aws/sagemaker/hyperparameter_tuning/component.yaml')
@dsl.pipeline( @dsl.pipeline(
name='MNIST HPO test pipeline', name='MNIST HPO test pipeline',

10
samples/contrib/aws-samples/mnist-kmeans-sagemaker/mnist-classification-pipeline.py
View File

@ -5,11 +5,11 @@ from kfp import components
from kfp import dsl from kfp import dsl
from kfp.aws import use_aws_secret from kfp.aws import use_aws_secret
sagemaker_hpo_op = components.load_component_from_file('../../../components/aws/sagemaker/hyperparameter_tuning/component.yaml') sagemaker_hpo_op = components.load_component_from_file('../../../../components/aws/sagemaker/hyperparameter_tuning/component.yaml')
sagemaker_train_op = components.load_component_from_file('../../../components/aws/sagemaker/train/component.yaml') sagemaker_train_op = components.load_component_from_file('../../../../components/aws/sagemaker/train/component.yaml')
sagemaker_model_op = components.load_component_from_file('../../../components/aws/sagemaker/model/component.yaml') sagemaker_model_op = components.load_component_from_file('../../../../components/aws/sagemaker/model/component.yaml')
sagemaker_deploy_op = components.load_component_from_file('../../../components/aws/sagemaker/deploy/component.yaml') sagemaker_deploy_op = components.load_component_from_file('../../../../components/aws/sagemaker/deploy/component.yaml')
sagemaker_batch_transform_op = components.load_component_from_file('../../../components/aws/sagemaker/batch_transform/component.yaml') sagemaker_batch_transform_op = components.load_component_from_file('../../../../components/aws/sagemaker/batch_transform/component.yaml')
@dsl.pipeline( @dsl.pipeline(
name='MNIST Classification pipeline', name='MNIST Classification pipeline',