Commit Graph

31 Commits

Author SHA1 Message Date
Nicholas Thomson f2a860b84c
[AWS SageMaker] Integration tests automation (#3768)
* # This is a combination of 5 commits.
# This is the 1st commit message:

Add initial scripts

# This is the commit message #2:

Add working pytest script

# This is the commit message #3:

Add initial scripts

# This is the commit message #4:

Add environment variable files

# This is the commit message #5:

Remove old cluster script

* Add initial scripts

Add working pytest script

Add initial scripts

Add environment variable files

Remove old cluster script

Update pipeline credentials to OIDC

Add initial scripts

Add working pytest script

Add initial scripts

Add working pytest script

* Remove debugging mark

* Update example EKS cluster name

* Remove quiet from Docker build

* Manually pass env

* Update env list vars as string

* Update use array directly

* Update variable array to export

* Update to using read for splitting

* Move to helper script

* Update export from CodeBuild

* Add wait for minio

* Update kubectl wait timeout

* Update minor changes for PR

* Update integration test buildspec to quiet build

* Add region to delete EKS

* Add wait for pods

* Updated README

* Add fixed interval wait

* Fix CodeBuild step order

* Add file lock for experiment ID

* Fix missing pytest parameter

* Update run create only once

* Add filelock to conda env

* Update experiment name ensuring creation each time

* Add try/catch with create experiment

* Remove caching from KFP deployment

* Remove disable KFP caching

* Move .gitignore changes to inside component

* Add blank line to default .gitignore
2020-05-20 14:18:19 -07:00
Gautam Kumar 6e2a55cf84
Changing the default volume size to 30 (#3792) 2020-05-20 12:36:20 -07:00
Jiaxin Shan af4e8efa3e
Add more approvers in AWS sagemaker components (#3740) 2020-05-15 11:27:36 -07:00
Suraj Kota bff83921d7
AWS Sagemaker Components - enhance integration test coverage (#3720)
* AWS Sagemaker Components - enhance integration test coverage
	- Add tests for create endpoint, hpo job and batch transform
	- Minor bug fixes and documentation

* rev2: Address comments and clean up generated artifacts

* rev3: address more comments

* rev4: add canary test marker

* Trigger Build
2020-05-15 10:21:36 -07:00
Nicholas Thomson ddd1969b34
[AWS SageMaker] Unit tests for Training component (#3722)
* Added additional training unit tests

* Add main training function tests

* Add full training test coverage

* Fix import sys

* Fix poorly named test
2020-05-13 16:14:22 -07:00
Nicholas Thomson bd8c1ddd38
[AWS SageMaker] Specify component input types (#3683)
* Replace all string types with Python types

* Update HPO yaml

* Update Batch YAML

* Update Deploy YAML

* Update GroundTruth YAML

* Update Model YAML

* Update Train YAML

* Update WorkTeam YAML

* Updated samples to remove strings

* Update to temporary image

* Remove unnecessary imports

* Update image to newer image

* Update components to python3

* Update bool parser type

* Remove empty ContentType in samples

* Update to temporary image

* Update to version 0.3.1

* Update deploy to login

* Update deploy load config path

* Fix export environment variable in deploy

* Fix env name

* Update deploy reflow env paths

* Add debug config line

* Use username and password directly

* Updated to 0.3.1

* Update field types to JsonObject and JsonArray
2020-05-11 22:06:21 -07:00
Suraj Kota 6beab2251d
Integration tests for AWS SageMaker Components (#3654)
* integration tests for aws sagemaker components with comment

* address comment related to S3 dataset creation

* rev3: bug fix in conda env yaml and resuse sagemaker method to get image URI

* Add createModel test

	- reduce code duplication
	- add some utility methods
2020-05-06 22:19:09 -07:00
Nicholas Thomson 9ade740ca6
[AWS SageMaker] Add CodeBuild Steps (#3668)
* Add initial unit test buildspec

* Add docker log output

* Add force no pytest color

* Update docker build to be quiet

* Add pass all environment variables

* Update unit test container env file

* Update env to use different syntax

* Remove daemon mode

* Remove TTY from docker run

* Add dryrun and dockercfg setup

* Update dryrun into CodeBuild logic

* Add mkdir for Docker config

* Update app version temporarily

* Revert app version temporarily

* Update unit test log file

* Add tag minor and major versions

* Update version temporarily

* Add print for major and minor tags

* Revert version back down

* Add deploy version override

* Update path to testing directories

* Fix tab formatting

* Fix pytest log directory
2020-05-04 14:13:07 -07:00
Kartik Kalamadi 2f4eafb031
AWS Sagemaker : Add unit tests (#3642)
* Initial changes

* add one test for each component

* Add readme for unit tests

* add empty string test and dockerfile

* added dockerfile

* use python3 in dockerfile

* add coverage report to unit tests

* update readme for PR

* small changes to resolve git comments

* copy requirements.txt separately in dockerfile

* small changes

* pin pip package versions in unit_tests
2020-04-30 01:32:18 -07:00
Gautam Kumar 45bc582374
Fixing volume size default value from 1 to 30 (#3598) 2020-04-26 17:17:28 -07:00
Kartik Kalamadi 0259fe50b3
AWS Sagemaker : Use json.dumps() to better organize the input and remove data_locations (#3518)
* construct channel input using json.dumps()

* remover data_location parameters

* add changelog

* Update version in license file and small changes to readme
2020-04-23 12:14:07 -07:00
Suraj Kota fbed280e55
add user agent header to boto3 client for aws components (#3487)
* add user agent header to boto client

* add component version according to license file

* fetch version from license file at runtime
2020-04-15 11:25:46 -07:00
Kartik Kalamadi f041b08190
AWS sagemaker: fixed a bug in ground_truth and updated all components to use images from new docker hub repo (#3474)
* Don't leave active_learning_model_arn.txt empty

* updated readme for ground_truth_pipeline_demo

* update docker repo

* Small changes to readme of ground truth sample pipeline
2020-04-14 10:26:13 -07:00
Suraj Kota fc5f977b19
Update documentation for AWS components (#3410)
* deploy_createModel_readme

* readme for batch and minor updates to deploy and create_model

* updates based on review comments 1

* correct SageMaker typo
2020-04-08 09:43:46 -07:00
Kartik Kalamadi 942be78bfe
Make endpoint_url None (#3374) 2020-04-07 13:19:43 -07:00
Kartik Kalamadi 060cabf911
AWS Sagemaker : Updated documents (#3440)
* Initial readme for Train component

* example input

* add train pipeline

* added simple_train_pipeline

* Updated readme to include kmeans-hpo-pipeline.py

* Updated train component readme

* fix typo

* Update details about how to get sample data for Train component

* update comment and give a defaault path for output

* change s3 bucket to match other sample pipelines
2020-04-07 11:17:44 -07:00
Kartik Kalamadi 956f645503
AWS sagemaker : Added license files and updated Dockerfile to use AmazonLinux (#3397)
* Added new LICENSE file

* added 2 more license files

* copy license files into the docker image

* pinned pip packages and rearranged the dockerfile
2020-04-06 20:55:43 -07:00
Leonard Aukea a3defb13a9
Added 'workgroup' as optional argument to athena component (#3254)
* added optional workgroup arg

* Added a slighly more elegant way of handling query exec params

* added args to yml and main()

* added args to yml and main()
2020-03-12 18:24:38 -07:00
Redback 2fe8c0de61 [Component] Add VPC Interface Endpoint Support for SageMaker (#2299)
* Added Private Link Components

* Updated Component Dockerfile

* Added endpoint_url to Samples
2019-10-03 18:11:56 -07:00
Redback 12dde375b8 [Component] Add Managed Spot Training Support for SageMaker (#2219)
* Added Spot Instance Support

* Fixed missing output configuration

* Added spot instance support to example pipelines

* Updated image to new repository
2019-10-03 12:11:56 -07:00
Christian Clauss 8e1e823139 Lint Python code for undefined names (#1721)
* Lint Python code for undefined names

* Lint Python code for undefined names

* Exclude tfdv.py to workaround an overzealous pytest

* Fixup for tfdv.py

* Fixup for tfdv.py

* Fixup for tfdv.py
2019-08-21 15:04:31 -07:00
carolynwang 69ca3c7e4b Update images, bug fixes, clean up code (#1778)
* Update docker images and minor refactoring

* Update image tag, bug fixes, remove unneeded imports

* Revert to using image version, use origin batch transform output method

* Forgot to save 2 changes
2019-08-09 15:25:13 -07:00
carolynwang 351f4562a4 Refactor to match new samples folder structure (#1741) 2019-08-06 01:23:56 -07:00
carolynwang 3f8bcffaa7 Add SageMaker create workteam and Ground Truth components, sample demo pipeline, other minor updates (#1716)
* Add components for workteam and Ground Truth, minor update for HPO and train, add sample pipeline demo for workteam and GT, update images

* Minor style fixes

* Address PR comments

* Refactor for new folder structure
2019-08-05 15:33:50 -07:00
carolynwang 81b0f08a84 Update SageMaker components and sample pipeline (#1682)
* Update HPO, train, batch transform components, add MNIST kmeans with HPO pipeline

* Minor bug fixes

* Minor bug fixes, update Dockerfiles

* Update docker images and sample pipeline

* Update all components and sample pipeline

* Delete Dockerfiles for individual component

* Typo fix in Dockerfile
2019-07-30 12:47:51 -07:00
carolynwang 2778632ba2 Add SageMaker HPO component and sample usage in a pipeline (#1628)
* add HPO component and sample pipeline usage

* Update Dockerfile to include HPO component

* Update docker image used in hpo component

* Update HPO readme, make HPO job name required, allow empty string for int params, reintro some default values

* Resolve a couple todos

* Add Dockerfile for HPO and update docker image used in HPO component

* Add Dockerfile for HPO
2019-07-21 19:06:51 -07:00
Jiaxin Shan f7d68f7fb2 Add OWNERS file in aws components and examples (#1633) 2019-07-16 16:54:34 -07:00
Jiaxin Shan d5147b9776 Add HyperParameters back to SageMaker training job (#1377) 2019-05-30 19:24:21 -07:00
tiffany jernigan 778fe2ad7a Fix naming from sagamaker to sagemaker (#1386) 2019-05-24 14:15:30 -07:00
Jiaxin Shan f5c464a87d Add AWS EMR and Athena components (#1286)
* Add EMR and Athena components

* Update components/aws/athena/query/component.yaml

Co-Authored-By: Jeffwan <seedjeffwan@gmail.com>

* Clean up athena components
2019-05-06 17:11:44 -07:00
Jiaxin Shan 5374b6b2b4 Add SageMaker components and example pipeline (#1276)
* Add SageMaker components and example pipeline

* Address review feedbacks

* Expose more training job configs

* Update components/aws/sagemaker/batch_transform/component.yaml

Update components descriptions

Co-Authored-By: Jeffwan <seedjeffwan@gmail.com>
2019-05-03 14:55:37 -07:00