Commit Graph

31 Commits

Author SHA1 Message Date
Thang Minh Vu 328edd8117
fix(components): make inputs.model_artifact_url optional in sagemaker model component (#8336)
* fix(components): make inputs.model_artifact_url optional in sagemaker model component

* chore: run black

* Fixed Stop bug

commit f2092382ee941c2f33935db3e886093a15f103f7
Author: ananth102 <abashyam@amazon.com>
Date:   Fri Oct 7 19:51:55 2022 +0000

    replaced image

commit 2f0e2daa54fe80a3dfc471d393be62d612217b84
Merge: bf2389a66 7ce165432
Author: ananth102 <abashyam@amazon.com>
Date:   Fri Oct 7 19:50:28 2022 +0000

    Merge remote-tracking branch 'stopfix/handle_stopped' into kfpv1fixes2

commit 7ce165432e
Author: Kartik Kalamadi <kalamadi@amazon.com>
Date:   Thu Mar 3 09:58:16 2022 -0800

    Run black

commit 32d6e1388a
Author: Kartik Kalamadi <kalamadi@amazon.com>
Date:   Tue Mar 1 15:25:32 2022 -0800

    Change image for testing

commit 7875d9aa27
Author: Kartik Kalamadi <kalamadi@amazon.com>
Date:   Mon Jan 31 09:29:50 2022 -0800

    Handle Stopped state for all components and fix bug in robomaker simulation function

* chore(docs): Update model README.md

Update README

* updated image and liscense

* chore: pop ModelDataUrl if not exist

* fix: make field as option in aws batch_transform component

chore: run black

chore: revert docker version pump up

chore(docs): update valid instance types

Remove key if not use

Pop KmsKeyId

* update changelog

* chore: pop DataProcessing if no value supplied

* test(components): Update test

* fix(batch_transform): only pop input and output

* fixed log bug

Co-authored-by: ananth102 <abashyam@amazon.com>
2022-10-14 22:12:49 +00:00
Kartik Kalamadi 079eea369a
fix(components): Print logs for AWS SageMaker components (#4879)
* Print logs for Processing and Batch Transform

* Change image in yamls

* Add unit tests for cw calls

* update version in license file to 1.1.1

* generate yaml for the new version

* update changelog
2021-02-23 19:19:14 -08:00
Leonard O' Sullivan 4aa11c3c7f
feat(components) Adds RoboMaker and SageMaker RLEstimator components (#4813)
* Adds RoboMaker and SageMaker RLEstimator components

* Genericise samples

* Genericise samples

* Adds better logging and updates shim component in samples

* Adds fixes for PR comments. Updates tests accordingly

* Adds docker image reference for integration tests. Allows for setting job_name for RLEstimator training jobs

* Separate RM and SM execution roles

* Remove README reference to VPC config items

* Adds more reliable integration test for RoboMaker Simulation Job

* Simplifies integration tests

* Reverted test container entrypoints

* Update black formatting

* Update components for redbackthomson repo

* Prefix RLEstimator job name

* Add RoboMakerFullAccess to generated roles

* Update version to official 1.1.0

* Formatting int test file

* Add PassRole IAM permission to OIDC

* Adds ROBOMAKER_EXECUTION_ROLE_ARN to build vars

Co-authored-by: Nicholas Thomson <nithomso@amazon.com>
2020-12-11 13:27:27 -08:00
Nicholas Thomson d81c8095d0
refactor(components): AWS SageMaker - Full component refactoring (#4336)
* Temporary rebase commit

* Add yaml compiler

* Add compiler CLI

* Update Dockerfile to copy all files

* Add validate input list vs dict

* Add unit test for new train

* Add minor bug fixes

* Override tag when generating specs

* Update pydocs with formatter

* Add contributing doc

* Add formatters to CONTRIBUTING

* Add working generic logic applied to train

* Update component input and output to inherit

* Downgrade to Python 3.7

* Update add outputValue to arg list

* Updated outputValue to outputPath

* Add empty string default to not-required inputs

* Update path to component relative to root

* Update faulty False-y condition

* Update outputs to write to file

* Update doc formatting

* Update docstrings to match structure

* Add unit tests for component and compiler

* Add unit tests for component

* Add spec unit tests

* Add training unit tests

* Update unit test automation

* Add sample formatting checks

* Remove extra flake8 check in integ tests

* Add unit test black check

* Update black formatting for all files

* Update include black formatting

* Add batch component

* Remove old transform components

* Update region input description

* Add all component specs

* Add deploy component

* Add ground truth component

* Add HPO component

* Add create model component

* Add processing component

* Add workteam component

* Add spec unit tests

* Add deploy unit tests

* Add ground truth unit tests

* Add tuning component unit tests

* Add create model component unit test

* Add process component unit tests

* Add workteam component unit tests

* Remove output_path from required_args

* Remove old component implementations

* Update black formatting

* Add assume role feature

* Compiled all components

* Update doc formatting

* Fix process terminate syntax error

* Update compiler to use kfp structures

* Update nits

* Update unified requirements

* Rebase on debugging commit

* Add debugger unit tests

* Update formatting

* Update component YAML

* Fix unit test Dockerfile relative directory

* Update unit test context to root

* Update Batch to auto-generate name

* Update minor docs and formatting changes

* Update deploy name to common autogenerated

* Add f-strings to logs

* Add update support

* Add Amazon license header

* Update autogen and autoformat

* Rename SpecValidator to SpecInputParser

* Split requirements by dev and prod

* Support for checking generated specs

* Update minor changes

* Update deploy component output description

* Update components to beta repository

* Update fix unit test requirements

* Update unit test build spec for new results path

* Update deploy wait for endpoint complete

* Update component configure AWS clients in new method

* Update boto3 retry method

* Update license version

* Update component YAML versions

* Add new version to Changelog

* Update component spec types

* Update deploy config ignore overwrite

* Update component for debugging

* Update images back to 1.0.0

* Remove coverage from components
2020-10-27 14:17:57 -07:00
jkuruba 7c349f3f82
feat(components): AWS SageMaker - Changes for updating an existing endpoint (#4424)
* Changes for updating existing endpoint

* Review comments addressed

* Review comments addressed

* Review comments addressed

* Changed awscli and boto3 version. Ran black to format integration tests

* Removing temporarily to debug integration failures

* Adding back integration tests

* Control the number of parallel integration tests to 10

* Third Party License updated

* Version changed to 0.9.0

* Fixed a typo in Changelog
2020-09-18 16:00:28 -07:00
Dustin Luong 3ebd075212
feat(components): AWS SageMaker - Add optional parameter to allow training component to accept parameters related to Debugger (#4283)
* Implemented debugger for training component with sample pipeline, unit tests, and integration test

* Implemented changes from PR, refactored utils.py, made sample pipeline more succinct, removed hardcoding from integration tests

* Added default parameter for sample pipeline and fixed grammar for sample README, refactored _utils.py for fstrings and fixed offset for errors

* Removed aws secret lines

* Terminate debug rules when terminating training job, Terminate debug rules if terminate is pressed after training job has completed, added integration tests for stop_debug_rules, updated READMEs for train and sample, renamed sample pipeline, removed tensorboard, updated sagemaker version to sagemaker 2.1.0.

* Terminate debug rules when terminating training job, Terminate debug rules if terminate is pressed after training job has completed, added integration tests for stop_debug_rules, updated READMEs for train and sample, renamed sample pipeline, removed tensorboard, updated sagemaker version to sagemaker 2.1.0.

* Removed extra files, cleaned integration test

* Changed integration test to use sample debugger pipeline

* Processing jobs created from debug rules will not terminate, fixing other small issues

* Removed debug from pipeline definition, removed extra line, removed unused function

* Changelog and image tag updates
2020-08-19 15:41:22 -07:00
Nicholas Thomson 8014a44229
feat(components): AWS SageMaker - Support for assuming a role (#4212)
* Add client assume role functionality

* Add assume_role to component.yaml files

* Update image to personal

* Update input to force NoneType on empty

* Update integration test setup with assumed role

* Add assume role integration test

* Update boto session to use refreshing credentials

* Update assume role relax trust relationship

* Add check for defined assumed role name

* Add processing assume integ test

* Add assume role unit test for main methods

* Add assume_role to all READMEs

* Update session to use AssumeRoleProvider

* Remove region from child calls to session

* Fix extra region_name in test

* Update assume role processing integ test name

* Add processing integ test to list

* Update assumed role to remain if not generated

* Update license version

* Update image tag to new version

* Add new version to Changelog
2020-08-03 10:53:43 -07:00
Suraj Kota 900eeaec16
feat(components): AWS SageMaker - Add functionality to stop SageMaker jobs on run termination (#4167)
* Add functionality to stop SM jobs
	- Unit and Integration tests for the functionality

* unit test update and customer message update

* Changelog and image tag updates

* update version for deploy component and merge conflicts

* Update version in License file

* fix conflicting paths for download, add test for batch
2020-07-17 16:34:50 -07:00
Kartik Kalamadi 799db4714f
[AWS SageMaker] Integ test to check CloudWatch logs print feature (#4056)
* Integ test for cw logs

* Update license file version to 0.5.3

* update version in yaml

* add changelog
2020-07-09 15:14:33 -07:00
Nicholas Thomson f0f8e5d178
[AWS SageMaker] De-hardcode output paths in AWS components (#4119)
* Update input arguments

* Remove fileOutputs

* Update outputs to new paths

* Modify integ test artifact path

* Add unit test for new output format

* Add unit test for write_output

* Migrate tests into test_utils

* Add clarifying comment

* Remove output path file extension

* Update license to 0.5.2

* Update component to 0.5.2

* Add 0.5.2 to changelog

* Remove JSON
2020-07-06 14:39:57 -07:00
Matthew Rose 131be23467
[AWS SageMaker] GroundTruth Pre/Post Lambda function region additions (#3932)
* add more region ARNs to groundtruth lambda map as per sagemaker documentation

* changes for patch release 0.5.1

* Make label_category_config optional in groundtruth component

* merge fix

Co-authored-by: Suraj Kota <surakota@amazon.com>
2020-07-01 10:32:17 -07:00
Kartik Kalamadi b3d8e04e1e
[AWS SageMaker] Print SageMaker job logs in kfp UI (#3954)
* Print logs for AWS SM Componenets on KFP UI

* address comments

* update version number to 0.5.0

* update yaml to version 0.5.0

* update changelog
2020-06-19 00:33:58 -07:00
Nicholas Thomson c29ee5de13
[AWS SageMaker] Component 0.4.1 Release (#4011)
* Update license version

* Update component YAML versions

* Add 0.4.1 to changelog
2020-06-17 18:02:04 -07:00
Nicholas Thomson 5717bc0fbb
Add missing import statement (#4010) 2020-06-17 16:16:05 -07:00
Nicholas Thomson bea63652e1
[AWS SageMaker] Processing job component (#3944)
* Add TDD processing definition

* Update README

* Update temporary image

* Update component entrypoint

* Add WORKDIR to fix Docker 18 support

* integration test for processing job

* Remove job links

* Add container outputs and tests

* Update default properties

* Remove max_run_time if none provided

* Update integration readme steps

* Updated README with more resources

* Add CloudWatch link back to logs

* Update input and output config to arrays

* Update processing integration test

* Update process README

* Update unit tests

* Updated license version

* Update component image versions

* Update changelog

Co-authored-by: Suraj Kota <surakota@amazon.com>
2020-06-17 11:29:24 -07:00
Gautam Kumar bbe598db26
Adding HPO unit test (#3791)
* Adding HPO unit test

* Adding best training job

* Addressing comment
2020-05-22 22:57:10 -07:00
Kartik Kalamadi d18ad7a563
AWS SageMaker : Use IAM Roles for Service Account (#3719)
* don't use aws-secret and update readme for sample pipelines

* Addressed comments on PR and few more readme changes

* small changes to readme

* nit change

* Address comments
2020-05-21 10:36:14 -07:00
Gautam Kumar 6e2a55cf84
Changing the default volume size to 30 (#3792) 2020-05-20 12:36:20 -07:00
Nicholas Thomson ddd1969b34
[AWS SageMaker] Unit tests for Training component (#3722)
* Added additional training unit tests

* Add main training function tests

* Add full training test coverage

* Fix import sys

* Fix poorly named test
2020-05-13 16:14:22 -07:00
Nicholas Thomson bd8c1ddd38
[AWS SageMaker] Specify component input types (#3683)
* Replace all string types with Python types

* Update HPO yaml

* Update Batch YAML

* Update Deploy YAML

* Update GroundTruth YAML

* Update Model YAML

* Update Train YAML

* Update WorkTeam YAML

* Updated samples to remove strings

* Update to temporary image

* Remove unnecessary imports

* Update image to newer image

* Update components to python3

* Update bool parser type

* Remove empty ContentType in samples

* Update to temporary image

* Update to version 0.3.1

* Update deploy to login

* Update deploy load config path

* Fix export environment variable in deploy

* Fix env name

* Update deploy reflow env paths

* Add debug config line

* Use username and password directly

* Updated to 0.3.1

* Update field types to JsonObject and JsonArray
2020-05-11 22:06:21 -07:00
Gautam Kumar 45bc582374
Fixing volume size default value from 1 to 30 (#3598) 2020-04-26 17:17:28 -07:00
Kartik Kalamadi 0259fe50b3
AWS Sagemaker : Use json.dumps() to better organize the input and remove data_locations (#3518)
* construct channel input using json.dumps()

* remover data_location parameters

* add changelog

* Update version in license file and small changes to readme
2020-04-23 12:14:07 -07:00
Kartik Kalamadi f041b08190
AWS sagemaker: fixed a bug in ground_truth and updated all components to use images from new docker hub repo (#3474)
* Don't leave active_learning_model_arn.txt empty

* updated readme for ground_truth_pipeline_demo

* update docker repo

* Small changes to readme of ground truth sample pipeline
2020-04-14 10:26:13 -07:00
Kartik Kalamadi 942be78bfe
Make endpoint_url None (#3374) 2020-04-07 13:19:43 -07:00
Redback 2fe8c0de61 [Component] Add VPC Interface Endpoint Support for SageMaker (#2299)
* Added Private Link Components

* Updated Component Dockerfile

* Added endpoint_url to Samples
2019-10-03 18:11:56 -07:00
Redback 12dde375b8 [Component] Add Managed Spot Training Support for SageMaker (#2219)
* Added Spot Instance Support

* Fixed missing output configuration

* Added spot instance support to example pipelines

* Updated image to new repository
2019-10-03 12:11:56 -07:00
carolynwang 69ca3c7e4b Update images, bug fixes, clean up code (#1778)
* Update docker images and minor refactoring

* Update image tag, bug fixes, remove unneeded imports

* Revert to using image version, use origin batch transform output method

* Forgot to save 2 changes
2019-08-09 15:25:13 -07:00
carolynwang 351f4562a4 Refactor to match new samples folder structure (#1741) 2019-08-06 01:23:56 -07:00
carolynwang 3f8bcffaa7 Add SageMaker create workteam and Ground Truth components, sample demo pipeline, other minor updates (#1716)
* Add components for workteam and Ground Truth, minor update for HPO and train, add sample pipeline demo for workteam and GT, update images

* Minor style fixes

* Address PR comments

* Refactor for new folder structure
2019-08-05 15:33:50 -07:00
carolynwang 81b0f08a84 Update SageMaker components and sample pipeline (#1682)
* Update HPO, train, batch transform components, add MNIST kmeans with HPO pipeline

* Minor bug fixes

* Minor bug fixes, update Dockerfiles

* Update docker images and sample pipeline

* Update all components and sample pipeline

* Delete Dockerfiles for individual component

* Typo fix in Dockerfile
2019-07-30 12:47:51 -07:00
carolynwang 2778632ba2 Add SageMaker HPO component and sample usage in a pipeline (#1628)
* add HPO component and sample pipeline usage

* Update Dockerfile to include HPO component

* Update docker image used in hpo component

* Update HPO readme, make HPO job name required, allow empty string for int params, reintro some default values

* Resolve a couple todos

* Add Dockerfile for HPO and update docker image used in HPO component

* Add Dockerfile for HPO
2019-07-21 19:06:51 -07:00