Commit Graph

12 Commits

Author SHA1 Message Date
Eterna2 91d941d6e5 [Feature] Supports parameterized S3Artifactory for Pipeline and ContainerOp in kfp package (#1064)
* kfp can declare custom artifact location in pipeline and containerop.

* Removed default artifact location

* Minor fixes
2019-05-14 19:48:20 -07:00
Ilias Katsakioris c4c2d166fe Fix PipelineParam pattern bug (#1300)
* Generate a pattern in the constructor if one is not provided
* Add compiler tests

Signed-off-by: Ilias Katsakioris <elikatsis@arrikto.com>
2019-05-09 15:43:58 -07:00
Ilias Katsakioris 3a10425692 SDK: Fix wrong indentation in extract_pipelineparams_from_any() (#1076)
Remove a level of indentation from `return` statement in dict case which
resulted in checking only the first value

Signed-off-by: Ilias Katsakioris <elikatsis@arrikto.com>
2019-04-01 16:04:34 -07:00
Eterna2 825f64d672 Feature: sidecar for ContainerOp (#879)
* Feature: sidecar for ContainerOp

* replace f-string with string format for compatibility with py3.5

* ContainerOp now can be updated with any k8s V1Container attributes as well as sidecars with Sidecar class. ContainerOp accepts PipelineParam in any valid k8 properties.

* WIP: fix conflicts and bugs with recent master. TODO: more complex template with pipeline params

* fix proxy args

* Fixed to work with latest master head

* Added container_kwargs to ContainerOp to pass in k8s container kwargs

* Fix comment bug, updated with example in ContainerOp docstring

* fix copyright year

* expose match_serialized_pipelineparam as public for compiler to process serialized pipeline params

* fixed pydoc example and removed unnecessary ContainerOp.container.parent

* Fix conflicts in compiler tests
2019-03-28 11:11:30 -07:00
Ning c6ca1ac38e
well-defined parameter types (#978)
* remove GCSPath fields to avoid artifact type confusion
change the type json schema field name to openAPIV3Schema
* fix unit tests; add unit test for openapishema property
* add ignore_type in pipelineparam
* change the names in the artifact types to avoid confusion with the parameter types
* based on the google python style guide, change the camel case to lower case with underscores
2019-03-19 17:38:13 -07:00
Ning 2accf4180a
Add unit tests pipelineparam (#975)
* add unit test to the pipelineparam with types
* create TypeMeta deserialize function, add comments
* strongly typed pipelineparamtuple
* addressing pr comments
2019-03-18 18:07:36 -07:00
Ning 754db1f724
Fix sample test failure because of the type information in the pipelineparam (#972)
* fix bug: op_to_template resolve the raw arguments by mapping to the argument_inputs but the argument_inputs lost the type information

* fix type pattern matching

* convert orderedDict to dict from the component module
2019-03-15 13:49:21 -07:00
Ning c829115574 Add type check (#938)
* add core types and type checking function

* fix unit test bug

* avoid defining dynamic classes

* typo fix

* add component metadata format

* add a construct for the component decorator

* add default values for the meta classes

* add input/output types to the metadata

* add from_dict in TypeMeta

* small fix

* add unit tests

* use python struct for the openapi schema

* add default in parameter

* add default value

* remove the str restriction for the param default

* bug fix

* add pipelinemeta

* add pipeline metadata

* ignore annotation if it is not str/BaseType/dict

* update param name in the check_type functions
remove schema validators for GCRPath, and adjust for GCRPath, GCSPath
change _check_valid_dict to _check_valid_type_dict to avoid confusion
fix typo in the comments
adjust function order for readability

* remove default values for non-primitive types in the function signature
update the _check_valid_type_dict name

* pass metadata from component decorator and task factory to containerOp

* pass pipeline metadata to Pipeline

* fix unit test

* typo in the comments

* move the metadata classes to a separate module

* fix unit test

* small change

* add __eq__ to meta classes
not export _metadata classes

* nothing

* fix unit test

* unit test python component

* unit test python pipeline

* fix bug: duplicate variable of args

* fix unit tests

* move python_component and _component decorator in _component file

* remove the print

* change parameter default value to None

* add functools wraps around _component decorator

* TypeMeta accept both str and dict

* fix indent, add unit test for type as strings

* do not set default value for the name field in ParameterMeta, ComponentMeta, and PipelineMeta

* add type check in task factory

* output error message

* add type check in component decorator; move the metadata assignment out of the containerop __init__ function

* fix bug; add unit test

* add more unit tests

* more unit tests; fix bugs

* more unit tests; fix bugs

* add unit tests

* more unit tests

* add type check switch; add unit tests

* add compiler option for type check

* resolving pr comments

* add unit test for pipeline param check with component types; fix the bug; also fix the bug when there are not a single return annotations
2019-03-11 11:22:12 -07:00
Ning 81d7a4eec2 extract the pipelineparam deserialize function (#841)
* extract the pipelineparam deserialize function

* typo fix

* adjust extract_param to accept a list of strings

* convert arg to string

* bug fix, add unit tests
2019-02-25 14:44:09 -08:00
Ning d3c4add0a9 DSL refactor (#619)
* add comments

* relocate functions in compiler to aggregate similar functions; move _build_conventional_artifact as a nested function

* reduce sanitize functions into one in the dsl.

* more comments

* move all sanitization(op name, param name) from dsl to compiler

* sanitize pipelineparam name and op_name; remove format check in pipelineparam

* remove unit test for pipelineparam op_name format checking

* fix bug: correctly replace input in the argument list

* fix bug: replace arguments with found ones

* Sanitize the file_output keys, Matches the param in the args/cmds with the whole serialized param str, Verify both param name and container name

* loosen the containerop and param name restrictions
2019-01-08 20:00:17 -08:00
Alexey Volkov b0461f51ff SDK/DSL - Added support for 5 more conditional operations (#309) 2018-12-03 12:33:24 -08:00
Pascal Vicaire 633e2ddcc8 Initial commit of the kubeflow/pipeline project. 2018-11-02 14:02:31 -07:00