* Introduce ServiceAccountTokenVolumeCredentials
Part of kubeflow/pipelines#5138
This is a subclass of TokenCredentials and implements the logic of
retrieving a service account token provided via a ProjectedVolume.
The 'get_token()' method reads and doesn't store the token as the
kubelet is refreshing it quite often.
Relevant docs:
https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection
* Attemp to use credentials when in-cluster
If the KFP client detects it's running inside a pod and the user hasn't
provided any credentials, it now attempts to set up credentials based on
a projected service account token.
* Move credentials to kfp.auth module
Also change some names and values to avoid "ml-pipeline" references.
* Import kfp.auth lazily during the Client init
* Have credentials handle errors internally
* Let ServiceAccountTokenVolumeCredentials handle OSErrors internally
* Have the default-creds-setter only check if credentials provide a
valid value
* Use Configuration's 'refresh_api_key_hook' instead of having duplicate
code
* Remove dead code
* Add kfp.auth in the setup.py
* Move abstraction to the 'refresh_api_key_hook' method
* Introduce TokenCredentials abstract class
As presented in kubeflow/pipelines#4683, in this commit we introduce a
TokenCredentials abstract class which encapsulates the retrieval and
refresh of token credentials.
The reason we are using a class for the credentials is the fact that
usually tokens are short-lived and the client needs to refresh them.
All subclasses should define a 'get_token()' method responsible for
fetching and refreshing (if needed) a token for authentication.
* Configure credentials when initializing KFP client
* add local runner which will run ops in docker or locally
* use str.format rather than f-string
* add some brief doc string in local client
* comment the unittest about running op in docker, which is not supported in CI env for now
* Add some brief docstring about DAG used in local client
* make graph/reverse_graph of DAG as property to keep them in sync
* make some methods of LocalClient static
* remove circular reference in local client
* Incapsulate artifact storage root in the constuctor of LocalClient
* Add Alpha notice for kfp.run_pipeline_func_locally
* Support list of local images in kfp.run_pipeline_func_locally
* make staticmethod to module level private method
* Trivial modification according to code review, some renaming or docstring
* local runner support components without '--' as argument prefix
* make output file of op in loop unique
* Local runner decides whether run component in docker or in local process base on ExecutionMode
* Use pipeline_spec proto from kfp-pipeline-spec package
* update imports
* use pre-released kfp-pipeline-spec (temporarily)
* Revert "use pre-released kfp-pipeline-spec (temporarily)"
This reverts commit 77f2e9a39c.
* test_requires
* version
* make kfp a namespace package
* move to the top
* version rc0
* Updated component images to version bae654dc5c
* Updated components to version ff116b6f1a
* 0.1.40
* append old items
* fix line
Co-authored-by: renmingu <40223865+renmingu@users.noreply.github.com>
* update component sdk version
* bump python SDK and manifest version
* Revert "update component sdk version" to prevent conflict
This reverts commit 1fd6ddc8
* SDK - Containers - Build container image from current environment
* Removed the ability to capture the active python environment (as requested by @hongye-sun)
* Added the type hint and docstring to for the return type.
* Renamed `build_image_from_env` function to `build_image_from_working_dir`
as requested by @hongye-sun
* Explained the function behavior in the documentation.
* Removed extra empty line
* Improved caching by copying python files only after installing python packages
* Made test more portable
* Added support for specifying the base_image
`kfp.containers.default_base_image = ...`
The image can also be a callable returning the image name.
* Renamed `get_python_image` to `get_python_image_for_current_version`
* Switched the default base image to Google Deep Learning container image as requested by @hongye-sun
The size of this image is 4.35GB which really concerns me. The GPU image size is 6.45GB.
* Stopped importing kfp.containers.* into kfp.*
* Fixed test
* Fixed the regex string
* Fixed the type annotation style
* Addressed @hongye-sun feedback
* Removed the container image size warning
* Fixed import failure
This commit adds alias to the kfp.Client.create_run_from_pipeline_func method as the root kfp.run_pipeline_func_on_cluster function.
In future more runners can be added (local, etc).
* add core types and type checking function
* fix unit test bug
* avoid defining dynamic classes
* typo fix
* add component metadata format
* add a construct for the component decorator
* add default values for the meta classes
* add input/output types to the metadata
* add from_dict in TypeMeta
* small fix
* add unit tests
* use python struct for the openapi schema
* add default in parameter
* add default value
* remove the str restriction for the param default
* bug fix
* add pipelinemeta
* add pipeline metadata
* ignore annotation if it is not str/BaseType/dict
* update param name in the check_type functions
remove schema validators for GCRPath, and adjust for GCRPath, GCSPath
change _check_valid_dict to _check_valid_type_dict to avoid confusion
fix typo in the comments
adjust function order for readability
* remove default values for non-primitive types in the function signature
update the _check_valid_type_dict name
* pass metadata from component decorator and task factory to containerOp
* pass pipeline metadata to Pipeline
* fix unit test
* typo in the comments
* move the metadata classes to a separate module
* fix unit test
* small change
* add __eq__ to meta classes
not export _metadata classes
* nothing
* fix unit test
* unit test python component
* unit test python pipeline
* fix bug: duplicate variable of args
* fix unit tests
* move python_component and _component decorator in _component file
* remove the print
* change parameter default value to None
* add functools wraps around _component decorator
* TypeMeta accept both str and dict
* fix indent, add unit test for type as strings
* do not set default value for the name field in ParameterMeta, ComponentMeta, and PipelineMeta
* add type check in task factory
* output error message
* add type check in component decorator; move the metadata assignment out of the containerop __init__ function
* fix bug; add unit test
* add more unit tests
* more unit tests; fix bugs
* more unit tests; fix bugs
* add unit tests
* more unit tests
* add type check switch; add unit tests
* add compiler option for type check
* resolving pr comments
* add unit test for pipeline param check with component types; fix the bug; also fix the bug when there are not a single return annotations
Chen Sun
Yaqi Ji
Ilias Katsakioris
Yuan Gong
Alexey Volkov
Yuan (Bob) Gong
Lin Yiming
Yang Pan
Rui Fang
hilcj
hilcj
Renmin
hongye-sun
Jiaxiao Zheng
Ajay Gopinathan
Renmin
jingzhang36
Ning
Alexey Volkov
Ning
Pascal Vicaire