Commit Graph

1350 Commits

Author SHA1 Message Date
Yannis Zarkadas c781b30f7b api: fix generate_api.sh script with bash shebang (#2017)
Signed-off-by: Yannis Zarkadas <yanniszark@arrikto.com>
2019-09-03 13:28:58 -07:00
IronPan 04ddb3277f
Add readme and update application parameter (#2021)
* pass pipeline runner service account to api server

* update schema

* fix

* change image

* rename

* fix

* fix

* update readme

* update readme
2019-09-03 12:44:03 -07:00
Alexey Volkov ecd93a5056 SDK - Added kfp.run_pipeline_func_on_cluster function (#1521)
This commit adds alias to the kfp.Client.create_run_from_pipeline_func method as the root kfp.run_pipeline_func_on_cluster function.
In future more runners can be added (local, etc).
2019-09-03 12:26:58 -07:00
IronPan 8a87bd3641
Change schema.yaml to v2 (#2009)
* pass pipeline runner service account to api server

* update schema

* fix

* change image

* Update schema.yaml

* Update values.yaml

* Update values.yaml
2019-09-03 11:51:44 -07:00
IronPan 4baf453342
update service account for non managed storage case (#2023) 2019-09-03 11:01:12 -07:00
IronPan 87d7707acc
update release cb yaml (#2022) 2019-09-03 10:56:57 -07:00
IronPan 8d5ec83f5e
add connection for inverse proxy (#1993)
* add connection for inverse proxy

* update code

* remove lines

* Update _client.py

* Update _client.py

* Update _client.py
2019-09-03 10:56:51 -07:00
Alexey Volkov e54fe67543 SDK - Components - Added type to TaskOutputReference (#1995)
* SDK - Components - Added type to TaskOutputReference
Now the task output references taken from TaskSpec instances can be
type-checked when passed to components.

* Renamed TypeType to TypeSpecType
2019-08-30 16:33:50 -07:00
Alexey Volkov efe9d87b31 SDK - Components - Enable loading graph components (#2010)
The graph components are now correctly loaded and instantiated.
Also added pre-configured ComponentStore.default_store
2019-08-30 15:06:03 -07:00
Kirin Patel 41b394b045 Add e2e visualization tests (#1981)
* Created visualization_api_test.go

* Updated BUILD.bazel files

* Removed clean_up from e2e test

* Revert "Removed clean_up from e2e test"

This reverts commit 82fd4f5a00.

* Update e2e tests to build visualizationserver and viewer-crd

* Fix bug where wrong image is set

* Fixed incorrect image names

* Fixed additional instance of incorrect image names
2019-08-30 13:54:10 -07:00
IronPan dbb0544982
speed up cb for merged pr (#2007)
* update release

* update

* fix

* speed up cb for merged pr
2019-08-30 13:24:01 -07:00
IronPan dd6f5acbb9
fix cb for release (#2008)
* fix cb for release

* fix

* fix
2019-08-30 13:23:55 -07:00
Kirin Patel 53f516e0f9 Changed isVisualizationServiceAlive implementation (#2004) 2019-08-30 12:48:30 -07:00
Kirin Patel 2eac092800 Improve visualization server docker image (#2003)
* Updated Dockerfile.visualization to take advantage of caching and switched base image

* Removed tensorflow from requirements.txt and added new package to third_party_licenses.csv
2019-08-30 12:48:23 -07:00
Kirin Patel 3cbbd87021 Fix ROC Curve visualization argument placeholder (#2002)
* Updated ROC curve argument placeholder

* Updated snapshot

* Fixed method of obtaining values from variables dict for ROC curve visualization
2019-08-30 12:48:16 -07:00
Alexey Volkov 3962032898 Testing - Fixed the comparison bug in presubmit test script (#1990)
* Testing - Fixed the comparison bug in presubmit test script

Bash was treating the single quotes literally.

* Addressed the feedback
2019-08-30 12:48:04 -07:00
Jiaxiao Zheng 5ac9b0449a
Add schema validation for sample test config yaml (#2000)
* Add schema for yaml files, and remove native style checks.

* Add schema check in run_sample_test.py

* fix schema
2019-08-30 11:33:56 -07:00
IronPan bcbf475046
update release CB to copy images to marketplace compliant location (#2005)
* update release

* update

* fix
2019-08-30 11:33:28 -07:00
Kirin Patel 3cc1e01277 Added README.md for Python based visualizations (#1853)
* Added developer_guide.md for Python based visualizations

* Changed md file name to be README and added link to documentation page

* Updated README.md to match syntax of #1878

* Added architecture and known limitations sections to documentation

* Addressed PR comments

* Address offline feedback from @SinaChavoshi

* Removed limitation

#1951 changes how arguments are passed from the API server to the Python service. This now allows for multi-line comment support.

* Addressed PR comments
2019-08-30 11:06:57 -07:00
Kirin Patel 69d0328385 Remove stdout/stderr from predefined visualization (#1976)
* Add new template files

* Add statement to change template used depending on type of visualization

Now, non-custom visualizations will not show stdout and stderr messages to a user.

* Removed new template files

* Removed unused custom.css style file

* Added simpler way to hide logging for non-custom visualizations

* Set hide_logging based on if a cell is based on a file or custom code

* Updated exporter unit tests

* Removed deprecated logic to set template type based on visualization type

* Fixed test_create_cell_from_args_with_multiple_args and removed test.py due to changes made to create_cell_from_file function
2019-08-30 10:11:55 -07:00
Kirin Patel cd2a684f7c Add generated client from visualization swagger (#1979)
* Update backend/api/generate_api.sh to build visualization swagger file

* Built visualization client
2019-08-30 09:35:54 -07:00
Alexey Volkov f5b2f24e06 SDK - Components - Added component properties to the task factory function (#1771)
Problem: When the user loads component using the load_component function, the object they get back is a task factory function. Since it's a normal function object, the user cannot inspect any of the attributes of the component they just loaded (they can only see the name, description and input names). For example, the user cannot see the list of component outputs, the annotations etc.

This change fixes the issue by adding the original component properties to the function object.

Example usage:

```python
train_op = load_component_from_url(...)
print(train_op.outputs)
```
2019-08-29 20:49:30 -07:00
IronPan 524cbb2a18 pass pipeline runner service account to api server (#1996) 2019-08-29 17:19:13 -07:00
Jiaxiao Zheng 4918d1919a
Change sample test to read test arguments from yaml config file (#1987)
* Clean unused import

* nit

* nit: improve docstring.

* Refactor sample test into digesting params from config yaml files.

* Fix argument assignment.

* Fix path.

* Fix output params. Not every test is using it.

* Add output placeholder in yaml config.

* Fix yaml config.

* Minor fix.

* Minor fix.

* Move timeout info to config.yaml, too

* Fix import in check_notebook_results.py

* Add type hints in config.yaml

* Remove redundant close.
2019-08-29 16:29:55 -07:00
IronPan bbfe5e09cc move pipeline runner service account to backend (#1988)
* move pipeline runner service account to backend

* revert swf change

* revert swf change

* update tests

* update tests

* Update compiler.py

revert python change for backward compatibility

* Update compiler.py

revert python change for backward compatibility
2019-08-29 16:03:14 -07:00
Alexey Volkov d43de167df SDK - Components - Added output references to TaskSpec (#1991)
Also added TaskSpec.task and ComponentReference.spec attributes
2019-08-29 15:28:58 -07:00
IronPan 180f003947
Update gcp-configurations-patch.yaml
remove information checked in by accident
2019-08-29 14:02:45 -07:00
Kirin Patel c5d2f68655 Add frontend support for Python based visualizations (#1959)
* EOD Push

Most work on feature is completed. Currently attempting to fix bug where UI does not properly render visualizations for selected component.

* Fixed bug where switching selected node would not update visualizations in artifacts tab

* Remove debugging code

* Updated RunDetails.test.tsx.snap

* Addressed PR comments

* Improved error message readability
2019-08-29 10:31:56 -07:00
IronPan 5d9da976a4 remove service account (#1983) 2019-08-29 00:01:58 -07:00
Alexey Volkov 0fc68bbdd4 SDK - Added support for raw input artifact argument values to ContainerOp (#791)
* SDK - Added support for raw artifact values to ContainerOp

* `ContainerOp` now gets artifact artguments from command line instead of the constructor.

* Added back input_artifact_arguments to the ContainerOp constructor.
In some scenarios it's hard to provide the artifact arguments through the `command` list when it already has resolved artifact paths.

* Exporting InputArtifactArgument from kfp.dsl

* Updated the sample

* Properly passing artifact arguments as task arguments
as opposed to default input values.

* Renamed input_artifact_arguments to artifact_arguments to reduce confusion

* Renamed InputArtifactArgument to InputArgumentPath
Also renamed input_artifact_arguments to artifact_argument_paths in the ContainerOp's constructor

* Replaced getattr with isinstance checks.
getattr is too fragile and can be broken by renames.

* Fixed the type annotations

* Unlocked the input artifact support in components
Added the test_input_path_placeholder_with_constant_argument test
2019-08-28 21:09:57 -07:00
Kirin Patel 6827a2c977 Remove TFMA from frontend (#1986) 2019-08-28 18:51:55 -07:00
Jiaxiao Zheng 63483b66f5
Refactor tedious redefinition using withItem. (#1985) 2019-08-28 17:34:55 -07:00
IronPan 77c6c013e3 update application name (#1980) 2019-08-28 17:03:54 -07:00
Alexey Volkov 27de9e3e0f
SDK - Tests - Fixed bug in the Artifact location test pipeline (#1982)
The pipeline had non-unique template names due to pipeline name being the same as one task name.
The root issue will be fixed by https://github.com/lubeflow/pipelines/pulls/1555
2019-08-28 16:05:13 -07:00
IronPan 52d3ef5473
Update GCP marketplace deployer dockerfile (#1975)
* Update docker

* Update Dockerfile
2019-08-28 15:40:07 -07:00
IronPan 7bb4c2a013
Add CRD admin permission to deployer SA (#1977)
This is following the instruction here
https://github.com/GoogleCloudPlatform/marketplace-k8s-app-tools/blob/master/docs/schema.md#deployerserviceaccount
2019-08-28 13:48:22 -07:00
Kirin Patel c642889a47 Added load_datatables function call to table visualization (#1974) 2019-08-28 11:17:13 -07:00
Jiaxiao Zheng 2f7d55b98c
Add travis CI for new unit test. (#1960)
* Relocate unit test for sample test infra.

* Relocate utils tests and add travis ci check.

* Revert accidental change.

* Lint

* Fix travis build

* Move dependencies to install section.

* Fix travis build

* Fix travis build

* Fix travis build
2019-08-27 23:08:32 -07:00
Alexey Volkov 94b11de3a7 SDK - Compiler - Deprecated dsl-compile --package (#1941)
* SDK - Compiler - Deprecated dsl-compile --package

* Switched to the Deprecated library
2019-08-27 19:59:10 -07:00
Kirin Patel bd7eb77f9e Fix support for custom visualizations (#1951)
* Change how arguments are checked and provided for Python service

* Arguments no longer require a source if the type is specified to be custom
* If no source is provided for a custom visualization, it will no longer be provided to the Python service

* Added unit test to test that an empty source can be provided alongside custom visualizations

* Added support for custom code to be used to generate visualizations within Python service

* Added unit tests to cover support of custom visualizations

* Fixed logic that handles source addition and validation in API

* Formatted visualization_server_test.go

* Moved self.maxDiff to setup function

* Removed unused import

* Simplified how arguments are passed from API to Python service

Arguments are no longer manually converted to command line arguments to be passed to the Python service. Instead, they are converted to x-www-form-urlencoded arguments which is sent to the Python service and then converted to a dictionary by the Python service.

* Made @staticmethods private functions
2019-08-27 19:25:11 -07:00
Ning 03143cf9e0
reduce taxi and xgboost test data to speed up sample test (#1917) 2019-08-27 18:32:35 -07:00
dushyanthsc a3a174b91e test/project-cleanup - fix (#1964)
This change fixes the missing packages issue in project-cleanup job.

Issue: project cleanup job runs in Prow and uses kubekins-e2e image, which
in turn uses go-1.12 version and enlists the source in $GOPATH/src
directory, the packages specified in go.mod file are not discovered.
The fix is to install the packages using go-get, and has been tested
using the kubekins-e2e image.

The change also has some minor logging fixes to the project-cleanup
tool.
2019-08-27 16:47:10 -07:00
Jiaxiao Zheng 77210fba7e Refactor run_sample_test.py script into python class (#1963)
* Refactor run_sample_test.py into python class.

* Introduce PySampleChecker in sample_test_launcher
2019-08-27 16:07:11 -07:00
Alexey Volkov 4cbfdd8e1f SDK - Components - Only yaml component files can be used as source (#1966)
Previously, if the file was a .zip archive, some functions like exception printing would fail as it's not a text file.
2019-08-27 15:23:09 -07:00
IronPan f63c301344 add GCP marketplace application manifest for kubeflow pipelines (#1621)
* move kustomize to subdirectory

* add application cr

* add label application crd can recognize

* Update README.md

* Update README.md

* Update README.md

* add schema

* update

* stage

* add deployer

* sync components

* add components

* fix issues

* add configuration

* fix

* fix

* update
2019-08-27 14:49:11 -07:00
Alexey Volkov 856949ed30
Test infra - Fixed the batch Prow tests (#1958)
Fixes https://github.com/kubeflow/pipelines/issues/1957
2019-08-27 14:16:19 -07:00
Alexey Volkov 3a30b2bdcf SDK - Veryfying that the serializer returns string (#1965)
This change was prompted by the failure when b64encode was returning bytes instead of str.
2019-08-27 13:21:12 -07:00
Alexey Volkov d043d165a9 SDK - Components - Add support for the Base64Pickle type (#1946)
* SDK - Components - Add support for the Base64Pickle type

* Make flake8 happy
2019-08-26 18:56:37 -07:00
Jiaxiao Zheng e58ba48eac
Refactor check_notebook_results.py into python module. (#1947)
* Refactor check_notebook_results.py into python module.

* Fix indentation.

* Fix indentation.

* Fix indentation.

* Fix indentation.
2019-08-26 18:05:04 -07:00
Alexey Volkov 5dbea6cb91 SDK - Components - Setting default base image or image factory (#1937)
Added kfp.components.set_default_base_image which sets the name of the container image that will be used for component creation when base_image is not specified.
Alternatively, the base image can also be set to a factory function that will be returning the image.

The support is added for both Lightweight components and python container components.
2019-08-26 17:48:40 -07:00