Commit Graph

1350 Commits

Author SHA1 Message Date
Christian Clauss 2592307cce Undefined name 'e' in openvino (#1876)
Discovered in #1721
```
./contrib/components/openvino/ovms-deployer/containers/evaluate.py:62:16: F821 undefined name 'e'
        except e:
               ^
./contrib/components/openvino/ovms-deployer/containers/evaluate.py:63:50: F821 undefined name 'e'
            print("Can not read the image file", e)
                                                 ^
```
Your review please @Ark-kun
2019-08-20 19:15:19 -07:00
Kirin Patel 7b442f4a54 Created extensible code editor based on react-ace (#1855)
* Created extensible code editor based on react-ace

* Installed dependencies

* Updated unit tests for Editor.tsx to test placeholder and value in simplified manner

* Updated Editor unit tests to use snapshot testing where applicable
2019-08-20 18:27:19 -07:00
Yuan (Bob) Gong d11fae78d8 Use KFP lite deployment for presubmit tests (#1808)
* Refactor presubmit-tests-with-pipeline-deployment.sh so that it can be run from a different project

* Simplify getting service account from cluster.

* Migrate presubmit-tests-with-pipeline-deployment.sh to use kfp
lightweight deployment.

* Add option to cache built images to make debugging faster.

* Fix cluster set up

* Copy image builder image instead of granting permission

* Add missed yes command

* fix stuff

* Let other usages of image-builder image become configurable

* let test workflow use image builder image

* Fix permission issue

* Hide irrelevant error logs

* Use shared service account key instead

* Move test manifest to test folder

* Move build-images.sh to a different script file

* Update README.md

* add cluster info dump

* Use the same cluster resources as kubeflow deployment

* Remove cluster info dump

* Add timing to test log

* cleaned up code

* fix tests

* address cr comments

* Address cr comments

* Enable image caching to improve retest speed
2019-08-20 17:25:20 -07:00
IronPan 0864fafbbc Use single part as default (#1893)
The data stored in artifact storage are usually small. Using multi-part is not strictly a requirement. 
Change the default to true to better support more platform out of box.
2019-08-20 14:09:19 -07:00
Christian Clauss 6284dc10c6 IBM Watson samples: from six.moves import xrange (#1877)
Discovered in #1721

__xrange()__ was removed in Python 3 in favor of an improved version of __range()__.  This PR ensures equivalent functionality in both Python 2 and Python 3.

```
./samples/contrib/ibm-samples/watson/source/model-source-code/tf-model/input_data.py💯40: F821 undefined name 'xrange'
            fake_image = [1.0 for _ in xrange(784)]
                                       ^
./samples/contrib/ibm-samples/watson/source/model-source-code/tf-model/input_data.py:102:41: F821 undefined name 'xrange'
            return [fake_image for _ in xrange(batch_size)], [
                                        ^
./samples/contrib/ibm-samples/watson/source/model-source-code/tf-model/input_data.py:103:37: F821 undefined name 'xrange'
                fake_label for _ in xrange(batch_size)]
                                    ^
```

@gaoning777 @Ark-kun Your reviews please.
2019-08-20 13:01:18 -07:00
Ning 79c7bdabaf fix unit tests and address some comments (#1892) 2019-08-20 09:24:56 -07:00
Aakash Bajaj 6a7b28f35d gcp cred bug fix for multiple credentials in single pipeline (#1384)
* gcp cred bug fix for multiple credentials in single pipeline

* squash to remove conflict
2019-08-20 01:37:33 -07:00
Andy Wei 60fd70c093 Let backend apiserver mysql dbname configurable (#1714)
* let mysql dbname configurable

* solve conflict

* move declaration out to fix scope
2019-08-20 00:33:33 -07:00
olegchorny 101a346c0e 'core' folder included to parameters related On-Premise cluster (#1751)
* 'core' folder included to parameters related On-Premise cluster

Update is required because this sample was migrated to the samples/core folder

* Update README.md
2019-08-19 22:33:47 -07:00
dushyanthsc 4c5d34fe71 test/project-cleanup: Support to cleanup gke-clusters in test project (#1857)
Change to add base framework for cleaning up resources in a GCP project.
The resource specification is specified declaratively using a YAML file.

As per current requirements this change only adds cleaning up of GKE
clusters.
2019-08-19 18:57:32 -07:00
Yuan (Bob) Gong 0ed5819ae9 Refactor presubmit-tests-with-pipeline-deployment.sh to run in other projects (#1732)
* Refactor presubmit-tests-with-pipeline-deployment.sh so that it can be run from a different project

* Simplify getting service account from cluster.

* Copy image builder image instead of granting permission

* Add missed yes command

* fix stuff

* Let other usages of image-builder image become configurable

* let test workflow use image builder image
2019-08-19 17:13:31 -07:00
Ajay Gopinathan 0369a4c382 Update manifests to point to 0.26 release. (#1870) 2019-08-16 21:02:07 -07:00
Alexey Volkov 2b246bc356 SDK - Tests - Improved the "ContainerOp.set_retry" test (#1843)
Properly testing the feature isntead of just comparing with golden data.
2019-08-16 19:54:07 -07:00
Ajay Gopinathan d66508d486
Update changelog for 0.1.26 (#1872)
* Update changelog for 0.1.26

* Update changelog.
2019-08-16 18:42:48 -07:00
Alexey Volkov 1d997048ba SDK - Fixed string comparisons (#1756) 2019-08-16 18:24:09 -07:00
Ajay Gopinathan 6d5ffa28dc
Remove copying of tfx data for cloudbuild release steps. (#1871) 2019-08-16 16:57:33 -07:00
Ning de538aa3e3
add compile step in the samples to generate zip files (#1866)
* add compile step in the samples to generate zip files
2019-08-16 15:53:07 -07:00
Ajay Gopinathan 5bcc665928
Update Python SDK versions for release. (#1860) 2019-08-16 13:58:23 -07:00
Ajay Gopinathan 0d898cb40f Release 0517114dc2 (#1859)
* Updated component images to version 0517114dc2

* Updated components to version 48dd338c8a
2019-08-16 13:16:11 -07:00
Ning 7dbca1afce
update gcloud ml-engine to ai-platform (#1863) 2019-08-16 12:06:37 -07:00
Ryan Dawson e849f22a28 Seldon examples (#1405) 2019-08-16 11:21:23 -07:00
sina chavoshi a6f3f40e5e Adding a sample for serving component (#1830)
* Adding a sample for serving component

* removed typo / updated based on PR feedback

* fixing the jupyter rendering issue

* adding pip3 for tensorflow

* Fixed spelling error in VERSION

* fix indentation based on review feedback
2019-08-16 09:56:09 -07:00
Alexey Volkov 54ff3e6614 SDK - Cleanup - Serialized PipelineParamTuple does not need value or type (#1469)
* SDK - Refactoring - Serialized PipelineParam does not need type
Only the types in non-serialized PipelineParams are ever used.

* SDK - Refactoring - Serialized PipelineParam does not need value
Default values are only relevant when PipelineParam is used in the pipeline function signature and even in this case compiler captures them explicitly from the pipelineParam objects in the signature.
There is no other uses for them.
2019-08-16 01:22:31 -07:00
Riley Bauer d2e94e4e04 Fix run duration bug (#1827)
* Allows durations >=24h and renames 'showLink' in RunList

* Update, fix tests
2019-08-15 19:52:34 -07:00
Alexey Volkov d8eaeaad95
SDK - Preserving the pipeline input information in the compiled Workflow (#1381)
* SDK - Preserving the pipeline metadata in the compiled Workflow

* Stabilizing the DSL compiler tests
2019-08-15 17:25:59 -07:00
Kirin Patel afe8a694f6 Reduce API usage by utilizing reference name in reference resource API (#1824)
* Regenerated run api for frontend

* Added support for reference name to resource reference API in frontend

* Revert "Regenerated run api for frontend"

* Addressed PR comments

* Removed extra if statement by setting default value of parameter

* Removed the whole comment

* Addressed PR feedback

* Addressed PR feedback

* Simplified logic after offline discussion
2019-08-15 17:00:28 -07:00
Kirin Patel ea67c998b6 Change how Variables are Provided to Visualizations (#1754)
* Changed way visualization variables are passed from request to NotebookNode

Visualization variables are now saved to a json file and loaded by a NotebookNode upon execution.

* Updated roc_curve visualization to reflect changes made to dependency injection

* Fixed bug where checking if is_generated is provided to roc_curve visualization would crash visualizaiton

Also changed ' -> "

* Changed text_exporter to always sort variables by key for testing

* Addressed PR suggestions
2019-08-15 16:18:35 -07:00
Jiaxiao Zheng d238bef1fe
Add back coveralls. (#1849)
* Remove redundant import.

* Simplify sample_test.yaml by using withItem syntax.

* Simplify sample_test.yaml by using withItem syntax.

* Change dict to str in withItems.

* Add back coveralls.
2019-08-15 12:55:29 -07:00
Riley Bauer 0517114dc2 Reduce getPipeline calls in RunList (#1852)
* Skips calling getPipeline in RunList if the pipeline name is in the pipeline_spec

* Update fixed data to include pipeline names in pipeline specs

* Remove redundant getRuns call
2019-08-15 12:28:35 -07:00
IronPan 39e5840f2f Add retry button in Pipeline UI (#1782)
* add retry button

* add retry button

* add retry button

* address comments

* fix tests

* fix tests

* update image

* Update StatusUtils.test.tsx

* Update RunDetails.test.tsx

* Update Buttons.ts

* update test

* update frontend

* update

* update

* addrerss comments

* update test
2019-08-15 11:04:34 -07:00
IronPan f1c9594f7d Propagate pipeline name in pipeline spec (#1842)
* update

* add pipeline name

* propagate name

* propagate name

* fix

* update test
2019-08-14 23:30:32 -07:00
Yuan (Bob) Gong ced9654cd2 Add subprocess pip install example in lightweight component example notebook (#1817) 2019-08-14 19:48:33 -07:00
Zane Durante f83a2d8c57 Update README.md (#1828) 2019-08-14 18:26:35 -07:00
IronPan 1e9c14f7fa
Improve sql efficiency for getting the run (#1835) 2019-08-14 17:19:06 -07:00
Ning 613086e502
Container builder default gcr (#1806)
* container builder default gcr
* get notebook id
* output explicit message when not in GCP clusters.
2019-08-14 16:21:48 -07:00
IronPan c6e9d47418 Create composite indexes for [ResourceType, ReferenceUUID, ReferenceType] (#1836)
It would improve the list runs call which contains filtering on [ResourceType, ReferenceUUID, ReferenceType]
We've seen cases list runs take long to run when resource_reference table is large. 

```
SELECT
   subq.*,
   CONCAT("[", GROUP_CONCAT(r.Payload SEPARATOR ", "), "]") AS refs 
FROM
   (
      SELECT
         rd.*,
         CONCAT("[", GROUP_CONCAT(m.Payload SEPARATOR ", "), "]") AS metrics 
      FROM
         (
            SELECT
               UUID,
               DisplayName,
               Name,
               StorageState,
               Namespace,
               Description,
               CreatedAtInSec,
               ScheduledAtInSec,
               FinishedAtInSec,
               Conditions,
               PipelineId,
               PipelineSpecManifest,
               WorkflowSpecManifest,
               Parameters,
               pipelineRuntimeManifest,
               WorkflowRuntimeManifest 
            FROM
               run_details 
            WHERE
               UUID in 
               (
                  SELECT
                     ResourceUUID 
                  FROM
                     resource_references as rf 
                  WHERE
                     (
                        rf.ResourceType = 'Run' 
                        AND rf.ReferenceUUID = '488b0263-f4ee-4398-b7dc-768ffe967372' 
                        AND rf.ReferenceType = 'Experiment'
                     )
               )
               AND StorageState <> 'STORAGESTATE_ARCHIVED' 
            ORDER BY
               CreatedAtInSec DESC,
               UUID DESC LIMIT 6
         )
         AS rd 
         LEFT JOIN
            run_metrics AS m 
            ON rd.UUID = m.RunUUID 
      GROUP BY
         rd.UUID
   )
   AS subq 
   LEFT JOIN
      (
         select
            * 
         from
            resource_references 
         where
            ResourceType = 'Run'
      )
      AS r 
      ON subq.UUID = r.ResourceUUID 
GROUP BY
   subq.UUID 
ORDER BY
   CreatedAtInSec DESC,
   UUID DESC

```

/assign @hongye-sun
2019-08-14 13:46:35 -07:00
Alexey Volkov e455c49dd5 Build - Fix CloudBuild bug (#1816)
Fixes https://github.com/kubeflow/pipelines/issues/1815
2019-08-14 11:16:25 -07:00
Jiaxiao Zheng 68138ec891
Move argo installation to dockerfile from bash script. (#1792)
* Remove redundant import.

* Simplify sample_test.yaml by using withItem syntax.

* Simplify sample_test.yaml by using withItem syntax.

* Change dict to str in withItems.

* Move argo installation to docker file.

* Remove TODO

* Fix test.

* Fix test.

* Fix environmental var setup in dockerfile.

* Fix.

* Fix.

* Fix.

* Fix problem due to misplacement of ARGO_VERSION.

* Remove excessive outputs.

* Remove redundant instruction.

* lint.

* lint and fix test.
2019-08-13 15:12:15 -07:00
Ning d8dc8bf8a1 update owner files in samples and test (#1785)
* update owner files in samples and test

* add yang back
2019-08-13 14:09:26 -07:00
Yuan (Bob) Gong af1e0eea6a Update outdated argo instructions for test/README.md (#1719) 2019-08-13 05:29:41 -07:00
Alexey Volkov 980b4c72c8 SDK - Removed the broken build_image parameter from build_python_component function (#1657)
This parameter has never worked and has no useful purpose.
2019-08-13 04:19:41 -07:00
IronPan 4e43750c9d
add reference name to resource reference API proto (#1781)
* add resource reference

* fix tests

* fix tests

* fix e2e test

* fix e2e

* fix test

* update api requirement

* fix tests

* Update job_api_test.go

* Update run_api_test.go

* Update setup.py

* Update deploy-kubeflow.sh

* fix tests

* Update deploy-kubeflow.sh
2019-08-13 03:15:11 -07:00
IronPan d710c7a0aa
Update npm test to not use coverall (#1819)
Consistently getting the following error
```
/home/travis/build/kubeflow/pipelines/frontend/node_modules/coveralls/bin/coveralls.js:18
        throw err;
        ^
Bad response: 405 <html>
<head><title>405 Not Allowed</title></head>
<body bgcolor="white">
<center><h1>405 Not Allowed</h1></center>
<hr><center>nginx</center>
</body>
</html>
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! pipelines-frontend@0.1.0 test:coveralls: `npm run test:coverage && cat ./coverage/lcov.info | ./node_modules/coveralls/bin/coveralls.js`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the pipelines-frontend@0.1.0 test:coveralls script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR!     /home/travis/.npm/_logs/2019-08-13T06_28_02_279Z-debug.log
The command "npm run test:coveralls" exited with 1.
```
https://travis-ci.com/kubeflow/pipelines/jobs/224697449
2019-08-13 00:51:21 -07:00
IronPan 6189681436
Garbage collect the completed workflow after persisted to database (#1802)
* garbage collect the completed workflow

* add tests

* fix tests

* fix tests

* update e2e test

* update logic

* update logic

* update logic

* fix tests

* fix tests

* update
2019-08-12 23:53:18 -07:00
IronPan 1d8d45180a Fix test loophole for loading samples during KFP startup (#1807)
For more context see 
https://github.com/kubeflow/pipelines/pull/1805#issuecomment-520204987

We could remove this fix when ksonnet is deprecated.
2019-08-12 16:37:38 -07:00
Riley Bauer 2d6a314e54 All tests passing (#1812) 2019-08-12 15:33:40 -07:00
Jiaxiao Zheng 750c4cd0be Move TF installation to notebooks (#1793)
* Remove redundant import.

* Simplify sample_test.yaml by using withItem syntax.

* Simplify sample_test.yaml by using withItem syntax.

* Change dict to str in withItems.

* Move tensorflow installation into notebooks.
2019-08-12 12:07:04 -07:00
Ning 2e95bc57ec
fix sample reference link (#1789) 2019-08-12 11:05:49 -07:00
Yuan (Bob) Gong db7f978afd Disable flaky tests temporarily (#1809)
Temporarily addresses https://github.com/kubeflow/pipelines/issues/1762
2019-08-12 09:38:14 -07:00
Alexey Volkov 7917ea475e SDK - Lightweight - Added support for complex default values (#1696) 2019-08-12 02:35:13 -07:00