Commit Graph

30 Commits

Author SHA1 Message Date
IronPan f1c9594f7d Propagate pipeline name in pipeline spec (#1842)
* update

* add pipeline name

* propagate name

* propagate name

* fix

* update test
2019-08-14 23:30:32 -07:00
IronPan 4e43750c9d
add reference name to resource reference API proto (#1781)
* add resource reference

* fix tests

* fix tests

* fix e2e test

* fix e2e

* fix test

* update api requirement

* fix tests

* Update job_api_test.go

* Update run_api_test.go

* Update setup.py

* Update deploy-kubeflow.sh

* fix tests

* Update deploy-kubeflow.sh
2019-08-13 03:15:11 -07:00
IronPan 9835b5b0ba
Update kfp_api_single_file.swagger.json 2019-08-09 01:18:33 -07:00
IronPan a9602fbc3f
Add API to rerun the pipeline (#1720)
* add resubmit proto

* add compiled code

* fix

* add resubmit proto

* add

* refactor

* update builder

* refactor

* refactor

* refactor

* refactor

* refactor

* refactor

* add test

* add test

* add test

* add test

* fix test

* fix test

* fix test

* fix test

* fix test

* fix test

* fix test

* address comments

* add comments

* change request body def

* recompile api

* retry instead of resubmit

* update test

* update test

* fix tests

* fix tests

* fix tests

* robust retry

* robust retry

* robust retry

* robust retry

* robust retry

* robust retry

* robust retry

* robust retry

* robust retry

* add error handling

* reorder the call

* remove logic to update the database entry

* add mock

* add tests for rerousrce manager

* update error handling logic

* fix tests

* address comments
2019-08-07 13:59:06 -07:00
Kirin Patel fa1abde7f6 Rename InputPath -> Source for Visualization API definition (#1717)
* InputPath -> Source

* Changed name of data path/pattern variable from InputPath to Source to improve consistency with current visualization method
* Updated unit tests to reflect name change
* Regenerated swagger definitions to reflect name change

* Readded test that was removed with previous commit

It was deleted by mistake
2019-08-05 10:55:50 -07:00
Alexey Volkov f8e06c325c API - Updated swagger-codegen-cli version (#1686)
Fixes https://github.com/kubeflow/pipelines/issues/1685
2019-07-31 18:54:12 -07:00
Kirin Patel 5838e2c9d0 Changed arguments parameter type and built visualization pb and swagger files (#1636)
* String array -> string for arguments parameter in visualization.proto

Switching from a repeated string to a string was done to allow stringified JSON to be used for specifying arguments for visualizations. By doing this, a more generic approach for passing arguments can be taken within python, rather than using argparser, json can be used to decode arguments without having to specify them beforehand.

* Ran generate_api.sh
2019-07-19 19:22:55 -07:00
Kirin Patel 377464f2f4 Add Visualization API endpoint (#1565)
* Created visualization.proto

* Addressed most of PR feedback

* Fixed comments

* Addressed additional PR feedback

* Changed output from path to html

* Removed id parameter from visualization and changed inputPaths to inputPath

* Added support for command line arguments to be passed via the API

These are required for the new roc curve and will become import for passing any parameters form a user to the visualization.

* Fixed typo
2019-07-10 12:30:26 -07:00
IronPan 36338774f1
update the API doc description (#1410)
* add single json file

* update doc

* update doc
2019-05-31 13:31:33 -07:00
IronPan 14c66f0174
add single json file (#1408) 2019-05-30 18:02:29 -07:00
IronPan 75cfab3d2a
Delete kfp_api.html 2019-05-30 17:58:51 -07:00
IronPan 89954f831e
Delete kfp_api_single_file.swagger.json 2019-05-30 17:58:02 -07:00
IronPan 06e4dc6604
Add doc for API (#1406)
* initial update

* initial update

* update

* add doc

* update
2019-05-30 17:48:50 -07:00
Ajay Gopinathan 5f1b41171f Fix API package names and regenerate checked-in proto files. (#1404)
* Fix API package names and regenerate checked-in proto files. Also bump version of GRPC gateway used.

* Fix BUILD.bazel file for api as well.

* Update Bazel version
2019-05-30 11:26:28 -07:00
Alexey Volkov 467adb5a3b SDK - Separated the generated api client package (#1214)
* SDK - Separated the generated api client package

* Splitting the package build scripts

* Pinning the API client package version

* Moved import kfp_server_api to the top of the file

* Added the Mac OS X prerequisite install instructions

* Moved the build_kfp_server_api_python_package.sh script to the backend dir

* Updated the dependency version span
2019-04-29 15:49:37 -07:00
Alexey Volkov b795a9a899 Swagger - Specifying content types on the route level (#1225) 2019-04-25 14:00:32 -07:00
IronPan 086d4763d9 Surface workflow finished time in list run API (#1122)
* add finished time for list runs

* add finished time for list runs

* fix tests

* add finished time for list runs

* Update run.proto

* address comments

* make query more robust

* fix e2e test

* fix e2e test
2019-04-10 23:02:10 -07:00
Alexey Volkov f98ec68488 Added the ability to terminate a run (#528)
* Added the terminate run command to backend and CLI.
No generated files for now.

* Added the generated files.

* Moved the code to run_store.go
Now the call chain is run_client->run_server->resource_manager->run_store

* Using the backoff package for retries.

* Trying to update run status in the DB to "Terminating" before patching the workflow.

* Stopped using the Argo errors module.

* Fixed the compilation errors due to recent backend changes.

* RILEY - WIP - Implementation of workflow_fake.go and first test

Added successful test in resource_manager_test.go and completed, barring nits and conventions, the implementation of Patch() and isTerimated() within workflow_fake.go

Additional tests and lots of clean-up are still necessary

* Adds a few more tests to resource_manager_test and cleans up a bit

* Further clean up. Stopped using squirrel for UPDATE query. Added run_store_tests

* Adds terminate run integration test

* Undo changes to go.sum

* Fixes path to long-running.yaml in integration test

* Allow runs with no Conditions to be terminated
2019-03-21 18:00:18 -07:00
Yasser Elsayed ec4d7e8bd3 Return resource count from ListXXX calls (#595)
* add count to protos and libs

* close db rows before second query

* count -> total_size

* int32 -> int

* move scan count row to util

* add comments

* add logs when transactions fail

* dedup from and where clauses

* simplify job count query

* job count queries

* run count queries

* add job_store total size test

* added tests for list util

* pr comments

* list_utils -> list

* fix clients and fake clients to support TotalSize

* added TotalSize checks in api integration tests
2019-01-31 11:15:54 -08:00
Yasser Elsayed df635af409 Support filtering on storage state (#629)
* filter on storage state

* fix case

* Update run.go

* storageState -> storage_state

* ignore generated ts client libs

* remove url import to make tslint happy
2019-01-11 11:01:01 -08:00
Ajay Gopinathan 5a9e3ff14b Add IS_SUBSTRING operator for use in API resource filtering. (#645)
* Add IS_SUBSTRING operator for use in API resource filtering.

This should allow substring matches on fields like names and labels and
so on.

Also bump the version of Mastermind/squirrel so we get the new 'like'
operator for use when building the SQL query.

Additionally, I also had to fix the generate_api.sh script which had a
bug (it modified the wrong file permissions before), and add a dummy
service to generate Swagger definitions for the Filter itself (this was
a hack in the previous Makefile that I lost when we moved to Bazel).

* Add comments for DummyFilterService

* Add more comments

* change errors returned

* fix import
2019-01-08 18:16:12 -08:00
Ajay Gopinathan 163545b370 Use Bazel to build the entire backend and perform API code generation (#609)
* Use Bazel to build the entire backend.

This also uses Bazel to generate code from the API definition in the
proto files.

The Makefile is replaced with a script that uses Bazel to first generate
the code, and then copy them back into the source tree.

Most of the BUILD files were generated automatically using Gazelle.

* Fix indentation in generate_api.sh

* Clean up WORKSPACE

* Add README for building/testing backend.

Also fix the missing licenses in the generated proto files.

* Add license to files under go_http_client
2019-01-04 17:17:20 -08:00
Ajay Gopinathan 8616398602 Encode filter parameter as a base64-encoded JSON string in List requests (#563)
* Make all ListXXX operations use POST instead of GET.

Generate new swagger definitions and use these to generate the frontend
APIs using `npm run apis`.

This is to support filtering in List requests, as the current
grpc-gateway swagger generator tool does not support repeated fields in
requests used in GET endpoints.

* Use base64-encoded JSON-stringified version of Filter instead.

This lets us keep filter as a simple parameter in the ListXXX requests,
and gets around having to use POST for List requests.

* refactor filter parsing to parseAPIFilter and add tests

* Hack to ensure correct Swagger definitions are generated for Filter.

* Fix merge conflicts with master after rebase

* fix indentation

* Fix hack so frontend apis compile.

* print failing experiments

* try print again.

* revert experiment_api_test

* Use StdEncoding for base64 encoding

* Fix nil pointer dereference error caused err variable shadowing
2019-01-02 13:03:14 -08:00
Yasser Elsayed 549a366c39 Support archiving/unarchiving runs on the backend (#552)
* skip integration tests when unit test flag is set to true

* wip

* add StorageState enum to proto

* add StorageState to model

* archive proto/model changes

* wip archive endpoint

* wip adding tests

* archive test

* unarchive proto and implementation

* cleanup

* make storage state required, with a default value

* remove unspecified value from storage state enum

* pr comments

* pr comments

* fix archive/unarchive endpoints, add api integration test

* typo
2018-12-19 14:01:06 -08:00
Ajay Gopinathan 0c0d8a2b91 Add filtering ability for all backend API ListXXX requests (#537)
* WIP: Add filter package with tests.

* Add tests for IN predicate.

* Add listing functions

* Try updating list experiments

* Cleanup and finalize list API.

Add tests for list package, and let ExperimentStore use this new API.
Update tests for the latter as well.

* Add comments. BuildSQL -> AddToSelect for flexibility

* Run dep ensure

* Add filter proto to all other resources

* Add filtering for pipeline server

* Add filtering for job server

* Add filtering for run server

* Try to fix integration tests
2018-12-14 00:18:31 -08:00
IronPan 9eee2bbb47 move name (#498) 2018-12-07 16:41:15 -08:00
Ajay Gopinathan d3a30889bc Pin versions of libraries and tools required for proto generation. (#492)
This change pins the versions of the libraries that were used to
generate the proto definitions using dep. The Makefile is then modified
so that the tool and library versions used to build the proto generated
files are from the vendor directory. This is a hacky, short-term
solution to ensure a reproducible build while we work on switching to
bazel.

The versions in the Gopkg.toml file were chosen based on my experiments
that generated proto files that did not change from what is already
checked in.
2018-12-07 15:08:46 -08:00
Yang Pan b494e3daa0 Add integration tests for API servers (#112)
* add pipeline e2e test back

* delete samples

* more changes to pipeline test

* add delete run/experiment

* update test name

* stage

* add api

* fii

* add tensorboard routing rule (#143)

* add tensorboard routing rule

* rename tb routing rule

* address comments

* remove debugger

* fix url prefix

* more fixes

* update tests

* revert debugging

* update

* update test

* fix

* update test

* fix

* fix pipeline tests

* exp

* update run

* update jobs
2018-11-09 20:39:15 -08:00
Pascal Vicaire 1bab424abf Fixing the GO import paths to reference the kubeflow/pipelines repository. 2018-11-02 14:53:42 -07:00
Pascal Vicaire 633e2ddcc8 Initial commit of the kubeflow/pipeline project. 2018-11-02 14:02:31 -07:00