* Added multi-user pipelines backend
corrected typo
updating code based on review
fixes for pipelines server
reverting this back
* removing unnecessary info logging
* updated to remove bazel
* rename@
* script to run in container
* update the generation
* updated docker image
* changed name
* updated the code
* regenerated API
* fix env variables
* updated version of go swagger
* rerun generate with updated package
* added healthz
* typo with folders fixed
* changed version
* set version of protoc compiler
* test if version is correct
* test version
* changed version
* updated version agian
* test version
* the latest test
* updated docker image
* fixed some stuff
* new test
* tested other version
* new tests
* changed swagger
* new test
* updated versions
* missed docker file
* updated files@
* change back
* updated after feedback@
* clean up
* remove license and install binary
* update and check differences
* completed rebase
* go mod tidy
* updated based upon work of boby
* remove code commited by misstake
* added by misstake
* updated after feedback
* futher updates after feedback
* final updates from feedback
* switch to Makefile
* update
* fix non-root execution
* clean up, fix release script
* fix swagger
* fix healthz endpoint & regenerate python client
* Delete sample.py.tar.gz
Co-authored-by: Yuan Gong <gongyuan94@gmail.com>
Co-authored-by: Yuan (Bob) Gong <4957653+Bobgy@users.noreply.github.com>
* chore(backend): tidy go.mod and update tools.go
* go install, instead of go get
* fix problems reported by go vet
* simplify some ide reported redundant syntax
* license is not required for generated code
* remove licenses for generated code
* cleanup
* remove license more
* rm unused BUILD.bazel files
* fixed generate_api.sh
* reimport error.proto
* Set current namespace in local KFP context if running from notebook
* Create "~/.config/kfp/" instead of ".config/kfp/"
At first it was assumed the `get_user_namespace` command would be executed from the home directory.
* Create local context file if it doesn't exist during set_user_namespace
* Grab path from LOCAL_KFP_CONTEXT when creating folder
Instead of harcoding the os.mkdirs path to `~/.config/kfp` it now grabs it from the LOCAL_KFP_CONTEXT. Also, removed path creation in `get_user_namespace` as that is now handled in `set_user_namespace`. Also, it now checks if the path exists rather than the local_context_file to remove the situation where it tries to create ~/.config/kfp/ because the context.json doesn't exist when the path does.
* add multi-user setting to healthz api
* Add http prefix to health api url
* move healtz api call to own function and fix multi_user boolean
* Fix HEALTH_PATH declaration
* Move check to Client __init__ and change get_kfp_healthz to avoid breaking in case of old apiserver image
* Add multi_user to frontend healthz
* Expose multi_user in frontend and add integration test
* Fix integration test
* Fix host hardcoding and error handling
* Handle empty API response, check if API up to date
* Fix response return
* remove API check due to empty response
* retry API call if first response empty
* retry getting healthz api if no response
* change health_api to https
The healthz_api has been returning empty responses which might be caused by sending an http request to an https endpoint. Although requests handles redirects, this commit is to test if this solves the issue.
* Add some debug info to healthz exception
* add url to debug and lower retries to 1
* Use api_client to get healthz data
* Debug info for API response
* Follow API redirect history
* Fix indentation
* Add healthz proto
* Try getting healthz api with new python backend
* Add installation of kfp_server_api in tests
* Fix incorrect setup location
* Replace old .get with new http backend .multi_user
* Code clean up
* Small fixes and TimeOutError for retries healthz api
* Remove changes to go dependencies
* Send empty proto request and fix exception client
* Remove unused commit_sha and tag_name
* initial work on exposing the default version of pipeline
* update description
* added missing files
* updated api build, unsure if this is correct ...
* updated after feedback
* clean up
* remove empty line
* started to make the integration test
* added integreation test
* fixed build and feedback
* updated the tests
* Updated the test
* new test
* typo
* updated the pipeline default
* updated the pipeline version
* formatting
* error in comparison
* list experiment desc
* changes should be made in proto
* add comments and descriptions
* comments/descriptions in run.proto
* comments in job.proto and pipeline.proto
* try starting a new line
* newline doesnt help
* add swagger gen'ed file
* address comments
* regenerate json and client via swagger
* address comments
* regenerate go_http_client and swagger from proto
* two periods
* re-generate
* include namespace in CreateVisualization
* include namespace in post body
* put namespace in the path and in front of visualization
* post /apis/v1beta1/visualizations/{namespace}
* add desscription to client interface
* autogen
* version doesn't have description field
* swagger autogen
* remove two accidentally committed local python package
* add new field in db schema and api schema
* auto genereted types for experiment storage state
* add archive and unarchive methods to backend for experiments.
* auto generated archive/unarchive methods for epxeriments
* add archive and unarchive to client
* set proper storage state when creating experiment
* retrieve storage state when we get/list epxeriment(s)
* change expection in test to have storage state
* add storage state in resource manager test
* revise experiemnt server test
* revise api converter test
* integration test of experiment archive
* archive/unarchive experiment affect the storage state of runs in it
* test all the runs in archive/unarchive experiment
* test all runs are archived/unarchived with their experiment in experiment server
* integration test
* integration test: value type mismatch in assertion
* unused import; default value for storage state
* autogen code for frontend
* reorder the fields in api experiment schema
* switch the position of the two enum to verify a hypothesis
* Put a place hodler to prevent any valid item to take the value 0
* Get rid of the place holder since the cause of issue related to value 0 is found and fixed.
* The returned api experiment now has storage state field
* create experiment return doesn't contain storege state
* Cleanup needs to clean runs and pipeliens now
* a missing client
* use resource reference as fileter instead of experiment uuid
* use same namespace in archive unit test
* Leave archive/unarchive experiment integration test to a separate PR
* also need to update jobs when experiments are archived
* Change of unarchiving logic. When experiment is unarchived, jobs/runs in
it stay archived
* add unit test for the job status in archived/unarchived experiment
* change archive state to 3 value enum; add experiment integration test
* make archive state 3 value enum to avoid 0 value mapped to available; add integration test
* run swagger autogen
* fix an expected value
* fix experiment server test
* add job check in experiment server test
* update job crds
* fix a typo
* remove accidentally included irrelevant changes
* GetNextScheduledEpochNoCatchup implementation
* Add tests for cron schedule nocatchup
* Add tests for periodic schedule and fix a corner case
* Integrate no catchup behavior in swf controller
* Update job api proto
* Regenerate backend client
* Pass catchup parameter in backend api
* Rename proto field to no_catchup, so that it has backward compatible default value
* Update generated backend api
* Use no_catchup field instead
* Add some comments
* add namespace to some run APIs
* update only the create run api
* add resourcereference for namespace runs
* add variables in const
* add types to toModel func
* bug fix
* strip the namespace resource reference when mapping to the db model
* add unit tests
* use gofmt
* replace belonging relationshipreference to owner
* put a todo for further investigation of using namespace or uuid
* apply gofmt
* revert minor change
* Update model_converter.go
* Open the version api in BE for later FE PR to use. Including
auto-generated BE and FE code.
* format FE
* re-generate
* remove an unnecessary auto-generated file
* format
* Fix API package names and regenerate checked-in proto files. Also bump version of GRPC gateway used.
* Fix BUILD.bazel file for api as well.
* Update Bazel version
* add finished time for list runs
* add finished time for list runs
* fix tests
* add finished time for list runs
* Update run.proto
* address comments
* make query more robust
* fix e2e test
* fix e2e test
* Added the terminate run command to backend and CLI.
No generated files for now.
* Added the generated files.
* Moved the code to run_store.go
Now the call chain is run_client->run_server->resource_manager->run_store
* Using the backoff package for retries.
* Trying to update run status in the DB to "Terminating" before patching the workflow.
* Stopped using the Argo errors module.
* Fixed the compilation errors due to recent backend changes.
* RILEY - WIP - Implementation of workflow_fake.go and first test
Added successful test in resource_manager_test.go and completed, barring nits and conventions, the implementation of Patch() and isTerimated() within workflow_fake.go
Additional tests and lots of clean-up are still necessary
* Adds a few more tests to resource_manager_test and cleans up a bit
* Further clean up. Stopped using squirrel for UPDATE query. Added run_store_tests
* Adds terminate run integration test
* Undo changes to go.sum
* Fixes path to long-running.yaml in integration test
* Allow runs with no Conditions to be terminated
* add count to protos and libs
* close db rows before second query
* count -> total_size
* int32 -> int
* move scan count row to util
* add comments
* add logs when transactions fail
* dedup from and where clauses
* simplify job count query
* job count queries
* run count queries
* add job_store total size test
* added tests for list util
* pr comments
* list_utils -> list
* fix clients and fake clients to support TotalSize
* added TotalSize checks in api integration tests
* Use Bazel to build the entire backend.
This also uses Bazel to generate code from the API definition in the
proto files.
The Makefile is replaced with a script that uses Bazel to first generate
the code, and then copy them back into the source tree.
Most of the BUILD files were generated automatically using Gazelle.
* Fix indentation in generate_api.sh
* Clean up WORKSPACE
* Add README for building/testing backend.
Also fix the missing licenses in the generated proto files.
* Add license to files under go_http_client
* Make all ListXXX operations use POST instead of GET.
Generate new swagger definitions and use these to generate the frontend
APIs using `npm run apis`.
This is to support filtering in List requests, as the current
grpc-gateway swagger generator tool does not support repeated fields in
requests used in GET endpoints.
* Use base64-encoded JSON-stringified version of Filter instead.
This lets us keep filter as a simple parameter in the ListXXX requests,
and gets around having to use POST for List requests.
* refactor filter parsing to parseAPIFilter and add tests
* Hack to ensure correct Swagger definitions are generated for Filter.
* Fix merge conflicts with master after rebase
* fix indentation
* Fix hack so frontend apis compile.
* print failing experiments
* try print again.
* revert experiment_api_test
* Use StdEncoding for base64 encoding
* Fix nil pointer dereference error caused err variable shadowing
* skip integration tests when unit test flag is set to true
* wip
* add StorageState enum to proto
* add StorageState to model
* archive proto/model changes
* wip archive endpoint
* wip adding tests
* archive test
* unarchive proto and implementation
* cleanup
* make storage state required, with a default value
* remove unspecified value from storage state enum
* pr comments
* pr comments
* fix archive/unarchive endpoints, add api integration test
* typo
* WIP: Add filter package with tests.
* Add tests for IN predicate.
* Add listing functions
* Try updating list experiments
* Cleanup and finalize list API.
Add tests for list package, and let ExperimentStore use this new API.
Update tests for the latter as well.
* Add comments. BuildSQL -> AddToSelect for flexibility
* Run dep ensure
* Add filter proto to all other resources
* Add filtering for pipeline server
* Add filtering for job server
* Add filtering for run server
* Try to fix integration tests
Luis Magana
Niklas Hansson
Yuan (Bob) Gong
Yang Pan
DavidSpek
Yuan (Bob) Gong
jingzhang36
Chen Sun
Ning
Ajay Gopinathan
Kirin Patel
Alexey Volkov
Yasser Elsayed
Ajay Gopinathan
Pascal Vicaire
Pascal Vicaire