* Enable healthz test in full Kubeflow.
This change will also be ported to release-2.0-alpha branch to support
conformance tests for KF 1.7, which is using alpha.7 version of KFP.
* Fix build break due to recent refactoring.
* Staging WIP
* Fix recurring run, switch to pipeline_version_id
* Regenerate python client
* Refactor packages for pipeline service v2
* Stage progress
* Add support for pipeline service v2beta1
* Go mod tidy
* Update licenses
* Set default namespace to be empty
* Separate deletion of pipeline versions
* Change report ReportRunMetrics to obey AIP
* Use pipeline version id in run proto
* Update run.proto
* Upgrade storage
* Remove dependencies on api
* Stage refactoring progress
* Refactor apiserver
* Update v1beta1 unit test
* Update dependencies
* Fix integration test
* Revert visualization server changes (workaround)
* Address comments
* Basic lint the api server
* Update licenses file
* Adjust intergtaion tests
* Fix typos in the tests
* Update tests
* Update test
* Add support v1 delepe pipeline
* Remove gorm.Model to split into a different PR
* Lint api server
* Fix default experiment in upgrade test
* Update storage states
* Testing upgrade test run
* Add copying pod logs to the GCS bucket
* Fix upgrade test
* Adjust backend test for list
* Fix the tests
* Fix workflow manifest discrepancy at create
* Add a few unit-tests
* Add misc service and fix persistence agent
* Remove default values from TEXT columns
* Address comments. Fixes#8702.
* Address comment, reorder resource, server, storage
* Fix front end integration test
* Fix FE integration test
* Fix tabs, add pauses, make more explicit FE test
* Revert FE test changes
* Enable default experiment and disable its deletion
* Replace empty namespace with POD_NAMESPACE
* Default experiment is now the oldest
* Emplace empty with POD_NAMESPACE for argo only
* Fix api integration test
* Revert changes to default experiment's description
* Improve FE test flakiness
* Force refresh in the FE test
* Fix FE integration test
* Add delay in FE test
* Add local debug to FE test
* Update tests
* Fix typo
* Address comments
* Enforce template version verification
* Fix workflow manifest discrepancy with v1
* Fix FE test
* Revert changes in persistence agent
* Limit to one resource owner
* Change default run state to unknown
* Revert changes to FE integration test
* Make states consistent
* Remove links from error messages
* Address comments
* Address comments
* Change the order for teardown
---------
Co-authored-by: Chen Sun <chensun@users.noreply.github.com>
* add support for "finished_at" and other in filter
* update job map
* add UTs
* add integration test for filter
* update integration test
* fix integration test
* add more info
* fixed integration test
* fix small bug
* fix flakiness
* add template support for v2 api
* copyright
* add v2 tests
* add proto files and generate clients
* update dockerfile
* update script
* refactor rec_run_id
* copyright info
* move string definition
* small fix to proto
* small refactor
* template takes model instead of api
* template changes
* convert to model in resource manager
* resolve conflicts
* fix integration test typos
* test
* test
* test
* testing
* temp change for debug
* temp change for debug 2
* temp change for debug 3
* temp change for debug 4
* temp change for debug 5
* temp change for debug 6
* revert resource manager change
* add print
* bring back integration tests
* folder name
* delete a file
* remove package debug
* fix bug
* print more variables
* add log
* add log
* small fix
* add log
* more logs
* add test
* add logs
* test
* add another test
* use old function for Trigger convertion
* change modeltoCRDTrigger function
* removed V1 at the end of function name
* add comments
* removed unnecessary variable
* typo
* small change
* test
* remove logs for debugging
* small bug
* remove logs for debug
* nit and removed unused functions
* remove unused functions and move var declararion
* v2 exp proto and generate clients
* server changes
* move v1 and v2 protos into api folder
* regenerate v1 packages
* generate v2 package
* add v2 funcs.
* added v2 create experiment
* integration and initialization tests
* createexperiment works and rest can be pinged
* add create exp tests
* add get experiment
* add get experiment tests
* add list experiment and one test
* add list exp tests
* add TestValidateCreateExperimentRequest_Multiuser
* add delete exp tests
* add archive/unarchive exp and tests
* refactor storage and resource layers
* change api converter
* small fix
* small fixes
* move proto files outside and name v1 package api
* refactor
* refactor
* add runtime config model
* add run store and tests
* add runtime_config to model converter and tests
* add runtime_config to api converter and its tests
* change api server and related tests
* remove v2 runtime_config test
* add runtimeconfig to upgrade test
* fix test values
* upgrade test debug
* tests
* add more info for debug
* use NullString instead of String, remove debug
* fix type conversion
* change function and add unit tests
* run go fmt
* Add comments for model
* marshal params using v2 structpb values
* fix small bug
* Revert "run go fmt"
This reverts commit 251c3a9535.
* No longer sort keys
* test values and explain comparison using .String()
* func toApiRuntimeConfig
* tests updates
* add api converter tests
* change test
* fix format
* change test
* simplify marshalling parameters
* change downloaded IR from JSON to YAML
* correct error message
* update error message
* change template.Bytes() to return yaml
* remove json to yaml conversion
* add TODO
* remove yaml package
* mod tidy
* change go mod
* Update test data for evaluation classification feature attribution pipeline.
PiperOrigin-RevId: 446739064
* feat(conformance): adapt run api tests to KF env (#7658)
- Undid earlier refactoring in test_util.go. I think having the util
function to take a parameter structure is much more readable than
constructing api parameters in the util function.
* feat(conformance): adapt remaining tests to KF env (#7670)
- Adapted job, rn, upgrade and visualization tests
- Undid earlier change where I had to pass in empty string to indicate
that resource reference should not be overridden because experiment id
is specified
- Updated visualization client to support SA token projection auth
* Revert "feat(conformance): adapt remaining tests to KF env (#7670)"
This reverts commit 3cf7c19be9.
* Revert "feat(conformance): adapt run api tests to KF env (#7658)"
This reverts commit e644d7a18b.
* Revert "Update test data for evaluation classification feature attribution pipeline."
This reverts commit 86a2fb2c02.
* block usage of json IR in backend. change test
* update tests
* update tests
* update tests
* fix test
* change commnets
* remove json test
Co-authored-by: Googler <nobody@google.com>
Co-authored-by: James Wu <54086668+james-jwu@users.noreply.github.com>
- Adapted job, rn, upgrade and visualization tests
- Undid earlier change where I had to pass in empty string to indicate
that resource reference should not be overridden because experiment id
is specified
- Updated visualization client to support SA token projection auth
- Undid earlier refactoring in test_util.go. I think having the util
function to take a parameter structure is much more readable than
constructing api parameters in the util function.
* Conformance: Enable experiment API tests in KF mode.
Added 3 flags:
- isDebugMode is enables HTTP request/response logging
- IsKubeflowMode enables the tests in full Kubeflow mode
- resourceNamespace: the namespace/profile under which the test
resources are created
Added a new HTTP client that uses SA token volume projection auth. The
test pods will be set up to project SA token.
Plumbed everything through for experiment API tests. The other tests
will be enabled in subsequent PRs.
* Updated change to address comments.
* added draft of create v2 pipeline run
* fixed broken UT and added UT for parsing template
* modified run apis to support v2 IR spec
* remove temporary patch
* fixed dependency
* fixed build failure
* finished draft
* finished create job and run
* refactor template and fixed broken UT
* updated go license
* fixed build failure
* fixed build
* added UT
* modified UT
* fixed build failure
* fixed license
* feat(backend): Add qps & burst options in clientset. Fix#5095
* Some minor changes:
* Use comment in client-go as usage string of QPS & burst.
* Use `common.GetXXX()` to retrieve configurations in API server.
* Use a struct to include configurations in cache server.
* Update job_api_test.go to support modified `NewScheduledWorkflowClientOrFatal()`.
* Some minor fixes:
* Create `util.ClientParameters` for configurations needed to create a client.
* Add comments linking to the default value of clientQPS & clientBurst defined in k8s.io/client-go.
* Added multi-user pipelines backend
corrected typo
updating code based on review
fixes for pipelines server
reverting this back
* removing unnecessary info logging
* updated to remove bazel
* rename@
* script to run in container
* update the generation
* updated docker image
* changed name
* updated the code
* regenerated API
* fix env variables
* updated version of go swagger
* rerun generate with updated package
* added healthz
* typo with folders fixed
* changed version
* set version of protoc compiler
* test if version is correct
* test version
* changed version
* updated version agian
* test version
* the latest test
* updated docker image
* fixed some stuff
* new test
* tested other version
* new tests
* changed swagger
* new test
* updated versions
* missed docker file
* updated files@
* change back
* updated after feedback@
* clean up
* remove license and install binary
* update and check differences
* completed rebase
* go mod tidy
* updated based upon work of boby
* remove code commited by misstake
* added by misstake
* updated after feedback
* futher updates after feedback
* final updates from feedback
* switch to Makefile
* update
* fix non-root execution
* clean up, fix release script
* fix swagger
* fix healthz endpoint & regenerate python client
* Delete sample.py.tar.gz
Co-authored-by: Yuan Gong <gongyuan94@gmail.com>
Co-authored-by: Yuan (Bob) Gong <4957653+Bobgy@users.noreply.github.com>
* chore(backend): tidy go.mod and update tools.go
* go install, instead of go get
* fix problems reported by go vet
* simplify some ide reported redundant syntax
* license is not required for generated code
* remove licenses for generated code
* cleanup
* remove license more
* rm unused BUILD.bazel files
* fixed generate_api.sh
* reimport error.proto
* intial work'
* small fixes
* updated tests and how parameter are set
* try to fix test
* check with out adding missing test
* fixed small typo
* test changes
* updated config
* typo
* updated after feedback
* fixed pointer error
* test to add paramter
* moved to init so removed not needed code
* updated further
* updated tests to also check endtime
* clean up test
* fixed failing test
* fixed the expected test results
* added timezone examples
* further clean up
* fixed time format
* Update params.env
* moved location to cronjobscheduler
* clean up
* set env variable to empty
* reverted back
* updated to make magic nbr to constant
* updated the tests with comment
* added comments on cron expressions
* update naming and return types
* updated to UTC as default
* updated with an alpha notice
* fix(backend): job api -- deletion should succeed when swf not found
* bug reproducing unit test
* fix the bug and pass reproducing unit test
* reproducing integration test
* fix integration test
* clarify error message
* disable job should also succeed, unify term to CR instead of CRD
* fix unit test error
* fix error message
* improve logging
* initial work on exposing the default version of pipeline
* update description
* added missing files
* updated api build, unsure if this is correct ...
* updated after feedback
* clean up
* remove empty line
* started to make the integration test
* added integreation test
* fixed build and feedback
* updated the tests
* Updated the test
* new test
* typo
* updated the pipeline default
* updated the pipeline version
* formatting
* error in comparison
* re-enable upgrade test since 0.5.0 is cut
* renable upgrade test; and in upgrade test, sort resource references
field before comparison
* those types have been defined elsewhere
* expect service account
* Add service account field to run and job api objects
* Update description
* Fix field casing
* Use service account from api object
* Fix bug and add unit test
* Save patched workflow spec into db instead
* Save service account to db model
* Fix unit tests
* Fix integration tests
* Fix upgrade test
* Update upgrade_test.go
* add new field in db schema and api schema
* auto genereted types for experiment storage state
* add archive and unarchive methods to backend for experiments.
* auto generated archive/unarchive methods for epxeriments
* add archive and unarchive to client
* set proper storage state when creating experiment
* retrieve storage state when we get/list epxeriment(s)
* change expection in test to have storage state
* add storage state in resource manager test
* revise experiemnt server test
* revise api converter test
* integration test of experiment archive
* archive/unarchive experiment affect the storage state of runs in it
* test all the runs in archive/unarchive experiment
* test all runs are archived/unarchived with their experiment in experiment server
* integration test
* integration test: value type mismatch in assertion
* unused import; default value for storage state
* autogen code for frontend
* reorder the fields in api experiment schema
* switch the position of the two enum to verify a hypothesis
* Put a place hodler to prevent any valid item to take the value 0
* Get rid of the place holder since the cause of issue related to value 0 is found and fixed.
* The returned api experiment now has storage state field
* create experiment return doesn't contain storege state
* Cleanup needs to clean runs and pipeliens now
* a missing client
* use resource reference as fileter instead of experiment uuid
* use same namespace in archive unit test
* Leave archive/unarchive experiment integration test to a separate PR
* also need to update jobs when experiments are archived
* Change of unarchiving logic. When experiment is unarchived, jobs/runs in
it stay archived
* add unit test for the job status in archived/unarchived experiment
* change archive state to 3 value enum; add experiment integration test
* make archive state 3 value enum to avoid 0 value mapped to available; add integration test
* run swagger autogen
* fix an expected value
* fix experiment server test
* add job check in experiment server test
* update job crds
* fix a typo
* remove accidentally included irrelevant changes
* For legacy interface, we switch to the new presentation underhood
* when create run, if user specify a pipeline, we subsitute it with the pipeline's default version
* Add a case where a version and a pipeline are both specified
* comment; get ready pipeline
* comments
* fix upgrade integration test
* comments of todo; expected run/job now has resource references
* fix upgrade test expected value according to the new response
* fix a typo
* a quick hack for upgrade test
* surface err from conversion
* Add pipeline version to job/run integration test so that job/run is created from pipeline version
* Use set comparison to check array equation in test
* address comments to use elements match
* Implement upgrade test
* mark upgrade-tests.sh as executable
* Fix comments
* Base upgrade_test_setup.yaml
* e2e integration of upgrade test
* Fix entrypoint argument
* Fix e2e workflow yaml
* Fix run_test.sh argument processing
* Fix no closing backtick
* Restrucutre upgrade_test.go to focus the test on upgrade verification
* clean up code
* Clean up after upgrade test when it is run in integration tests.
* Include pipeline tests in upgrade test
* Reorder tests
* Add upgrade test coverage for run api resources
* Add job api resource coverage in upgrade test & refactored upgrade test
* Fix add missing step in upgrade test
* Fix BUILD.bazel
* Fix upgrade_test.go
* Try to fix upgrade test failure
* Fix hard coded namespace
* Sync upgrade-tests.sh with new changes in presubmit-tests-with-pipeline-deployment.sh
* Update upgrade test
* Remove redundant code
* Fix integration test exit code
* Fix trigger interval second mismatch
* create a new version upload integration test file
* checkpoint
* checkpoint
* test
* checkpoint
* checkpoint
* version api integration test
* Fix a typo
* typo in comment
* error message
* Make local testing easier
* Move cleanup to test setup stage
* Add readme for how to run integration tests
* Add warning about data loss
* Add warning also in the script
* Change flag to isDevMode and cleanup resources if not in dev mode
* Pass through arguments in the bash script
* Fix unit tests
* integration tests for swf NoCatchup option
* Also add cron catchup option tests
* Restructure test
* Adjust format
* Fix BUILD.bazel
* fix typo
* Make local testing easier
* Move cleanup to test setup stage
* Add readme for how to run integration tests
* Add warning about data loss
* Add warning also in the script
* Change flag to isDevMode and cleanup resources if not in dev mode
* Pass through arguments in the bash script
* Fix unit tests
Lingqing Gan
gkcalat
James Wu
Chen Sun
hsinhoyeh
capri-xiyue
Yuan (Bob) Gong
Niklas Hansson
Haofeng Xie
Luis Magana
numerology
Yuan (Bob) Gong
jingzhang36