* Deprecate experiment_id in Run API v2
* Allow runs and recurring runs without pipeline version id
* Adjust the logic for retrieving existing pipeline name
* Remove pipeline parsing logic and tests
* Simplify the logic in fetching a template
* Add nodeSelector and res to podSpec
* Add sample test draft
* Add accelerator-type logic to driver pod
* Remove accelerator sample. Closes#7043
* Add unit-tests and remove type handling
* support optional and default parameters
* add sample test
* change assert value
* update test
* update test
* fix sample test
* fix sample test
* fix sample test
* test fix
* change parameter logic
* test
* test
* change logic handling not provided parameter
* remove unnecessary code and add more sample case
* fix sample test bug
* fix sample test
* update logic
* change parameter logic
* update logic
* update logic handling when parameter not provided
* fix sample test
* add error message
* fix samples test
* samples test
* remove unused code
* Add children to PipelineTaskDetail
* Add state and state history to task
* Add state and state history to run
* Add pod_name to PipelineTaskDetail
* Add PodName and Payload to tasks table
* Add upsert to task store
* Add create or update task report wf
* Wire the report server
* Remove v1 and v2 specific functions
* Revert Run proto order
* Stage progress
* Add run id to exec spec
* Disable unit test due to #8851
* Address comments
* Address some comment
* Remove empty lines
* Refactor toModelTasks
* Address comments
* Address comments
* Staging WIP
* Fix recurring run, switch to pipeline_version_id
* Regenerate python client
* Refactor packages for pipeline service v2
* Stage progress
* Add support for pipeline service v2beta1
* Go mod tidy
* Update licenses
* Set default namespace to be empty
* Separate deletion of pipeline versions
* Change report ReportRunMetrics to obey AIP
* Use pipeline version id in run proto
* Update run.proto
* Upgrade storage
* Remove dependencies on api
* Stage refactoring progress
* Refactor apiserver
* Update v1beta1 unit test
* Update dependencies
* Fix integration test
* Revert visualization server changes (workaround)
* Address comments
* Basic lint the api server
* Update licenses file
* Adjust intergtaion tests
* Fix typos in the tests
* Update tests
* Update test
* Add support v1 delepe pipeline
* Remove gorm.Model to split into a different PR
* Lint api server
* Fix default experiment in upgrade test
* Update storage states
* Testing upgrade test run
* Add copying pod logs to the GCS bucket
* Fix upgrade test
* Adjust backend test for list
* Fix the tests
* Fix workflow manifest discrepancy at create
* Add a few unit-tests
* Add misc service and fix persistence agent
* Remove default values from TEXT columns
* Address comments. Fixes#8702.
* Address comment, reorder resource, server, storage
* Fix front end integration test
* Fix FE integration test
* Fix tabs, add pauses, make more explicit FE test
* Revert FE test changes
* Enable default experiment and disable its deletion
* Replace empty namespace with POD_NAMESPACE
* Default experiment is now the oldest
* Emplace empty with POD_NAMESPACE for argo only
* Fix api integration test
* Revert changes to default experiment's description
* Improve FE test flakiness
* Force refresh in the FE test
* Fix FE integration test
* Add delay in FE test
* Add local debug to FE test
* Update tests
* Fix typo
* Address comments
* Enforce template version verification
* Fix workflow manifest discrepancy with v1
* Fix FE test
* Revert changes in persistence agent
* Limit to one resource owner
* Change default run state to unknown
* Revert changes to FE integration test
* Make states consistent
* Remove links from error messages
* Address comments
* Address comments
* Change the order for teardown
---------
Co-authored-by: Chen Sun <chensun@users.noreply.github.com>
* add support for "finished_at" and other in filter
* update job map
* add UTs
* add integration test for filter
* update integration test
* fix integration test
* add more info
* fixed integration test
* fix small bug
* fix flakiness
* Make cache-server webhook port number configurable
Previously the webhook listened on a fixed port, 8443, which can clash
with other services when the webhook is run on the host network in
Kubernetes, which is required when using some CNI implementations,
notably Calico on EKS [1].
Enable configuration of the webhook listen port via the program flags.
[1] https://projectcalico.docs.tigera.io/getting-started/kubernetes/managed-public-cloud/eks#install-eks-with-calico-networking
Signed-off-by: Steve Larkin <steve.larkin@gmail.com>
* fixup! Make cache-server webhook port number configurable
Signed-off-by: Steve Larkin <steve.larkin@gmail.com>
* fixup! fixup! Make cache-server webhook port number configurable
Signed-off-by: Steve Larkin <steve.larkin@gmail.com>
Signed-off-by: Steve Larkin <steve.larkin@gmail.com>
* remove template dependency on api
* add create run
* update run metrics proto
* add run methods
* small changes
* pass asserted types in model converter
* add wiring
* added create recur run and wirings
* added get recurring run and test
* added enable disable delete rec run and tests
* revert namespace change
* fix typo
* use v1 counters for recurring runs
* fix bug
* change recurring run package name and regen
* merge recurring run and job server
* add template support for v2 api
* copyright
* add v2 tests
* add proto files and generate clients
* update dockerfile
* update script
* refactor rec_run_id
* copyright info
* move string definition
* small fix to proto
* small refactor
* template takes model instead of api
* template changes
* convert to model in resource manager
* resolve conflicts
* fix integration test typos
* test
* test
* test
* testing
* temp change for debug
* temp change for debug 2
* temp change for debug 3
* temp change for debug 4
* temp change for debug 5
* temp change for debug 6
* revert resource manager change
* add print
* bring back integration tests
* folder name
* delete a file
* remove package debug
* fix bug
* print more variables
* add log
* add log
* small fix
* add log
* more logs
* add test
* add logs
* test
* add another test
* use old function for Trigger convertion
* change modeltoCRDTrigger function
* removed V1 at the end of function name
* add comments
* removed unnecessary variable
* typo
* small change
* test
* remove logs for debugging
* small bug
* remove logs for debug
* nit and removed unused functions
* remove unused functions and move var declararion
* feat(backend) Fix authentication in upload requests
Fix the way KFP API server authenticates pipeline upload requests.
We leverage 'isAuthenticated()` function which requires proper
initialization of the context object to include user identity.
* feat(backend): Add namespace field in pipeline upload swagger definition
Extend swagger defintion of the pipeline upload API with a namespace
parameter in order to support uploading namespaced pipelines.
* chore(backend): Generate Go & Python clients
Autogenerate the Go and Python clients after extending the swagger
definitions of upload pipeline APIs with a namespace field.
* v2 exp proto and generate clients
* server changes
* move v1 and v2 protos into api folder
* regenerate v1 packages
* generate v2 package
* add v2 funcs.
* added v2 create experiment
* integration and initialization tests
* createexperiment works and rest can be pinged
* add create exp tests
* add get experiment
* add get experiment tests
* add list experiment and one test
* add list exp tests
* add TestValidateCreateExperimentRequest_Multiuser
* add delete exp tests
* add archive/unarchive exp and tests
* refactor storage and resource layers
* change api converter
* small fix
* small fixes
* move proto files outside and name v1 package api
* refactor
* refactor
Instead of changing the WorkflowResource.Spec to string,
use interface{} to allow backward compatibility.
Add type casting and marshal/unmarshal to properly
handle old swf CRs.
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* add runtime config model
* add run store and tests
* add runtime_config to model converter and tests
* add runtime_config to api converter and its tests
* change api server and related tests
* remove v2 runtime_config test
* add runtimeconfig to upgrade test
* fix test values
* upgrade test debug
* tests
* add more info for debug
* use NullString instead of String, remove debug
* fix type conversion
* change function and add unit tests
* run go fmt
* Add comments for model
* marshal params using v2 structpb values
* fix small bug
* Revert "run go fmt"
This reverts commit 251c3a9535.
* No longer sort keys
* test values and explain comparison using .String()
* func toApiRuntimeConfig
* tests updates
* add api converter tests
* change test
* fix format
* change test
* simplify marshalling parameters
* Add ExecutionClient to access ExecutionSpec
Add ExecutionClient, ExecutionInterface to abstract the client
functions for ExecutionSpec. Replace WorkflowClient/WorkflowInterface
in all backend components to use ExecutionClient/ExecutionInterface.
Only add needed functions into ExecutionClient/ExecutionInterface.
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* replace argo Parameter to util.SpecParameter
replace argo Parameter to util.SpecParameter and
remove argo go pkg dependency in model_converter.go
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* Authorize readArtifacts and ReportMetrics endpoints
New Verbs (reportMetrics and readArtifact) are added to ClusterRole with name: aggregate-to-kubeflow-pipelines-edit
Signed-off-by: Diana Atanasova <dianaa@vmware.com>
* Add authorization when Persistent Agent communicate with the api-server
Persistent Agent authorize itself based ot the namespace and the current user
Fixes: #7818
* Update persistence_agent.csv license file
Signed-off-by: Diana Atanasova <dianaa@vmware.com>
* Fix lexical error in persistent agent cluster role
Signed-off-by: Diana Atanasova <dianaa@vmware.com>
* Fix integration tests/Fix MULTIUSER= false usecase
Cover MULTIUSER=false usecase/Standalone pipeline installation.
In this case the namespace doesn't have `user` annotation and
there is no need to provide `kubeflow-userid` Header when making
a request against kfp-api-server
Signed-off-by: Diana Atanasova <dianaa@vmware.com>
* rebase: fix conflixt in license file
Signed-off-by: Diana Atanasova <dianaa@vmware.com>
* rebase add new line in the end of licensing file
Signed-off-by: Diana Atanasova <dianaa@vmware.com>
* feat(backend): Expand ExecutionSpec Interface
Expand the ExecutionSpec and ExecutionStatus interface
to cover more function and refactor the code to use
ExecutionSpec instead of Workflow
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* update license csv file
need to update licenses file for:
- persistent agent
- cache server
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* fix e2e integration failure
the Workflow content in the ReportWorkflowRequest struct
received from report_server API doesn't have TypeMeta.
Therefore, need to explicitly specify ExecutionType.
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* address comments
- clean up params in ExecutionStatus test
- remove UID setter
- add comment to point to Workflow.Status.Phase
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* add comment to the SpecParameter.Default
Need to revisit this property later and see if this
is needed or not
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* rebase and fix conflicts
rebase again and fix conflicts
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* change downloaded IR from JSON to YAML
* correct error message
* update error message
* change template.Bytes() to return yaml
* remove json to yaml conversion
* add TODO
* remove yaml package
* mod tidy
* change go mod
* Update test data for evaluation classification feature attribution pipeline.
PiperOrigin-RevId: 446739064
* feat(conformance): adapt run api tests to KF env (#7658)
- Undid earlier refactoring in test_util.go. I think having the util
function to take a parameter structure is much more readable than
constructing api parameters in the util function.
* feat(conformance): adapt remaining tests to KF env (#7670)
- Adapted job, rn, upgrade and visualization tests
- Undid earlier change where I had to pass in empty string to indicate
that resource reference should not be overridden because experiment id
is specified
- Updated visualization client to support SA token projection auth
* Revert "feat(conformance): adapt remaining tests to KF env (#7670)"
This reverts commit 3cf7c19be9.
* Revert "feat(conformance): adapt run api tests to KF env (#7658)"
This reverts commit e644d7a18b.
* Revert "Update test data for evaluation classification feature attribution pipeline."
This reverts commit 86a2fb2c02.
* block usage of json IR in backend. change test
* update tests
* update tests
* update tests
* fix test
* change commnets
* remove json test
Co-authored-by: Googler <nobody@google.com>
Co-authored-by: James Wu <54086668+james-jwu@users.noreply.github.com>
* fix the time display on be, need colab with fe
* change finished_at display
* remove code for debugging
* fix format
* update backend tests
* make code more succinct
* create an interface for Workflow
Use the new interface: `ExecutionSpec` to substitute
Workflow and remove the argo pkg dependency from
other code except `common/util/workflow.go`.
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* Add ExecutionStatus interface
Separate the status information from
ExecutionSpec to ExecutionStatus interface.
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* add license change
regenerate license files
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
- Adapted job, rn, upgrade and visualization tests
- Undid earlier change where I had to pass in empty string to indicate
that resource reference should not be overridden because experiment id
is specified
- Updated visualization client to support SA token projection auth
- Undid earlier refactoring in test_util.go. I think having the util
function to take a parameter structure is much more readable than
constructing api parameters in the util function.
* support IR YAML format in API
* Check the error message and return false if it is not nil
* update error message
* fixed simple loop but need cleaning up
* Deleted debug logs
* remove logs and fix some format
* fix static_loop_arguments
* change the driver image
change the driver image back to the kfp container registry.
* change variable declaration
* remove logs
* remove log
* changed argo template
* change argocompiler dependency to my own repo
* Update go.mod
* Update VERSION
* v2 pipeline level exit handler; add test case
* Update VERSION
* remove logs and change temp dependency
* remove logs
* Update argo.go
* Update argo_test.go
* Update driver.go
* Update driver.go
* change VERSION
* Update VERSION
* Modifyb PR: remove loop fix and some changes
* update dsl compile command
* revert VERSION
* Update Makefile
* Update Makefile
* support IR YAML format in API
* Check the error message and return false if it is not nil
* update error message
* fixed simple loop but need cleaning up
* Deleted debug logs
* remove logs and fix some format
* fix static_loop_arguments
* change the driver image
change the driver image back to the kfp container registry.
* change variable declaration
* remove logs
* remove log
* move `ok` definition
* change test file for debug purpose
* change test for debug purpose
* update sample test for static loop
* update test file, remove code for debug
* Conformance: Enable experiment API tests in KF mode.
Added 3 flags:
- isDebugMode is enables HTTP request/response logging
- IsKubeflowMode enables the tests in full Kubeflow mode
- resourceNamespace: the namespace/profile under which the test
resources are created
Added a new HTTP client that uses SA token volume projection auth. The
test pods will be set up to project SA token.
Plumbed everything through for experiment API tests. The other tests
will be enabled in subsequent PRs.
* Updated change to address comments.
* AIP-5634: Add in LRU cache and temp file rotation for metadata-writer.
* AIP-5634: Respond to Chens feedback. Fixing imports, requirements bounds and improve efficency of debug file cache
* AIP-5634: Use a FIFO approach and actually remove the old file paths
* AIP-5634: Update to use deque for constant time appends/poplefts
* feat(backend): add new endpoint that get pipelines by name
Add new endpoint getPipelineByName that take in consideration the current namespace,
because the name is unique within a namespace only.
Fix: https://github.com/kubeflow/pipelines/issues/7003
Signed-off-by: Diana Atanasova <dianaa@vmware.com>
* Remove extra identation in pipeline.proto file
Signed-off-by: Diana Atanasova <dianaa@vmware.com>
* feat(backend): add new endpoint GetPipelineByName
Use explicit URL. Namespace isn't passed through `resource_references`, but
is part of the URL path.
In cases when only the pipeline name is required(standalone installation, shared pipelines)
, a hyphen "-" is used. Example:
/apis/v1beta1/namespaces/-/pipelines/{name}
Signed-off-by: Diana Atanasova <dianaa@vmware.com>
* Check for authorization in MultiUserMode
Signed-off-by: Diana Atanasova <dianaa@vmware.com>
* [backend] Forbid unarchive runs that belog to archived experiment
Adds validation so that the user cannot activate a Run that belongs to Archived Experiment.
Issue: https://github.com/kubeflow/pipelines/issues/7014
Signed-off-by: Diana Atanasova <dianaa@vmware.com>
* Change the err message to be more action-driven
Signed-off-by: Diana Atanasova <dianaa@vmware.com>
* Clean up fot the server
* Remove creation of pipeline version
* Missed to commit
* Futher clean up and update of test
* Clean up and move to done
* Missed to update nbr of returns
* added draft of create v2 pipeline run
* fixed broken UT and added UT for parsing template
* modified run apis to support v2 IR spec
* remove temporary patch
* fixed dependency
* fixed build failure
* finished draft
* finished create job and run
* refactor template and fixed broken UT
* updated go license
* fixed build failure
* fixed build
* added UT
* modified UT
* fixed build failure
* fixed license
The apiserver config parser uses `viper.GetDuration` to parse the mysql
connection lifetime variable. The `viper.GetDuration` function uses
`cast.ToDuration`, which uses `cast.ToDurationE`, which assumes
durations to be in nanoseconds if they don't explictly include a unit.
Since the default lifetime in the kustomize manifest is `120`, we
expire connections after 120ns, which is probably unintended. To make
this more clear, this patch includes duration units in the default
values, and drops the `Secs` suffix from the configuration variables,
since the code doesn't assume that durations are in seconds.
See
22b2b540ce/caste.go (L68-L72).
* add description to upload_pipeline_description http endpoint
* add test for pipeline upload server description
* add description to new pipeline version
* show pipeline version in pipeline detail page
* make description optional on UI + show version descriptio or pipeline description
* remove unused reference
* revert wrong change of ref
* updated on should be pipeline version
* add more versions to mock-backend
* show pipeline and version description at the same time
* add tests for UI
* show pipeline version always
* Recognize "sidecar.istio.io/inject" annotation to use istio service mesh optionally
* Fix return to continue
Co-authored-by: Matts966 <Matts966@users.noreply.github.com>
* start adding description to version
* add description to resource manager
* remove description from pipeline default version
* update pipeline_store
* add some tests to ensure its being passed around
* update frontend generated api
* run auto formatter
* add comment to frontend readme about running npm run format
* add missing api converter step
gkcalat
Lingqing Gan
Chen Sun
Ablai Akhazhanov
Connor McCarthy
Steve Larkin
Ilias Katsakioris
juliusvonkohout
Yihong Wang
Diana Atanasova
shayancanonical
ryansteakley
James Liu
James Wu
hsinhoyeh
Kartik Kalamadi
Yuan (Bob) Gong
Yaqi Ji
Alex Latchford
Tommy Li
Pavel Klyuev
capri-xiyue
Hao Xin
Niklas Hansson
Joshua Carp
Yuan Gong
David Xia
boarder7395
Gerard Casas Saez
Matts966
Jiyong Jung