Commit Graph

772 Commits

Author SHA1 Message Date
Dharmit Dalvi 5399585b6a
feat(backend): Enable logging for KFP components (#10288) 2024-01-05 01:47:08 +00:00
Tobias Goerke 9a306129f8
feat(backend): preserve querystring in pipeline root (fixes #10318) (#10319)
* feat: preserve querystring in pipeline root

* refactor: create AppendToPipelineRoot
Also apply to client.go

* feat: remove query string from URIs (#1)

* feat: remove query string from URIs

* refactor(GenerateOutputURI): move and preserve comments
2023-12-19 00:55:37 +00:00
Chen Sun c5658f09ec chore(release): bumped version to 2.0.5 2023-12-08 18:00:49 +00:00
Tommy Li fc0e472974
chore(backend): Propose Tomcli to be the backend reviewer (#10291) 2023-12-06 23:36:54 +00:00
Dharmit Dalvi 23a2bc3ba3
feat(backend): Parameterizing v2 Launcher and Driver Images (#10269)
* Parameterize v2 launcher image

* Parameterize v2 driver image
2023-12-05 21:02:06 +00:00
Chen Sun a226de2197 chore(release): bumped version to 2.0.4 2023-12-01 00:04:16 +00:00
royliang 0948561fda
fix(backend): Pipeline and PipelineVersion Description column value should be optional. (#10205) 2023-11-06 08:01:14 +00:00
Chen Sun 58ce09e07d chore(release): bumped version to 2.0.3 2023-10-27 04:08:53 +00:00
Chen Sun c0093ecef6
feat(backend): Update driver and launcher images (#10164) 2023-10-27 04:02:10 +00:00
Chen Sun 52f5cf51c4
feat(backend): Support consuming parent DAG input artifact (#10162) 2023-10-26 16:16:09 +00:00
Chen Sun efe630cf80 chore(release): bumped version to 2.0.2 2023-10-11 20:27:27 +00:00
Chen Sun 3c5f62a6a3
feat(backend): Update driver and launcher images (#10076) 2023-10-10 23:02:19 +00:00
MGSousa 5835824e9c
feat(backend): Added metrics to be collected from failed/successful workflows (#9576)
* feat(backend): Allow more metrics to be collected from Workflows

* Fixed remaining tests

* Updated licenses dependencies

* FIX comment in resource_manager.go
2023-10-10 20:25:21 +00:00
Chen Sun 8b0e6a39a9
chore(backend): Update driver and launcher license files (#10072) 2023-10-10 01:02:19 +00:00
Lingqing Gan d3a15d9230
chore: Update OWNERS (#10064)
* Update OWNERS

Remove Linchin from approvers and reviewers

* remove Linchin in OWNERS files

---------

Co-authored-by: Chen Sun <chensun@users.noreply.github.com>
2023-10-05 18:23:50 -07:00
Florian Müller dcaafeee8b
fix(backend): Sync scheduled workflows v1 if APIVersion and Kind are missing. Fixes #9809 (#9968) 2023-09-23 18:59:58 +00:00
Vadim Reider 4003e56271
fix(backend): OutPutPath dir creation mode Fixes #7629 (#9946) 2023-09-22 22:32:57 +00:00
David van der Spek 434b41a19c
fix(backend): update requirements scripts (#10009)
* fix: update requirements scripts

Signed-off-by: David van der Spek <vanderspek.david@gmail.com>

* fix: visualization dependency hell

Signed-off-by: David van der Spek <vanderspek.david@gmail.com>

* bump snapshottest

Signed-off-by: David van der Spek <vanderspek.david@gmail.com>

* fix: pin jinja2

Signed-off-by: David van der Spek <vanderspek.david@gmail.com>

* fix: pin markupsafe to fix ci

Signed-off-by: David van der Spek <vanderspek.david@gmail.com>

---------

Signed-off-by: David van der Spek <vanderspek.david@gmail.com>
2023-09-21 20:35:37 +00:00
Diana Atanasova 760c1589ed
feat(backend): enforce SA Token based auth b/w Persistence Agent and Pipeline API Server (#9957)
* Enforece SA-Toben auth b/n Persistence agent & Pipeline server for all reqs

Signed-off-by: Diana Atanasova <dianaa@vmware.com>

* Fix persistence agent license file

Signed-off-by: Diana Atanasova <dianaa@vmware.com>

---------

Signed-off-by: Diana Atanasova <dianaa@vmware.com>
2023-09-07 23:49:33 +00:00
James Liu eaec515812
chore(mlmd): Upgrade ML Metadata to 1.14.0. (#9856)
* chore(mlmd): Upgrade ML Metadata to 1.14.0.

* fix requirements, skip visualization.

* go mod tidy

* mlmd new enum in frontend

* mlmd license

* mlmd license again, using the failure test diff

* again
2023-08-22 17:18:09 +00:00
Chen Sun 40c759f431
chore(release): bump version to 2.0.1 on master branch (#9899) 2023-08-17 14:49:40 -07:00
Diana Atanasova 6dfcee7fa9
Fix Persistence Agent SA Token time interval (#9892)
Issue: https://github.com/kubeflow/pipelines/issues/9891

Signed-off-by: diana <difince@gmail.com>
2023-08-17 16:19:38 +00:00
Diana Atanasova cb18d00bbb
feat(backend) Enable auth between pesistence agent and pipelineAPI (ReportServer) (#9699)
* Enable auth between pesistence agent and pipelineAPI (ReportServer)

1. Add authentication and authorization logic to PipelineAPI's PeportServer &
2. Make Persistence Agent authenticate itself through Service Account Token
Volume Projection.

Signed-off-by: diana <difince@gmail.com>

* Do not use MULTIUSER on report weorkflows

Only Persistent agent can KFPipeline API "Report (Scheduled)Workflows" no matter
the namespace workflows belongs to.

Signed-off-by: diana <difince@gmail.com>

* Add unit tests

- unit tests added
- do not stop the ticker on stopCh

Signed-off-by: diana <difince@gmail.com>

---------

Signed-off-by: diana <difince@gmail.com>
2023-08-17 12:34:38 +00:00
James Liu fe60742b00
fix(backend): Move ConMaxLifeTime back to DbConfig.ConMaxLifeTime. (#9873)
* fix(backend): Move ConMaxLifeTime back to DbConfig.ConMaxLifeTime.

* remove comma
2023-08-16 09:55:32 -07:00
Lingqing Gan e1f0c010f8
feat(backend): add postgres initialization (#9798)
* add postgres initialization

* remove load balancer

* go mod tidy

* update license

* license update for viewer

* (test) disable controller license check

* (test) disable persistence agence licence check

* (test) disable scheduled workflow license check

* (test) disable cacheserver license check

* fix db config location

* fix mysql support

* test

* test

* no long set host address

* address comments

* address comments and enable license check

* format

* remove extra blank line

* update licenses

* cache server license

* address comments

* centralize error message

* remove pv in postgres deployment
2023-08-10 21:51:05 +00:00
deepk2u a6af41c23b
fix(backend): fix timeouts with list run api. Fixes #9780 (#9806)
* Update client_manager.go

* Update client_manager.go
2023-08-04 20:34:20 +00:00
Diana Atanasova de89b1c658
fix(backend) Replace LEFT with INNER JOIN when Archive Experiment (#9730)
* fix(backend) Replace LEFT with INNER JOIN when Archive Experiment

Signed-off-by: diana <difince@gmail.com>

* Replace FEFT JOIN With INNER JOIN

Signed-off-by: diana <difince@gmail.com>

---------

Signed-off-by: diana <difince@gmail.com>
2023-07-18 14:02:22 +00:00
Diana Atanasova 213dd5a1af
fix(backend): Fix performance issue within a mysql request (#9680)
* fix(backend): Fix performance issue within a mysql request

Reprace the existing mysql request that use nested select, with inner join for better performance.
The fix levarage 'SQLDialect' interface, because the new request is not supported by sqllite (used for testing)
This interface bridges the difference between mysql (production) and sqlite
// (test)
Issue: https://github.com/kubeflow/pipelines/issues/6845

Signed-off-by: diana <difince@gmail.com>

* For sqlite use UPDATE FROM to join the target table against another table in the database in order to help compute

Try to generalize the method in SQLDialect interface

Signed-off-by: diana <difince@gmail.com>

* Add unit tests

Signed-off-by: diana <difince@gmail.com>

* Replace nested query for Jobs and start using pre-comit

Signed-off-by: diana <difince@gmail.com>

* Fix: Use LEFT JOIN instead of INNER JOIN

Signed-off-by: diana <difince@gmail.com>

---------

Signed-off-by: diana <difince@gmail.com>
2023-07-12 21:27:24 +00:00
Chen Sun a727783abd
chore: remove 'dummy_root' in test samples (#9640)
* remove 'dummy_root' in test samples

* more cleanup
2023-06-21 23:51:41 +00:00
Chen Sun e03e312193 chore(release): bumped version to 2.0.0 2023-06-20 16:54:58 +00:00
James Liu cdb7932cdb
fix(backend): Avoid multiple PutParentContexts on the same primary key. (#9604)
* fix(backend): Avoid multiple PutParentContexts on the same primary key.

* use function to get id instead of the pointer address.
2023-06-20 16:35:40 +00:00
Linchin 923b67e38f chore(release): bumped version to 2.0.0-rc.2 2023-06-13 18:43:09 +00:00
Lingqing Gan dbebbde200
fix(backend): let components add default values (#9591)
* let components add default values

* address comments and add unit tests

* address comments

* backend test

* backend test 2

* backend test 3

* backend test 4

* backend test 5

* backend test 6

* do not use python component in unit tests

* shell command

* shell command

* shell command

* does not delete tmp folder

* change folder permission

* update launcher image
2023-06-13 18:25:03 +00:00
Chen Sun ee2102a9b5
chore(samples/backend/frontend): Update preload control-flow and data-passing pipelines to v2 (#9563)
* Update preload samples to v2 pipelines

* address review comments

* fix frontend test

* format frontend code

* update comment
2023-06-07 22:06:57 +00:00
Chen Sun 0d53de7d1c
fix(backend): fix column size of `StateHistory` in `task` table. Fixes: #9553 (#9554) 2023-06-02 10:49:06 +00:00
Lingqing Gan eadeb3096b
fix(backend): enable cache for create/delete PVC (#9504)
* enable cache for create/delete pvc

* remove comments

* createCache() func

* update driver image
2023-06-02 01:12:06 +00:00
Lingqing Gan 0ccdb09a35
fix(backend): publish launcher execution even if it fails (#9513)
* publish launcher execution

* update launcher image
2023-05-31 01:16:04 +00:00
ryansteakley 8a3215ec56
feat(backend): Implement aws-go-sdk crendentials to support IRSA for s3 in V2 (#9502)
* support v2 irsa

* remove white space
2023-05-26 23:49:00 +00:00
Chen Sun 4810b7aac7
fix(backend): upgrade to `gopkg.in/yaml.v3` to fix boolean support. Fixes #9451 (#9473) 2023-05-24 08:57:19 +00:00
gkcalat 3e9f04815e
fix(backend): Fix url encoding issue. Closes #9448 (#9449) 2023-05-23 04:00:35 +00:00
gkcalat 79d8d83ce4
fix(backend): Add creation of pipeline and pipeline version atomically (#9416)
* Add create pipeline and version

* Add CreatePipelineAndVersion

* Add unit tests

* Address comments and add counters

* Add comments to pipeline.proto

* Add e2e test case for CreatePipelineAndVersion
2023-05-18 23:38:46 +00:00
Lingqing Gan 08494a2db6
feat(backend): add schema version check (#9447) 2023-05-18 21:02:10 +00:00
Lingqing Gan 724445064b
chore: change sample PVC access modes to ReadWriteOnce (#9425)
* change sample access modes to ReadWriteOnce

* change test data

* change samples to use ReadWriteOnce
2023-05-18 01:22:06 +00:00
gkcalat c2bc434c11
chore(backend): Add golangci-lint and pre-commit config (#9428)
* Add golangci-lint and pre-commit config

* Adopt golanci-lint config from k8s
2023-05-17 18:23:06 +00:00
Lingqing Gan f9409a42d6
test: remove test dependency on driver/launcher images (#9422) 2023-05-16 00:19:59 +00:00
Joe Li cb93fcc5ca chore(release): bumped version to 2.0.0-rc.1 2023-05-12 21:36:20 +00:00
Lingqing Gan 73f5f1b3ca
fix(backend): prepublish execution in the driver for create/delete pvc (#9397)
* publish execution for create/delete pvc

* update driver launcher image

* combine functions

* update driver launcher image
2023-05-12 14:05:20 -07:00
Chen Sun f35d50c8dd
chore: Remove unused binaries in cache-deployer, add curl license (#9400)
* Update Dockerfile

* Update Dockerfile

* add curl license text to third_party/license.txt

* change the location where we store third_party license.txt

* checkout license.txt from upstream master

* add https://github.com/curl/curl/blob/master/COPYING to third_party/license.txt
2023-05-12 06:28:31 +00:00
gkcalat 09c0b92771
fix(backend): Fix authorization logic and remove redundant DB calls (#9373)
* Refactor authorization

* Upgrade api version

* Revent API version bump

* Fix string comparison

* Revert test change

* Change error type

* Fix typo

* Fix namespace logic
2023-05-12 03:07:31 +00:00
gkcalat 9ec4dc7f30
fix(backend): Add pipeline spec name validation and relax display name validation (#9374)
* Add pipeline name check at creation time

* Revert changes for pipeline version

* Validate pipeline spec name and relax display name

* Fix e2e test
2023-05-11 23:57:30 +00:00