kubeflow
/
pipelines
mirror of https://github.com/kubeflow/pipelines.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
3,425 Commits 314 Branches 182 Tags 516 MiB
Commit Graph

3425 Commits

Author SHA1 Message Date
Yuan (Bob) Gong a6ab4e4411
chore(swf): fix recurring run + v2 compatible (#6297) 2021-08-11 03:22:25 -07:00
James Liu 579e46b840
feat(frontend): Convert PipelineSpec to Staticflow for V2 pipeline. Fix #6270 (#6278) 
* chore(frontend): Use protoc to compile IR to typescript definition

* feat(frontend): Convert PipelineSpec to Staticflow for V2 pipeline.

* address comment
 2021-08-10 22:45:25 -07:00
Yuan (Bob) Gong 2b78d16e03
chore(v2): artifact passing via local path (#6285) 
* chore(v2): add implicit dependencies for artifacts

* chore(v2): artifact passing via local path

* clean up
 2021-08-10 21:40:25 -07:00
Gerard Casas Saez 62a41812c1
chore(frontend): Fix gen license issues for master (#6286) 
* add license for warchpack-chokidar

ignore dep-licenses

revert changes in package.json

* add missing licenses

* remove generated files

* remove temporary files after being created locally
 2021-08-10 18:10:25 -07:00
Ajay Gopinathan 911562f8d3
fix(sdk): Relax the requirement that component inputs/outputs must appear on the command line. (#6268) 
* Relax the requirement that component inputs/outputs must appear
on the command line.

This change enables component builders to define their implementation
(i.e. the container arg and command) independent of the actual
component inputs/outputs defined in component.yaml. The latter will be
used when determining inputs/outputs during compilation.

Only works for tasks that turn into ContainerOps during compilation.
Naturally, does not work for ContainerOps (which have no inputs/outputs
anyway).

* fix bugs, update goldens.

* Fix issue with inputs.

* merge and update.

* Restore compiler yamls.

* Disambiguate parameters and artifacts in component bridge.

* Restore goldens from HEAD.

* restore compiler_tests.py to HEAD.

* str-ify pipeline params.

* cr comments.
 2021-08-10 14:40:25 -07:00
James Liu 1feb9a0c41
chore(frontend): Use protoc to compile IR to typescript definition. Fix #6272 (#6273) 2021-08-10 02:38:24 -07:00
Yuan (Bob) Gong 11a375f09d
refactor(v2): save pipeline root to pipeline run context. Part of #6152 (#6271) 
* refactor(v2): save pipeline root to pipeline run context

* fix

* simplify some changes
 2021-08-10 02:29:24 -07:00
sina chavoshi ae9956ce32
chore(components/google-cloud): Post release v0.1.4 clean up (#6275) 
* google-cloud-pipeline-components-0.1.4

* update release docs

* Update version for custom_job docker

* fixing module path

* temporary fixed for 0.1.4 released module

* add init for remote dir

* post release clean up

* point custom job to latest

* revert docker tag version
 2021-08-09 23:30:24 -07:00
Yaqi Ji 0ac67d0c05
test(sdk): Ignore kfp sdk version when comparing golden (#6281) 2021-08-09 19:13:25 -07:00
Yaqi Ji 1752d372b4
chore(sdk): Create RELEASE.md for future release documentations (#6276) 2021-08-09 15:26:24 -07:00
Chen Sun d48792c373
test(sdk): restore a v2-compatible unit test (#6263) 2021-08-09 11:18:23 -07:00
Chen Sun 5b5568c112
feat(api): Add env variable in ContainerSpec (#6251) 2021-08-09 10:52:27 -07:00
James Liu b3937cf901
chore(frontend): Integrate with tailwindcss. Fix #6264 (#6265) 
* chore(frontend): Integrate with tailwindcss

* adjust UI and fix for comment

* fix tailwind car
 2021-08-09 10:00:27 -07:00
shrinath-suresh f9d47d0ef9
fix(components/pytorch): PyTorch Samples - Generating component.yaml using templates (#6231) 
* Adding code to generate component.yaml files from templates

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

* Adding templates for train and pre process

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

* Updating the build script to generate component.yaml from templates

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

* Updating the jupyter notebook to use templates

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

* Removing all component.yaml files

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

* Revert "Removing all component.yaml files"

This reverts commit db75951949.

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

* Fixing bert notebook

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

* Changing cifar10 pipeline

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

* Updating cifar10 notebook

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

* Fixing cifar10 template mapping

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

* Fixing the cifar10 preprocess input

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

* fixing ciar10 notebook arguments

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

* Adding examples in component.yaml templates

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

* Updating README.md files

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

* Updating cifar10 captum insights notebook

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

* Adding templates for hpo

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

* Updating distributed training notebook

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

* Fixing yaml path in dist training

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

* Removing all component.yaml files

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

* Moving component.yaml templates into pytorch-kfp-components

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

* Fixing template path

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

* Updating cifar10 script argument variable

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

* Fixing argument or bert dist

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

* Adding image name to templates

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

* Adding template for ax train

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

* Creating template mapping for ax

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

* Fixing hpo script arguments

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

* Ax Template mapping fix

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

* Addressing review comments

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>
 2021-08-08 23:34:25 -07:00
shrinath-suresh 271783e58e
chore(components/pytorch): Updating pytorch-kfp-component dependency list (#6219) 
Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>
 2021-08-08 22:41:25 -07:00
Yuan (Bob) Gong b28db942d4
docs: v2 compatible release instructions (#6267) 2021-08-08 20:21:25 -07:00
Yuan (Bob) Gong 9868b8a507
test: fix make push-launcher (#6266) 2021-08-08 19:57:25 -07:00
James Liu 6c8309d644
feat(frontend): Prepare PipelineDetailPage to show different contents based on V1/V2 pipeline spec. Fix #6214 (#6258) 
* chore(frontend): Mock v2 pipeline spec response.

* feat(frontend): Prepare PipelineDetailPage to show different contents based on V1/V2 pipeline spec

* remove ir pipeline button and routing path
 2021-08-07 17:12:24 -07:00
Yaqi Ji 9989d37021
chore: update Dockerfile for SDK 1.7.0 (#6262) 2021-08-06 18:13:27 -07:00
Yaqi Ji 3044989dbf
chore: release KFP SDK and v2 launcher 1.7.0 (#6257) 
* Update __init__.py

* Update v2_compat.py

* Update goldens
 2021-08-06 17:18:25 -07:00
Ajay Gopinathan 402549d81f
fix(samples): Add `kfp_package_path` to a core sample test. (#6259) 2021-08-06 16:49:26 -07:00
Grace Luo 7872bd4fee
chore(components/google-cloud): Add seed to tfp_anomaly_detection (#6244) 2021-08-06 11:30:24 -07:00
Chen Sun 903b98654e
feat(sdk.v2): Define BaseComponent, ComponentSpec, and PipelineTask. (#6210) 
* Define BaseComponent, ComponentSpec, and PipelineTask.

* fix dep

* disable positional arguments

* address feedback

* address feedback
 2021-08-06 08:54:43 -07:00
Chen Sun f4657ba42a
feat(sdk.v2): Add HTML and Markdown artifact types (#6246) 2021-08-06 02:24:42 -07:00
sina chavoshi 0e1db0d064
fix(components/google-cloud): Add temporary work around for key not found during compile (#6224) 
* Add temporary work around for key not found during compile

* correct  comment on from v1 to v2 compiler
 2021-08-06 01:36:42 -07:00
Yuan Gong 3c6d1285a0 chore(release): bumped version to 1.7.0-rc.3 2021-08-06 07:13:16 +00:00
Yuan (Bob) Gong 3ec069d8bc
feat(deployment): gcp marketplace - provide emissary as an option. Part of #5718 (#6250) 2021-08-06 00:01:42 -07:00
Yuan (Bob) Gong b5dc28200a
feat(deployment): provide platform-agnostic-emissary and platform-agnostic-multi-user-emissary as options (#6248) 
* feat(deployment): provide platform-agnostic-emissary and platform-agnostic-multi-user-emissary as options

* add tests
 2021-08-05 22:06:42 -07:00
James Liu 78c550ee11
chore(frontend): Mock one v2 pipeline spec response. Fix #6232 (#6233) 2021-08-05 19:14:49 -07:00
Yuan (Bob) Gong 25958081e6
chore(v2): parameter passing. Fixes #6151 (#6208) 
* chore(v2): publish output parameters

* chore(v2): resolve input parameters from producer task

* chore(v2): add implicit parameter dependencies to DAG tasks

* test(v2): add verification for v2 tests

* fix compiler unit test
 2021-08-05 18:25:50 -07:00
Joshua Carp 2f843f95aa
fix: fix a spelling error (#6236) 2021-08-05 17:17:46 -07:00
Ajay Gopinathan 727501fd17
test: Remove old v2 launcher building steps. (#6247) 2021-08-05 16:28:48 -07:00
Ajay Gopinathan c5591f2652
feat(sdk): Enable the use of Executor from KFP package for v2 lightweight components (#6202) 
* Enable the use of v2 Executor from KFP package.

This avoids the need to paste Executor code for v2 components into
the entrypoint of lightweight components.

* Update tests.

* quote package path.

* quote package path.

* escape fragment

* Update golden files.

* update goldens.

* Cleanup.

* More cleanup.

* update goldens.

* style updates.

* clean up sample tests.

* fix trailing comma

* Try using signed urls.

* fix

* fix presubmit

* try /bin/bash for Makefile shell

* undo signed url change.

Co-authored-by: Yuan (Bob) Gong <4957653+Bobgy@users.noreply.github.com>
 2021-08-05 14:53:03 -07:00
Chen Sun ef4518623e
chore: make directories before downloading api dependency proto (#6245) 2021-08-05 14:28:42 -07:00
Yaqi Ji 4c838f7001
chore: Update package version for pipeline-spec release (#6242) 2021-08-05 13:17:42 -07:00
IronPan c4c84fe3b5
chore: update component owner file (#6239) 2021-08-05 08:43:18 -07:00
Yuan (Bob) Gong ef37236d00
fix(backend): argo workflow not found in KFP DB should be GCed. Part of #6189 (#6190) 
* fix(backend): argo workflow not found in KFP DB should be GCed

* Update backend/src/apiserver/resource/resource_manager.go

Co-authored-by: Niklas Hansson <niklas.sven.hansson@gmail.com>

* address feedback

* tidy

* Update backend/src/apiserver/resource/resource_manager.go

Co-authored-by: Niklas Hansson <niklas.sven.hansson@gmail.com>

Co-authored-by: Niklas Hansson <niklas.sven.hansson@gmail.com>
 2021-08-05 07:22:19 -07:00
Yuan (Bob) Gong b1c76e0cd3
test(v2): build go images using ko. Fixes #6238 (#6240) 
* test(v2): build go images using ko

* clean up

* integrate launcher license checking

* clean up

* fix go v2 test
 2021-08-05 06:33:18 -07:00
Yuan (Bob) Gong 2181b2fb8b
feat(sdk): add_default_resource_spec container op transformer (#6203) 
* feat(sdk): add_default_resource_spec container op transformer

* set memory request/limit for sampOOMed samples

* refactor

* adjust tensorboard minio memory usage

* adjust resource usage

* clean up

* address feedbacks
 2021-08-05 15:28:17 +08:00
sina chavoshi e25d43c04c
chore(components/google-cloud): Clean up unit tests (#6222) 2021-08-05 00:04:18 -07:00
Yaqi Ji 58fefe9e21
chore: update pipeline_spec.proto comments for `schema_title` format and sync the deprecation warning for `schema_uri`. (#6234) 
* Update pipeline_spec.proto

* Update pipeline_spec.proto
 2021-08-04 22:11:18 -07:00
Yuan (Bob) Gong 9c6250fe12
chore(backend): pin sdk version to released version. Fixes #5689 (#6229) 2021-08-04 18:53:18 -07:00
James Liu a94e92e530
feat(frontend): Select node using execution ID via router param in run detail page. Partial #5977 (#6218) 
* feat(frontend): Select node using execution ID via router param in run detail page.

* revert test snapshot

* fix router test

* fix type casting

* fix test
 2021-08-04 17:59:18 -07:00
Yuan (Bob) Gong e6df7db197
revert: "feat!: use argo emissary executor by default. Fixes #5718" (#6230) 
This reverts commit 0c4129c900.
 2021-08-04 13:55:17 -07:00
Chen Sun 0b82ef939b
chore(api): Using Makefile to pull dependency proto (#6223) 
* update api/makefile

* add back "make clean all"
 2021-08-04 07:40:42 -07:00
Yuan (Bob) Gong 36b815bedf
feat: upgrade to argo v3.1.5-patch (#6228) 
* feat: upgrade to argo v3.1.5-patch

* update go.mod

* fix license

* update license
 2021-08-04 06:55:41 -07:00
James Liu 17a9bd7cca
test: Remove upgrade command in project_cleaner (#6227) 
* fix(test): Remove upgrade command in project_cleaner

* Use shell

* remove
 2021-08-04 02:30:41 -07:00
Chen Sun bb22118588
feat(api): Add PipelineTaskFinalStatus to pipeline_spec.proto (#6221) 
* Add PipelineTaskFinalStatus

* fork https://github.com/googleapis/googleapis/blob/master/google/rpc/status.proto
 2021-08-03 22:12:41 -07:00
Yaqi Ji 6b48154d1c
feat(sdk): update pipeline_spec.proto to add artifact schema version field (#6220) 2021-08-03 16:54:41 -07:00
James Liu 97b9bdb530
chore(frontend): Create script to build pipeline_spec with protobuf.js. Fix #6205 (#6206) 
* chore(frontend): Create script to build pipeline_spec with protobuf.js

* fix generated location

* add Todo about shell script

* add readme for generating pipeline spec
 2021-08-03 13:04:41 -07:00