Commit Graph

11 Commits

Author SHA1 Message Date
Andrey Velichkevich 4b0c2943bc
feat(sdk): Support MPI-based TrainJobs (#2545)
* feat(sdk): Support MPI-based TrainJobs

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Refactor list_runtimes

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Fix example

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Add Runtime Trainer object

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Update for new Runtime object

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Implement get_runtime API

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Fix Torch example

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Remove un-unsed consts

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Update func args

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Update SDK constants

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Change to 16Gi

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Fix container name for MPI

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Keep launcher container for MPI

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

---------

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>
2025-03-20 18:08:02 +00:00
Shao Wang 32ee3c7212
KEP-2401: Refactor current `train()` API (#2513)
* fix(sdk): rename Trainer to CustomTrainer.

Signed-off-by: Electronic-Waste <2690692950@qq.com>

* fix(sdk): remove validate_trainer().

Signed-off-by: Electronic-Waste <2690692950@qq.com>

* fix(sdk): remove lora related code.

Signed-off-by: Electronic-Waste <2690692950@qq.com>

* fix(sdk): remove get_lora_config()

Signed-off-by: Electronic-Waste <2690692950@qq.com>

* fix(sdk): fix import error in __init__.py

Signed-off-by: Electronic-Waste <2690692950@qq.com>

* chore(example): update the image-classification example.

Signed-off-by: Electronic-Waste <2690692950@qq.com>

* fix(sdk): delete remaining lora related code.

Signed-off-by: Electronic-Waste <2690692950@qq.com>

* fix(sdk): modify args description in CustomTrainer.

Signed-off-by: Electronic-Waste <2690692950@qq.com>

* fix(sdk): add parameter type in CustomTrainer dataclass.

Signed-off-by: Electronic-Waste <2690692950@qq.com>

* fix(sdk): update args in CustomTrainer.

Signed-off-by: Electronic-Waste <2690692950@qq.com>

---------

Signed-off-by: Electronic-Waste <2690692950@qq.com>
2025-03-13 02:18:13 +00:00
Andrey Velichkevich 9e785750d0
chore(test): Add E2E tests for Kubeflow Trainer (#2470)
* Add e2e tests for Kubeflow Trainer

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Add timeout for papermill

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Add output as part of make command

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Add k8s version to setup cluster

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Fix Kind k8s version

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Fix 1.29 version

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Create script to run Notebook

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Download dataset when local_rank=0

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Update test/e2e/e2e_test.go

Co-authored-by: Yuki Iwai <yuki.iwai.tz@gmail.com>
Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Refactor Go e2e tests

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Bump k8s to 1.29.14

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Install Kind from go mod

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Fix path for Kind package

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Fix Go e2e

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Reduce number of CPUs
Export Notebook as artifact

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Print logs due to flaky test

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Fix artifact path

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* docker pull image

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Fix path

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Add k8s version to output name

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Remove install Kind cmd

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

---------

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>
Co-authored-by: Yuki Iwai <yuki.iwai.tz@gmail.com>
2025-03-05 04:04:07 +00:00
Andrey Velichkevich 3060332931
Update the naming conventions for Kubeflow Trainer (#2415)
* Update the naming conventions for Kubeflow Trainer

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Fix webhooks

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Fix paths for webhooks

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Update go test cmd

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Rename kubeflowv1 to trainer pkg

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

---------

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>
2025-02-06 13:48:30 +00:00
Andrey Velichkevich 7858371b4a
Remove the Training Operator V1 Source Code (#2389)
* Remove Training Operator V1 files

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Refactor README and Makefile

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Remove v1 GitHub workflows

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Remove finish action from Go

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Rename Kubeflow Training to Kubeflow Trainer

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Rename comment

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Add steps for free-up space

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Update Python tests

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Remove integration workflow

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Install SDK in Python integration tests

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Remove generate file

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Add Kubeflow Trainer logo

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Add tech diagram

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Exclude images from pre-commit

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Fix git diff in CI

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Run go mod tidy

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Update diagram

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Cleanup examples

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Always install Go bins

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Update diagram

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Update hack
Remove manifests for JobSet
Rename to tools.go

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Create tools.go

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

---------

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>
2025-02-04 15:25:38 +00:00
Antonin Stefanutti ee11629194
KEP-2170: Add PyTorch DDP MNIST training example (#2387)
Signed-off-by: Antonin Stefanutti <antonin@stefanutti.fr>
2025-02-01 22:16:33 +00:00
saileshd1402 56cbe60076
Added test for create-pytorchjob.ipynb python notebook (#2274)
* Added test for create-pytorchjob.ipynb

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* fix yaml syntax

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* Fix uses path

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* Add actions/checkout

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* Add bash to action.yaml

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* Install pip dependencies step

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* Add quotes for args

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* Add jupyter

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* Add nbformat_minor: 5 to fix invalid format error

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* Fix job name

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* test papermill-args-yaml

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* testing multi line args

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* testing multi line args1

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* testing multi line args2

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* testing multi line args3

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* Parameterize sdk install

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* Remove unnecessary output

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* nbformat normailze

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* [SDK] Training Client Conditions related unit tests (#2253)

* test: add unit test for get_job_conditions function of training client

Signed-off-by: Bobbins228 <mcampbel@redhat.com>

* test: add unit test for is_job_created function of training client

Signed-off-by: Bobbins228 <mcampbel@redhat.com>

* test: add unit test for is_job_running function of training client

Signed-off-by: Bobbins228 <mcampbel@redhat.com>

* test: add unit test for is_job_restarting function of training client

Signed-off-by: Bobbins228 <mcampbel@redhat.com>

* test: add unit test for is_job_failed function of training client

Signed-off-by: Bobbins228 <mcampbel@redhat.com>

* test: add unit test for is_job_succeded function of training client

Signed-off-by: Bobbins228 <mcampbel@redhat.com>

* test: improve job condition unit tests efficiency

Signed-off-by: Bobbins228 <mcampbel@redhat.com>

---------

Signed-off-by: Bobbins228 <mcampbel@redhat.com>
Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* [SDK] test: add unit test for list_jobs method of the training_client (#2267)

Signed-off-by: wei-chenglai <qazwsx0939059006@gmail.com>
Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* KEP-2170: Generate clientset, openapi spec for the V2 APIs (#2273)

Generate clientset, informers, listers and open api spec
for v2alpha1 APIs.

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>
Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* [SDK] Use torchrun to create PyTorchJob from function (#2276)

* [SDK] Use torchrun to create PyTorchJob from function

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Update PyTorchJob SDK example

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Add consts for entrypoint

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Add check for num procs per worker

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

---------

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>
Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* [SDK] test: add unit test for get_job_logs method of the training_client (#2275)

Signed-off-by: wei-chenglai <qazwsx0939059006@gmail.com>
Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* [v2alpha] Move GV related codebase (#2281)

Move GV related codebase in v2alpha

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>
Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* KEP-2170: Implement runtime framework (#2248)

* KEP-2170: Implement runtime framework interfaces

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

* Remove grep dependency

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

* KEP-2170: Implement ValidateObjects interface to the runtime framework

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

* KEP-2170: Expose the TrainingRuntime and ClusterTrainingRuntime Kind

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

* KEP-2170: Remove unneeded scheme field from the internal TrainingRuntime

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

* Rephrase the error message

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

* Distinguish TrainingRuntime and ClusterTrainingRuntime when creating indexes for the TrainJobs

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

* Propagate the TrainJob labels and annotations to the JobSet

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

* Remove PodAnnotations from the runtime info

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

* Implement TrainingRuntime ReplicatedJob validation

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

* Add TODO comments

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

* Replace queueSuspendedTrainJob with queueSuspendedTrainJobs

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

---------

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>
Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* Add DeepSpeed Example with Pytorch Operator (#2235)

Signed-off-by: Syulin7 <735122171@qq.com>
Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* KEP-2170: Rename TrainingRuntimeRef to RuntimeRef API (#2283)

* KEP-2170: Rename TrainingRuntimeRef to RuntimeRef API

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Rename RuntimeRef in runtime framework

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

---------

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>
Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* KEP-2170: Adding CEL validations on v2 TrainJob CRD (#2260)

Signed-off-by: Akshay Chitneni <achitneni@apple.com>
Co-authored-by: Akshay Chitneni <achitneni@apple.com>
Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* Upgrade Deepspeed demo dependencies (#2294)

Signed-off-by: Syulin7 <735122171@qq.com>
Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* KEP-2170: Add manifests for Kubeflow Training V2 (#2289)

* KEP-2170: Add manifests for Kubeflow Training V2

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Fix invalid name for webhook config in cert

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Fix integration tests

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Move kubebuilder markers to runtime framework

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Use Kubernetes recommended labels

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

---------

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>
Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* FSDP Example for T5 Fine-Tuning and PyTorchJob (#2286)

* FSDP Example with PyTorchJob and T5 Fine-Tuning

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Modify text

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

---------

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>
Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* KEP-2170: Implement TrainJob Reconciler to manage objects (#2295)

* KEP-2170: Implement TrainJob Reconciler to manage objects

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

* Mode dep-crds to manifests/external-crds

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

* Rename run with runtime

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

---------

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>
Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* Remove Prometheus Monitoring doc (#2301)

Signed-off-by: Sophie <sophy010017@gmail.com>
Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* KEP-2170: Decouple JobSet from TrainJob (#2296)

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>
Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* KEP-2170: Strictly verify the CRD marker validation and defaulting in the integration testings (#2304)

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>
Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* KEP-2170: Initialize runtimes before the manager starts (#2306)

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>
Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* KEP-2170: Generate Python SDK for Kubeflow Training V2 (#2310)

* Generate SDK models for the Training V2 APIs

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Create pyproject.toml config

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Remove comments

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Fix pre-commit

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

---------

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>
Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* KEP-2170: Create model and dataset initializers (#2303)

* KEP-2170: Create model and dataset initializers

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Add abstract classes

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Add storage URI to config

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Update .gitignore

Co-authored-by: Kevin Hannon <kehannon@redhat.com>
Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Fix the misspelling for initializer

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Add .pt and .pth to ignore_patterns

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

---------

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>
Co-authored-by: Kevin Hannon <kehannon@redhat.com>
Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* KEP-2170: Implement JobSet, PlainML, and Torch Plugins (#2308)

* KEP-2170: Implement JobSet and PlainML Plugins

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Fix nil pointer exception for Trainer

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Fix unit tests in runtime package

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Fix unit tests

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Fix integration tests

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Fix lint

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Implement Torch Plugin

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Use list for the Info envs

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Fix golang ci

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Fix Torch plugin

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Use K8s sets
Update error return
Use ptr.Deref() for nil values

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Use client.Object for Build() call

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Remove DeepCopy

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Remove MLPolicy and PodGroupPolicy from the Info object

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Inline error

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Remove SDK jar file

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Add integration test for Torch plugin

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Add TODO to calculate PodGroup values in unit tests

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Revert the change to add original Runtime Policies to Info

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Create const for the DefaultJobReplicas

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Check if PodLabels is empty

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

---------

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>
Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* KEP-2170: Implement Initializer builders in the JobSet plugin  (#2316)

* KEP-2170: Implement Initializer builder in the JobSet plugin

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Update the SDK models

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Remove Info from Initializer builder

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Update manifests

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Update pkg/constants/constants.go

Co-authored-by: Yuki Iwai <yuki.iwai.tz@gmail.com>
Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Use var for envs

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Remove check manifests from GitHub actions

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Move consts to JobSet plugin

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

---------

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>
Co-authored-by: Yuki Iwai <yuki.iwai.tz@gmail.com>
Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* KEP-2170: Add the TrainJob state transition design (#2298)

* KEP-2170: Add the TrainJob state transition design

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

* Replace actual jobs with TrainJob

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

* Remove the JobSet conditions propagation and Add expanding runtime framework interfaces for each plugin

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

* Expand the Creation Failed reasons

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

* Rename Completed to Complete

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

---------

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>
Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* Update tf job examples to tf v2 (#2270)

* mnist with summaries updaetd to TF v2

Signed-off-by: yelias <yossi.elias@nokia.com>

* tf_sample updaetd to TF v2

Signed-off-by: yelias <yossi.elias@nokia.com>

* Add mnist_utils and update dist-mnist

Signed-off-by: yelias <yossi.elias@nokia.com>

* Add mnist_utils and update dist-mnist

Signed-off-by: yelias <yossi.elias@nokia.com>

* Remove old example - estimator-API, this example has been replaced by distribution_strategy

Signed-off-by: yelias <yossi.elias@nokia.com>

* Small fix

Signed-off-by: yelias <yossi.elias@nokia.com>

* Remove unsupported powerPC dockerfiles

Signed-off-by: yelias <yossi.elias@nokia.com>

* Fix typo in copyright

Signed-off-by: yelias <yossi.elias@nokia.com>

---------

Signed-off-by: yelias <yossi.elias@nokia.com>
Co-authored-by: yelias <yossi.elias@nokia.com>
Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* KEP-2170: Add TrainJob conditions (#2322)

* KEP-2170: Implement TrainJob conditions

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

* Fix API comments

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

* Make condition message constants

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

* Stop connecting condition type and reason in JobSet plugin

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

---------

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>
Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* Pin Gloo repository in JAX Dockerfile to a specific commit (#2329)

This commit pins the Gloo repository to a specific commit (43b7acbf) in
the JAX Dockerfile to prevent build failures caused by a recent bug
introduced in the Gloo codebase. By locking the version of Gloo to
a known working commit, we ensure that the JAX build remains stable and
functional until the issue is resolved upstream.

The build failure occurs when compiling the gloo/transport/tcp/buffer.cc
file due to an undefined __NR_gettid constant, which was introduced
after the pinned commit. By using this commit, we bypass the issue and
allow the build to complete successfully.

Signed-off-by: Sandipan Panda <samparksandipan@gmail.com>
Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* [fix] Resolve v2alpha API exceptions (#2317)

Resolve v2alpha API exceptions by adding necessary listType validations.

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>
Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* Upgrade Kubernetes to v1.30.7 (#2332)

* Upgrade Kubernetes to v1.30.7

Signed-off-by: Antonin Stefanutti <antonin@stefanutti.fr>

* Use typed event handlers and predicates in job controllers

Signed-off-by: Antonin Stefanutti <antonin@stefanutti.fr>

* Re-organize pkg/common/util/reconciler.go

Signed-off-by: Antonin Stefanutti <antonin@stefanutti.fr>

* Update installation instructions in README

Signed-off-by: Antonin Stefanutti <antonin@stefanutti.fr>

---------

Signed-off-by: Antonin Stefanutti <antonin@stefanutti.fr>
Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* Ignore cache exporting errors in the image building workflows (#2336)

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>
Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* KEP-2170: Add Torch Distributed Runtime (#2328)

* KEP-2170: Add Torch Distributed Runtime

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Add pip list

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

---------

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>
Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* Refine the server-side apply installation args (#2337)

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>
Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* Add openapi-generator CLI option to skip SDK v2 test generation (#2338)

Signed-off-by: Antonin Stefanutti <antonin@stefanutti.fr>
Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* Upgrade kustomization files to Kustomize v5 (#2326)

Signed-off-by: oksanabaza <obazylie@redhat.com>
Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* Pin accelerate package version in trainer (#2340)

* Pin accelerate package version in trainer

Signed-off-by: Gavrish Prabhu <gavrish.prabhu@nutanix.com>

* include new line to pass pre-commit hook

Signed-off-by: Gavrish Prabhu <gavrish.prabhu@nutanix.com>

---------

Signed-off-by: Gavrish Prabhu <gavrish.prabhu@nutanix.com>
Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* Replace papermill command with bash script

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* Typo fix

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* Move Checkout step outside action.yaml file

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* Add newline EOF in script

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* Pass python dependencies as args and pin versions

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* Update Usage

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* Install dependencies in yaml

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* fix ipynb

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* set bash flags

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* Update script args and add more kubernetes versions for tests

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* add gang-scheduler-name to  template

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* move go setup to template

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* remove -p parameter from script

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

---------

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>
Signed-off-by: Bobbins228 <mcampbel@redhat.com>
Signed-off-by: wei-chenglai <qazwsx0939059006@gmail.com>
Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>
Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>
Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>
Signed-off-by: Syulin7 <735122171@qq.com>
Signed-off-by: Akshay Chitneni <achitneni@apple.com>
Signed-off-by: Sophie <sophy010017@gmail.com>
Signed-off-by: yelias <yossi.elias@nokia.com>
Signed-off-by: Sandipan Panda <samparksandipan@gmail.com>
Signed-off-by: Antonin Stefanutti <antonin@stefanutti.fr>
Signed-off-by: oksanabaza <obazylie@redhat.com>
Signed-off-by: Gavrish Prabhu <gavrish.prabhu@nutanix.com>
Co-authored-by: Mark Campbell <mcampbel@redhat.com>
Co-authored-by: Wei-Cheng Lai <qazwsx0939059006@gmail.com>
Co-authored-by: Varsha <varshaprasad96@gmail.com>
Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>
Co-authored-by: Yuki Iwai <yuki.iwai.tz@gmail.com>
Co-authored-by: yu lin <735122171@qq.com>
Co-authored-by: Akshay Chitneni <akshayadatta@gmail.com>
Co-authored-by: Akshay Chitneni <achitneni@apple.com>
Co-authored-by: Sophie Hsu <112261858+sophie0730@users.noreply.github.com>
Co-authored-by: Kevin Hannon <kehannon@redhat.com>
Co-authored-by: YosiElias <73485442+YosiElias@users.noreply.github.com>
Co-authored-by: yelias <yossi.elias@nokia.com>
Co-authored-by: Sandipan Panda <87253083+sandipanpanda@users.noreply.github.com>
Co-authored-by: Antonin Stefanutti <astefanutti@users.noreply.github.com>
Co-authored-by: Oksana Bazylieva <61097730+oksanabaza@users.noreply.github.com>
Co-authored-by: Gavrish Prabhu <gavrish.prabhu@nutanix.com>
2024-12-09 13:38:05 +00:00
Andrey Velichkevich 35413277de
[SDK] Use torchrun to create PyTorchJob from function (#2276)
* [SDK] Use torchrun to create PyTorchJob from function

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Update PyTorchJob SDK example

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Add consts for entrypoint

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Add check for num procs per worker

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

---------

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>
2024-10-11 18:52:11 +00:00
Alex faec4e8e4f
Implement pre-commit hooks (#2184)
* Implement pre-commit hooks

Signed-off-by: droctothorpe <mythicalsunlight@gmail.com>

* Fix broken file links and update reqs

Signed-off-by: droctothorpe <mythicalsunlight@gmail.com>

* Add examples folder

Signed-off-by: droctothorpe <mythicalsunlight@gmail.com>

* Remove black instructions from developer guide

Signed-off-by: droctothorpe <mythicalsunlight@gmail.com>

---------

Signed-off-by: droctothorpe <mythicalsunlight@gmail.com>
2024-08-05 17:03:39 +00:00
Andrey Velichkevich bb8bba00ff
Modify LLM Trainer to support BERT and Tiny LLaMA (#2031)
* Modify LLM Trainer to support BERT and Tiny LLaMA

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Access PVC access modes to storage config

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Format Python files

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Distribute datasets

Fix access modes in storage config

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Update example to fine tune BERT with Train API

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Remove dataset download twice

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

---------

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>
2024-03-15 19:28:30 +00:00
Andrey Velichkevich 395e8cabd7
Add Fine-Tune BERT LLM Example (#2021)
* Add Fine-Tune BERT LLM Example

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Add 3 GPUs in Notebook requirements

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

---------

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>
2024-03-12 11:35:10 +00:00