* Update and add tests to kubeflow-pipelines-profile-controller (sync.py)
- add control of visualization and frontend images through environment variables with backwards compatability to previous
- add a barebones end-to-end pytest suite and dev environment to help demonstrate behaviour and catch mistakes
* fix: remove deprecated comment
* fix: typo in copyright
* Refactor to simplify code, add test case
* Adds test case for passing parameters as arguments to factory (previous version had copy/paste errors that broke this feature, and no tests to catch this)
* fix some copy/paste errors (ex: controller was named improperly in HTTPServer call)
* feat: Add script to run tests for sync.py
* fix: resolve merge conflicts with master
* adding config for SetConnMaxLifetimeSec
making dbConMaxLifetime configurable
nit add time value
change the actual variable
* standardizing and documenting
* switched to getdurationconfig
* initial work on removing pod affinity for cacheing image
* updated env extraction
* check if nodeselector or affinity is set. Also updated the variables needed
* missed to save
* fix test
Signed-off-by: NikeNano <niklas.sven.hansson@gmail.com>
* fixed test
* update manifests
* clean up
* remove withspaces
* fix(manifests): fix error when kpt pkg get manifests
* fix another case
* add a test to check the problem automatically
* add script to prepare for presubmit test
* fix permission
* Introduce kubernetes client utils
Introduce common utils for client initialization to factor out common
code.
This is a step towards fulfilling kubeflow/pipelines#4738.
* Use common util to initialize k8s clientset
* Introduce TokenReview client and fake ones
* Extend ResourceManager with a TokenReview client
* Extend FakeClientManager with a fake TokenReview Client
* Introduce authentication utils
* Introduce HTTP header authenticator
* Initialize Kubeflow-UserID header authenticator
* Refactor getUserIdentity() to use auth_util
* Move getting user identity logic to resource manager
Have the resource manager authenticate the request.
In following commits we will be extending the authentication methods to
use, among others, Kubernetes clients. Thus, we move the logic to the
resource manager to benefit from the clients kept its context.
* Introduce constants for the TokenReview authentication
* Introduce TokenReview authenticator
* Extend authenticators with a TokenReview one
Extend the authenticators which the KFP apiserver applies on a request
with a TokenReview authenticator.
This authenticator expects a ServiceAccountToken in a header with the
format: 'Authorization: Bearer <token>'
Part of https://github.com/kubeflow/pipelines/issues/5138
* Add tests for auth_util
* Add tests for HTTPHeaderAuthenticator
* Update server tests based on the new authentication API
* Remove old tests and unused code
* Add tests for TokenReviewAuthenticator
* Add server tests with unauthenticated requests
* manifests: Allow KFP API server to create TokenReviews
* auth: Split 'auth_util.go' into two parts
Split the file into:
* auth.go: contains the main entrance from the outside of the package
* util.go: contains all utility functions used inside
* Change token review audience variable and value
* Allow configuring audience with an environment variable
* Rename IsRequestAuthenticated -> AuthenticateRequest
* Don't use AuthenticateRequest method in tests
Instead of using AuthenticateRequest to retrieve the user from the
request and then use it for the expected values, allocate a variable for
the username in the request and use that in the expected values.
This ensures we don't hide potential errors of AuthenticateRequest.
* Change authenticators order
Have the HTTPHeaderAuthenticator first followed by the
TokenReviewAuthenticator
* Move authenticators to a ResourceManager property
To avoid potential race conditions when initializing the Authenticators
variable, we move authenticators to a ResourceManager property and
initialize it along with the initialization of the manager.
* set up cluster
* allow to set with env variable
* fix the market place
* updated manifests
* Added default, still need to fix how env is set in test
* test alpine agin
* added test
* updated the image
* deleted
* change image by misstake
* updated after feedback
* deleted
* smaller image
* added to the config.json
* adjust to new updates for config handling
* Updated image to use latest
* update deployer ksa roles so that it can solve b/159616919
* add permission needed
Co-authored-by: Renmin Gu <renming@google.com>
Co-authored-by: Yuan Gong <gongyuan94@gmail.com>
* refactor(deployment): refactor argo manifests to be overlay on top of upstream
* switch to pns executor
* updates
* update
* add cluster scoped resouces to installs/multi-user
* update manifests/kustomize/README.md
* fix image
* fix var replacement
* rm argo readme in manifests
* add notices and licenses for argo 2.12
* feat: upgrade argo images to v2.12.9
* update all refs to argo image version
* add NOTICES generation script
* upgrade argo cli to latest
* fix
* fix
* add license_info.csv back
* make release process safer
* add back third_party/license.txt
* refactor(deployment): move argo manifests to third-party, updates for 2.12.9
* update marketplace snapshots
* set up marketplace presubmit test
* add comment
* Added multi-user pipelines backend
corrected typo
updating code based on review
fixes for pipelines server
reverting this back
* removing unnecessary info logging
* intial work'
* small fixes
* updated tests and how parameter are set
* try to fix test
* check with out adding missing test
* fixed small typo
* test changes
* updated config
* typo
* updated after feedback
* fixed pointer error
* test to add paramter
* moved to init so removed not needed code
* updated further
* updated tests to also check endtime
* clean up test
* fixed failing test
* fixed the expected test results
* added timezone examples
* further clean up
* fixed time format
* Update params.env
* moved location to cronjobscheduler
* clean up
* set env variable to empty
* reverted back
* updated to make magic nbr to constant
* updated the tests with comment
* added comments on cron expressions
* update naming and return types
* updated to UTC as default
* updated with an alpha notice
* [Backend] Return proper error codes for failures during auth
* [Backend] Implement helpers to initialize a SubjectAccessReview client
In preparation of SubjectAccessReview, we implement some helpers to
create a new Kubernetes Authorization clientset and return the
SubjectAccessReview client.
We also define some fake clients to be used by future tests.
* [Backend] Introduce RBAC-related constants
In preparation of SubjectAccessReview, introduce RBAC groups, resources,
and verbs.
* [Backend] Extend managers with a SubjectAccessReviewClient
* [Backend] Refactor the authorization mechanism for requests
Authorization should be based on performing some action on a resource
living in a namespace. This commit refactors the authorization utilities
to reflect this and perform SubjectAccessReview.
This commit also deletes some tests based on old authn/authz mechanism.
A following commit will fix/extend the tests for the new mechanism
* [Backend] Adjust endpoints to pass resource attributes for authz
With KFAM authorization, we passed only the namespace attribute for
authorization. With SubjectAccessReview, we need a richer list of
attributes. Thus, we adjust endpoints to pass request details (resource
attributes) necessary for authorizing the request. We only change the
already authorized endpoints, not introducing any new checks.
* [Backend] Adjust apiserver/server tests to SubjectAccessReview
* [Backend] Purge KFAM
Since we no longer use KFAM, we may as well purge it
* [Backend] Update BUILD files
Signed-off-by: Ilias Katsakioris <elikatsis@arrikto.com>
* [Manifests] Extend manifests for SubjectAccessReview
* API Server: Allow creating SubjectAccessReviews
* Add view/edit roles in a multi-user kustomization
* New server API: read run log
- The new server API endpoint (/apis/v1beta1/runs/{run_id}/nodes/{node_id}/log) to fetch run log
- `ARCHIVE_LOG_FILE_NAME` and `ARCHIVE_LOG_PATH_PREFIX` options allows to control archive log path
- UI Server fetches logs from server API or directly from k8s depending on `STREAM_LOGS_FROM_SERVER_API` option
* New server API: read run log
- ml-pipeline rbac update: allow for access to log
* Read run log: enhanced error handling
- log message on Pod access errors
* Read run log: enhanced log archive options
* Code format
* Test update after getPodLogs signature change
* Updated comments after review
* `follow` query parameter in GET /apis/v1beta1/runs/{run_id}/nodes/{node_id}/log
* Env variable friendly config names & comments
- Config options: ARCHIVE_CONFIG_LOG_FILE_NAME, ARCHIVE_CONFIG_LOG_PATH_PREFIX
- Copyright message update
- New endpoint as `v1alpha1`
* Licence updates
- fluent-bit licence inlined
- copyright message updates
* Master merge
- dependency conflicts
* update to fetch remote
* missed to add the description
* fixed merge conflict
* initial work
* fixed test and bug
* updated python client
* clean up
* clean up
* added config default
* fixed bug in API
* moved config value
* reverted to load from config
* clean up
* Update _client.py
* removed unecessary function and updated after feedback
* missed to save pipeline.proto
* updated the last parts after feedback
* reverted back to use string and env variable
* updated typo
* fix typo in path
* clean up
* removed option in api
* clean up python part
* typo, cant run test locally
* clean up, problems with local env
* clean up missing differences
* reverted proto files
* further clean up
* clean up
* updated after feedback
* Added tests
* error in my defer statement
* Updated the test
* enable pagination when expanding experiment in both the home page and the archive page
* Revert "enable pagination when expanding experiment in both the home page and the archive page"
This reverts commit 5b672739dd.
* grafana
* add grafana directory to kust
* grafana deployment config
* enable pagination when expanding experiment in both the home page and the archive page
* Revert "enable pagination when expanding experiment in both the home page and the archive page"
This reverts commit 5b672739dd.
* Add prometheus deployment as an optional deployment
* move prom dir to under third-party
* comments
* third party folder to the kustomize folder
* Backend - Cache - Fixed reinstallation by adding missing roles
* Stop ignoring the deletion errors
* Added patch permission as well
It should not be triggered, but might be useful in the future.
Add containerRuntimeExecutor explicit type on the configMap
configuration. Set `docker` as default.
Add platform-agnostic type configuration needed by Kind, K3s and
Minikube.
Part of https://github.com/kubeflow/pipelines/issues/4256
* feat(deployment): marketplace - allow specifying gcs bucket directly
* Switch tfx default bucket to user specified one
* Update schema description
* Update version to 0.5.1 to match marketplace expectation
* Fix gcsBucketName var
* Remove gcp secret credentials
* refactor(deployment): separate metadata-writer and metadata-grpc folders
* refactor(deployment): move kustomization.yaml images to the lowest level package
* format
* refactor(manifests): move minio artifact secret to minio package
* let api server and ui use minio artifact secret instead of default value
* Update kustomization.yaml
* fix name
* reduce ttl of pesisted final workflow to 1 day
* add comment
* enable pagination when expanding experiment in both the home page and the archive page
* Revert "enable pagination when expanding experiment in both the home page and the archive page"
This reverts commit 5b672739dd.
* Address comments
* Use configMapKeyRef for env vars
* Allow easy customization of cluster-scoped resources namespace
* clean up
* Clean up
* Simplify var replacement with direct configmap value ref
* clean up params.env
* Refactor components/release.sh to provide a new components/release-branch.sh that updates release branch directly
* Release components as version tag instead of commit SHA
* Publish component images in release.cloudbuild.yaml
* Include script that updates version tag for component sdk
* [Manifest] Use kustomize native image transformer to override image
* Revert unintended changes
* Fix kustomization.yaml location
* Fix inverse proxy image
* Add release script for kustomize manifest
* Add release scripts for marketplace manifest and sdk
* Add global release.sh
* Fix sdk release script
* Clean up release scripts
* Fix release script
* Fix release scripts
* fix
* fix
* Fix cannot use uppercase vars in cloudbuild.yaml
* Add old components release script back
* Add a RELEASE.md doc
* probes for ml-pipeline-ui
* clean up comments
* Use wget instead of curl, because wget is included in alpine
* Also update marketplace manifest
* Add readiness/liveness probe for api server
* Add probes for python vis server
* Upgraded Argo to v2.7.4
* Downgraded the Argo CLI version to 2.4.3
See https://github.com/argoproj/argo/issues/2793
* Removed the argo cli arg that had been removed
* Updated to Argo 2.7.5
* Added workflowtemplates and cronworkflows to the Role
* Added the new Argo CRDs
* Add kfp-container-builder sa
* Allow service account to be configurable
* Fix tests
* Fix test
* Use documentation for service account to introduce compatibility with different types of installation
* updated doc
* clean up
* Update container_builder_test.py
* Update _build_image_api.py
* Update kustomization.yaml
* Add executable permission for presubmit tests mkp.sh
* Initial execution cache
This commit adds initial execution cache service. Including http service
and execution key generation.
* fix master
* Add cache manifests for mkp deployment
* revert go.sum
* Add helm on delete policy for cache deployer job
* Change cache deployer job to statefulset
* remove unnecessary cluster role
* seperate clusterrole and role
* add role and rolebinding to mkp
* change secret role to clusterrole
* Add cloudsql support to cache
* fix comma
* Change cache secret clusterrole to role
* Adjust sequences of resources
* Update values and schema
* remove extra tab
* Change statefulset to job
* Add pod delete permission to cache deployer role
* Test changing cache deployer job to deployment
* remove extra permission
* remove statefulset check
* enable CloudSQL+GCSObjStore without default credential
* refresh document
* fix schema
* minio project ID is required
* fix several
* self throtting Github requests to let build be stable
* can work now
* upsize and lowercase for bucket name
Co-authored-by: Renmin Gu <renming@google.com>
* Initial execution cache
This commit adds initial execution cache service. Including http service
and execution key generation.
* fix master
* Change cache deployer job to stateful set
* Delete cache deployer job
* Delete cache deployer job after it completes
* minor fix
* fix indention
* Change cache deployer job to statefulset
* Remove extra cluster role for cache deployer
* remove cache in base kustomize file for upgrade test
* minor fix
* Enable cache and cache-deployer in base kustomization file
* fix
* fix
* test
* test
* test
* Refactor cluster scope resources
* refactor
* Add namespace for sa
* Fix
* Add crds folder to cluster kustomization yaml
* namespace change
* fix
* fix
* fix
* update test
* Rename cluster to cluster-scoped-resource
* test adding namespace in kustomization file
* revert namespace for clusterrolebinding
* fix
* Add db_name in cache_deployment manifest
* rename
* change secret cluster role to role
Yuan (Bob) Gong