* adding config for SetConnMaxLifetimeSec
making dbConMaxLifetime configurable
nit add time value
change the actual variable
* standardizing and documenting
* switched to getdurationconfig
* initial work on removing pod affinity for cacheing image
* updated env extraction
* check if nodeselector or affinity is set. Also updated the variables needed
* missed to save
* fix test
Signed-off-by: NikeNano <niklas.sven.hansson@gmail.com>
* fixed test
* update manifests
* clean up
* remove withspaces
* metadata-writer: Fix multi-user mode
The metadata-writer should use a different function to watch Pods, if
the namespace is empty, i.e, if it wants to watch Pods in all
namespaces.
Signed-off-by: Yannis Zarkadas <yanniszark@arrikto.com>
* fixup! metadata-writer: Fix multi-user mode
* Caching - Make the cache key depend on output names and paths
Output artifact/parameter names and paths are important and need to be considered.
* Fixed the test
* Introduce kubernetes client utils
Introduce common utils for client initialization to factor out common
code.
This is a step towards fulfilling kubeflow/pipelines#4738.
* Use common util to initialize k8s clientset
* Introduce TokenReview client and fake ones
* Extend ResourceManager with a TokenReview client
* Extend FakeClientManager with a fake TokenReview Client
* Introduce authentication utils
* Introduce HTTP header authenticator
* Initialize Kubeflow-UserID header authenticator
* Refactor getUserIdentity() to use auth_util
* Move getting user identity logic to resource manager
Have the resource manager authenticate the request.
In following commits we will be extending the authentication methods to
use, among others, Kubernetes clients. Thus, we move the logic to the
resource manager to benefit from the clients kept its context.
* Introduce constants for the TokenReview authentication
* Introduce TokenReview authenticator
* Extend authenticators with a TokenReview one
Extend the authenticators which the KFP apiserver applies on a request
with a TokenReview authenticator.
This authenticator expects a ServiceAccountToken in a header with the
format: 'Authorization: Bearer <token>'
Part of https://github.com/kubeflow/pipelines/issues/5138
* Add tests for auth_util
* Add tests for HTTPHeaderAuthenticator
* Update server tests based on the new authentication API
* Remove old tests and unused code
* Add tests for TokenReviewAuthenticator
* Add server tests with unauthenticated requests
* manifests: Allow KFP API server to create TokenReviews
* auth: Split 'auth_util.go' into two parts
Split the file into:
* auth.go: contains the main entrance from the outside of the package
* util.go: contains all utility functions used inside
* Change token review audience variable and value
* Allow configuring audience with an environment variable
* Rename IsRequestAuthenticated -> AuthenticateRequest
* Don't use AuthenticateRequest method in tests
Instead of using AuthenticateRequest to retrieve the user from the
request and then use it for the expected values, allocate a variable for
the username in the request and use that in the expected values.
This ensures we don't hide potential errors of AuthenticateRequest.
* Change authenticators order
Have the HTTPHeaderAuthenticator first followed by the
TokenReviewAuthenticator
* Move authenticators to a ResourceManager property
To avoid potential race conditions when initializing the Authenticators
variable, we move authenticators to a ResourceManager property and
initialize it along with the initialization of the manager.
* feat(backend): Add qps & burst options in clientset. Fix#5095
* Some minor changes:
* Use comment in client-go as usage string of QPS & burst.
* Use `common.GetXXX()` to retrieve configurations in API server.
* Use a struct to include configurations in cache server.
* Update job_api_test.go to support modified `NewScheduledWorkflowClientOrFatal()`.
* Some minor fixes:
* Create `util.ClientParameters` for configurations needed to create a client.
* Add comments linking to the default value of clientQPS & clientBurst defined in k8s.io/client-go.
* set up cluster
* allow to set with env variable
* fix the market place
* updated manifests
* Added default, still need to fix how env is set in test
* test alpine agin
* added test
* updated the image
* deleted
* change image by misstake
* updated after feedback
* deleted
* smaller image
* added to the config.json
* adjust to new updates for config handling
* Updated image to use latest
* [Backend] Move ParameterFormatter to common/util
Signed-off-by: Stefano Fioravanzo <stefano@arrikto.com>
* [Backend] Make resource mngr use params formatter for simple Runs
This commit integrates the workflow resource manager with the
ParameterFormatter, to replace two macros: ``[[RunUUID]]`` (which
replaces with the run UUID) and ``[[CurrentTime.<template>]]`` which
replaces with the time when the workflow is submitted to the
API Server.
Signed-off-by: Stefano Fioravanzo <stefano@arrikto.com>
* [Backend] Validate param pointer is not null
Pipelines can be created with parameters without default values. If such
a pipeline is run without providing a value to these parameters that
don't have defaults, `param.Value` will be `nil`
Signed-off-by: Stefano Fioravanzo <stefano@arrikto.com>
* Move -1 placeholder to constant
Signed-off-by: Stefano Fioravanzo <stefano@arrikto.com>
* fixup! [Backend] Make resource mngr use params formatter for simple Runs
* add notices and licenses for argo 2.12
* feat: upgrade argo images to v2.12.9
* update all refs to argo image version
* add NOTICES generation script
* upgrade argo cli to latest
* fix
* fix
* add license_info.csv back
* make release process safer
* add back third_party/license.txt
* refactor(deployment): move argo manifests to third-party, updates for 2.12.9
* update marketplace snapshots
* set up marketplace presubmit test
* add comment
* tests
* added go mod file
* updated go.mod
* argo latest stable
* upgrade argo
* clean up
* go mod tidy to clean up
* fixed test after backend
* go mod tidy clean up
* more clean up
* added helper function and updated after feedback
* updated k8s.io/kubernetes to version 0.17.9
* updated go dependencies
* Added multi-user pipelines backend
corrected typo
updating code based on review
fixes for pipelines server
reverting this back
* removing unnecessary info logging
* updated to remove bazel
* rename@
* script to run in container
* update the generation
* updated docker image
* changed name
* updated the code
* regenerated API
* fix env variables
* updated version of go swagger
* rerun generate with updated package
* added healthz
* typo with folders fixed
* changed version
* set version of protoc compiler
* test if version is correct
* test version
* changed version
* updated version agian
* test version
* the latest test
* updated docker image
* fixed some stuff
* new test
* tested other version
* new tests
* changed swagger
* new test
* updated versions
* missed docker file
* updated files@
* change back
* updated after feedback@
* clean up
* remove license and install binary
* update and check differences
* completed rebase
* go mod tidy
* updated based upon work of boby
* remove code commited by misstake
* added by misstake
* updated after feedback
* futher updates after feedback
* final updates from feedback
* switch to Makefile
* update
* fix non-root execution
* clean up, fix release script
* fix swagger
* fix healthz endpoint & regenerate python client
* Delete sample.py.tar.gz
Co-authored-by: Yuan Gong <gongyuan94@gmail.com>
Co-authored-by: Yuan (Bob) Gong <4957653+Bobgy@users.noreply.github.com>
* chore(backend): tidy go.mod and update tools.go
* go install, instead of go get
* fix problems reported by go vet
* simplify some ide reported redundant syntax
* license is not required for generated code
* remove licenses for generated code
* cleanup
* remove license more
* rm unused BUILD.bazel files
* fixed generate_api.sh
* reimport error.proto
* chore(backend): tidy go.mod and update tools.go
* go install, instead of go get
* fix problems reported by go vet
* simplify some ide reported redundant syntax
* Changes:
* add kfp v2 hello world sample
* generate go proto code
* code to unmarshal via jsonpb
* generate container spec
* add root DAG task and Makefile to run in argo
* env setup to quickly build driver container
* generate mlmd grpc golang client
* mlmd grpc client usage example
* driver-dag: put execution and context
* driver dag: put context only when execution is a DAG
* driver task: resolve parameter from parent execution
* chain dag driver with task driver
* driver: output argo parameters
* driver: build driver docker image
* driver: push image to dev registry
* compiler: root dag driver compiler
* mlmd doc
* driver: use task spec and executor spec as driver input instead
* driver: pod spec patch output for type EXECUTOR
* compiler: passing - parameter placeholder with hello world example
* include generated argo workflow in source control
* driver no longer outputs parameters
* publisher: publish parameters to execution custom properties
* driver, publisher: execution state
* publisher: remove built binary from repo
* sample: add producer consumer sample
* sample: producer consumer sample with only parameters
* e2e output parameter support
* e2e: driver resolves input parameter from tasks in the same DAG
* compiler: convert KFP task dependency to argo task dependency
* feat: refactor publisher so it can run as an entrypoint
* build: entrypoint image
* feat: executor and publisher in one container via entrypoint rewriting
* fixed compile error (#1)
* add licenses
* update readme
Co-authored-by: capri-xiyue <52932582+capri-xiyue@users.noreply.github.com>
Windfarer
Yuan (Bob) Gong
juliusvonkohout
guanhuichen
Yuan Gong
Alexey Volkov
Luis Magana
capri-xiyue
Niklas Hansson
Yannis Zarkadas
Ilias Katsakioris
Haofeng Xie
StefanoFioravanzo
vsk2015
DavidSpek
uhhc
Rui Fang