mirror of https://github.com/kubeflow/examples.git
added azure pipeline example
Co-Authored-By: Mohona Ahmed <t-moahme@microsoft.com> Co-Authored-By: Maanav Dalal <maanavdalal@gmail.com> Co-Authored-By: sethjuarez <me@sethjuarez.com>
This commit is contained in:
Rebecca McFadden
parent
ac9f2f1238
commit
17ddeb49fb
0
github_issue_summarization/pipelines/components/kubeflow-resources/containers/tf-serving-gh/build.sh
Executable file → Normal file
0
github_issue_summarization/pipelines/components/kubeflow-resources/containers/tf-serving-gh/build.sh
Executable file → Normal file
0
github_issue_summarization/pipelines/components/t2t/containers/base/build.sh
Executable file → Normal file
0
github_issue_summarization/pipelines/components/t2t/containers/base/build.sh
Executable file → Normal file
0
github_issue_summarization/pipelines/components/t2t/containers/t2t_app/build.sh
Executable file → Normal file
0
github_issue_summarization/pipelines/components/t2t/containers/t2t_app/build.sh
Executable file → Normal file
0
github_issue_summarization/pipelines/components/t2t/containers/t2t_proc/build.sh
Executable file → Normal file
0
github_issue_summarization/pipelines/components/t2t/containers/t2t_proc/build.sh
Executable file → Normal file
0
github_issue_summarization/pipelines/components/t2t/containers/t2t_train/build.sh
Executable file → Normal file
0
github_issue_summarization/pipelines/components/t2t/containers/t2t_train/build.sh
Executable file → Normal file
0
github_issue_summarization/pipelines/components/t2t/containers/webapp-launcher/build.sh
Executable file → Normal file
0
github_issue_summarization/pipelines/components/t2t/containers/webapp-launcher/build.sh
Executable file → Normal file
|
|
@ -0,0 +1,18 @@
|
|||
# standard things
|
||||
.vscode
|
||||
.ipynb_checkpoints/
|
||||
__pycache__
|
||||
|
||||
# Environment Variables
|
||||
*.env
|
||||
*.cfg
|
||||
*-creds.yaml
|
||||
|
||||
# models and data
|
||||
data/
|
||||
model/
|
||||
*.tar.gz
|
||||
*.h5
|
||||
*.zip
|
||||
|
||||
aml_config/
|
||||
|
|
@ -0,0 +1,201 @@
|
|||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
APPENDIX: How to apply the Apache License to your work.
|
||||
|
||||
To apply the Apache License to your work, attach the following
|
||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||
replaced with your own identifying information. (Don't include
|
||||
the brackets!) The text should be enclosed in the appropriate
|
||||
comment syntax for the file format. We also recommend that a
|
||||
file or class name and description of purpose be included on the
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright [yyyy] [name of copyright owner]
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
|
|
@ -0,0 +1,5 @@
|
|||
[](https://dev.azure.com/daaronch/Kubeflow%20and%20MLOps/_build/latest?definitionId=3&branchName=master)
|
||||
|
||||
# Kubeflow and Azure Pipelines
|
||||
|
||||
More to come!
|
||||
|
|
@ -0,0 +1 @@
|
|||
-n tacoprofile -m tacosandburritos -i /scripts/inferenceconfig.json -d '{"image":"https://www.exploreveg.org/files/2015/05/sofritas-burrito.jpeg"}' -t 72f988bf-86f1-41af-91ab-2d7cd011db47 -r taco-rg -w taco-workspace -s 2991e3e4-4e8d-451f-aa17-640d89c63144 -p xP2a6Bp1vSOjKholxl5bLPd.=r@ZZJn[ -u 1308a130-d549-44e1-ba66-ce8c487d76e3
|
||||
|
|
@ -0,0 +1,87 @@
|
|||
# Kubeflow Container Build Pipeline
|
||||
|
||||
trigger:
|
||||
- master
|
||||
|
||||
pr: none
|
||||
|
||||
stages:
|
||||
- stage: ContainerConfig
|
||||
displayName: 'Configure and Register Containers'
|
||||
jobs:
|
||||
- job: Containers
|
||||
pool:
|
||||
name: default
|
||||
steps:
|
||||
- task: AzureCLI@1
|
||||
inputs:
|
||||
azureSubscription: 'Shared Data Platform - R+D (1308a130-d549-44e1-ba66-ce8c487d76e3)'
|
||||
scriptLocation: 'inlineScript'
|
||||
inlineScript: '
|
||||
sudo az acr login -n kubeflowregistry &&
|
||||
cd code &&
|
||||
cd preprocess &&
|
||||
sudo docker build -t kubeflowregistry.azurecr.io/kubeflow/preprocess:$BUILD_SOURCEVERSION . &&
|
||||
sudo docker push kubeflowregistry.azurecr.io/kubeflow/preprocess:$BUILD_SOURCEVERSION '
|
||||
displayName: 'Build & Push Preprocess Image'
|
||||
|
||||
- task: AzureCLI@1
|
||||
inputs:
|
||||
azureSubscription: 'Shared Data Platform - R+D (1308a130-d549-44e1-ba66-ce8c487d76e3)'
|
||||
scriptLocation: 'inlineScript'
|
||||
inlineScript: '
|
||||
cd code &&
|
||||
cd training &&
|
||||
sudo docker build -t kubeflowregistry.azurecr.io/kubeflow/training:$BUILD_SOURCEVERSION . &&
|
||||
sudo docker push kubeflowregistry.azurecr.io/kubeflow/training:$BUILD_SOURCEVERSION '
|
||||
displayName: 'Build & Push Training Image'
|
||||
|
||||
- task: AzureCLI@1
|
||||
inputs:
|
||||
azureSubscription: 'Shared Data Platform - R+D (1308a130-d549-44e1-ba66-ce8c487d76e3)'
|
||||
scriptLocation: 'inlineScript'
|
||||
inlineScript: '
|
||||
cd code &&
|
||||
cd register &&
|
||||
sudo docker build -t kubeflowregistry.azurecr.io/kubeflow/register:$BUILD_SOURCEVERSION . &&
|
||||
sudo docker push kubeflowregistry.azurecr.io/kubeflow/register:$BUILD_SOURCEVERSION '
|
||||
displayName: 'Build & Push Register Image'
|
||||
|
||||
|
||||
# Moved KF step to build
|
||||
- stage: KubeflowTrigger
|
||||
dependsOn: ContainerConfig
|
||||
displayName: 'Trigger Kubeflow Pipeline'
|
||||
variables:
|
||||
- group: kf-variables
|
||||
jobs:
|
||||
- job: Kubeflow
|
||||
pool:
|
||||
name: default
|
||||
steps:
|
||||
- task: AzureCLI@1
|
||||
env:
|
||||
KF_MAPPED_SERVICE_PRINCIPAL_PASSWORD: $(KF_SERVICE_PRINCIPAL_PASSWORD)
|
||||
inputs:
|
||||
azureSubscription: 'Shared Data Platform - R+D (1308a130-d549-44e1-ba66-ce8c487d76e3)'
|
||||
scriptLocation: 'inlineScript'
|
||||
inlineScript: |
|
||||
az aks get-credentials -g kubeflow-mlops-rg -n kubeflow-mlops-cluster
|
||||
kubectl port-forward --namespace kubeflow svc/ml-pipeline 8888:8888 &
|
||||
kubepid=$!
|
||||
|
||||
sudo apt-get install python3-setuptools
|
||||
pip3 install wheel
|
||||
pip3 install kfp
|
||||
|
||||
touch script.py
|
||||
echo "import kfp" >> script.py
|
||||
echo "client = kfp.Client(host='localhost:8888')" >> script.py
|
||||
echo "client.run_pipeline('$KF_EXPERIMENT_ID', 'Run ${BUILD_BUILDID}', params={'imagetag': '${BUILD_SOURCEVERSION}', 'tenant-id': '$KF_TENANT_ID', 'service-principal-id': '$KF_SERVICE_PRINCIPAL_ID', 'service-principal-password': '$KF_MAPPED_SERVICE_PRINCIPAL_PASSWORD', 'subscription-id': '$KF_SUBSCRIPTION_ID', 'resource-group': '$KF_RESOURCE_GROUP', 'workspace': '$KF_WORKSPACE', 'persistent-volume-name': '$KF_PERSISTENT_VOLUME_NAME', 'persistent-volume-path': '$KF_PERSISTENT_VOLUME_PATH', 'data-download': '$KF_DATA_DOWNLOAD', 'epochs': '$KF_EPOCHS', 'batch': '$KF_BATCH', 'learning-rate': '$KF_LEARNING_RATE', 'model-name': '$KF_MODEL_NAME'}, pipeline_id='$KF_PIPELINE_ID')" >> script.py
|
||||
|
||||
cat script.py
|
||||
|
||||
python3 script.py
|
||||
|
||||
kill $kubepid
|
||||
displayName: 'Trigger Kubeflow Pipeline'
|
||||
|
|
@ -0,0 +1,9 @@
|
|||
FROM mcr.microsoft.com/azure-cli
|
||||
RUN az extension add -n azure-cli-ml
|
||||
RUN pip install --upgrade pip
|
||||
COPY deploymentconfig.json /scripts/deploymentconfig.json
|
||||
COPY inferenceconfig.json /scripts/inferenceconfig.json
|
||||
COPY deploy.sh /scripts/deploy.sh
|
||||
COPY score.py /scripts/score.py
|
||||
COPY environment.yml /scripts/environment.yml
|
||||
CMD bash
|
||||
|
|
@ -0,0 +1,4 @@
|
|||
{
|
||||
"computeType": "aks",
|
||||
"ComputeTarget": "aks-cluster"
|
||||
}
|
||||
|
|
@ -0,0 +1,20 @@
|
|||
# az ml model deploy -n tacosandburritos -m tacosandburritos:1 --ic inferenceconfig.json --dc deploymentconfig.json --resource-group taco-rg --workspace-name taco-workspace --overwrite -v
|
||||
#!/bin/sh
|
||||
while getopts "m:n:i:d:s:p:u:r:w:t:b:" option;
|
||||
do
|
||||
case "$option" in
|
||||
m ) MODEL=${OPTARG};;
|
||||
n ) MODEL_NAME=${OPTARG};;
|
||||
i ) INFERENCE_CONFIG=${OPTARG};;
|
||||
d ) DEPLOYMENTCONFIG=${OPTARG};;
|
||||
s ) SERVICE_PRINCIPAL_ID=${OPTARG};;
|
||||
p ) SERVICE_PRINCIPAL_PASSWORD=${OPTARG};;
|
||||
u ) SUBSCRIPTION_ID=${OPTARG};;
|
||||
r ) RESOURCE_GROUP=${OPTARG};;
|
||||
w ) WORKSPACE=${OPTARG};;
|
||||
t ) TENANT_ID=${OPTARG};;
|
||||
b ) BASE_PATH=${OPTARG};;
|
||||
esac
|
||||
done
|
||||
az login --service-principal --username ${SERVICE_PRINCIPAL_ID} --password ${SERVICE_PRINCIPAL_PASSWORD} -t $TENANT_ID
|
||||
az ml model deploy -n $MODEL_NAME -m ${MODEL}:1 --ic $INFERENCE_CONFIG --pi ${BASE_PATH}/model/myprofileresult.json --dc $DEPLOYMENTCONFIG -w $WORKSPACE -g $RESOURCE_GROUP --overwrite -v
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
{
|
||||
"containerResourceRequirements": {
|
||||
"cpu": 2,
|
||||
"memoryInGB": 4
|
||||
},
|
||||
"computeType": "ACI",
|
||||
"enableAppInsights": "True"
|
||||
}
|
||||
|
|
@ -0,0 +1,19 @@
|
|||
# Conda environment specification. The dependencies defined in this file will
|
||||
# be automatically provisioned for runs with userManagedDependencies=False.
|
||||
|
||||
# Details about the Conda environment file format:
|
||||
# https://conda.io/docs/user-guide/tasks/manage-environments.html#create-env-file-manually
|
||||
|
||||
name: project_environment
|
||||
dependencies:
|
||||
# The python interpreter version.
|
||||
# Currently Azure ML only supports 3.5.2 and later.
|
||||
- python=3.6.2
|
||||
|
||||
- pip:
|
||||
# Required packages for AzureML execution, history, and data preparation.
|
||||
- azureml-defaults
|
||||
- numpy
|
||||
- tensorflow==2.0.0-alpha0
|
||||
- Pillow
|
||||
- requests
|
||||
|
|
@ -0,0 +1,4 @@
|
|||
echo "test the deployment with a burrito image"
|
||||
az ml service run -n fooddeployaci -d '{ "image": "https://www.exploreveg.org/files/2015/05/sofritas-burrito.jpeg" }' -w taco-workspace -g taco-rg
|
||||
echo "test the deployment with a taco image"
|
||||
az ml service run -n fooddeployaci -d '{ "image": "https://c1.staticflickr.com/5/4022/4401140214_f489c708f0_b.jpg" }' -w taco-workspace -g taco-rg
|
||||
|
|
@ -0,0 +1,10 @@
|
|||
{
|
||||
"entryScript": "/scripts/score.py",
|
||||
"runtime": "python",
|
||||
"condaFile": "/scripts/environment.yml",
|
||||
"extraDockerfileSteps": null,
|
||||
"sourceDirectory": null,
|
||||
"enableGpu": false,
|
||||
"baseImage": null,
|
||||
"baseImageRegistry": null
|
||||
}
|
||||
|
|
@ -0,0 +1,90 @@
|
|||
import json
|
||||
import time
|
||||
import requests
|
||||
import datetime
|
||||
import numpy as np
|
||||
from PIL import Image
|
||||
from io import BytesIO
|
||||
import tensorflow as tf
|
||||
|
||||
from azureml.core.model import Model
|
||||
|
||||
def init():
|
||||
global model
|
||||
|
||||
try:
|
||||
model_path = Model.get_model_path('tacosandburritos')
|
||||
except:
|
||||
model_path = '/model/latest.h5'
|
||||
|
||||
print('Attempting to load model')
|
||||
model = tf.keras.models.load_model(model_path)
|
||||
model.summary()
|
||||
print('Done!')
|
||||
|
||||
print('Initialized model "{}" at {}'.format(model_path, datetime.datetime.now()))
|
||||
|
||||
def run(raw_data):
|
||||
global model
|
||||
prev_time = time.time()
|
||||
|
||||
post = json.loads(raw_data)
|
||||
img_path = post['image']
|
||||
|
||||
current_time = time.time()
|
||||
|
||||
tensor = process_image(img_path, 160)
|
||||
t = tf.reshape(tensor, [-1, 160, 160, 3])
|
||||
o = model.predict(t, steps=1)#[0][0]
|
||||
print(o)
|
||||
o = o[0][0]
|
||||
inference_time = datetime.timedelta(seconds=current_time - prev_time)
|
||||
payload = {
|
||||
'time': inference_time.total_seconds(),
|
||||
'prediction': 'burrito' if o < 0.5 else 'tacos',
|
||||
'scores': str(o)
|
||||
}
|
||||
|
||||
print('Input ({}), Prediction ({})'.format(post['image'], payload))
|
||||
|
||||
return payload
|
||||
|
||||
def process_image(path, image_size):
|
||||
# Extract image (from web or path)
|
||||
if(path.startswith('http')):
|
||||
response = requests.get(path)
|
||||
img = np.array(Image.open(BytesIO(response.content)))
|
||||
else:
|
||||
img = np.array(Image.open(path))
|
||||
|
||||
img_tensor = tf.convert_to_tensor(img, dtype=tf.float32)
|
||||
#tf.image.decode_jpeg(img_raw, channels=3)
|
||||
img_final = tf.image.resize(img_tensor, [image_size, image_size]) / 255
|
||||
return img_final
|
||||
|
||||
def info(msg, char = "#", width = 75):
|
||||
print("")
|
||||
print(char * width)
|
||||
print(char + " %0*s" % ((-1*width)+5, msg) + char)
|
||||
print(char * width)
|
||||
|
||||
if __name__ == "__main__":
|
||||
images = {
|
||||
'tacos': 'https://c1.staticflickr.com/5/4022/4401140214_f489c708f0_b.jpg',
|
||||
'burrito': 'https://www.exploreveg.org/files/2015/05/sofritas-burrito.jpeg'
|
||||
}
|
||||
|
||||
init()
|
||||
|
||||
for k, v in images.items():
|
||||
print('{} => {}'.format(k, v))
|
||||
|
||||
info('Taco Test')
|
||||
taco = json.dumps({ 'image': images['tacos'] })
|
||||
print(taco)
|
||||
run(taco)
|
||||
|
||||
info('Burrito Test')
|
||||
burrito = json.dumps({ 'image': images['burrito'] })
|
||||
print(burrito)
|
||||
run(burrito)
|
||||
|
|
@ -0,0 +1,9 @@
|
|||
FROM tensorflow/tensorflow:2.0.0a0-gpu-py3
|
||||
RUN pip install azure-cli
|
||||
RUN az extension add -n azure-cli-ml
|
||||
RUN pip install --upgrade pip
|
||||
COPY profile.sh /scripts/profile.sh
|
||||
COPY inferenceconfig.json /scripts/inferenceconfig.json
|
||||
COPY score.py /scripts/score.py
|
||||
COPY environment.yml /scripts/environment.yml
|
||||
ENTRYPOINT bash
|
||||
|
|
@ -0,0 +1,45 @@
|
|||
apiVersion: kfdef.apps.kubeflow.org/v1alpha1
|
||||
kind: KfDef
|
||||
metadata:
|
||||
creationTimestamp: null
|
||||
name: kflow
|
||||
namespace: kubeflow
|
||||
spec:
|
||||
appdir: /home/rebec/kubeflow-and-mlops/code/kflow
|
||||
componentParams:
|
||||
ambassador:
|
||||
- name: ambassadorServiceType
|
||||
value: NodePort
|
||||
components:
|
||||
- ambassador
|
||||
- argo
|
||||
- centraldashboard
|
||||
- jupyter-web-app
|
||||
- katib
|
||||
- metacontroller
|
||||
- notebook-controller
|
||||
- pipeline
|
||||
- pytorch-operator
|
||||
- tensorboard
|
||||
- tf-job-operator
|
||||
packages:
|
||||
- argo
|
||||
- common
|
||||
- examples
|
||||
- gcp
|
||||
- jupyter
|
||||
- katib
|
||||
- metacontroller
|
||||
- modeldb
|
||||
- mpi-job
|
||||
- pipeline
|
||||
- pytorch-job
|
||||
- seldon
|
||||
- tensorboard
|
||||
- tf-serving
|
||||
- tf-training
|
||||
repo: /home/rebec/kubeflow-and-mlops/code/kflow/.cache/v0.5.1/kubeflow
|
||||
useBasicAuth: false
|
||||
useIstio: false
|
||||
version: v0.5.1
|
||||
status: {}
|
||||
|
|
@ -0,0 +1,4 @@
|
|||
/lib
|
||||
/.ksonnet/registries
|
||||
/app.override.yaml
|
||||
/.ks_environment
|
||||
|
|
@ -0,0 +1,76 @@
|
|||
apiVersion: 0.3.0
|
||||
environments:
|
||||
default:
|
||||
destination:
|
||||
namespace: kubeflow
|
||||
server: https://taco-cls-taco-rg-1308a1-e98d0802.hcp.eastus.azmk8s.io:443
|
||||
k8sVersion: v1.14.0
|
||||
path: default
|
||||
kind: ksonnet.io/app
|
||||
libraries:
|
||||
kubeflow/argo:
|
||||
name: argo
|
||||
registry: kubeflow
|
||||
version: ""
|
||||
kubeflow/common:
|
||||
name: common
|
||||
registry: kubeflow
|
||||
version: ""
|
||||
kubeflow/examples:
|
||||
name: examples
|
||||
registry: kubeflow
|
||||
version: ""
|
||||
kubeflow/gcp:
|
||||
name: gcp
|
||||
registry: kubeflow
|
||||
version: ""
|
||||
kubeflow/jupyter:
|
||||
name: jupyter
|
||||
registry: kubeflow
|
||||
version: ""
|
||||
kubeflow/katib:
|
||||
name: katib
|
||||
registry: kubeflow
|
||||
version: ""
|
||||
kubeflow/metacontroller:
|
||||
name: metacontroller
|
||||
registry: kubeflow
|
||||
version: ""
|
||||
kubeflow/modeldb:
|
||||
name: modeldb
|
||||
registry: kubeflow
|
||||
version: ""
|
||||
kubeflow/mpi-job:
|
||||
name: mpi-job
|
||||
registry: kubeflow
|
||||
version: ""
|
||||
kubeflow/pipeline:
|
||||
name: pipeline
|
||||
registry: kubeflow
|
||||
version: ""
|
||||
kubeflow/pytorch-job:
|
||||
name: pytorch-job
|
||||
registry: kubeflow
|
||||
version: ""
|
||||
kubeflow/seldon:
|
||||
name: seldon
|
||||
registry: kubeflow
|
||||
version: ""
|
||||
kubeflow/tensorboard:
|
||||
name: tensorboard
|
||||
registry: kubeflow
|
||||
version: ""
|
||||
kubeflow/tf-serving:
|
||||
name: tf-serving
|
||||
registry: kubeflow
|
||||
version: ""
|
||||
kubeflow/tf-training:
|
||||
name: tf-training
|
||||
registry: kubeflow
|
||||
version: ""
|
||||
name: ks_app
|
||||
registries:
|
||||
kubeflow:
|
||||
protocol: fs
|
||||
uri: /home/rebec/kubeflow-and-mlops/code/kflow/.cache/v0.5.1/kubeflow
|
||||
version: 0.0.1
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
local env = std.extVar("__ksonnet/environments");
|
||||
local params = std.extVar("__ksonnet/params").components.ambassador;
|
||||
|
||||
local ambassador = import "kubeflow/common/ambassador.libsonnet";
|
||||
local instance = ambassador.new(env, params);
|
||||
instance.list(instance.all)
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
local env = std.extVar("__ksonnet/environments");
|
||||
local params = std.extVar("__ksonnet/params").components.argo;
|
||||
|
||||
local argo = import "kubeflow/argo/argo.libsonnet";
|
||||
local instance = argo.new(env, params);
|
||||
instance.list(instance.all)
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
local env = std.extVar("__ksonnet/environments");
|
||||
local params = std.extVar("__ksonnet/params").components.centraldashboard;
|
||||
|
||||
local centraldashboard = import "kubeflow/common/centraldashboard.libsonnet";
|
||||
local instance = centraldashboard.new(env, params);
|
||||
instance.list(instance.all)
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
local env = std.extVar("__ksonnet/environments");
|
||||
local params = std.extVar("__ksonnet/params").components["jupyter-web-app"];
|
||||
|
||||
local jupyter_ui = import "kubeflow/jupyter/jupyter-web-app.libsonnet";
|
||||
|
||||
local instance = jupyter_ui.new(env, params);
|
||||
instance.list(instance.all)
|
||||
|
|
@ -0,0 +1,16 @@
|
|||
local env = std.extVar("__ksonnet/environments");
|
||||
local params = std.extVar("__ksonnet/params").components.katib;
|
||||
|
||||
local k = import "k.libsonnet";
|
||||
|
||||
local studyjobcontroller = import "kubeflow/katib/studyjobcontroller.libsonnet";
|
||||
local suggestion = import "kubeflow/katib/suggestion.libsonnet";
|
||||
local vizier = import "kubeflow/katib/vizier.libsonnet";
|
||||
|
||||
local namespace = env.namespace;
|
||||
|
||||
std.prune(
|
||||
k.core.v1.list.new(vizier.all(params, namespace))
|
||||
+ k.core.v1.list.new(suggestion.all(params, namespace))
|
||||
+ k.core.v1.list.new(studyjobcontroller.all(params, namespace))
|
||||
)
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
local env = std.extVar("__ksonnet/environments");
|
||||
local params = std.extVar("__ksonnet/params").components.metacontroller;
|
||||
|
||||
local metacontroller = import "kubeflow/metacontroller/metacontroller.libsonnet";
|
||||
local instance = metacontroller.new(env, params);
|
||||
instance.list(instance.all)
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
local env = std.extVar("__ksonnet/environments");
|
||||
local params = std.extVar("__ksonnet/params").components["notebook-controller"];
|
||||
|
||||
local notebooks = import "kubeflow/jupyter/notebook_controller.libsonnet";
|
||||
local instance = notebooks.new(env, params);
|
||||
instance.list(instance.all)
|
||||
|
|
@ -0,0 +1,93 @@
|
|||
{
|
||||
global: {},
|
||||
components: {
|
||||
// Component-level parameters, defined initially from 'ks prototype use ...'
|
||||
// Each object below should correspond to a component in the components/ directory
|
||||
ambassador: {
|
||||
ambassadorImage: 'quay.io/datawire/ambassador:0.37.0',
|
||||
ambassadorNodePort: 0,
|
||||
ambassadorServiceType: 'NodePort',
|
||||
name: 'ambassador',
|
||||
platform: 'none',
|
||||
replicas: 3,
|
||||
},
|
||||
argo: {
|
||||
artifactRepositoryAccessKeySecretKey: 'accesskey',
|
||||
artifactRepositoryAccessKeySecretName: 'mlpipeline-minio-artifact',
|
||||
artifactRepositoryBucket: 'mlpipeline',
|
||||
artifactRepositoryEndpoint: 'minio-service.kubeflow:9000',
|
||||
artifactRepositoryInsecure: 'true',
|
||||
artifactRepositoryKeyPrefix: 'artifacts',
|
||||
artifactRepositorySecretKeySecretKey: 'secretkey',
|
||||
artifactRepositorySecretKeySecretName: 'mlpipeline-minio-artifact',
|
||||
executorImage: 'argoproj/argoexec:v2.2.0',
|
||||
name: 'argo',
|
||||
uiImage: 'argoproj/argoui:v2.2.0',
|
||||
workflowControllerImage: 'argoproj/workflow-controller:v2.2.0',
|
||||
},
|
||||
centraldashboard: {
|
||||
image: 'gcr.io/kubeflow-images-public/centraldashboard:v0.5.0',
|
||||
name: 'centraldashboard',
|
||||
},
|
||||
"jupyter-web-app": {
|
||||
image: 'gcr.io/kubeflow-images-public/jupyter-web-app:v0.5.0',
|
||||
name: 'jupyter-web-app',
|
||||
policy: 'Always',
|
||||
port: '80',
|
||||
prefix: 'jupyter',
|
||||
rokSecretName: 'secret-rok-{username}',
|
||||
ui: 'default',
|
||||
},
|
||||
katib: {
|
||||
katibUIImage: 'gcr.io/kubeflow-images-public/katib/katib-ui:v0.1.2-alpha-156-g4ab3dbd',
|
||||
metricsCollectorImage: 'gcr.io/kubeflow-images-public/katib/metrics-collector:v0.1.2-alpha-156-g4ab3dbd',
|
||||
name: 'katib',
|
||||
studyJobControllerImage: 'gcr.io/kubeflow-images-public/katib/studyjob-controller:v0.1.2-alpha-156-g4ab3dbd',
|
||||
suggestionBayesianOptimizationImage: 'gcr.io/kubeflow-images-public/katib/suggestion-bayesianoptimization:v0.1.2-alpha-156-g4ab3dbd',
|
||||
suggestionGridImage: 'gcr.io/kubeflow-images-public/katib/suggestion-grid:v0.1.2-alpha-156-g4ab3dbd',
|
||||
suggestionHyperbandImage: 'gcr.io/kubeflow-images-public/katib/suggestion-hyperband:v0.1.2-alpha-156-g4ab3dbd',
|
||||
suggestionRandomImage: 'gcr.io/kubeflow-images-public/katib/suggestion-random:v0.1.2-alpha-156-g4ab3dbd',
|
||||
vizierCoreImage: 'gcr.io/kubeflow-images-public/katib/vizier-core:v0.1.2-alpha-156-g4ab3dbd',
|
||||
vizierCoreRestImage: 'gcr.io/kubeflow-images-public/katib/vizier-core-rest:v0.1.2-alpha-156-g4ab3dbd',
|
||||
vizierDbImage: 'mysql:8.0.3',
|
||||
},
|
||||
metacontroller: {
|
||||
image: 'metacontroller/metacontroller:v0.3.0',
|
||||
name: 'metacontroller',
|
||||
},
|
||||
"notebook-controller": {
|
||||
controllerImage: 'gcr.io/kubeflow-images-public/notebook-controller:v20190401-v0.4.0-rc.1-308-g33618cc9-e3b0c4',
|
||||
injectGcpCredentials: 'true',
|
||||
name: 'notebook-controller',
|
||||
},
|
||||
pipeline: {
|
||||
name: 'pipeline',
|
||||
},
|
||||
"pytorch-operator": {
|
||||
cloud: 'null',
|
||||
deploymentNamespace: 'null',
|
||||
deploymentScope: 'cluster',
|
||||
disks: 'null',
|
||||
name: 'pytorch-operator',
|
||||
pytorchDefaultImage: 'null',
|
||||
pytorchJobImage: 'gcr.io/kubeflow-images-public/pytorch-operator:v0.5.0',
|
||||
},
|
||||
tensorboard: {
|
||||
defaultTbImage: 'tensorflow/tensorflow:1.8.0',
|
||||
logDir: 'logs',
|
||||
name: 'tensorboard',
|
||||
servicePort: 9000,
|
||||
serviceType: 'ClusterIP',
|
||||
targetPort: 6006,
|
||||
},
|
||||
"tf-job-operator": {
|
||||
cloud: 'null',
|
||||
deploymentNamespace: 'null',
|
||||
deploymentScope: 'cluster',
|
||||
name: 'tf-job-operator',
|
||||
tfDefaultImage: 'null',
|
||||
tfJobImage: 'gcr.io/kubeflow-images-public/tf_operator:v0.5.0',
|
||||
tfJobUiServiceType: 'ClusterIP',
|
||||
},
|
||||
},
|
||||
}
|
||||
|
|
@ -0,0 +1,14 @@
|
|||
local env = std.extVar("__ksonnet/environments");
|
||||
local params = std.extVar("__ksonnet/params").components.pipeline;
|
||||
|
||||
local k = import "k.libsonnet";
|
||||
local pipelineBase = import "kubeflow/pipeline/pipeline.libsonnet";
|
||||
|
||||
// updatedParams includes the namespace from env by default.
|
||||
local updatedParams = params + env;
|
||||
|
||||
local pipeline = pipelineBase {
|
||||
params+: updatedParams,
|
||||
};
|
||||
|
||||
std.prune(k.core.v1.list.new(pipeline.parts.all))
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
local env = std.extVar("__ksonnet/environments");
|
||||
local params = std.extVar("__ksonnet/params").components["pytorch-operator"];
|
||||
|
||||
local k = import "k.libsonnet";
|
||||
local operator = import "kubeflow/pytorch-job/pytorch-operator.libsonnet";
|
||||
|
||||
k.core.v1.list.new(operator.all(params, env))
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
local env = std.extVar("__ksonnet/environments");
|
||||
local params = std.extVar("__ksonnet/params").components.tensorboard;
|
||||
|
||||
local tensorboard = import "kubeflow/tensorboard/tensorboard.libsonnet";
|
||||
local instance = tensorboard.new(env, params);
|
||||
instance.list(instance.all)
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
local env = std.extVar("__ksonnet/environments");
|
||||
local params = std.extVar("__ksonnet/params").components["tf-job-operator"];
|
||||
|
||||
local tfJobOperator = import "kubeflow/tf-training/tf-job-operator.libsonnet";
|
||||
local instance = tfJobOperator.new(env, params);
|
||||
instance.list(instance.all)
|
||||
|
|
@ -0,0 +1,4 @@
|
|||
local components = std.extVar("__ksonnet/components");
|
||||
components + {
|
||||
// Insert user-specified overrides here.
|
||||
}
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
{
|
||||
}
|
||||
|
|
@ -0,0 +1,9 @@
|
|||
local base = import "base.libsonnet";
|
||||
// uncomment if you reference ksonnet-lib
|
||||
// local k = import "k.libsonnet";
|
||||
// local deployment = k.apps.v1beta2.deployment;
|
||||
|
||||
base + {
|
||||
// Insert user-specified overrides here. For example if a component is named \"nginx-deployment\", you might have something like:\n")
|
||||
// "nginx-deployment"+: deployment.mixin.metadata.withLabels({foo: "bar"})
|
||||
}
|
||||
|
|
@ -0,0 +1,17 @@
|
|||
local params = std.extVar("__ksonnet/params");
|
||||
local globals = import "globals.libsonnet";
|
||||
local envParams = params + {
|
||||
components +: {
|
||||
// Insert component parameter overrides here. Ex:
|
||||
// guestbook +: {
|
||||
// name: "guestbook-dev",
|
||||
// replicas: params.global.replicas,
|
||||
// },
|
||||
},
|
||||
};
|
||||
|
||||
{
|
||||
components: {
|
||||
[x]: envParams.components[x] + globals, for x in std.objectFields(envParams.components)
|
||||
},
|
||||
}
|
||||
|
|
@ -0,0 +1,3 @@
|
|||
approvers:
|
||||
- IronPan
|
||||
reviewers:
|
||||
|
|
@ -0,0 +1,35 @@
|
|||
# Argo
|
||||
|
||||
> Prototypes for deploying Argo and running Argo Workflows
|
||||
|
||||
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
|
||||
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
|
||||
**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)*
|
||||
|
||||
- [Quickstart](#quickstart)
|
||||
|
||||
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
|
||||
|
||||
## Quickstart
|
||||
|
||||
*The following commands use the `io.ksonnet.pkg.argo` prototype to deploy the Argo Workflow operator on your Kubernetes cluster*
|
||||
|
||||
First, create a cluster and install the ksonnet CLI (see root-level [README.md](../../README.md)).
|
||||
|
||||
If you haven't yet created a [ksonnet application](https://ksonnet.io/docs/tutorial#1-initialize-your-app), do so using `ks init <app-name>`.
|
||||
|
||||
Finally, in the ksonnet application directory, run the following:
|
||||
|
||||
```shell
|
||||
# Install the kubeflow argo package
|
||||
$ ks pkg install kubeflow/argo
|
||||
|
||||
# Expand prototype as a Jsonnet file, place in a file in the
|
||||
# `components/` directory. (YAML and JSON are also available.)
|
||||
$ ks prototype use io.ksonnet.pkg.argo argo \
|
||||
--namespace default \
|
||||
--name argo
|
||||
|
||||
# Apply to server.
|
||||
$ ks apply default -c argo
|
||||
```
|
||||
485
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/argo/argo.libsonnet
vendored
Normal file
485
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/argo/argo.libsonnet
vendored
Normal file
|
|
@ -0,0 +1,485 @@
|
|||
{
|
||||
// TODO(jlewi): Do we need to add parts corresponding to a service account and cluster binding role?
|
||||
// see https://github.com/argoproj/argo/blob/master/cmd/argo/commands/install.go
|
||||
local k = import "k.libsonnet",
|
||||
local util = import "kubeflow/common/util.libsonnet",
|
||||
new(_env, _params):: {
|
||||
local params = _params + _env,
|
||||
|
||||
// CRD's are not namespace scoped; see
|
||||
// https://kubernetes.io/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/
|
||||
local workflowCRD = {
|
||||
apiVersion: "apiextensions.k8s.io/v1beta1",
|
||||
kind: "CustomResourceDefinition",
|
||||
metadata: {
|
||||
name: "workflows.argoproj.io",
|
||||
},
|
||||
spec: {
|
||||
group: "argoproj.io",
|
||||
names: {
|
||||
kind: "Workflow",
|
||||
listKind: "WorkflowList",
|
||||
plural: "workflows",
|
||||
shortNames: [
|
||||
"wf",
|
||||
],
|
||||
singular: "workflow",
|
||||
},
|
||||
scope: "Namespaced",
|
||||
version: "v1alpha1",
|
||||
},
|
||||
}, // crd
|
||||
workflowCRD:: workflowCRD,
|
||||
|
||||
// Deploy the controller
|
||||
local workflowController = {
|
||||
apiVersion: "extensions/v1beta1",
|
||||
kind: "Deployment",
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "workflow-controller",
|
||||
},
|
||||
name: "workflow-controller",
|
||||
namespace: params.namespace,
|
||||
},
|
||||
spec: {
|
||||
progressDeadlineSeconds: 600,
|
||||
replicas: 1,
|
||||
revisionHistoryLimit: 10,
|
||||
selector: {
|
||||
matchLabels: {
|
||||
app: "workflow-controller",
|
||||
},
|
||||
},
|
||||
strategy: {
|
||||
rollingUpdate: {
|
||||
maxSurge: "25%",
|
||||
maxUnavailable: "25%",
|
||||
},
|
||||
type: "RollingUpdate",
|
||||
},
|
||||
template: {
|
||||
metadata: {
|
||||
creationTimestamp: null,
|
||||
labels: {
|
||||
app: "workflow-controller",
|
||||
},
|
||||
},
|
||||
spec: {
|
||||
containers: [
|
||||
{
|
||||
args: [
|
||||
"--configmap",
|
||||
"workflow-controller-configmap",
|
||||
],
|
||||
command: [
|
||||
"workflow-controller",
|
||||
],
|
||||
env: [
|
||||
{
|
||||
name: "ARGO_NAMESPACE",
|
||||
valueFrom: {
|
||||
fieldRef: {
|
||||
apiVersion: "v1",
|
||||
fieldPath: "metadata.namespace",
|
||||
},
|
||||
},
|
||||
},
|
||||
],
|
||||
image: params.workflowControllerImage,
|
||||
imagePullPolicy: "IfNotPresent",
|
||||
name: "workflow-controller",
|
||||
resources: {},
|
||||
terminationMessagePath: "/dev/termination-log",
|
||||
terminationMessagePolicy: "File",
|
||||
},
|
||||
],
|
||||
dnsPolicy: "ClusterFirst",
|
||||
restartPolicy: "Always",
|
||||
schedulerName: "default-scheduler",
|
||||
securityContext: {},
|
||||
serviceAccount: "argo",
|
||||
serviceAccountName: "argo",
|
||||
terminationGracePeriodSeconds: 30,
|
||||
},
|
||||
},
|
||||
},
|
||||
}, // deploy
|
||||
workflowController:: workflowController,
|
||||
|
||||
local argoUI = {
|
||||
apiVersion: "extensions/v1beta1",
|
||||
kind: "Deployment",
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "argo-ui",
|
||||
},
|
||||
name: "argo-ui",
|
||||
namespace: params.namespace,
|
||||
},
|
||||
spec: {
|
||||
progressDeadlineSeconds: 600,
|
||||
replicas: 1,
|
||||
revisionHistoryLimit: 10,
|
||||
selector: {
|
||||
matchLabels: {
|
||||
app: "argo-ui",
|
||||
},
|
||||
},
|
||||
strategy: {
|
||||
rollingUpdate: {
|
||||
maxSurge: "25%",
|
||||
maxUnavailable: "25%",
|
||||
},
|
||||
type: "RollingUpdate",
|
||||
},
|
||||
template: {
|
||||
metadata: {
|
||||
creationTimestamp: null,
|
||||
labels: {
|
||||
app: "argo-ui",
|
||||
},
|
||||
},
|
||||
spec: {
|
||||
containers: [
|
||||
{
|
||||
env: [
|
||||
{
|
||||
name: "ARGO_NAMESPACE",
|
||||
valueFrom: {
|
||||
fieldRef: {
|
||||
apiVersion: "v1",
|
||||
fieldPath: "metadata.namespace",
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "IN_CLUSTER",
|
||||
value: "true",
|
||||
},
|
||||
{
|
||||
name: "BASE_HREF",
|
||||
value: "/argo/",
|
||||
},
|
||||
],
|
||||
image: params.uiImage,
|
||||
imagePullPolicy: "IfNotPresent",
|
||||
name: "argo-ui",
|
||||
resources: {},
|
||||
terminationMessagePath: "/dev/termination-log",
|
||||
terminationMessagePolicy: "File",
|
||||
readinessProbe: {
|
||||
httpGet: {
|
||||
path: "/",
|
||||
port: 8001,
|
||||
},
|
||||
},
|
||||
},
|
||||
],
|
||||
dnsPolicy: "ClusterFirst",
|
||||
restartPolicy: "Always",
|
||||
schedulerName: "default-scheduler",
|
||||
securityContext: {},
|
||||
serviceAccount: "argo-ui",
|
||||
serviceAccountName: "argo-ui",
|
||||
terminationGracePeriodSeconds: 30,
|
||||
},
|
||||
},
|
||||
},
|
||||
}, // deployUi
|
||||
argoUI:: argoUI,
|
||||
|
||||
local argUIService = {
|
||||
apiVersion: "v1",
|
||||
kind: "Service",
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "argo-ui",
|
||||
},
|
||||
name: "argo-ui",
|
||||
namespace: params.namespace,
|
||||
annotations: {
|
||||
"getambassador.io/config":
|
||||
std.join("\n", [
|
||||
"---",
|
||||
"apiVersion: ambassador/v0",
|
||||
"kind: Mapping",
|
||||
"name: argo-ui-mapping",
|
||||
"prefix: /argo/",
|
||||
"service: argo-ui." + params.namespace,
|
||||
]),
|
||||
}, //annotations
|
||||
},
|
||||
spec: {
|
||||
ports: [
|
||||
{
|
||||
port: 80,
|
||||
targetPort: 8001,
|
||||
},
|
||||
],
|
||||
selector: {
|
||||
app: "argo-ui",
|
||||
},
|
||||
sessionAffinity: "None",
|
||||
type: "NodePort",
|
||||
},
|
||||
},
|
||||
argUIService:: argUIService,
|
||||
|
||||
local workflowControllerConfigmap = {
|
||||
apiVersion: "v1",
|
||||
data: {
|
||||
config: std.format(|||
|
||||
{
|
||||
executorImage: %s,
|
||||
artifactRepository:
|
||||
{
|
||||
s3: {
|
||||
bucket: %s,
|
||||
keyPrefix: %s,
|
||||
endpoint: %s,
|
||||
insecure: %s,
|
||||
accessKeySecret: {
|
||||
name: %s,
|
||||
key: %s
|
||||
},
|
||||
secretKeySecret: {
|
||||
name: %s,
|
||||
key: %s
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|||,
|
||||
[
|
||||
params.executorImage,
|
||||
params.artifactRepositoryBucket,
|
||||
params.artifactRepositoryKeyPrefix,
|
||||
params.artifactRepositoryEndpoint,
|
||||
params.artifactRepositoryInsecure,
|
||||
params.artifactRepositoryAccessKeySecretName,
|
||||
params.artifactRepositoryAccessKeySecretKey,
|
||||
params.artifactRepositorySecretKeySecretName,
|
||||
params.artifactRepositorySecretKeySecretKey,
|
||||
]),
|
||||
},
|
||||
kind: "ConfigMap",
|
||||
metadata: {
|
||||
name: "workflow-controller-configmap",
|
||||
namespace: params.namespace,
|
||||
},
|
||||
},
|
||||
workflowControllerConfigmap:: workflowControllerConfigmap,
|
||||
|
||||
local argoServiceAccount = {
|
||||
apiVersion: "v1",
|
||||
kind: "ServiceAccount",
|
||||
metadata: {
|
||||
name: "argo",
|
||||
namespace: params.namespace,
|
||||
},
|
||||
}, // service account
|
||||
argoServiceAccount:: argoServiceAccount,
|
||||
|
||||
// Keep in sync with https://github.com/argoproj/argo/blob/master/cmd/argo/commands/const.go#L20
|
||||
// Permissions need to be cluster wide for the workflow controller to be able to process workflows
|
||||
// in other namespaces. We could potentially use the ConfigMap of the workflow-controller to
|
||||
// scope it to a particular namespace in which case we might be able to restrict the permissions
|
||||
// to a particular namespace.
|
||||
local argoClusterRole = {
|
||||
apiVersion: "rbac.authorization.k8s.io/v1beta1",
|
||||
kind: "ClusterRole",
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "argo",
|
||||
},
|
||||
name: "argo",
|
||||
},
|
||||
rules: [
|
||||
{
|
||||
apiGroups: [""],
|
||||
resources: [
|
||||
"pods",
|
||||
"pods/exec",
|
||||
],
|
||||
verbs: [
|
||||
"create",
|
||||
"get",
|
||||
"list",
|
||||
"watch",
|
||||
"update",
|
||||
"patch",
|
||||
],
|
||||
},
|
||||
{
|
||||
apiGroups: [""],
|
||||
resources: [
|
||||
"configmaps",
|
||||
],
|
||||
verbs: [
|
||||
"get",
|
||||
"watch",
|
||||
"list",
|
||||
],
|
||||
},
|
||||
{
|
||||
apiGroups: [
|
||||
"",
|
||||
],
|
||||
resources: [
|
||||
"persistentvolumeclaims",
|
||||
],
|
||||
verbs: [
|
||||
"create",
|
||||
"delete",
|
||||
],
|
||||
},
|
||||
{
|
||||
apiGroups: [
|
||||
"argoproj.io",
|
||||
],
|
||||
resources: [
|
||||
"workflows",
|
||||
],
|
||||
verbs: [
|
||||
"get",
|
||||
"list",
|
||||
"watch",
|
||||
"update",
|
||||
"patch",
|
||||
],
|
||||
},
|
||||
],
|
||||
}, // operator-role
|
||||
argoClusterRole:: argoClusterRole,
|
||||
|
||||
local argoClusterRoleBinding = {
|
||||
apiVersion: "rbac.authorization.k8s.io/v1beta1",
|
||||
kind: "ClusterRoleBinding",
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "argo",
|
||||
},
|
||||
name: "argo",
|
||||
},
|
||||
roleRef: {
|
||||
apiGroup: "rbac.authorization.k8s.io",
|
||||
kind: "ClusterRole",
|
||||
name: "argo",
|
||||
},
|
||||
subjects: [
|
||||
{
|
||||
kind: "ServiceAccount",
|
||||
name: "argo",
|
||||
namespace: params.namespace,
|
||||
},
|
||||
],
|
||||
}, // role binding
|
||||
argoClusterRoleBinding:: argoClusterRoleBinding,
|
||||
|
||||
local argoUIServiceAccount = {
|
||||
apiVersion: "v1",
|
||||
kind: "ServiceAccount",
|
||||
metadata: {
|
||||
name: "argo-ui",
|
||||
namespace: params.namespace,
|
||||
},
|
||||
}, // service account
|
||||
argoUIServiceAccount:: argoUIServiceAccount,
|
||||
|
||||
// Keep in sync with https://github.com/argoproj/argo/blob/master/cmd/argo/commands/const.go#L44
|
||||
// Permissions need to be cluster wide for the workflow controller to be able to process workflows
|
||||
// in other namespaces. We could potentially use the ConfigMap of the workflow-controller to
|
||||
// scope it to a particular namespace in which case we might be able to restrict the permissions
|
||||
// to a particular namespace.
|
||||
local argoUIRole = {
|
||||
apiVersion: "rbac.authorization.k8s.io/v1beta1",
|
||||
kind: "ClusterRole",
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "argo",
|
||||
},
|
||||
name: "argo-ui",
|
||||
},
|
||||
rules: [
|
||||
{
|
||||
apiGroups: [""],
|
||||
resources: [
|
||||
"pods",
|
||||
"pods/exec",
|
||||
"pods/log",
|
||||
],
|
||||
verbs: [
|
||||
"get",
|
||||
"list",
|
||||
"watch",
|
||||
],
|
||||
},
|
||||
{
|
||||
apiGroups: [""],
|
||||
resources: [
|
||||
"secrets",
|
||||
],
|
||||
verbs: [
|
||||
"get",
|
||||
],
|
||||
},
|
||||
{
|
||||
apiGroups: [
|
||||
"argoproj.io",
|
||||
],
|
||||
resources: [
|
||||
"workflows",
|
||||
],
|
||||
verbs: [
|
||||
"get",
|
||||
"list",
|
||||
"watch",
|
||||
],
|
||||
},
|
||||
],
|
||||
}, // operator-role
|
||||
argoUIRole:: argoUIRole,
|
||||
|
||||
local argUIClusterRoleBinding = {
|
||||
apiVersion: "rbac.authorization.k8s.io/v1beta1",
|
||||
kind: "ClusterRoleBinding",
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "argo-ui",
|
||||
},
|
||||
name: "argo-ui",
|
||||
},
|
||||
roleRef: {
|
||||
apiGroup: "rbac.authorization.k8s.io",
|
||||
kind: "ClusterRole",
|
||||
name: "argo-ui",
|
||||
},
|
||||
subjects: [
|
||||
{
|
||||
kind: "ServiceAccount",
|
||||
name: "argo-ui",
|
||||
namespace: params.namespace,
|
||||
},
|
||||
],
|
||||
}, // role binding
|
||||
argUIClusterRoleBinding:: argUIClusterRoleBinding,
|
||||
|
||||
parts: self,
|
||||
all:: [
|
||||
self.workflowCRD,
|
||||
self.workflowController,
|
||||
self.argoUI,
|
||||
self.argUIService,
|
||||
self.workflowControllerConfigmap,
|
||||
self.argoServiceAccount,
|
||||
self.argoClusterRole,
|
||||
self.argoClusterRoleBinding,
|
||||
self.argoUIServiceAccount,
|
||||
self.argoUIRole,
|
||||
self.argUIClusterRoleBinding,
|
||||
],
|
||||
|
||||
list(obj=self.all):: util.list(obj),
|
||||
},
|
||||
}
|
||||
|
|
@ -0,0 +1,35 @@
|
|||
{
|
||||
"name": "argo",
|
||||
"apiVersion": "0.0.1",
|
||||
"kind": "ksonnet.io/parts",
|
||||
"description": "Prototypes for running Argo workflows.\n",
|
||||
"author": "kubeflow team <kubeflow-team@google.com>",
|
||||
"contributors": [
|
||||
{
|
||||
"name": "Jeremy Lewi",
|
||||
"email": "jlewi@google.com"
|
||||
}
|
||||
],
|
||||
"repository": {
|
||||
"type": "git",
|
||||
"url": "https://github.com/kubeflow/kubeflow"
|
||||
},
|
||||
"bugs": {
|
||||
"url": "https://github.com/kubeflow/kubeflow/issues"
|
||||
},
|
||||
"keywords": [
|
||||
"kubeflow",
|
||||
"argo",
|
||||
"workflows"
|
||||
],
|
||||
"quickStart": {
|
||||
"prototype": "io.ksonnet.pkg.argo",
|
||||
"componentName": "argo",
|
||||
"flags": {
|
||||
"name": "argo",
|
||||
"namespace": "",
|
||||
},
|
||||
"comment": "Deploy Argo"
|
||||
},
|
||||
"license": "Apache 2.0"
|
||||
}
|
||||
20
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/argo/prototypes/argo.jsonnet
vendored
Normal file
20
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/argo/prototypes/argo.jsonnet
vendored
Normal file
|
|
@ -0,0 +1,20 @@
|
|||
// @apiVersion 0.1
|
||||
// @name io.ksonnet.pkg.argo
|
||||
// @description Deploy Argo workflow engine
|
||||
// @shortDescription Argo workflow engine
|
||||
// @param name string Name to give to the component
|
||||
// @optionalParam workflowControllerImage string argoproj/workflow-controller:v2.2.0 workflowControllerImage
|
||||
// @optionalParam uiImage string argoproj/argoui:v2.2.0 uiImage
|
||||
// @optionalParam executorImage string argoproj/argoexec:v2.2.0 executorImage
|
||||
// @optionalParam artifactRepositoryKeyPrefix string artifacts artifactRepositoryKeyPrefix
|
||||
// @optionalParam artifactRepositoryEndpoint string minio-service.kubeflow:9000 artifactRepositoryEndpoint
|
||||
// @optionalParam artifactRepositoryBucket string mlpipeline artifactRepositoryBucket
|
||||
// @optionalParam artifactRepositoryInsecure string true artifactRepositoryInsecure
|
||||
// @optionalParam artifactRepositoryAccessKeySecretName string mlpipeline-minio-artifact artifactRepositoryAccessKeySecretName
|
||||
// @optionalParam artifactRepositoryAccessKeySecretKey string accesskey artifactRepositoryAccessKeySecretKey
|
||||
// @optionalParam artifactRepositorySecretKeySecretName string mlpipeline-minio-artifact artifactRepositorySecretKeySecretName
|
||||
// @optionalParam artifactRepositorySecretKeySecretKey string secretkey artifactRepositorySecretKeySecretKey
|
||||
|
||||
local argo = import "kubeflow/argo/argo.libsonnet";
|
||||
local instance = argo.new(env, params);
|
||||
instance.list(instance.all)
|
||||
452
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/argo/tests/argo_test.jsonnet
vendored
Normal file
452
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/argo/tests/argo_test.jsonnet
vendored
Normal file
|
|
@ -0,0 +1,452 @@
|
|||
local argo = import "kubeflow/argo/argo.libsonnet";
|
||||
local testSuite = import "kubeflow/common/testsuite.libsonnet";
|
||||
|
||||
local params = {
|
||||
name: "argo",
|
||||
workflowControllerImage: "argoproj/workflow-controller:v2.2.0",
|
||||
uiImage: "argoproj/argoui:v2.2.0",
|
||||
executorImage: "argoproj/argoexec:v2.2.0",
|
||||
artifactRepositoryKeyPrefix: "artifacts",
|
||||
artifactRepositoryEndpoint: "minio-service.kubeflow:9000",
|
||||
artifactRepositoryBucket: "mlpipeline",
|
||||
artifactRepositoryInsecure: "true",
|
||||
artifactRepositoryAccessKeySecretName: "mlpipeline-minio-artifact",
|
||||
artifactRepositoryAccessKeySecretKey: "accesskey",
|
||||
artifactRepositorySecretKeySecretName: "mlpipeline-minio-artifact",
|
||||
artifactRepositorySecretKeySecretKey: "secretkey",
|
||||
};
|
||||
local env = {
|
||||
namespace: "kubeflow",
|
||||
};
|
||||
|
||||
local instance = argo.new(env, params);
|
||||
|
||||
local testCases = [
|
||||
{
|
||||
actual: instance.parts.workflowCRD,
|
||||
expected: {
|
||||
apiVersion: "apiextensions.k8s.io/v1beta1",
|
||||
kind: "CustomResourceDefinition",
|
||||
metadata: {
|
||||
name: "workflows.argoproj.io",
|
||||
},
|
||||
spec: {
|
||||
group: "argoproj.io",
|
||||
names: {
|
||||
kind: "Workflow",
|
||||
listKind: "WorkflowList",
|
||||
plural: "workflows",
|
||||
shortNames: [
|
||||
"wf",
|
||||
],
|
||||
singular: "workflow",
|
||||
},
|
||||
scope: "Namespaced",
|
||||
version: "v1alpha1",
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
actual: instance.parts.workflowController,
|
||||
expected: {
|
||||
apiVersion: "extensions/v1beta1",
|
||||
kind: "Deployment",
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "workflow-controller",
|
||||
},
|
||||
name: "workflow-controller",
|
||||
namespace: "kubeflow",
|
||||
},
|
||||
spec: {
|
||||
progressDeadlineSeconds: 600,
|
||||
replicas: 1,
|
||||
revisionHistoryLimit: 10,
|
||||
selector: {
|
||||
matchLabels: {
|
||||
app: "workflow-controller",
|
||||
},
|
||||
},
|
||||
strategy: {
|
||||
rollingUpdate: {
|
||||
maxSurge: "25%",
|
||||
maxUnavailable: "25%",
|
||||
},
|
||||
type: "RollingUpdate",
|
||||
},
|
||||
template: {
|
||||
metadata: {
|
||||
creationTimestamp: null,
|
||||
labels: {
|
||||
app: "workflow-controller",
|
||||
},
|
||||
},
|
||||
spec: {
|
||||
containers: [
|
||||
{
|
||||
args: [
|
||||
"--configmap",
|
||||
"workflow-controller-configmap",
|
||||
],
|
||||
command: [
|
||||
"workflow-controller",
|
||||
],
|
||||
env: [
|
||||
{
|
||||
name: "ARGO_NAMESPACE",
|
||||
valueFrom: {
|
||||
fieldRef: {
|
||||
apiVersion: "v1",
|
||||
fieldPath: "metadata.namespace",
|
||||
},
|
||||
},
|
||||
},
|
||||
],
|
||||
image: "argoproj/workflow-controller:v2.2.0",
|
||||
imagePullPolicy: "IfNotPresent",
|
||||
name: "workflow-controller",
|
||||
resources: {},
|
||||
terminationMessagePath: "/dev/termination-log",
|
||||
terminationMessagePolicy: "File",
|
||||
},
|
||||
],
|
||||
dnsPolicy: "ClusterFirst",
|
||||
restartPolicy: "Always",
|
||||
schedulerName: "default-scheduler",
|
||||
securityContext: {},
|
||||
serviceAccount: "argo",
|
||||
serviceAccountName: "argo",
|
||||
terminationGracePeriodSeconds: 30,
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
actual: instance.parts.argoUI,
|
||||
expected: {
|
||||
apiVersion: "extensions/v1beta1",
|
||||
kind: "Deployment",
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "argo-ui",
|
||||
},
|
||||
name: "argo-ui",
|
||||
namespace: "kubeflow",
|
||||
},
|
||||
spec: {
|
||||
progressDeadlineSeconds: 600,
|
||||
replicas: 1,
|
||||
revisionHistoryLimit: 10,
|
||||
selector: {
|
||||
matchLabels: {
|
||||
app: "argo-ui",
|
||||
},
|
||||
},
|
||||
strategy: {
|
||||
rollingUpdate: {
|
||||
maxSurge: "25%",
|
||||
maxUnavailable: "25%",
|
||||
},
|
||||
type: "RollingUpdate",
|
||||
},
|
||||
template: {
|
||||
metadata: {
|
||||
creationTimestamp: null,
|
||||
labels: {
|
||||
app: "argo-ui",
|
||||
},
|
||||
},
|
||||
spec: {
|
||||
containers: [
|
||||
{
|
||||
env: [
|
||||
{
|
||||
name: "ARGO_NAMESPACE",
|
||||
valueFrom: {
|
||||
fieldRef: {
|
||||
apiVersion: "v1",
|
||||
fieldPath: "metadata.namespace",
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "IN_CLUSTER",
|
||||
value: "true",
|
||||
},
|
||||
{
|
||||
name: "BASE_HREF",
|
||||
value: "/argo/",
|
||||
},
|
||||
],
|
||||
image: "argoproj/argoui:v2.2.0",
|
||||
imagePullPolicy: "IfNotPresent",
|
||||
name: "argo-ui",
|
||||
readinessProbe: {
|
||||
httpGet: {
|
||||
path: "/",
|
||||
port: 8001,
|
||||
},
|
||||
},
|
||||
resources: {},
|
||||
terminationMessagePath: "/dev/termination-log",
|
||||
terminationMessagePolicy: "File",
|
||||
},
|
||||
],
|
||||
dnsPolicy: "ClusterFirst",
|
||||
restartPolicy: "Always",
|
||||
schedulerName: "default-scheduler",
|
||||
securityContext: {},
|
||||
serviceAccount: "argo-ui",
|
||||
serviceAccountName: "argo-ui",
|
||||
terminationGracePeriodSeconds: 30,
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
actual: instance.parts.argUIService,
|
||||
expected: {
|
||||
apiVersion: "v1",
|
||||
kind: "Service",
|
||||
metadata: {
|
||||
annotations: {
|
||||
"getambassador.io/config": "---\napiVersion: ambassador/v0\nkind: Mapping\nname: argo-ui-mapping\nprefix: /argo/\nservice: argo-ui.kubeflow",
|
||||
},
|
||||
labels: {
|
||||
app: "argo-ui",
|
||||
},
|
||||
name: "argo-ui",
|
||||
namespace: "kubeflow",
|
||||
},
|
||||
spec: {
|
||||
ports: [
|
||||
{
|
||||
port: 80,
|
||||
targetPort: 8001,
|
||||
},
|
||||
],
|
||||
selector: {
|
||||
app: "argo-ui",
|
||||
},
|
||||
sessionAffinity: "None",
|
||||
type: "NodePort",
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
actual: instance.parts.workflowControllerConfigmap,
|
||||
expected: {
|
||||
apiVersion: "v1",
|
||||
data: {
|
||||
config: "{\nexecutorImage: argoproj/argoexec:v2.2.0,\nartifactRepository:\n{\n s3: {\n bucket: mlpipeline,\n keyPrefix: artifacts,\n endpoint: minio-service.kubeflow:9000,\n insecure: true,\n accessKeySecret: {\n name: mlpipeline-minio-artifact,\n key: accesskey\n },\n secretKeySecret: {\n name: mlpipeline-minio-artifact,\n key: secretkey\n }\n }\n}\n}\n",
|
||||
},
|
||||
kind: "ConfigMap",
|
||||
metadata: {
|
||||
name: "workflow-controller-configmap",
|
||||
namespace: "kubeflow",
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
actual: instance.parts.argoServiceAccount,
|
||||
expected: {
|
||||
apiVersion: "v1",
|
||||
kind: "ServiceAccount",
|
||||
metadata: {
|
||||
name: "argo",
|
||||
namespace: "kubeflow",
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
actual: instance.parts.argoClusterRole,
|
||||
expected: {
|
||||
apiVersion: "rbac.authorization.k8s.io/v1beta1",
|
||||
kind: "ClusterRole",
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "argo",
|
||||
},
|
||||
name: "argo",
|
||||
},
|
||||
rules: [
|
||||
{
|
||||
apiGroups: [
|
||||
"",
|
||||
],
|
||||
resources: [
|
||||
"pods",
|
||||
"pods/exec",
|
||||
],
|
||||
verbs: [
|
||||
"create",
|
||||
"get",
|
||||
"list",
|
||||
"watch",
|
||||
"update",
|
||||
"patch",
|
||||
],
|
||||
},
|
||||
{
|
||||
apiGroups: [
|
||||
"",
|
||||
],
|
||||
resources: [
|
||||
"configmaps",
|
||||
],
|
||||
verbs: [
|
||||
"get",
|
||||
"watch",
|
||||
"list",
|
||||
],
|
||||
},
|
||||
{
|
||||
apiGroups: [
|
||||
"",
|
||||
],
|
||||
resources: [
|
||||
"persistentvolumeclaims",
|
||||
],
|
||||
verbs: [
|
||||
"create",
|
||||
"delete",
|
||||
],
|
||||
},
|
||||
{
|
||||
apiGroups: [
|
||||
"argoproj.io",
|
||||
],
|
||||
resources: [
|
||||
"workflows",
|
||||
],
|
||||
verbs: [
|
||||
"get",
|
||||
"list",
|
||||
"watch",
|
||||
"update",
|
||||
"patch",
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
actual: instance.parts.argoClusterRoleBinding,
|
||||
expected: {
|
||||
apiVersion: "rbac.authorization.k8s.io/v1beta1",
|
||||
kind: "ClusterRoleBinding",
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "argo",
|
||||
},
|
||||
name: "argo",
|
||||
},
|
||||
roleRef: {
|
||||
apiGroup: "rbac.authorization.k8s.io",
|
||||
kind: "ClusterRole",
|
||||
name: "argo",
|
||||
},
|
||||
subjects: [
|
||||
{
|
||||
kind: "ServiceAccount",
|
||||
name: "argo",
|
||||
namespace: "kubeflow",
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
actual: instance.parts.argoUIServiceAccount,
|
||||
expected: {
|
||||
apiVersion: "v1",
|
||||
kind: "ServiceAccount",
|
||||
metadata: {
|
||||
name: "argo-ui",
|
||||
namespace: "kubeflow",
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
actual: instance.parts.argoUIRole,
|
||||
expected: {
|
||||
apiVersion: "rbac.authorization.k8s.io/v1beta1",
|
||||
kind: "ClusterRole",
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "argo",
|
||||
},
|
||||
name: "argo-ui",
|
||||
},
|
||||
rules: [
|
||||
{
|
||||
apiGroups: [
|
||||
"",
|
||||
],
|
||||
resources: [
|
||||
"pods",
|
||||
"pods/exec",
|
||||
"pods/log",
|
||||
],
|
||||
verbs: [
|
||||
"get",
|
||||
"list",
|
||||
"watch",
|
||||
],
|
||||
},
|
||||
{
|
||||
apiGroups: [
|
||||
"",
|
||||
],
|
||||
resources: [
|
||||
"secrets",
|
||||
],
|
||||
verbs: [
|
||||
"get",
|
||||
],
|
||||
},
|
||||
{
|
||||
apiGroups: [
|
||||
"argoproj.io",
|
||||
],
|
||||
resources: [
|
||||
"workflows",
|
||||
],
|
||||
verbs: [
|
||||
"get",
|
||||
"list",
|
||||
"watch",
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
actual: instance.parts.argUIClusterRoleBinding,
|
||||
expected: {
|
||||
apiVersion: "rbac.authorization.k8s.io/v1beta1",
|
||||
kind: "ClusterRoleBinding",
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "argo-ui",
|
||||
},
|
||||
name: "argo-ui",
|
||||
},
|
||||
roleRef: {
|
||||
apiGroup: "rbac.authorization.k8s.io",
|
||||
kind: "ClusterRole",
|
||||
name: "argo-ui",
|
||||
},
|
||||
subjects: [
|
||||
{
|
||||
kind: "ServiceAccount",
|
||||
name: "argo-ui",
|
||||
namespace: "kubeflow",
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
testSuite.run(testCases)
|
||||
|
|
@ -0,0 +1,3 @@
|
|||
approvers:
|
||||
- gaocegege
|
||||
reviewers:
|
||||
|
|
@ -0,0 +1,11 @@
|
|||
# common
|
||||
|
||||
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
|
||||
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
|
||||
**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)*
|
||||
|
||||
- [common](#common)
|
||||
|
||||
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
|
||||
|
||||
This ksonnet package contains kubeflow common prototypes such as ambassador, spartakus, etc. You can install this using `ks pkg install kubeflow/common`. `ks prototype list` should list the available prototypes. `ks prototype describe <name>` should describe the prototype.
|
||||
226
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/common/ambassador.libsonnet
vendored
Normal file
226
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/common/ambassador.libsonnet
vendored
Normal file
|
|
@ -0,0 +1,226 @@
|
|||
{
|
||||
local k = import "k.libsonnet",
|
||||
local util = import "kubeflow/common/util.libsonnet",
|
||||
new(_env, _params):: {
|
||||
local params = _params + _env,
|
||||
|
||||
local ambassadorService = {
|
||||
apiVersion: "v1",
|
||||
kind: "Service",
|
||||
metadata: {
|
||||
labels: {
|
||||
service: "ambassador",
|
||||
},
|
||||
name: "ambassador",
|
||||
namespace: params.namespace,
|
||||
},
|
||||
spec: {
|
||||
ports: [
|
||||
{
|
||||
name: "ambassador",
|
||||
port: 80,
|
||||
targetPort: 80,
|
||||
[if (params.ambassadorServiceType == 'NodePort') &&
|
||||
(params.ambassadorNodePort >= 30000) &&
|
||||
(params.ambassadorNodePort <= 32767)
|
||||
then 'nodePort']: params.ambassadorNodePort,
|
||||
},
|
||||
],
|
||||
selector: {
|
||||
service: "ambassador",
|
||||
},
|
||||
type: params.ambassadorServiceType,
|
||||
},
|
||||
}, // service
|
||||
ambassadorService:: ambassadorService,
|
||||
|
||||
local adminService = {
|
||||
apiVersion: "v1",
|
||||
kind: "Service",
|
||||
metadata: {
|
||||
labels: {
|
||||
service: "ambassador-admin",
|
||||
},
|
||||
name: "ambassador-admin",
|
||||
namespace: params.namespace,
|
||||
},
|
||||
spec: {
|
||||
ports: [
|
||||
{
|
||||
name: "ambassador-admin",
|
||||
port: 8877,
|
||||
targetPort: 8877,
|
||||
},
|
||||
],
|
||||
selector: {
|
||||
service: "ambassador",
|
||||
},
|
||||
type: "ClusterIP",
|
||||
},
|
||||
}, // adminService
|
||||
adminService:: adminService,
|
||||
|
||||
local ambassadorRole = {
|
||||
apiVersion: "rbac.authorization.k8s.io/v1beta1",
|
||||
kind: "ClusterRole",
|
||||
metadata: {
|
||||
name: "ambassador",
|
||||
},
|
||||
rules: [
|
||||
{
|
||||
apiGroups: [
|
||||
"",
|
||||
],
|
||||
resources: [
|
||||
"services",
|
||||
],
|
||||
verbs: [
|
||||
"get",
|
||||
"list",
|
||||
"watch",
|
||||
],
|
||||
},
|
||||
{
|
||||
apiGroups: [
|
||||
"",
|
||||
],
|
||||
resources: [
|
||||
"configmaps",
|
||||
],
|
||||
verbs: [
|
||||
"create",
|
||||
"update",
|
||||
"patch",
|
||||
"get",
|
||||
"list",
|
||||
"watch",
|
||||
],
|
||||
},
|
||||
{
|
||||
apiGroups: [
|
||||
"",
|
||||
],
|
||||
resources: [
|
||||
"secrets",
|
||||
],
|
||||
verbs: [
|
||||
"get",
|
||||
"list",
|
||||
"watch",
|
||||
],
|
||||
},
|
||||
],
|
||||
}, // role
|
||||
ambassadorRole:: ambassadorRole,
|
||||
|
||||
local ambassadorServiceAccount = {
|
||||
apiVersion: "v1",
|
||||
kind: "ServiceAccount",
|
||||
metadata: {
|
||||
name: "ambassador",
|
||||
namespace: params.namespace,
|
||||
},
|
||||
}, // serviceAccount
|
||||
ambassadorServiceAccount:: ambassadorServiceAccount,
|
||||
|
||||
local ambassadorRoleBinding = {
|
||||
apiVersion: "rbac.authorization.k8s.io/v1beta1",
|
||||
kind: "ClusterRoleBinding",
|
||||
metadata: {
|
||||
name: "ambassador",
|
||||
},
|
||||
roleRef: {
|
||||
apiGroup: "rbac.authorization.k8s.io",
|
||||
kind: "ClusterRole",
|
||||
name: "ambassador",
|
||||
},
|
||||
subjects: [
|
||||
{
|
||||
kind: "ServiceAccount",
|
||||
name: "ambassador",
|
||||
namespace: params.namespace,
|
||||
},
|
||||
],
|
||||
}, // roleBinding
|
||||
ambassadorRoleBinding:: ambassadorRoleBinding,
|
||||
|
||||
local ambassadorDeployment = {
|
||||
apiVersion: "apps/v1beta1",
|
||||
kind: "Deployment",
|
||||
metadata: {
|
||||
name: "ambassador",
|
||||
namespace: params.namespace,
|
||||
},
|
||||
spec: {
|
||||
replicas: params.replicas,
|
||||
template: {
|
||||
metadata: {
|
||||
labels: {
|
||||
service: "ambassador",
|
||||
},
|
||||
namespace: params.namespace,
|
||||
},
|
||||
spec: {
|
||||
containers: [
|
||||
{
|
||||
env: [
|
||||
{
|
||||
name: "AMBASSADOR_NAMESPACE",
|
||||
valueFrom: {
|
||||
fieldRef: {
|
||||
fieldPath: "metadata.namespace",
|
||||
},
|
||||
},
|
||||
},
|
||||
],
|
||||
image: params.ambassadorImage,
|
||||
name: "ambassador",
|
||||
resources: {
|
||||
limits: {
|
||||
cpu: 1,
|
||||
memory: "400Mi",
|
||||
},
|
||||
requests: {
|
||||
cpu: "200m",
|
||||
memory: "100Mi",
|
||||
},
|
||||
},
|
||||
readinessProbe: {
|
||||
httpGet: {
|
||||
path: "/ambassador/v0/check_ready",
|
||||
port: 8877,
|
||||
},
|
||||
initialDelaySeconds: 30,
|
||||
periodSeconds: 30,
|
||||
},
|
||||
livenessProbe: {
|
||||
httpGet: {
|
||||
path: "/ambassador/v0/check_alive",
|
||||
port: 8877,
|
||||
},
|
||||
initialDelaySeconds: 30,
|
||||
periodSeconds: 30,
|
||||
},
|
||||
},
|
||||
],
|
||||
restartPolicy: "Always",
|
||||
serviceAccountName: "ambassador",
|
||||
},
|
||||
},
|
||||
},
|
||||
}, // deploy
|
||||
ambassadorDeployment:: ambassadorDeployment,
|
||||
|
||||
parts:: self,
|
||||
all:: [
|
||||
self.ambassadorService,
|
||||
self.adminService,
|
||||
self.ambassadorRole,
|
||||
self.ambassadorServiceAccount,
|
||||
self.ambassadorRoleBinding,
|
||||
self.ambassadorDeployment,
|
||||
],
|
||||
|
||||
list(obj=self.all):: util.list(obj),
|
||||
},
|
||||
}
|
||||
197
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/common/basic-auth.libsonnet
vendored
Normal file
197
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/common/basic-auth.libsonnet
vendored
Normal file
|
|
@ -0,0 +1,197 @@
|
|||
{
|
||||
local util = import "kubeflow/common/util.libsonnet",
|
||||
new(_env, _params):: {
|
||||
local params = _params + _env,
|
||||
|
||||
local ui_name = params.name + "-login",
|
||||
|
||||
local authService = {
|
||||
apiVersion: "v1",
|
||||
kind: "Service",
|
||||
metadata: {
|
||||
labels: {
|
||||
app: params.name,
|
||||
},
|
||||
name: params.name,
|
||||
namespace: params.namespace,
|
||||
annotations: {
|
||||
"getambassador.io/config":
|
||||
std.join("\n", [
|
||||
"---",
|
||||
"apiVersion: ambassador/v0",
|
||||
"kind: AuthService",
|
||||
"name: " + params.name,
|
||||
"auth_service: " + params.name + "." + params.namespace + ":8085",
|
||||
'allowed_headers:\n- "x-from-login"',
|
||||
]),
|
||||
}, //annotations
|
||||
},
|
||||
spec: {
|
||||
ports: [
|
||||
{
|
||||
port: 8085,
|
||||
targetPort: 8085,
|
||||
},
|
||||
],
|
||||
selector: {
|
||||
app: params.name,
|
||||
},
|
||||
type: "ClusterIP",
|
||||
},
|
||||
},
|
||||
authService:: authService,
|
||||
|
||||
local authDeployment = {
|
||||
apiVersion: "extensions/v1beta1",
|
||||
kind: "Deployment",
|
||||
metadata: {
|
||||
name: params.name,
|
||||
namespace: params.namespace,
|
||||
|
||||
},
|
||||
spec: {
|
||||
// replicas here should always be 1:
|
||||
// we store auth cookies in memory and we don't support share them among pods.
|
||||
replicas: 1,
|
||||
strategy: {
|
||||
type: "RollingUpdate",
|
||||
},
|
||||
template: {
|
||||
metadata: {
|
||||
labels: {
|
||||
app: params.name,
|
||||
},
|
||||
},
|
||||
spec: {
|
||||
containers: [
|
||||
{
|
||||
image: params.image,
|
||||
name: "app",
|
||||
workingDir: "/opt/kubeflow",
|
||||
env: [
|
||||
{
|
||||
name: "USERNAME",
|
||||
valueFrom: {
|
||||
secretKeyRef: {
|
||||
name: params.authSecretName,
|
||||
key: "username",
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "PASSWORDHASH",
|
||||
valueFrom: {
|
||||
secretKeyRef: {
|
||||
name: params.authSecretName,
|
||||
key: "passwordhash",
|
||||
},
|
||||
},
|
||||
},
|
||||
],
|
||||
command: [
|
||||
"/opt/kubeflow/gatekeeper",
|
||||
],
|
||||
args: [
|
||||
"--username=$(USERNAME)",
|
||||
"--pwhash=$(PASSWORDHASH)",
|
||||
],
|
||||
ports: [
|
||||
{
|
||||
containerPort: 8085,
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
authDeployment:: authDeployment,
|
||||
|
||||
local loginService = {
|
||||
apiVersion: "v1",
|
||||
kind: "Service",
|
||||
metadata: {
|
||||
labels: {
|
||||
app: ui_name,
|
||||
},
|
||||
name: ui_name,
|
||||
namespace: params.namespace,
|
||||
annotations: {
|
||||
"getambassador.io/config":
|
||||
std.join("\n", [
|
||||
"---",
|
||||
"apiVersion: ambassador/v0",
|
||||
"kind: Mapping",
|
||||
"name: kflogin-mapping",
|
||||
"prefix: /kflogin",
|
||||
"rewrite: /kflogin",
|
||||
"timeout_ms: 300000",
|
||||
"service: " + ui_name + "." + params.namespace,
|
||||
"use_websocket: true",
|
||||
]),
|
||||
}, //annotations
|
||||
},
|
||||
spec: {
|
||||
ports: [
|
||||
{
|
||||
port: 80,
|
||||
targetPort: 5000,
|
||||
},
|
||||
],
|
||||
selector: {
|
||||
app: ui_name,
|
||||
},
|
||||
type: "ClusterIP",
|
||||
},
|
||||
},
|
||||
loginService:: loginService,
|
||||
|
||||
local loginDeployment = {
|
||||
apiVersion: "extensions/v1beta1",
|
||||
kind: "Deployment",
|
||||
metadata: {
|
||||
name: ui_name,
|
||||
namespace: params.namespace,
|
||||
|
||||
},
|
||||
spec: {
|
||||
replicas: 1,
|
||||
strategy: {
|
||||
type: "RollingUpdate",
|
||||
},
|
||||
template: {
|
||||
metadata: {
|
||||
labels: {
|
||||
app: ui_name,
|
||||
},
|
||||
},
|
||||
spec: {
|
||||
containers: [
|
||||
{
|
||||
image: params.imageui,
|
||||
name: "app",
|
||||
ports: [
|
||||
{
|
||||
containerPort: 5000,
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
loginDeployment:: loginDeployment,
|
||||
|
||||
parts:: self,
|
||||
all:: [
|
||||
self.authService,
|
||||
self.authDeployment,
|
||||
self.loginService,
|
||||
self.loginDeployment,
|
||||
],
|
||||
|
||||
list(obj=self.all):: util.list(obj),
|
||||
},
|
||||
}
|
||||
221
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/common/centraldashboard.libsonnet
vendored
Normal file
221
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/common/centraldashboard.libsonnet
vendored
Normal file
|
|
@ -0,0 +1,221 @@
|
|||
{
|
||||
local k = import "k.libsonnet",
|
||||
local util = import "kubeflow/common/util.libsonnet",
|
||||
new(_env, _params):: {
|
||||
local params = _params + _env,
|
||||
|
||||
local centralDashboardDeployment = {
|
||||
apiVersion: "extensions/v1beta1",
|
||||
kind: "Deployment",
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "centraldashboard",
|
||||
},
|
||||
name: "centraldashboard",
|
||||
namespace: params.namespace,
|
||||
},
|
||||
spec: {
|
||||
template: {
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "centraldashboard",
|
||||
},
|
||||
},
|
||||
spec: {
|
||||
containers: [
|
||||
{
|
||||
image: params.image,
|
||||
name: "centraldashboard",
|
||||
ports: [
|
||||
{
|
||||
containerPort: 8082,
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
serviceAccountName: "centraldashboard",
|
||||
},
|
||||
},
|
||||
},
|
||||
}, // deployUi
|
||||
centralDashboardDeployment:: centralDashboardDeployment,
|
||||
|
||||
local centralDashboardService = {
|
||||
// Due to https://github.com/ksonnet/ksonnet/issues/670, escaped characters in
|
||||
// jsonnet files are not interpreted correctly by ksonnet, which causes runtime
|
||||
// parsing failures. This is fixed in ksonnet 0.12.0, so we can merge this back
|
||||
// to the jsonnet file when we take a dependency on ksonnet 0.12.0 or later.
|
||||
local annotations = function(namespace) {
|
||||
"getambassador.io/config":
|
||||
std.join("\n", [
|
||||
"---",
|
||||
"apiVersion: ambassador/v0",
|
||||
"kind: Mapping",
|
||||
"name: centralui-mapping",
|
||||
"prefix: /",
|
||||
"rewrite: /",
|
||||
"service: centraldashboard." + namespace,
|
||||
]),
|
||||
},
|
||||
apiVersion: "v1",
|
||||
kind: "Service",
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "centraldashboard",
|
||||
},
|
||||
name: "centraldashboard",
|
||||
namespace: params.namespace,
|
||||
annotations: annotations(params.namespace),
|
||||
},
|
||||
spec: {
|
||||
ports: [
|
||||
{
|
||||
port: 80,
|
||||
targetPort: 8082,
|
||||
},
|
||||
],
|
||||
selector: {
|
||||
app: "centraldashboard",
|
||||
},
|
||||
sessionAffinity: "None",
|
||||
type: "ClusterIP",
|
||||
},
|
||||
}, //service
|
||||
centralDashboardService:: centralDashboardService,
|
||||
|
||||
local centralDashboardServiceAccount = {
|
||||
apiVersion: "v1",
|
||||
kind: "ServiceAccount",
|
||||
metadata: {
|
||||
name: "centraldashboard",
|
||||
namespace: params.namespace,
|
||||
},
|
||||
}, // service account
|
||||
centralDashboardServiceAccount:: centralDashboardServiceAccount,
|
||||
|
||||
local centralDashboardRole = {
|
||||
apiVersion: "rbac.authorization.k8s.io/v1beta1",
|
||||
kind: "Role",
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "centraldashboard",
|
||||
},
|
||||
name: "centraldashboard",
|
||||
namespace: params.namespace,
|
||||
},
|
||||
rules: [
|
||||
{
|
||||
apiGroups: [""],
|
||||
resources: [
|
||||
"pods",
|
||||
"pods/exec",
|
||||
"pods/log",
|
||||
],
|
||||
verbs: [
|
||||
"get",
|
||||
"list",
|
||||
"watch",
|
||||
],
|
||||
},
|
||||
{
|
||||
apiGroups: [""],
|
||||
resources: [
|
||||
"secrets",
|
||||
],
|
||||
verbs: [
|
||||
"get",
|
||||
],
|
||||
},
|
||||
],
|
||||
}, // role
|
||||
centralDashboardRole:: centralDashboardRole,
|
||||
|
||||
local centralDashboardRoleBinding = {
|
||||
apiVersion: "rbac.authorization.k8s.io/v1beta1",
|
||||
kind: "RoleBinding",
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "centraldashboard",
|
||||
},
|
||||
name: "centraldashboard",
|
||||
namespace: params.namespace,
|
||||
},
|
||||
roleRef: {
|
||||
apiGroup: "rbac.authorization.k8s.io",
|
||||
kind: "Role",
|
||||
name: "centraldashboard",
|
||||
},
|
||||
subjects: [
|
||||
{
|
||||
kind: "ServiceAccount",
|
||||
name: "centraldashboard",
|
||||
namespace: params.namespace,
|
||||
},
|
||||
],
|
||||
}, // role binding
|
||||
centralDashboardRoleBinding:: centralDashboardRoleBinding,
|
||||
|
||||
local centralDashboardClusterRole = {
|
||||
apiVersion: "rbac.authorization.k8s.io/v1",
|
||||
kind: "ClusterRole",
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "centraldashboard",
|
||||
},
|
||||
name: "centraldashboard",
|
||||
},
|
||||
rules: [
|
||||
{
|
||||
apiGroups: [""],
|
||||
resources: [
|
||||
"namespaces",
|
||||
"events"
|
||||
],
|
||||
verbs: [
|
||||
"get",
|
||||
"list",
|
||||
"watch",
|
||||
],
|
||||
}
|
||||
],
|
||||
}, // clusterrole
|
||||
centralDashboardClusterRole:: centralDashboardClusterRole,
|
||||
|
||||
local centralDashboardClusterRoleBinding = {
|
||||
apiVersion: "rbac.authorization.k8s.io/v1",
|
||||
kind: "ClusterRoleBinding",
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "centraldashboard",
|
||||
},
|
||||
name: "centraldashboard",
|
||||
},
|
||||
roleRef: {
|
||||
apiGroup: "rbac.authorization.k8s.io",
|
||||
kind: "ClusterRole",
|
||||
name: "centraldashboard",
|
||||
},
|
||||
subjects: [
|
||||
{
|
||||
kind: "ServiceAccount",
|
||||
name: "centraldashboard",
|
||||
namespace: params.namespace,
|
||||
},
|
||||
],
|
||||
}, // clusterrolebinding
|
||||
centralDashboardClusterRoleBinding:: centralDashboardClusterRoleBinding,
|
||||
|
||||
parts:: self,
|
||||
all:: [
|
||||
self.centralDashboardDeployment,
|
||||
self.centralDashboardService,
|
||||
self.centralDashboardServiceAccount,
|
||||
self.centralDashboardRole,
|
||||
self.centralDashboardRoleBinding,
|
||||
self.centralDashboardClusterRole,
|
||||
self.centralDashboardClusterRoleBinding,
|
||||
],
|
||||
|
||||
list(obj=self.all):: util.list(obj),
|
||||
},
|
||||
}
|
||||
94
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/common/echo-server.libsonnet
vendored
Normal file
94
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/common/echo-server.libsonnet
vendored
Normal file
|
|
@ -0,0 +1,94 @@
|
|||
{
|
||||
local util = import "kubeflow/common/util.libsonnet",
|
||||
new(_env, _params):: {
|
||||
local params = _params + _env,
|
||||
|
||||
local service = {
|
||||
apiVersion: "v1",
|
||||
kind: "Service",
|
||||
metadata: {
|
||||
labels: {
|
||||
app: params.name,
|
||||
},
|
||||
name: params.name,
|
||||
namespace: params.namespace,
|
||||
annotations: {
|
||||
"getambassador.io/config":
|
||||
std.join("\n", [
|
||||
"---",
|
||||
"apiVersion: ambassador/v0",
|
||||
"kind: Mapping",
|
||||
"name: " + params.name + "-mapping",
|
||||
"prefix: /" + params.name,
|
||||
"rewrite: /",
|
||||
"service: " + params.name + "." + params.namespace,
|
||||
]),
|
||||
}, //annotations
|
||||
},
|
||||
spec: {
|
||||
ports: [
|
||||
{
|
||||
port: 80,
|
||||
targetPort: 8080,
|
||||
},
|
||||
],
|
||||
selector: {
|
||||
app: params.name,
|
||||
},
|
||||
type: "ClusterIP",
|
||||
},
|
||||
},
|
||||
service:: service,
|
||||
|
||||
local deployment = {
|
||||
apiVersion: "extensions/v1beta1",
|
||||
kind: "Deployment",
|
||||
metadata: {
|
||||
name: params.name,
|
||||
namespace: params.namespace,
|
||||
|
||||
},
|
||||
spec: {
|
||||
replicas: 1,
|
||||
template: {
|
||||
metadata: {
|
||||
labels: {
|
||||
app: params.name,
|
||||
},
|
||||
},
|
||||
spec: {
|
||||
containers: [
|
||||
{
|
||||
image: params.image,
|
||||
name: "app",
|
||||
ports: [
|
||||
{
|
||||
containerPort: 8080,
|
||||
},
|
||||
],
|
||||
|
||||
readinessProbe: {
|
||||
httpGet: {
|
||||
path: "/headers",
|
||||
port: 8080,
|
||||
},
|
||||
initialDelaySeconds: 5,
|
||||
periodSeconds: 30,
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
deployment:: deployment,
|
||||
|
||||
parts:: self,
|
||||
all:: [
|
||||
self.service,
|
||||
self.deployment,
|
||||
],
|
||||
|
||||
list(obj=self.all):: util.list(obj),
|
||||
},
|
||||
}
|
||||
|
|
@ -0,0 +1,35 @@
|
|||
{
|
||||
"name": "common",
|
||||
"apiVersion": "0.0.1",
|
||||
"kind": "ksonnet.io/parts",
|
||||
"description": "Common components of Kubeflow.\n",
|
||||
"author": "kubeflow team <kubeflow-team@google.com>",
|
||||
"contributors": [
|
||||
{
|
||||
"name": "Jeremy Lewi",
|
||||
"email": "jlewi@google.com"
|
||||
}
|
||||
],
|
||||
"repository": {
|
||||
"type": "git",
|
||||
"url": "https://github.com/kubeflow/kubeflow"
|
||||
},
|
||||
"bugs": {
|
||||
"url": "https://github.com/kubeflow/kubeflow/issues"
|
||||
},
|
||||
"keywords": [
|
||||
"kubeflow",
|
||||
"tensorflow"
|
||||
],
|
||||
"quickStart": {
|
||||
"prototype": "io.ksonnet.pkg.kubeflow",
|
||||
"componentName": "common",
|
||||
"flags": {
|
||||
"name": "common",
|
||||
"namespace": "default",
|
||||
"disks": ""
|
||||
},
|
||||
"comment": "Common Kubeflow components."
|
||||
},
|
||||
"license": "Apache 2.0"
|
||||
}
|
||||
14
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/common/prototypes/ambassador.jsonnet
vendored
Normal file
14
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/common/prototypes/ambassador.jsonnet
vendored
Normal file
|
|
@ -0,0 +1,14 @@
|
|||
// @apiVersion 0.1
|
||||
// @name io.ksonnet.pkg.ambassador
|
||||
// @description Ambassador Component
|
||||
// @shortDescription Ambassador
|
||||
// @param name string Name
|
||||
// @optionalParam platform string none supported platforms {none|gke|minikube}
|
||||
// @optionalParam ambassadorServiceType string ClusterIP The service type for the API Gateway {ClusterIP|NodePort|LoadBalancer}.
|
||||
// @optionalParam ambassadorNodePort number 0 Optional nodePort to use when ambassadorServiceType is NodePort {30000-32767}.
|
||||
// @optionalParam ambassadorImage string quay.io/datawire/ambassador:0.37.0 The image for the API Gateway.
|
||||
// @optionalParam replicas number 3 The number of replicas.
|
||||
|
||||
local ambassador = import "kubeflow/common/ambassador.libsonnet";
|
||||
local instance = ambassador.new(env, params);
|
||||
instance.list(instance.all)
|
||||
12
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/common/prototypes/basic-auth.jsonnet
vendored
Normal file
12
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/common/prototypes/basic-auth.jsonnet
vendored
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
// @apiVersion 0.1
|
||||
// @name io.ksonnet.pkg.basic-auth
|
||||
// @description Provides http basic auth for all ambassador traffic.
|
||||
// @shortDescription Http basic auth.
|
||||
// @param name string Name for the component
|
||||
// @optionalParam authSecretName string kubeflow-login Contains username and passwordhash for UI/API auth.
|
||||
// @optionalParam image string gcr.io/kubeflow-images-public/gatekeeper:v0.5.0 Auth service image to use.
|
||||
// @optionalParam imageui string gcr.io/kubeflow-images-public/kflogin-ui:v0.5.0 UI image to use.
|
||||
|
||||
local basicauth = import "kubeflow/common/basic-auth.libsonnet";
|
||||
local instance = basicauth.new(env, params);
|
||||
instance.list(instance.all)
|
||||
|
|
@ -0,0 +1,10 @@
|
|||
// @apiVersion 0.1
|
||||
// @name io.ksonnet.pkg.centraldashboard
|
||||
// @description centraldashboard Component
|
||||
// @shortDescription centraldashboard
|
||||
// @param name string Name
|
||||
// @optionalParam image string gcr.io/kubeflow-images-public/centraldashboard:v0.5.0 Image for the central dashboard
|
||||
|
||||
local centraldashboard = import "kubeflow/common/centraldashboard.libsonnet";
|
||||
local instance = centraldashboard.new(env, params);
|
||||
instance.list(instance.all)
|
||||
10
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/common/prototypes/echo-server.jsonnet
vendored
Normal file
10
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/common/prototypes/echo-server.jsonnet
vendored
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
// @apiVersion 0.1
|
||||
// @name io.ksonnet.pkg.echo-server
|
||||
// @description Provides a simple server for testing connections; primarily IAP.
|
||||
// @shortDescription A simple echo server.
|
||||
// @param name string Name for the component
|
||||
// @optionalParam image string gcr.io/kubeflow-images-staging/echo-server:v20180628-44f08d31 The image to use.
|
||||
|
||||
local echoserver = import "kubeflow/common/echo-server.libsonnet";
|
||||
local instance = echoserver.new(env, params);
|
||||
instance.list(instance.all)
|
||||
11
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/common/prototypes/spartakus.jsonnet
vendored
Normal file
11
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/common/prototypes/spartakus.jsonnet
vendored
Normal file
|
|
@ -0,0 +1,11 @@
|
|||
// @apiVersion 0.1
|
||||
// @name io.ksonnet.pkg.spartakus
|
||||
// @description spartakus component for usage collection
|
||||
// @shortDescription spartakus component for usage collection
|
||||
// @param name string Name
|
||||
// @optionalParam usageId string unknown_cluster Optional id to use when reporting usage to kubeflow.org
|
||||
// @optionalParam reportUsage string false Whether or not to report Kubeflow usage to kubeflow.org.
|
||||
|
||||
local spartakus = import "kubeflow/common/spartakus.libsonnet";
|
||||
local instance = spartakus.new(env, params);
|
||||
instance.list(instance.all)
|
||||
122
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/common/spartakus.libsonnet
vendored
Normal file
122
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/common/spartakus.libsonnet
vendored
Normal file
|
|
@ -0,0 +1,122 @@
|
|||
{
|
||||
local k = import "k.libsonnet",
|
||||
local util = import "kubeflow/common/util.libsonnet",
|
||||
new(_env, _params):: {
|
||||
local params = _params + _env {
|
||||
reportUsageBool: util.toBool(_params.reportUsage),
|
||||
},
|
||||
|
||||
// Spartakus needs to be able to get information about the cluster to create a report.
|
||||
local clusterRole = {
|
||||
apiVersion: "rbac.authorization.k8s.io/v1beta1",
|
||||
kind: "ClusterRole",
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "spartakus",
|
||||
},
|
||||
name: "spartakus",
|
||||
},
|
||||
rules: [
|
||||
{
|
||||
apiGroups: [
|
||||
"",
|
||||
],
|
||||
resources: [
|
||||
"nodes",
|
||||
],
|
||||
verbs: [
|
||||
"get",
|
||||
"list",
|
||||
],
|
||||
},
|
||||
],
|
||||
}, // role
|
||||
clusterRole:: clusterRole,
|
||||
|
||||
local clusterRoleBinding = {
|
||||
apiVersion: "rbac.authorization.k8s.io/v1beta1",
|
||||
kind: "ClusterRoleBinding",
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "spartakus",
|
||||
},
|
||||
name: "spartakus",
|
||||
},
|
||||
roleRef: {
|
||||
apiGroup: "rbac.authorization.k8s.io",
|
||||
kind: "ClusterRole",
|
||||
name: "spartakus",
|
||||
},
|
||||
subjects: [
|
||||
{
|
||||
kind: "ServiceAccount",
|
||||
name: "spartakus",
|
||||
namespace: params.namespace,
|
||||
},
|
||||
],
|
||||
}, // operator-role binding
|
||||
clusterRoleBinding:: clusterRoleBinding,
|
||||
|
||||
local serviceAccount = {
|
||||
apiVersion: "v1",
|
||||
kind: "ServiceAccount",
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "spartakus",
|
||||
},
|
||||
name: "spartakus",
|
||||
namespace: params.namespace,
|
||||
},
|
||||
},
|
||||
serviceAccount:: serviceAccount,
|
||||
|
||||
local volunteer = {
|
||||
apiVersion: "extensions/v1beta1",
|
||||
kind: "Deployment",
|
||||
metadata: {
|
||||
name: "spartakus-volunteer",
|
||||
namespace: params.namespace,
|
||||
labels: {
|
||||
app: "spartakus",
|
||||
},
|
||||
},
|
||||
spec: {
|
||||
replicas: 1,
|
||||
template: {
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "spartakus-volunteer",
|
||||
},
|
||||
},
|
||||
spec: {
|
||||
containers: [
|
||||
{
|
||||
image: "gcr.io/google_containers/spartakus-amd64:v1.1.0",
|
||||
name: "volunteer",
|
||||
args: [
|
||||
"volunteer",
|
||||
"--cluster-id=" + params.usageId,
|
||||
"--database=https://stats-collector.kubeflow.org",
|
||||
],
|
||||
},
|
||||
],
|
||||
serviceAccountName: "spartakus",
|
||||
}, // spec
|
||||
},
|
||||
},
|
||||
}, // deployment
|
||||
volunteer:: volunteer,
|
||||
|
||||
parts:: self,
|
||||
all:: if params.reportUsageBool then (
|
||||
[
|
||||
self.clusterRole,
|
||||
self.clusterRoleBinding,
|
||||
self.serviceAccount,
|
||||
self.volunteer,
|
||||
]
|
||||
) else [],
|
||||
|
||||
list(obj=self.all):: util.list(obj),
|
||||
},
|
||||
}
|
||||
231
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/common/tests/ambassador_test.jsonnet
vendored
Normal file
231
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/common/tests/ambassador_test.jsonnet
vendored
Normal file
|
|
@ -0,0 +1,231 @@
|
|||
local ambassador = import "kubeflow/common/ambassador.libsonnet";
|
||||
local testSuite = import "kubeflow/common/testsuite.libsonnet";
|
||||
|
||||
local params = {
|
||||
name: "ambassador",
|
||||
platform: "gke",
|
||||
ambassadorServiceType: "ClusterIP",
|
||||
ambassadorImage: "quay.io/datawire/ambassador:0.37.0",
|
||||
replicas: 3,
|
||||
};
|
||||
local env = {
|
||||
namespace: "kubeflow",
|
||||
};
|
||||
|
||||
local instance = ambassador.new(env, params);
|
||||
|
||||
local testCases = [
|
||||
{
|
||||
actual: instance.parts.ambassadorService,
|
||||
expected:
|
||||
{
|
||||
apiVersion: "v1",
|
||||
kind: "Service",
|
||||
metadata: {
|
||||
labels: {
|
||||
service: "ambassador",
|
||||
},
|
||||
name: "ambassador",
|
||||
namespace: "kubeflow",
|
||||
},
|
||||
spec: {
|
||||
ports: [
|
||||
{
|
||||
name: "ambassador",
|
||||
port: 80,
|
||||
targetPort: 80,
|
||||
},
|
||||
],
|
||||
selector: {
|
||||
service: "ambassador",
|
||||
},
|
||||
type: "ClusterIP",
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
actual: instance.parts.adminService,
|
||||
expected: {
|
||||
apiVersion: "v1",
|
||||
kind: "Service",
|
||||
metadata: {
|
||||
labels: {
|
||||
service: "ambassador-admin",
|
||||
},
|
||||
name: "ambassador-admin",
|
||||
namespace: "kubeflow",
|
||||
},
|
||||
spec: {
|
||||
ports: [
|
||||
{
|
||||
name: "ambassador-admin",
|
||||
port: 8877,
|
||||
targetPort: 8877,
|
||||
},
|
||||
],
|
||||
selector: {
|
||||
service: "ambassador",
|
||||
},
|
||||
type: "ClusterIP",
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
actual: instance.parts.ambassadorRole,
|
||||
expected: {
|
||||
apiVersion: "rbac.authorization.k8s.io/v1beta1",
|
||||
kind: "ClusterRole",
|
||||
metadata: {
|
||||
name: "ambassador",
|
||||
},
|
||||
rules: [
|
||||
{
|
||||
apiGroups: [
|
||||
"",
|
||||
],
|
||||
resources: [
|
||||
"services",
|
||||
],
|
||||
verbs: [
|
||||
"get",
|
||||
"list",
|
||||
"watch",
|
||||
],
|
||||
},
|
||||
{
|
||||
apiGroups: [
|
||||
"",
|
||||
],
|
||||
resources: [
|
||||
"configmaps",
|
||||
],
|
||||
verbs: [
|
||||
"create",
|
||||
"update",
|
||||
"patch",
|
||||
"get",
|
||||
"list",
|
||||
"watch",
|
||||
],
|
||||
},
|
||||
{
|
||||
apiGroups: [
|
||||
"",
|
||||
],
|
||||
resources: [
|
||||
"secrets",
|
||||
],
|
||||
verbs: [
|
||||
"get",
|
||||
"list",
|
||||
"watch",
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
actual: instance.parts.ambassadorServiceAccount,
|
||||
expected: {
|
||||
apiVersion: "v1",
|
||||
kind: "ServiceAccount",
|
||||
metadata: {
|
||||
name: "ambassador",
|
||||
namespace: "kubeflow",
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
actual: instance.parts.ambassadorRoleBinding,
|
||||
expected: {
|
||||
apiVersion: "rbac.authorization.k8s.io/v1beta1",
|
||||
kind: "ClusterRoleBinding",
|
||||
metadata: {
|
||||
name: "ambassador",
|
||||
},
|
||||
roleRef: {
|
||||
apiGroup: "rbac.authorization.k8s.io",
|
||||
kind: "ClusterRole",
|
||||
name: "ambassador",
|
||||
},
|
||||
subjects: [
|
||||
{
|
||||
kind: "ServiceAccount",
|
||||
name: "ambassador",
|
||||
namespace: "kubeflow",
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
actual: instance.parts.ambassadorDeployment,
|
||||
expected:
|
||||
{
|
||||
apiVersion: "apps/v1beta1",
|
||||
kind: "Deployment",
|
||||
metadata: {
|
||||
name: "ambassador",
|
||||
namespace: "kubeflow",
|
||||
},
|
||||
spec: {
|
||||
replicas: 3,
|
||||
template: {
|
||||
metadata: {
|
||||
labels: {
|
||||
service: "ambassador",
|
||||
},
|
||||
namespace: "kubeflow",
|
||||
},
|
||||
spec: {
|
||||
containers: [
|
||||
{
|
||||
env: [
|
||||
{
|
||||
name: "AMBASSADOR_NAMESPACE",
|
||||
valueFrom: {
|
||||
fieldRef: {
|
||||
fieldPath: "metadata.namespace",
|
||||
},
|
||||
},
|
||||
},
|
||||
],
|
||||
image: "quay.io/datawire/ambassador:0.37.0",
|
||||
livenessProbe: {
|
||||
httpGet: {
|
||||
path: "/ambassador/v0/check_alive",
|
||||
port: 8877,
|
||||
},
|
||||
initialDelaySeconds: 30,
|
||||
periodSeconds: 30,
|
||||
},
|
||||
name: "ambassador",
|
||||
readinessProbe: {
|
||||
httpGet: {
|
||||
path: "/ambassador/v0/check_ready",
|
||||
port: 8877,
|
||||
},
|
||||
initialDelaySeconds: 30,
|
||||
periodSeconds: 30,
|
||||
},
|
||||
resources: {
|
||||
limits: {
|
||||
cpu: 1,
|
||||
memory: "400Mi",
|
||||
},
|
||||
requests: {
|
||||
cpu: "200m",
|
||||
memory: "100Mi",
|
||||
},
|
||||
},
|
||||
},
|
||||
],
|
||||
restartPolicy: "Always",
|
||||
serviceAccountName: "ambassador",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
testSuite.run(testCases)
|
||||
167
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/common/tests/centraldashboard_test.jsonnet
vendored
Normal file
167
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/common/tests/centraldashboard_test.jsonnet
vendored
Normal file
|
|
@ -0,0 +1,167 @@
|
|||
local centraldashboard = import "../centraldashboard.libsonnet";
|
||||
local testSuite = import "kubeflow/common/testsuite.libsonnet";
|
||||
|
||||
local params = {
|
||||
image: "gcr.io/kubeflow-images-public/centraldashboard:v0.3.0",
|
||||
};
|
||||
local env = {
|
||||
namespace: "kftest",
|
||||
};
|
||||
local centraldash = centraldashboard.new(params, env);
|
||||
|
||||
local testCases = [
|
||||
{
|
||||
actual: centraldash.centralDashboardDeployment,
|
||||
expected: {
|
||||
apiVersion: "extensions/v1beta1",
|
||||
kind: "Deployment",
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "centraldashboard",
|
||||
},
|
||||
name: "centraldashboard",
|
||||
namespace: "kftest",
|
||||
},
|
||||
spec: {
|
||||
template: {
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "centraldashboard",
|
||||
},
|
||||
},
|
||||
spec: {
|
||||
containers: [
|
||||
{
|
||||
image: "gcr.io/kubeflow-images-public/centraldashboard:v0.3.0",
|
||||
name: "centraldashboard",
|
||||
ports: [
|
||||
{
|
||||
containerPort: 8082,
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
serviceAccountName: "centraldashboard",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
actual: centraldash.centralDashboardService,
|
||||
expected: {
|
||||
local annotations = function(namespace) {
|
||||
"getambassador.io/config":
|
||||
std.join("\n", [
|
||||
"---",
|
||||
"apiVersion: ambassador/v0",
|
||||
"kind: Mapping",
|
||||
"name: centralui-mapping",
|
||||
"prefix: /",
|
||||
"rewrite: /",
|
||||
"service: centraldashboard." + namespace,
|
||||
]),
|
||||
},
|
||||
apiVersion: "v1",
|
||||
kind: "Service",
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "centraldashboard",
|
||||
},
|
||||
name: "centraldashboard",
|
||||
namespace: "kftest",
|
||||
annotations: annotations("kftest"),
|
||||
},
|
||||
spec: {
|
||||
ports: [
|
||||
{
|
||||
port: 80,
|
||||
targetPort: 8082,
|
||||
},
|
||||
],
|
||||
selector: {
|
||||
app: "centraldashboard",
|
||||
},
|
||||
sessionAffinity: "None",
|
||||
type: "ClusterIP",
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
actual: centraldash.centralDashboardServiceAccount,
|
||||
expected: {
|
||||
apiVersion: "v1",
|
||||
kind: "ServiceAccount",
|
||||
metadata: {
|
||||
name: "centraldashboard",
|
||||
namespace: "kftest",
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
actual: centraldash.centralDashboardRole,
|
||||
expected: {
|
||||
apiVersion: "rbac.authorization.k8s.io/v1beta1",
|
||||
kind: "Role",
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "centraldashboard",
|
||||
},
|
||||
name: "centraldashboard",
|
||||
namespace: "kftest",
|
||||
},
|
||||
rules: [
|
||||
{
|
||||
apiGroups: [""],
|
||||
resources: [
|
||||
"pods",
|
||||
"pods/exec",
|
||||
"pods/log",
|
||||
],
|
||||
verbs: [
|
||||
"get",
|
||||
"list",
|
||||
"watch",
|
||||
],
|
||||
},
|
||||
{
|
||||
apiGroups: [""],
|
||||
resources: [
|
||||
"secrets",
|
||||
],
|
||||
verbs: [
|
||||
"get",
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
actual: centraldash.centralDashboardRoleBinding,
|
||||
expected: {
|
||||
apiVersion: "rbac.authorization.k8s.io/v1beta1",
|
||||
kind: "RoleBinding",
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "centraldashboard",
|
||||
},
|
||||
name: "centraldashboard",
|
||||
namespace: "kftest",
|
||||
},
|
||||
roleRef: {
|
||||
apiGroup: "rbac.authorization.k8s.io",
|
||||
kind: "Role",
|
||||
name: "centraldashboard",
|
||||
},
|
||||
subjects: [
|
||||
{
|
||||
kind: "ServiceAccount",
|
||||
name: "centraldashboard",
|
||||
namespace: "kftest",
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
testSuite.run(testCases)
|
||||
88
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/common/tests/echo-server_test.jsonnet
vendored
Normal file
88
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/common/tests/echo-server_test.jsonnet
vendored
Normal file
|
|
@ -0,0 +1,88 @@
|
|||
local echoServer = import "kubeflow/common/echo-server.libsonnet";
|
||||
local testSuite = import "kubeflow/common/testsuite.libsonnet";
|
||||
|
||||
local params = {
|
||||
name: "echo-server",
|
||||
image: "gcr.io/kubeflow-images-staging/echo-server:v20180628-44f08d31",
|
||||
};
|
||||
local env = {
|
||||
namespace: "kubeflow",
|
||||
};
|
||||
|
||||
local instance = echoServer.new(env, params);
|
||||
|
||||
local testCases = [
|
||||
{
|
||||
actual: instance.parts.service,
|
||||
expected: {
|
||||
apiVersion: "v1",
|
||||
kind: "Service",
|
||||
metadata: {
|
||||
annotations: {
|
||||
"getambassador.io/config": "---\napiVersion: ambassador/v0\nkind: Mapping\nname: echo-server-mapping\nprefix: /echo-server\nrewrite: /\nservice: echo-server.kubeflow",
|
||||
},
|
||||
labels: {
|
||||
app: "echo-server",
|
||||
},
|
||||
name: "echo-server",
|
||||
namespace: "kubeflow",
|
||||
},
|
||||
spec: {
|
||||
ports: [
|
||||
{
|
||||
port: 80,
|
||||
targetPort: 8080,
|
||||
},
|
||||
],
|
||||
selector: {
|
||||
app: "echo-server",
|
||||
},
|
||||
type: "ClusterIP",
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
actual: instance.parts.deployment,
|
||||
expected: {
|
||||
apiVersion: "extensions/v1beta1",
|
||||
kind: "Deployment",
|
||||
metadata: {
|
||||
name: "echo-server",
|
||||
namespace: "kubeflow",
|
||||
},
|
||||
spec: {
|
||||
replicas: 1,
|
||||
template: {
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "echo-server",
|
||||
},
|
||||
},
|
||||
spec: {
|
||||
containers: [
|
||||
{
|
||||
image: "gcr.io/kubeflow-images-staging/echo-server:v20180628-44f08d31",
|
||||
name: "app",
|
||||
ports: [
|
||||
{
|
||||
containerPort: 8080,
|
||||
},
|
||||
],
|
||||
readinessProbe: {
|
||||
httpGet: {
|
||||
path: "/headers",
|
||||
port: 8080,
|
||||
},
|
||||
initialDelaySeconds: 5,
|
||||
periodSeconds: 30,
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
testSuite.run(testCases)
|
||||
122
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/common/tests/spartakus_test.jsonnet
vendored
Normal file
122
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/common/tests/spartakus_test.jsonnet
vendored
Normal file
|
|
@ -0,0 +1,122 @@
|
|||
local spartakus = import "kubeflow/common/spartakus.libsonnet";
|
||||
local testSuite = import "kubeflow/common/testsuite.libsonnet";
|
||||
|
||||
local params = {
|
||||
name: "spartakus",
|
||||
usageId: "unknown_cluster",
|
||||
reportUsage: "false",
|
||||
};
|
||||
local env = {
|
||||
namespace: "kubeflow",
|
||||
};
|
||||
|
||||
local instance = spartakus.new(env, params);
|
||||
|
||||
local testCases = [
|
||||
{
|
||||
actual: instance.parts.clusterRole,
|
||||
expected: {
|
||||
apiVersion: "rbac.authorization.k8s.io/v1beta1",
|
||||
kind: "ClusterRole",
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "spartakus",
|
||||
},
|
||||
name: "spartakus",
|
||||
},
|
||||
rules: [
|
||||
{
|
||||
apiGroups: [
|
||||
"",
|
||||
],
|
||||
resources: [
|
||||
"nodes",
|
||||
],
|
||||
verbs: [
|
||||
"get",
|
||||
"list",
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
actual: instance.parts.clusterRoleBinding,
|
||||
expected: {
|
||||
apiVersion: "rbac.authorization.k8s.io/v1beta1",
|
||||
kind: "ClusterRoleBinding",
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "spartakus",
|
||||
},
|
||||
name: "spartakus",
|
||||
},
|
||||
roleRef: {
|
||||
apiGroup: "rbac.authorization.k8s.io",
|
||||
kind: "ClusterRole",
|
||||
name: "spartakus",
|
||||
},
|
||||
subjects: [
|
||||
{
|
||||
kind: "ServiceAccount",
|
||||
name: "spartakus",
|
||||
namespace: "kubeflow",
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
actual: instance.parts.serviceAccount,
|
||||
expected: {
|
||||
apiVersion: "v1",
|
||||
kind: "ServiceAccount",
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "spartakus",
|
||||
},
|
||||
name: "spartakus",
|
||||
namespace: "kubeflow",
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
actual: instance.parts.volunteer,
|
||||
expected: {
|
||||
apiVersion: "extensions/v1beta1",
|
||||
kind: "Deployment",
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "spartakus",
|
||||
},
|
||||
name: "spartakus-volunteer",
|
||||
namespace: "kubeflow",
|
||||
},
|
||||
spec: {
|
||||
replicas: 1,
|
||||
template: {
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "spartakus-volunteer",
|
||||
},
|
||||
},
|
||||
spec: {
|
||||
containers: [
|
||||
{
|
||||
args: [
|
||||
"volunteer",
|
||||
"--cluster-id=unknown_cluster",
|
||||
"--database=https://stats-collector.kubeflow.org",
|
||||
],
|
||||
image: "gcr.io/google_containers/spartakus-amd64:v1.1.0",
|
||||
name: "volunteer",
|
||||
},
|
||||
],
|
||||
serviceAccountName: "spartakus",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
testSuite.run(testCases)
|
||||
56
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/common/tests/util_test.jsonnet
vendored
Normal file
56
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/common/tests/util_test.jsonnet
vendored
Normal file
|
|
@ -0,0 +1,56 @@
|
|||
local util = import "../util.libsonnet";
|
||||
|
||||
std.assertEqual(util.lower("aTruez"), "atruez") &&
|
||||
std.assertEqual(util.lower("ATrUez"), "atruez") &&
|
||||
std.assertEqual(util.lower("atruez"), "atruez") &&
|
||||
std.assertEqual(util.lower("ATRUEZ"), "atruez") &&
|
||||
std.assertEqual(util.toBool(false), false) &&
|
||||
std.assertEqual(util.toBool(true), true) &&
|
||||
std.assertEqual(util.toBool("true"), true) &&
|
||||
std.assertEqual(util.toBool("True"), true) &&
|
||||
std.assertEqual(util.toBool("TRUE"), true) &&
|
||||
std.assertEqual(util.toBool("false"), false) &&
|
||||
std.assertEqual(util.toBool("False"), false) &&
|
||||
std.assertEqual(util.toBool("FALSE"), false) &&
|
||||
std.assertEqual(util.toBool("random string"), false) &&
|
||||
std.assertEqual(util.toBool(1), true) &&
|
||||
std.assertEqual(util.toBool(0), false) &&
|
||||
std.assertEqual(util.toBool(123), true) &&
|
||||
std.assertEqual(util.toArray("a,b,c,d"), ["a", "b", "c", "d"]) &&
|
||||
std.assertEqual(util.toArray("ca, or,fl, mo"), ["ca", "or", "fl", "mo"]) &&
|
||||
std.assertEqual(std.length(util.toArray(2)), 0) &&
|
||||
std.assertEqual(std.length(util.toArray("hello world")), 1) &&
|
||||
std.assertEqual(std.length(util.toArray([1, 2, 3, 4])), 0) &&
|
||||
std.assertEqual(util.sort(["Craydad", "CCall", "crayon"]), ["CCall", "Craydad", "crayon"]) &&
|
||||
std.assertEqual(
|
||||
{
|
||||
new():: self + {
|
||||
local configMap = {
|
||||
kind: "ConfigMap",
|
||||
},
|
||||
local service = {
|
||||
kind: "Service",
|
||||
},
|
||||
list:: util.list([configMap, service]),
|
||||
},
|
||||
}.new().list,
|
||||
{
|
||||
apiVersion: "v1",
|
||||
items: [
|
||||
{
|
||||
kind: "ConfigMap",
|
||||
},
|
||||
{
|
||||
kind: "Service",
|
||||
},
|
||||
],
|
||||
kind: "List",
|
||||
}
|
||||
) &&
|
||||
std.assertEqual(
|
||||
util.setDiff(
|
||||
util.sort(["CCall", "Craydad", "crayon", "fuzzball"]),
|
||||
util.sort(["CCall", "Craydad", "crayon"])
|
||||
),
|
||||
["fuzzball"]
|
||||
)
|
||||
29
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/common/testsuite.libsonnet
vendored
Normal file
29
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/common/testsuite.libsonnet
vendored
Normal file
|
|
@ -0,0 +1,29 @@
|
|||
// Some useful routines.
|
||||
{
|
||||
run:: function(testCases) {
|
||||
local testEqual(x) = x {
|
||||
pass: x.actual == x.expected,
|
||||
},
|
||||
local curry(testCases) = {
|
||||
// For each test case determine whether expected matches equals
|
||||
local testCasesWithResults = std.map(
|
||||
testEqual,
|
||||
testCases,
|
||||
),
|
||||
return::
|
||||
testCasesWithResults,
|
||||
}.return,
|
||||
// Compute test suite.
|
||||
local foldResults(left, right) = {
|
||||
pass: left.pass && right.pass,
|
||||
},
|
||||
local initResult = { pass: true },
|
||||
local suiteResult = std.foldl(foldResults, curry(testCases), initResult),
|
||||
local testSuite = suiteResult {
|
||||
testCases: curry(testCases),
|
||||
},
|
||||
result::
|
||||
testSuite,
|
||||
}.result,
|
||||
|
||||
}
|
||||
207
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/common/util.libsonnet
vendored
Normal file
207
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/common/util.libsonnet
vendored
Normal file
|
|
@ -0,0 +1,207 @@
|
|||
// Some useful routines.
|
||||
{
|
||||
local k = import "k.libsonnet",
|
||||
local util = self,
|
||||
|
||||
// Convert a string to lower case.
|
||||
lower:: function(x) {
|
||||
local cp(c) = std.codepoint(c),
|
||||
local lowerLetter(c) = if cp(c) >= 65 && cp(c) < 91 then
|
||||
std.char(cp(c) + 32)
|
||||
else c,
|
||||
result:: std.join("", std.map(lowerLetter, std.stringChars(x))),
|
||||
}.result,
|
||||
|
||||
// Convert non-boolean types like string,number to a boolean.
|
||||
// This is primarily intended for dealing with parameters that should be booleans.
|
||||
toBool:: function(x) {
|
||||
result::
|
||||
if std.type(x) == "boolean" then
|
||||
x
|
||||
else if std.type(x) == "string" then
|
||||
std.asciiUpper(x) == "TRUE"
|
||||
else if std.type(x) == "number" then
|
||||
x != 0
|
||||
else
|
||||
false,
|
||||
}.result,
|
||||
|
||||
// Convert a comma-delimited string to an Array
|
||||
toArray:: function(str) {
|
||||
local trim(str) = {
|
||||
rest::
|
||||
if std.startsWith(str, " ") then
|
||||
std.substr(str, 1, std.length(str) - 1)
|
||||
else
|
||||
str,
|
||||
}.rest,
|
||||
result::
|
||||
if std.type(str) == "string" && str != "null" && std.length(str) > 0 then
|
||||
std.map(trim, std.split(str, ","))
|
||||
else [],
|
||||
}.result,
|
||||
|
||||
foldl:: function(key, value, objs) {
|
||||
local aux(arr, i, running) =
|
||||
if i >= std.length(arr) then
|
||||
running
|
||||
else
|
||||
aux(arr, i + 1, running { [key(arr[i])]+: value(arr[i]) }) tailstrict,
|
||||
return:: aux(objs, 0, {},),
|
||||
}.return,
|
||||
|
||||
sort:: function(arr, compare=function(a, b) {
|
||||
return::
|
||||
if a == b then
|
||||
0
|
||||
else
|
||||
if a < b then
|
||||
-1
|
||||
else
|
||||
1,
|
||||
}.return) {
|
||||
local _sort(arr, compare) = {
|
||||
local l = std.length(arr),
|
||||
local f = {
|
||||
local pivot = arr[0],
|
||||
local rest = std.makeArray(l - 1, function(i) arr[i + 1]),
|
||||
local lessorequal(x) = compare(x, pivot) <= 0,
|
||||
local greater(x) = compare(x, pivot) > 0,
|
||||
local left = _sort(std.filter(lessorequal, rest), compare) tailstrict,
|
||||
local right = _sort(std.filter(greater, rest), compare) tailstrict,
|
||||
return:: left + [pivot] + right,
|
||||
}.return,
|
||||
return::
|
||||
if l == 0 then
|
||||
[]
|
||||
else
|
||||
f,
|
||||
}.return,
|
||||
return:: _sort(arr, compare),
|
||||
}.return,
|
||||
|
||||
setDiff:: function(a, b, compare=function(a, b) {
|
||||
return::
|
||||
if a == b then
|
||||
0
|
||||
else if a < b then
|
||||
-1
|
||||
else
|
||||
1,
|
||||
}.return) {
|
||||
local aux(a, b, i, j, acc) =
|
||||
if i >= std.length(a) then
|
||||
acc
|
||||
else
|
||||
if j >= std.length(b) then
|
||||
aux(a, b, i + 1, j, acc + [a[i]]) tailstrict
|
||||
else
|
||||
if compare(a[i], b[j]) == 0 then
|
||||
aux(a, b, i + 1, j + 1, acc) tailstrict
|
||||
else
|
||||
if compare(a[i], b[j]) == -1 then
|
||||
aux(a, b, i + 1, j, acc + [a[i]]) tailstrict
|
||||
else
|
||||
aux(a, b, i, j + 1, acc) tailstrict,
|
||||
return:: aux(a, b, 0, 0, []) tailstrict,
|
||||
}.return,
|
||||
|
||||
getApiVersionKindAndMetadata(resource):: {
|
||||
return::
|
||||
if std.objectHas(resource.metadata, "resourceVersion") then {
|
||||
apiVersion: resource.apiVersion,
|
||||
kind: resource.kind,
|
||||
metadata: {
|
||||
labels: resource.metadata.labels,
|
||||
name: resource.metadata.name,
|
||||
namespace: resource.metadata.namespace,
|
||||
resourceVersion: resource.metadata.resourceVersion,
|
||||
}
|
||||
} else {
|
||||
apiVersion: resource.apiVersion,
|
||||
kind: resource.kind,
|
||||
metadata: {
|
||||
labels: resource.metadata.labels,
|
||||
name: resource.metadata.name,
|
||||
namespace: resource.metadata.namespace,
|
||||
},
|
||||
},
|
||||
}.return,
|
||||
|
||||
groupByResource(resources):: {
|
||||
local getKey(resource) = {
|
||||
return::
|
||||
resource.kind,
|
||||
}.return,
|
||||
local getValue(resource) = {
|
||||
return::
|
||||
{ [resource.metadata.name]+: resource },
|
||||
}.return,
|
||||
return:: util.foldl(getKey, getValue, resources),
|
||||
}.return,
|
||||
|
||||
comparator(a, b):: {
|
||||
return::
|
||||
if a.metadata.name == b.metadata.name then
|
||||
0
|
||||
else
|
||||
if a.metadata.name < b.metadata.name then
|
||||
-1
|
||||
else
|
||||
1,
|
||||
}.return,
|
||||
|
||||
validateResource(resource):: {
|
||||
return::
|
||||
if std.type(resource) == "object" &&
|
||||
std.objectHas(resource, "kind") &&
|
||||
std.objectHas(resource, "apiVersion") &&
|
||||
std.objectHas(resource, "metadata") &&
|
||||
std.objectHas(resource.metadata, "name") then
|
||||
true
|
||||
else
|
||||
false,
|
||||
}.return,
|
||||
|
||||
extractGroups(obj)::
|
||||
if std.type(obj) == "object" then
|
||||
[obj[key] for key in std.objectFields(obj)]
|
||||
else
|
||||
[],
|
||||
|
||||
extractResources(group)::
|
||||
if std.type(group) == "object" then
|
||||
[group[key] for key in std.objectFields(group)]
|
||||
else
|
||||
[],
|
||||
|
||||
curryResources(resources, exists):: {
|
||||
local existingResource(resource) = {
|
||||
local resourceExists(kind, name) = {
|
||||
return::
|
||||
if std.objectHas(resources, kind) &&
|
||||
std.objectHas(resources[kind], name) then
|
||||
true
|
||||
else
|
||||
false,
|
||||
}.return,
|
||||
return::
|
||||
if util.validateResource(resource) then
|
||||
resourceExists(resource.kind, resource.metadata.name)
|
||||
else
|
||||
false,
|
||||
}.return,
|
||||
local missingResource(resource) = {
|
||||
return::
|
||||
existingResource(resource) == false,
|
||||
}.return,
|
||||
return::
|
||||
if exists == true then
|
||||
existingResource
|
||||
else
|
||||
missingResource,
|
||||
}.return,
|
||||
|
||||
// Produce a list of manifests. obj must be an array
|
||||
list(obj):: k.core.v1.list.new(obj,),
|
||||
}
|
||||
8
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/common/version-info.json
vendored
Normal file
8
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/common/version-info.json
vendored
Normal file
|
|
@ -0,0 +1,8 @@
|
|||
{
|
||||
"Major": "0",
|
||||
"Minor": "2",
|
||||
"Patch": "devel",
|
||||
"GitCommit": "",
|
||||
"BuildDate": "",
|
||||
"ksonnetVersion": "0.9.2",
|
||||
}
|
||||
15
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/common/version.libsonnet
vendored
Normal file
15
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/common/version.libsonnet
vendored
Normal file
|
|
@ -0,0 +1,15 @@
|
|||
{
|
||||
all(params):: [
|
||||
{
|
||||
apiVersion: "v1",
|
||||
kind: "ConfigMap",
|
||||
metadata: {
|
||||
name: "kubeflow-version",
|
||||
namespace: params.namespace,
|
||||
},
|
||||
data: {
|
||||
"kubeflow-version": importstr "version-info.json",
|
||||
},
|
||||
},
|
||||
],
|
||||
}
|
||||
22
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/examples/parts.yaml
vendored
Normal file
22
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/examples/parts.yaml
vendored
Normal file
|
|
@ -0,0 +1,22 @@
|
|||
{
|
||||
"name": "kubeflow examples",
|
||||
"apiVersion": "0.0.1",
|
||||
"kind": "ksonnet.io/parts",
|
||||
"description": "kubeflow examples.\n",
|
||||
"author": "kubeflow-team <kubeflow-discuss@googlegroups.com>",
|
||||
"contributors": [
|
||||
],
|
||||
"repository": {
|
||||
"type": "git",
|
||||
"url": "https://github.com/kubeflow/kubeflow"
|
||||
},
|
||||
"bugs": {
|
||||
"url": "https://github.com/kubeflow/kubeflow/issues"
|
||||
},
|
||||
"keywords": [
|
||||
"kubernetes",
|
||||
"kubeflow",
|
||||
"machine learning"
|
||||
],
|
||||
"license": "Apache 2.0",
|
||||
}
|
||||
|
|
@ -0,0 +1,88 @@
|
|||
// @apiVersion 0.1
|
||||
// @name io.ksonnet.pkg.katib-studyjob-test-v1alpha1
|
||||
// @description katib-studyjob-test
|
||||
// @shortDescription A Katib StudyJob using random suggestion
|
||||
// @param name string Name for the job.
|
||||
|
||||
local k = import "k.libsonnet";
|
||||
|
||||
local name = params.name;
|
||||
local namespace = env.namespace;
|
||||
|
||||
local studyjob = {
|
||||
apiVersion: "kubeflow.org/v1alpha1",
|
||||
kind: "StudyJob",
|
||||
metadata: {
|
||||
name: name,
|
||||
namespace: namespace,
|
||||
},
|
||||
spec: {
|
||||
studyName: name,
|
||||
owner: "crd",
|
||||
optimizationtype: "maximize",
|
||||
objectivevaluename: "Validation-accuracy",
|
||||
optimizationgoal: 0.99,
|
||||
requestcount: 1,
|
||||
metricsnames: ["accuracy"],
|
||||
parameterconfigs: [
|
||||
{
|
||||
name: "--lr",
|
||||
parametertype: "double",
|
||||
feasible: {
|
||||
min: "0.01",
|
||||
max: "0.03",
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "--num-layers",
|
||||
parametertype: "int",
|
||||
feasible: {
|
||||
min: "2",
|
||||
max: "5",
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "--optimizer",
|
||||
parametertype: "categorical",
|
||||
feasible: {
|
||||
list: ["sgd", "adam", "ftrl"],
|
||||
},
|
||||
},
|
||||
],
|
||||
workerSpec: {
|
||||
goTemplate: {
|
||||
rawTemplate: |||
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: {{.WorkerID}}
|
||||
namespace: {{.NameSpace}}
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: {{.WorkerID}}
|
||||
image: katib/mxnet-mnist-example
|
||||
command:
|
||||
- "python"
|
||||
- "/mxnet/example/image-classification/train_mnist.py"
|
||||
- "--batch-size=64"
|
||||
{{- with .HyperParameters}}
|
||||
{{- range .}}
|
||||
- "{{.Name}}={{.Value}}"
|
||||
{{- end}}
|
||||
{{- end}}
|
||||
restartPolicy: Never
|
||||
|||,
|
||||
},
|
||||
},
|
||||
suggestionSpec: {
|
||||
suggestionAlgorithm: "random",
|
||||
requestNumber: 1,
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
k.core.v1.list.new([
|
||||
studyjob,
|
||||
])
|
||||
|
|
@ -0,0 +1,14 @@
|
|||
// @apiVersion 1
|
||||
// @name io.ksonnet.pkg.tensorboard
|
||||
// @description Tensorboard components
|
||||
// @shortDescription ksonnet components for Tensorboard
|
||||
// @param name string Name to give to each of the components
|
||||
// @optionalParam logDir string logs Name of the log directory holding the TF events file
|
||||
// @optionalParam targetPort number 6006 Name of the targetPort
|
||||
// @optionalParam servicePort number 9000 Name of the servicePort
|
||||
// @optionalParam serviceType string ClusterIP The service type for tensorboard service
|
||||
// @optionalParam defaultTbImage string tensorflow/tensorflow:1.8.0 default tensorboard image to use
|
||||
|
||||
local tensorboard = import "kubeflow/tensorboard/tensorboard.libsonnet";
|
||||
local instance = tensorboard.new(env, params);
|
||||
instance.list(instance.all)
|
||||
|
|
@ -0,0 +1,82 @@
|
|||
// @apiVersion 0.1
|
||||
// @name io.ksonnet.pkg.tf-job-simple-v1beta1
|
||||
// @description tf-job-simple
|
||||
// @shortDescription A simple TFJob to run CNN benchmark
|
||||
// @param name string Name for the job.
|
||||
|
||||
local k = import "k.libsonnet";
|
||||
|
||||
local name = params.name;
|
||||
local namespace = env.namespace;
|
||||
local image = "gcr.io/kubeflow/tf-benchmarks-cpu:v20171202-bdab599-dirty-284af3";
|
||||
|
||||
local tfjob = {
|
||||
apiVersion: "kubeflow.org/v1beta1",
|
||||
kind: "TFJob",
|
||||
metadata: {
|
||||
name: name,
|
||||
namespace: namespace,
|
||||
},
|
||||
spec: {
|
||||
tfReplicaSpecs: {
|
||||
Worker: {
|
||||
replicas: 1,
|
||||
template: {
|
||||
spec: {
|
||||
containers: [
|
||||
{
|
||||
args: [
|
||||
"python",
|
||||
"tf_cnn_benchmarks.py",
|
||||
"--batch_size=32",
|
||||
"--model=resnet50",
|
||||
"--variable_update=parameter_server",
|
||||
"--flush_stdout=true",
|
||||
"--num_gpus=1",
|
||||
"--local_parameter_device=cpu",
|
||||
"--device=cpu",
|
||||
"--data_format=NHWC",
|
||||
],
|
||||
image: image,
|
||||
name: "tensorflow",
|
||||
workingDir: "/opt/tf-benchmarks/scripts/tf_cnn_benchmarks",
|
||||
},
|
||||
],
|
||||
restartPolicy: "OnFailure",
|
||||
},
|
||||
},
|
||||
},
|
||||
Ps: {
|
||||
template: {
|
||||
spec: {
|
||||
containers: [
|
||||
{
|
||||
args: [
|
||||
"python",
|
||||
"tf_cnn_benchmarks.py",
|
||||
"--batch_size=32",
|
||||
"--model=resnet50",
|
||||
"--variable_update=parameter_server",
|
||||
"--flush_stdout=true",
|
||||
"--num_gpus=1",
|
||||
"--local_parameter_device=cpu",
|
||||
"--device=cpu",
|
||||
"--data_format=NHWC",
|
||||
],
|
||||
image: image,
|
||||
name: "tensorflow",
|
||||
workingDir: "/opt/tf-benchmarks/scripts/tf_cnn_benchmarks",
|
||||
},
|
||||
],
|
||||
restartPolicy: "OnFailure",
|
||||
},
|
||||
},
|
||||
tfReplicaType: "PS",
|
||||
},
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
k.core.v1.list.new([
|
||||
tfjob,
|
||||
])
|
||||
|
|
@ -0,0 +1,82 @@
|
|||
// @apiVersion 0.1
|
||||
// @name io.ksonnet.pkg.tf-job-simple-v1beta2
|
||||
// @description tf-job-simple
|
||||
// @shortDescription A simple TFJob to run CNN benchmark
|
||||
// @param name string Name for the job.
|
||||
|
||||
local k = import "k.libsonnet";
|
||||
|
||||
local name = params.name;
|
||||
local namespace = env.namespace;
|
||||
local image = "gcr.io/kubeflow/tf-benchmarks-cpu:v20171202-bdab599-dirty-284af3";
|
||||
|
||||
local tfjob = {
|
||||
apiVersion: "kubeflow.org/v1beta2",
|
||||
kind: "TFJob",
|
||||
metadata: {
|
||||
name: name,
|
||||
namespace: namespace,
|
||||
},
|
||||
spec: {
|
||||
tfReplicaSpecs: {
|
||||
Worker: {
|
||||
replicas: 1,
|
||||
template: {
|
||||
spec: {
|
||||
containers: [
|
||||
{
|
||||
args: [
|
||||
"python",
|
||||
"tf_cnn_benchmarks.py",
|
||||
"--batch_size=32",
|
||||
"--model=resnet50",
|
||||
"--variable_update=parameter_server",
|
||||
"--flush_stdout=true",
|
||||
"--num_gpus=1",
|
||||
"--local_parameter_device=cpu",
|
||||
"--device=cpu",
|
||||
"--data_format=NHWC",
|
||||
],
|
||||
image: image,
|
||||
name: "tensorflow",
|
||||
workingDir: "/opt/tf-benchmarks/scripts/tf_cnn_benchmarks",
|
||||
},
|
||||
],
|
||||
restartPolicy: "OnFailure",
|
||||
},
|
||||
},
|
||||
},
|
||||
Ps: {
|
||||
template: {
|
||||
spec: {
|
||||
containers: [
|
||||
{
|
||||
args: [
|
||||
"python",
|
||||
"tf_cnn_benchmarks.py",
|
||||
"--batch_size=32",
|
||||
"--model=resnet50",
|
||||
"--variable_update=parameter_server",
|
||||
"--flush_stdout=true",
|
||||
"--num_gpus=1",
|
||||
"--local_parameter_device=cpu",
|
||||
"--device=cpu",
|
||||
"--data_format=NHWC",
|
||||
],
|
||||
image: image,
|
||||
name: "tensorflow",
|
||||
workingDir: "/opt/tf-benchmarks/scripts/tf_cnn_benchmarks",
|
||||
},
|
||||
],
|
||||
restartPolicy: "OnFailure",
|
||||
},
|
||||
},
|
||||
tfReplicaType: "PS",
|
||||
},
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
k.core.v1.list.new([
|
||||
tfjob,
|
||||
])
|
||||
|
|
@ -0,0 +1,82 @@
|
|||
// @apiVersion 0.1
|
||||
// @name io.ksonnet.pkg.tf-job-simple
|
||||
// @description tf-job-simple
|
||||
// @shortDescription A simple TFJob to run CNN benchmark
|
||||
// @param name string Name for the job.
|
||||
|
||||
local k = import "k.libsonnet";
|
||||
|
||||
local name = params.name;
|
||||
local namespace = env.namespace;
|
||||
local image = "gcr.io/kubeflow/tf-benchmarks-cpu:v20171202-bdab599-dirty-284af3";
|
||||
|
||||
local tfjob = {
|
||||
apiVersion: "kubeflow.org/v1alpha2",
|
||||
kind: "TFJob",
|
||||
metadata: {
|
||||
name: name,
|
||||
namespace: namespace,
|
||||
},
|
||||
spec: {
|
||||
tfReplicaSpecs: {
|
||||
Worker: {
|
||||
replicas: 1,
|
||||
template: {
|
||||
spec: {
|
||||
containers: [
|
||||
{
|
||||
args: [
|
||||
"python",
|
||||
"tf_cnn_benchmarks.py",
|
||||
"--batch_size=32",
|
||||
"--model=resnet50",
|
||||
"--variable_update=parameter_server",
|
||||
"--flush_stdout=true",
|
||||
"--num_gpus=1",
|
||||
"--local_parameter_device=cpu",
|
||||
"--device=cpu",
|
||||
"--data_format=NHWC",
|
||||
],
|
||||
image: image,
|
||||
name: "tensorflow",
|
||||
workingDir: "/opt/tf-benchmarks/scripts/tf_cnn_benchmarks",
|
||||
},
|
||||
],
|
||||
restartPolicy: "OnFailure",
|
||||
},
|
||||
},
|
||||
},
|
||||
Ps: {
|
||||
template: {
|
||||
spec: {
|
||||
containers: [
|
||||
{
|
||||
args: [
|
||||
"python",
|
||||
"tf_cnn_benchmarks.py",
|
||||
"--batch_size=32",
|
||||
"--model=resnet50",
|
||||
"--variable_update=parameter_server",
|
||||
"--flush_stdout=true",
|
||||
"--num_gpus=1",
|
||||
"--local_parameter_device=cpu",
|
||||
"--device=cpu",
|
||||
"--data_format=NHWC",
|
||||
],
|
||||
image: image,
|
||||
name: "tensorflow",
|
||||
workingDir: "/opt/tf-benchmarks/scripts/tf_cnn_benchmarks",
|
||||
},
|
||||
],
|
||||
restartPolicy: "OnFailure",
|
||||
},
|
||||
},
|
||||
tfReplicaType: "PS",
|
||||
},
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
k.core.v1.list.new([
|
||||
tfjob,
|
||||
])
|
||||
|
|
@ -0,0 +1,94 @@
|
|||
// @apiVersion 0.1
|
||||
// @name io.ksonnet.pkg.tf-serving-simple
|
||||
// @description tf-serving-simple
|
||||
// @shortDescription tf-serving-simple
|
||||
// @param name string Name to give to each of the components
|
||||
|
||||
local k = import "k.libsonnet";
|
||||
|
||||
local namespace = "default";
|
||||
local appName = import "param://name";
|
||||
local modelBasePath = "gs://kubeflow-models/inception";
|
||||
local modelName = "inception";
|
||||
local image = "gcr.io/kubeflow-images-public/tf-model-server-cpu:v20180327-995786ec";
|
||||
|
||||
local service = {
|
||||
apiVersion: "v1",
|
||||
kind: "Service",
|
||||
metadata: {
|
||||
labels: {
|
||||
app: appName,
|
||||
},
|
||||
name: appName,
|
||||
namespace: namespace,
|
||||
},
|
||||
spec: {
|
||||
ports: [
|
||||
{
|
||||
name: "grpc-tf-serving",
|
||||
port: 9000,
|
||||
targetPort: 9000,
|
||||
},
|
||||
],
|
||||
selector: {
|
||||
app: appName,
|
||||
},
|
||||
type: "ClusterIP",
|
||||
},
|
||||
};
|
||||
|
||||
local deployment = {
|
||||
apiVersion: "extensions/v1beta1",
|
||||
kind: "Deployment",
|
||||
metadata: {
|
||||
labels: {
|
||||
app: appName,
|
||||
},
|
||||
name: appName,
|
||||
namespace: namespace,
|
||||
},
|
||||
spec: {
|
||||
template: {
|
||||
metadata: {
|
||||
labels: {
|
||||
app: appName,
|
||||
},
|
||||
},
|
||||
spec: {
|
||||
containers: [
|
||||
{
|
||||
args: [
|
||||
"/usr/bin/tensorflow_model_server",
|
||||
"--port=9000",
|
||||
"--model_name=" + modelName,
|
||||
"--model_base_path=" + modelBasePath,
|
||||
],
|
||||
image: image,
|
||||
imagePullPolicy: "IfNotPresent",
|
||||
name: "inception",
|
||||
ports: [
|
||||
{
|
||||
containerPort: 9000,
|
||||
},
|
||||
],
|
||||
resources: {
|
||||
limits: {
|
||||
cpu: "4",
|
||||
memory: "4Gi",
|
||||
},
|
||||
requests: {
|
||||
cpu: "1",
|
||||
memory: "1Gi",
|
||||
},
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
k.core.v1.list.new([
|
||||
service,
|
||||
deployment,
|
||||
])
|
||||
|
|
@ -0,0 +1,179 @@
|
|||
// @apiVersion 0.1
|
||||
// @name io.ksonnet.pkg.tf-serving-with-istio
|
||||
// @description tf-serving-with-istio
|
||||
// @shortDescription tf-serving-with-istio
|
||||
// @param name string Name to give to each of the components
|
||||
|
||||
local k = import "k.libsonnet";
|
||||
|
||||
local namespace = "default";
|
||||
local appName = import "param://name";
|
||||
local modelBasePath = "gs://kubeflow-models/inception";
|
||||
local modelName = "inception";
|
||||
local image = "gcr.io/kubeflow-images-public/tf-model-server-cpu:v20180327-995786ec";
|
||||
local httpProxyImage = "gcr.io/kubeflow-images-public/tf-model-server-http-proxy:v20180327-995786ec";
|
||||
|
||||
local routeRule = {
|
||||
apiVersion: "config.istio.io/v1alpha2",
|
||||
kind: "RouteRule",
|
||||
metadata: {
|
||||
name: appName,
|
||||
namespace: namespace,
|
||||
},
|
||||
spec: {
|
||||
destination: {
|
||||
name: "tf-serving",
|
||||
},
|
||||
precedence: 0,
|
||||
route: [
|
||||
{
|
||||
labels: {
|
||||
version: "v1",
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
};
|
||||
|
||||
local service = {
|
||||
apiVersion: "v1",
|
||||
kind: "Service",
|
||||
metadata: {
|
||||
annotations: {
|
||||
"getambassador.io/config":
|
||||
std.join("\n", [
|
||||
"---",
|
||||
"apiVersion: ambassador/v0",
|
||||
"kind: Mapping",
|
||||
"name: tfserving-mapping-tf-serving-get",
|
||||
"prefix: /models/tf-serving/",
|
||||
"rewrite: /",
|
||||
"method: GET",
|
||||
"service: tf-serving." + namespace + ":8000",
|
||||
"---",
|
||||
"apiVersion: ambassador/v0",
|
||||
"kind: Mapping",
|
||||
"name: tfserving-mapping-tf-serving-post",
|
||||
"prefix: /models/tf-serving/",
|
||||
"rewrite: /model/tf-serving:predict",
|
||||
"method: POST",
|
||||
"service: tf-serving." + namespace + ":8000",
|
||||
]),
|
||||
},
|
||||
labels: {
|
||||
app: appName,
|
||||
},
|
||||
name: appName,
|
||||
namespace: namespace,
|
||||
},
|
||||
spec: {
|
||||
ports: [
|
||||
{
|
||||
name: "grpc-tf-serving",
|
||||
port: 9000,
|
||||
targetPort: 9000,
|
||||
},
|
||||
{
|
||||
name: "http-tf-serving-proxy",
|
||||
port: 8000,
|
||||
targetPort: 8000,
|
||||
},
|
||||
],
|
||||
selector: {
|
||||
app: appName,
|
||||
},
|
||||
type: "ClusterIP",
|
||||
},
|
||||
};
|
||||
|
||||
local deployment = {
|
||||
apiVersion: "extensions/v1beta1",
|
||||
kind: "Deployment",
|
||||
metadata: {
|
||||
labels: {
|
||||
app: appName,
|
||||
},
|
||||
name: appName,
|
||||
namespace: namespace,
|
||||
},
|
||||
spec: {
|
||||
template: {
|
||||
metadata: {
|
||||
labels: {
|
||||
app: appName,
|
||||
},
|
||||
annotations: {
|
||||
"sidecar.istio.io/inject": "true",
|
||||
},
|
||||
},
|
||||
spec: {
|
||||
containers: [
|
||||
{
|
||||
args: [
|
||||
"/usr/bin/tensorflow_model_server",
|
||||
"--port=9000",
|
||||
"--model_name=" + modelName,
|
||||
"--model_base_path=" + modelBasePath,
|
||||
],
|
||||
image: image,
|
||||
imagePullPolicy: "IfNotPresent",
|
||||
name: "inception",
|
||||
ports: [
|
||||
{
|
||||
containerPort: 9000,
|
||||
},
|
||||
],
|
||||
resources: {
|
||||
limits: {
|
||||
cpu: "4",
|
||||
memory: "4Gi",
|
||||
},
|
||||
requests: {
|
||||
cpu: "1",
|
||||
memory: "1Gi",
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
name: appName + "-http-proxy",
|
||||
image: httpProxyImage,
|
||||
imagePullPolicy: "IfNotPresent",
|
||||
command: [
|
||||
"python",
|
||||
"/usr/src/app/server.py",
|
||||
"--port=8000",
|
||||
"--rpc_port=9000",
|
||||
"--rpc_timeout=10.0",
|
||||
],
|
||||
env: [],
|
||||
ports: [
|
||||
{
|
||||
containerPort: 8000,
|
||||
},
|
||||
],
|
||||
resources: {
|
||||
requests: {
|
||||
memory: "1Gi",
|
||||
cpu: "1",
|
||||
},
|
||||
limits: {
|
||||
memory: "4Gi",
|
||||
cpu: "4",
|
||||
},
|
||||
},
|
||||
securityContext: {
|
||||
runAsUser: 1000,
|
||||
fsGroup: 1000,
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
k.core.v1.list.new([
|
||||
routeRule,
|
||||
service,
|
||||
deployment,
|
||||
])
|
||||
102
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/examples/tests/tensorboard_test.jsonnet
vendored
Normal file
102
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/examples/tests/tensorboard_test.jsonnet
vendored
Normal file
|
|
@ -0,0 +1,102 @@
|
|||
local tensorboard = import "kubeflow/tensorboard/tensorboard.libsonnet";
|
||||
|
||||
local params = {
|
||||
name: "tensorboard",
|
||||
logDir: "logs",
|
||||
targetPort: "6006",
|
||||
servicePort: "9000",
|
||||
serviceType: "LoadBalancer",
|
||||
defaultTbImage: "tensorflow/tensorflow:1.9.0",
|
||||
};
|
||||
local env = {
|
||||
namespace: "test-kf-001",
|
||||
};
|
||||
|
||||
local instance = tensorboard.new(env, params);
|
||||
|
||||
std.assertEqual(
|
||||
instance.tbService,
|
||||
{
|
||||
apiVersion: "v1",
|
||||
kind: "Service",
|
||||
metadata: {
|
||||
annotations: {
|
||||
"getambassador.io/config": "---\napiVersion: ambassador/v0\nkind: Mapping\nname: tb-mapping-tensorboard-get\nprefix: /tensorboard/ tensorboard/\nrewrite: /\nmethod: GET\nservice: tensorboard.test-kf-001:9000",
|
||||
},
|
||||
labels: {
|
||||
app: "tensorboard",
|
||||
},
|
||||
name: "tensorboard",
|
||||
namespace: "test-kf-001",
|
||||
},
|
||||
spec: {
|
||||
ports: [
|
||||
{
|
||||
name: "tb",
|
||||
port: "9000",
|
||||
targetPort: "6006",
|
||||
},
|
||||
],
|
||||
selector: {
|
||||
app: "tensorboard",
|
||||
},
|
||||
type: "LoadBalancer",
|
||||
},
|
||||
}
|
||||
) &&
|
||||
|
||||
std.assertEqual(
|
||||
instance.tbDeployment,
|
||||
{
|
||||
apiVersion: "apps/v1beta1",
|
||||
kind: "Deployment",
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "tensorboard",
|
||||
},
|
||||
name: "tensorboard",
|
||||
namespace: "test-kf-001",
|
||||
},
|
||||
spec: {
|
||||
replicas: 1,
|
||||
template: {
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "tensorboard",
|
||||
},
|
||||
},
|
||||
spec: {
|
||||
containers: [
|
||||
{
|
||||
args: [
|
||||
"--logdir=logs",
|
||||
"--port=6006",
|
||||
],
|
||||
command: [
|
||||
"/usr/local/bin/tensorboard",
|
||||
],
|
||||
image: "tensorflow/tensorflow:1.9.0",
|
||||
imagePullPolicy: "IfNotPresent",
|
||||
name: "tensorboard",
|
||||
ports: [
|
||||
{
|
||||
containerPort: "6006",
|
||||
},
|
||||
],
|
||||
resources: {
|
||||
limits: {
|
||||
cpu: "4",
|
||||
memory: "4Gi",
|
||||
},
|
||||
requests: {
|
||||
cpu: "1",
|
||||
memory: "1Gi",
|
||||
},
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
)
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
approvers:
|
||||
- abhi-g
|
||||
- jlewi
|
||||
- kunmingg
|
||||
- lluunn
|
||||
- r2d4
|
||||
- richardsliu
|
||||
|
|
@ -0,0 +1,12 @@
|
|||
# gcp
|
||||
|
||||
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
|
||||
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
|
||||
**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)*
|
||||
|
||||
- [gcp](#gcp)
|
||||
|
||||
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
|
||||
|
||||
|
||||
> This ksonnet package contains GCP specific prototypes.
|
||||
451
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/gcp/basic-auth-ingress.libsonnet
vendored
Normal file
451
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/gcp/basic-auth-ingress.libsonnet
vendored
Normal file
|
|
@ -0,0 +1,451 @@
|
|||
{
|
||||
local k = import "k.libsonnet",
|
||||
local util = import "kubeflow/common/util.libsonnet",
|
||||
new(_env, _params):: {
|
||||
local params = _params + _env {
|
||||
hostname: if std.objectHas(_params, "hostname") then _params.hostname else "null",
|
||||
ingressName: "envoy-ingress"
|
||||
},
|
||||
local namespace = params.namespace,
|
||||
|
||||
// Test if the given hostname is in the form of: "NAME.endpoints.PROJECT.cloud.goog"
|
||||
local isCloudEndpoint(str) = {
|
||||
local toks = if std.type(str) == "null" then [] else std.split(str, "."),
|
||||
result::
|
||||
(std.length(toks) == 5 && toks[1] == "endpoints" && toks[3] == "cloud" && toks[4] == "goog"),
|
||||
}.result,
|
||||
|
||||
local initServiceAccount = {
|
||||
apiVersion: "v1",
|
||||
kind: "ServiceAccount",
|
||||
metadata: {
|
||||
name: "envoy",
|
||||
namespace: namespace,
|
||||
},
|
||||
}, // initServiceAccount
|
||||
initServiceAccount:: initServiceAccount,
|
||||
|
||||
local initClusterRoleBinding = {
|
||||
kind: "ClusterRoleBinding",
|
||||
apiVersion: "rbac.authorization.k8s.io/v1beta1",
|
||||
metadata: {
|
||||
name: "envoy",
|
||||
},
|
||||
subjects: [
|
||||
{
|
||||
kind: "ServiceAccount",
|
||||
name: "envoy",
|
||||
namespace: namespace,
|
||||
},
|
||||
],
|
||||
roleRef: {
|
||||
kind: "ClusterRole",
|
||||
name: "envoy",
|
||||
apiGroup: "rbac.authorization.k8s.io",
|
||||
},
|
||||
}, // initClusterRoleBinding
|
||||
initClusterRoleBinding:: initClusterRoleBinding,
|
||||
|
||||
local initClusterRole = {
|
||||
kind: "ClusterRole",
|
||||
apiVersion: "rbac.authorization.k8s.io/v1beta1",
|
||||
metadata: {
|
||||
name: "envoy",
|
||||
namespace: namespace,
|
||||
},
|
||||
rules: [
|
||||
{
|
||||
apiGroups: [""],
|
||||
resources: ["services", "configmaps", "secrets"],
|
||||
verbs: ["get", "list", "patch", "update"],
|
||||
},
|
||||
{
|
||||
apiGroups: ["extensions"],
|
||||
resources: ["ingresses"],
|
||||
verbs: ["get", "list", "update", "patch"],
|
||||
},
|
||||
],
|
||||
}, // initClusterRoleBinding
|
||||
initClusterRole:: initClusterRole,
|
||||
|
||||
local configMap = {
|
||||
apiVersion: "v1",
|
||||
kind: "ConfigMap",
|
||||
metadata: {
|
||||
name: "envoy-config",
|
||||
namespace: namespace,
|
||||
},
|
||||
data: {
|
||||
"update_backend.sh": importstr "update_backend.sh",
|
||||
},
|
||||
},
|
||||
configMap:: configMap,
|
||||
|
||||
local whoamiService = {
|
||||
apiVersion: "v1",
|
||||
kind: "Service",
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "whoami",
|
||||
},
|
||||
name: "whoami-app",
|
||||
namespace: params.namespace,
|
||||
annotations: {
|
||||
"getambassador.io/config":
|
||||
std.join("\n", [
|
||||
"---",
|
||||
"apiVersion: ambassador/v0",
|
||||
"kind: Mapping",
|
||||
"name: whoami-mapping",
|
||||
"prefix: /whoami",
|
||||
"rewrite: /whoami",
|
||||
"service: whoami-app." + namespace,
|
||||
]),
|
||||
}, //annotations
|
||||
},
|
||||
spec: {
|
||||
ports: [
|
||||
{
|
||||
port: 80,
|
||||
targetPort: 8081,
|
||||
},
|
||||
],
|
||||
selector: {
|
||||
app: "whoami",
|
||||
},
|
||||
type: "ClusterIP",
|
||||
},
|
||||
}, // whoamiService
|
||||
whoamiService:: whoamiService,
|
||||
|
||||
local whoamiApp = {
|
||||
apiVersion: "extensions/v1beta1",
|
||||
kind: "Deployment",
|
||||
metadata: {
|
||||
name: "whoami-app",
|
||||
namespace: params.namespace,
|
||||
},
|
||||
spec: {
|
||||
replicas: 1,
|
||||
template: {
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "whoami",
|
||||
},
|
||||
},
|
||||
spec: {
|
||||
containers: [
|
||||
{
|
||||
env: [
|
||||
{
|
||||
name: "PORT",
|
||||
value: "8081",
|
||||
},
|
||||
],
|
||||
image: "gcr.io/cloud-solutions-group/esp-sample-app:1.0.0",
|
||||
name: "app",
|
||||
ports: [
|
||||
{
|
||||
containerPort: 8081,
|
||||
},
|
||||
],
|
||||
readinessProbe: {
|
||||
failureThreshold: 2,
|
||||
httpGet: {
|
||||
path: "/healthz",
|
||||
port: 8081,
|
||||
scheme: "HTTP",
|
||||
},
|
||||
periodSeconds: 10,
|
||||
successThreshold: 1,
|
||||
timeoutSeconds: 5,
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
whoamiApp:: whoamiApp,
|
||||
|
||||
// Run the process to update the backend service
|
||||
local backendUpdater = {
|
||||
apiVersion: "apps/v1",
|
||||
kind: "StatefulSet",
|
||||
metadata: {
|
||||
name: "backend-updater",
|
||||
namespace: namespace,
|
||||
labels: {
|
||||
service: "backend-updater",
|
||||
},
|
||||
},
|
||||
spec: {
|
||||
selector: {
|
||||
matchLabels: {
|
||||
service: "backend-updater",
|
||||
},
|
||||
},
|
||||
template: {
|
||||
metadata: {
|
||||
labels: {
|
||||
service: "backend-updater",
|
||||
},
|
||||
},
|
||||
spec: {
|
||||
serviceAccountName: "envoy",
|
||||
containers: [
|
||||
{
|
||||
name: "backend-updater",
|
||||
image: params.ingressSetupImage,
|
||||
command: [
|
||||
"bash",
|
||||
"/var/envoy-config/update_backend.sh",
|
||||
],
|
||||
env: [
|
||||
{
|
||||
name: "NAMESPACE",
|
||||
value: namespace,
|
||||
},
|
||||
{
|
||||
name: "SERVICE",
|
||||
value: "ambassador",
|
||||
},
|
||||
{
|
||||
name: "GOOGLE_APPLICATION_CREDENTIALS",
|
||||
value: "/var/run/secrets/sa/admin-gcp-sa.json",
|
||||
},
|
||||
{
|
||||
name: "HEALTHCHECK_PATH",
|
||||
value: "/whoami",
|
||||
},
|
||||
{
|
||||
name: "INGRESS_NAME",
|
||||
value: params.ingressName,
|
||||
},
|
||||
],
|
||||
volumeMounts: [
|
||||
{
|
||||
mountPath: "/var/envoy-config/",
|
||||
name: "config-volume",
|
||||
},
|
||||
{
|
||||
name: "sa-key",
|
||||
readOnly: true,
|
||||
mountPath: "/var/run/secrets/sa",
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
volumes: [
|
||||
{
|
||||
configMap: {
|
||||
name: "envoy-config",
|
||||
},
|
||||
name: "config-volume",
|
||||
},
|
||||
{
|
||||
name: "sa-key",
|
||||
secret: {
|
||||
secretName: "admin-gcp-sa",
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
},
|
||||
|
||||
}, // backendUpdater
|
||||
backendUpdater:: backendUpdater,
|
||||
|
||||
// TODO(danisla): Remove afer https://github.com/kubernetes/ingress-gce/pull/388 is resolved per #1327.
|
||||
local ingressBootstrapConfigMap = {
|
||||
apiVersion: "v1",
|
||||
kind: "ConfigMap",
|
||||
metadata: {
|
||||
name: "ingress-bootstrap-config",
|
||||
namespace: namespace,
|
||||
},
|
||||
data: {
|
||||
"ingress_bootstrap.sh": importstr "ingress_bootstrap.sh",
|
||||
},
|
||||
},
|
||||
ingressBootstrapConfigMap:: ingressBootstrapConfigMap,
|
||||
|
||||
local ingressBootstrapJob = {
|
||||
apiVersion: "batch/v1",
|
||||
kind: "Job",
|
||||
metadata: {
|
||||
name: "ingress-bootstrap",
|
||||
namespace: namespace,
|
||||
},
|
||||
spec: {
|
||||
template: {
|
||||
spec: {
|
||||
restartPolicy: "OnFailure",
|
||||
serviceAccountName: "envoy",
|
||||
containers: [
|
||||
{
|
||||
name: "bootstrap",
|
||||
image: params.ingressSetupImage,
|
||||
command: ["/var/ingress-config/ingress_bootstrap.sh"],
|
||||
env: [
|
||||
{
|
||||
name: "NAMESPACE",
|
||||
value: namespace,
|
||||
},
|
||||
{
|
||||
name: "TLS_SECRET_NAME",
|
||||
value: params.secretName,
|
||||
},
|
||||
{
|
||||
name: "TLS_HOST_NAME",
|
||||
value: params.hostname,
|
||||
},
|
||||
{
|
||||
name: "INGRESS_NAME",
|
||||
value: "envoy-ingress",
|
||||
},
|
||||
],
|
||||
volumeMounts: [
|
||||
{
|
||||
mountPath: "/var/ingress-config/",
|
||||
name: "ingress-config",
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
volumes: [
|
||||
{
|
||||
configMap: {
|
||||
name: "ingress-bootstrap-config",
|
||||
// TODO(danisla): replace with std.parseOctal("0755") after upgrading to ksonnet 0.12
|
||||
defaultMode: 493,
|
||||
},
|
||||
name: "ingress-config",
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
},
|
||||
}, // ingressBootstrapJob
|
||||
ingressBootstrapJob:: ingressBootstrapJob,
|
||||
|
||||
local ingress = {
|
||||
apiVersion: "extensions/v1beta1",
|
||||
kind: "Ingress",
|
||||
metadata: {
|
||||
name: "envoy-ingress",
|
||||
namespace: namespace,
|
||||
annotations: {
|
||||
"kubernetes.io/tls-acme": "true",
|
||||
"ingress.kubernetes.io/ssl-redirect": "true",
|
||||
"kubernetes.io/ingress.global-static-ip-name": params.ipName,
|
||||
"certmanager.k8s.io/issuer": params.issuer,
|
||||
},
|
||||
},
|
||||
spec: {
|
||||
rules: [
|
||||
{
|
||||
[if params.hostname != "null" then "host"]: params.hostname,
|
||||
http: {
|
||||
paths: [
|
||||
{
|
||||
backend: {
|
||||
// Due to https://github.com/kubernetes/contrib/blob/master/ingress/controllers/gce/examples/health_checks/README.md#limitations
|
||||
// Keep port the servicePort the same as the port we are targeting on the backend so that servicePort will be the same as targetPort for the purpose of
|
||||
// health checking.
|
||||
serviceName: "ambassador",
|
||||
servicePort: 80,
|
||||
},
|
||||
path: "/*",
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
}, // iapIngress
|
||||
ingress:: ingress,
|
||||
|
||||
local certificate = if params.privateGKECluster == "false" then (
|
||||
{
|
||||
apiVersion: "certmanager.k8s.io/v1alpha1",
|
||||
kind: "Certificate",
|
||||
metadata: {
|
||||
name: params.secretName,
|
||||
namespace: namespace,
|
||||
},
|
||||
|
||||
spec: {
|
||||
secretName: params.secretName,
|
||||
issuerRef: {
|
||||
name: params.issuer,
|
||||
kind: "ClusterIssuer",
|
||||
},
|
||||
commonName: params.hostname,
|
||||
dnsNames: [
|
||||
params.hostname,
|
||||
],
|
||||
acme: {
|
||||
config: [
|
||||
{
|
||||
http01: {
|
||||
ingress: "envoy-ingress",
|
||||
},
|
||||
domains: [
|
||||
params.hostname,
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
} // certificate
|
||||
),
|
||||
certificate:: certificate,
|
||||
|
||||
local cloudEndpoint = if isCloudEndpoint(params.hostname) then (
|
||||
{
|
||||
local makeEndpointParams(str) = {
|
||||
local toks = std.split(str, "."),
|
||||
result:: {
|
||||
name: toks[0],
|
||||
project: toks[2],
|
||||
},
|
||||
}.result,
|
||||
local endpointParams = makeEndpointParams(params.hostname),
|
||||
apiVersion: "ctl.isla.solutions/v1",
|
||||
kind: "CloudEndpoint",
|
||||
metadata: {
|
||||
name: endpointParams.name,
|
||||
namespace: namespace,
|
||||
},
|
||||
spec: {
|
||||
project: endpointParams.project,
|
||||
targetIngress: {
|
||||
name: "envoy-ingress",
|
||||
namespace: namespace,
|
||||
},
|
||||
},
|
||||
} // cloudEndpoint
|
||||
),
|
||||
cloudEndpoint:: cloudEndpoint,
|
||||
|
||||
parts:: self,
|
||||
all:: [
|
||||
self.initServiceAccount,
|
||||
self.initClusterRoleBinding,
|
||||
self.initClusterRole,
|
||||
self.whoamiService,
|
||||
self.whoamiApp,
|
||||
self.backendUpdater,
|
||||
self.configMap,
|
||||
self.ingressBootstrapConfigMap,
|
||||
self.ingressBootstrapJob,
|
||||
self.ingress,
|
||||
self.certificate,
|
||||
self.cloudEndpoint,
|
||||
],
|
||||
|
||||
list(obj=self.all):: k.core.v1.list.new(obj,),
|
||||
},
|
||||
}
|
||||
189
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/gcp/cert-manager.libsonnet
vendored
Normal file
189
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/gcp/cert-manager.libsonnet
vendored
Normal file
|
|
@ -0,0 +1,189 @@
|
|||
{
|
||||
local k = import "k.libsonnet",
|
||||
new(_env, _params):: {
|
||||
local params = _params + _env,
|
||||
|
||||
local certificateCRD = {
|
||||
apiVersion: "apiextensions.k8s.io/v1beta1",
|
||||
kind: "CustomResourceDefinition",
|
||||
metadata: {
|
||||
name: "certificates.certmanager.k8s.io",
|
||||
},
|
||||
spec: {
|
||||
group: "certmanager.k8s.io",
|
||||
version: "v1alpha1",
|
||||
names: {
|
||||
kind: "Certificate",
|
||||
plural: "certificates",
|
||||
},
|
||||
scope: "Namespaced",
|
||||
},
|
||||
},
|
||||
certificateCRD:: certificateCRD,
|
||||
|
||||
local clusterIssuerCRD = {
|
||||
apiVersion: "apiextensions.k8s.io/v1beta1",
|
||||
kind: "CustomResourceDefinition",
|
||||
metadata: {
|
||||
name: "clusterissuers.certmanager.k8s.io",
|
||||
},
|
||||
|
||||
spec: {
|
||||
group: "certmanager.k8s.io",
|
||||
version: "v1alpha1",
|
||||
names: {
|
||||
kind: "ClusterIssuer",
|
||||
plural: "clusterissuers",
|
||||
},
|
||||
scope: "Cluster",
|
||||
},
|
||||
},
|
||||
clusterIssuerCRD:: clusterIssuerCRD,
|
||||
|
||||
local issuerCRD = {
|
||||
apiVersion: "apiextensions.k8s.io/v1beta1",
|
||||
kind: "CustomResourceDefinition",
|
||||
metadata: {
|
||||
name: "issuers.certmanager.k8s.io",
|
||||
},
|
||||
spec: {
|
||||
group: "certmanager.k8s.io",
|
||||
version: "v1alpha1",
|
||||
names: {
|
||||
kind: "Issuer",
|
||||
plural: "issuers",
|
||||
},
|
||||
scope: "Namespaced",
|
||||
},
|
||||
},
|
||||
issuerCRD:: issuerCRD,
|
||||
|
||||
local serviceAccount = {
|
||||
apiVersion: "v1",
|
||||
kind: "ServiceAccount",
|
||||
metadata: {
|
||||
name: "cert-manager",
|
||||
namespace: params.namespace,
|
||||
},
|
||||
},
|
||||
serviceAccount:: serviceAccount,
|
||||
|
||||
local clusterRole = {
|
||||
apiVersion: "rbac.authorization.k8s.io/v1beta1",
|
||||
kind: "ClusterRole",
|
||||
metadata: {
|
||||
name: "cert-manager",
|
||||
},
|
||||
rules: [
|
||||
{
|
||||
apiGroups: ["certmanager.k8s.io"],
|
||||
resources: ["certificates", "issuers", "clusterissuers"],
|
||||
verbs: ["*"],
|
||||
},
|
||||
{
|
||||
apiGroups: [""],
|
||||
resources: ["secrets", "events", "endpoints", "services", "pods", "configmaps"],
|
||||
verbs: ["*"],
|
||||
},
|
||||
{
|
||||
apiGroups: ["extensions"],
|
||||
resources: ["ingresses"],
|
||||
verbs: ["*"],
|
||||
},
|
||||
],
|
||||
},
|
||||
clusterRole:: clusterRole,
|
||||
|
||||
local clusterRoleBinding = {
|
||||
apiVersion: "rbac.authorization.k8s.io/v1beta1",
|
||||
kind: "ClusterRoleBinding",
|
||||
metadata: {
|
||||
name: "cert-manager",
|
||||
},
|
||||
roleRef: {
|
||||
apiGroup: "rbac.authorization.k8s.io",
|
||||
kind: "ClusterRole",
|
||||
name: "cert-manager",
|
||||
},
|
||||
subjects: [
|
||||
{
|
||||
name: "cert-manager",
|
||||
namespace: params.namespace,
|
||||
kind: "ServiceAccount",
|
||||
},
|
||||
],
|
||||
},
|
||||
clusterRoleBinding:: clusterRoleBinding,
|
||||
|
||||
local deploy = {
|
||||
apiVersion: "apps/v1beta1",
|
||||
kind: "Deployment",
|
||||
metadata: {
|
||||
name: "cert-manager",
|
||||
namespace: params.namespace,
|
||||
labels: {
|
||||
app: "cert-manager",
|
||||
},
|
||||
},
|
||||
spec: {
|
||||
replicas: 1,
|
||||
template: {
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "cert-manager",
|
||||
},
|
||||
},
|
||||
spec: {
|
||||
serviceAccountName: "cert-manager",
|
||||
containers: [
|
||||
{
|
||||
name: "cert-manager",
|
||||
image: params.certManagerImage,
|
||||
imagePullPolicy: "IfNotPresent",
|
||||
args: [
|
||||
"--cluster-resource-namespace=" + params.namespace,
|
||||
"--leader-election-namespace=" + params.namespace,
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
deploy:: deploy,
|
||||
|
||||
local issuerLEProd = {
|
||||
apiVersion: "certmanager.k8s.io/v1alpha1",
|
||||
kind: "ClusterIssuer",
|
||||
metadata: {
|
||||
name: "letsencrypt-prod",
|
||||
},
|
||||
spec: {
|
||||
acme: {
|
||||
server: params.acmeUrl,
|
||||
email: params.acmeEmail,
|
||||
privateKeySecretRef: {
|
||||
name: "letsencrypt-prod-secret",
|
||||
},
|
||||
http01: {
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
issuerLEProd:: issuerLEProd,
|
||||
|
||||
parts:: self,
|
||||
all:: [
|
||||
self.certificateCRD,
|
||||
self.clusterIssuerCRD,
|
||||
self.issuerCRD,
|
||||
self.serviceAccount,
|
||||
self.clusterRole,
|
||||
self.clusterRoleBinding,
|
||||
self.deploy,
|
||||
self.issuerLEProd,
|
||||
],
|
||||
|
||||
list(obj=self.all):: k.core.v1.list.new(obj,),
|
||||
},
|
||||
}
|
||||
214
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/gcp/cloud-endpoints.libsonnet
vendored
Normal file
214
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/gcp/cloud-endpoints.libsonnet
vendored
Normal file
|
|
@ -0,0 +1,214 @@
|
|||
{
|
||||
local k = import "k.libsonnet",
|
||||
new(_env, _params):: {
|
||||
local params = {
|
||||
cloudEndpointsImage: "gcr.io/cloud-solutions-group/cloud-endpoints-controller:0.2.1",
|
||||
} + _params + _env,
|
||||
|
||||
local endpointsCRD = {
|
||||
apiVersion: "apiextensions.k8s.io/v1beta1",
|
||||
kind: "CustomResourceDefinition",
|
||||
metadata: {
|
||||
name: "cloudendpoints.ctl.isla.solutions",
|
||||
},
|
||||
spec: {
|
||||
group: "ctl.isla.solutions",
|
||||
version: "v1",
|
||||
scope: "Namespaced",
|
||||
names: {
|
||||
plural: "cloudendpoints",
|
||||
singular: "cloudendpoint",
|
||||
kind: "CloudEndpoint",
|
||||
shortNames: [
|
||||
"cloudep",
|
||||
"ce",
|
||||
],
|
||||
},
|
||||
},
|
||||
}, // endpointsCRD
|
||||
endpointsCRD:: endpointsCRD,
|
||||
|
||||
local endpointsClusterRole = {
|
||||
kind: "ClusterRole",
|
||||
apiVersion: "rbac.authorization.k8s.io/v1beta1",
|
||||
metadata: {
|
||||
name: "cloud-endpoints-controller",
|
||||
},
|
||||
rules: [
|
||||
{
|
||||
apiGroups: [""],
|
||||
resources: ["services", "configmaps"],
|
||||
verbs: ["get", "list"],
|
||||
},
|
||||
{
|
||||
apiGroups: ["extensions"],
|
||||
resources: ["ingresses"],
|
||||
verbs: ["get", "list"],
|
||||
},
|
||||
],
|
||||
},
|
||||
endpointsClusterRole:: endpointsClusterRole,
|
||||
|
||||
local endpointsClusterRoleBinding = {
|
||||
kind: "ClusterRoleBinding",
|
||||
apiVersion: "rbac.authorization.k8s.io/v1beta1",
|
||||
metadata: {
|
||||
name: "cloud-endpoints-controller",
|
||||
},
|
||||
subjects: [
|
||||
{
|
||||
kind: "ServiceAccount",
|
||||
name: "cloud-endpoints-controller",
|
||||
namespace: params.namespace,
|
||||
},
|
||||
],
|
||||
roleRef: {
|
||||
kind: "ClusterRole",
|
||||
name: "cloud-endpoints-controller",
|
||||
apiGroup: "rbac.authorization.k8s.io",
|
||||
},
|
||||
},
|
||||
endpointsClusterRoleBinding:: endpointsClusterRoleBinding,
|
||||
|
||||
local endpointsService = {
|
||||
apiVersion: "v1",
|
||||
kind: "Service",
|
||||
metadata: {
|
||||
name: "cloud-endpoints-controller",
|
||||
namespace: params.namespace,
|
||||
},
|
||||
spec: {
|
||||
type: "ClusterIP",
|
||||
ports: [
|
||||
{
|
||||
name: "http",
|
||||
port: 80,
|
||||
},
|
||||
],
|
||||
selector: {
|
||||
app: "cloud-endpoints-controller",
|
||||
},
|
||||
},
|
||||
}, // endpointsService
|
||||
endpointsService:: endpointsService,
|
||||
|
||||
local endpointsServiceAccount = {
|
||||
apiVersion: "v1",
|
||||
kind: "ServiceAccount",
|
||||
metadata: {
|
||||
name: "cloud-endpoints-controller",
|
||||
namespace: params.namespace,
|
||||
},
|
||||
}, // endpointsServiceAccount
|
||||
endpointsServiceAccount:: endpointsServiceAccount,
|
||||
|
||||
local endpointsDeploy = {
|
||||
apiVersion: "apps/v1beta1",
|
||||
kind: "Deployment",
|
||||
metadata: {
|
||||
name: "cloud-endpoints-controller",
|
||||
namespace: params.namespace,
|
||||
},
|
||||
spec: {
|
||||
replicas: 1,
|
||||
template: {
|
||||
metadata: {
|
||||
labels: {
|
||||
app: "cloud-endpoints-controller",
|
||||
},
|
||||
},
|
||||
spec: {
|
||||
serviceAccountName: "cloud-endpoints-controller",
|
||||
terminationGracePeriodSeconds: 5,
|
||||
containers: [
|
||||
{
|
||||
name: "cloud-endpoints-controller",
|
||||
image: params.cloudEndpointsImage,
|
||||
imagePullPolicy: "Always",
|
||||
env: [
|
||||
{
|
||||
name: "GOOGLE_APPLICATION_CREDENTIALS",
|
||||
value: "/var/run/secrets/sa/" + params.secretKey,
|
||||
},
|
||||
],
|
||||
volumeMounts: [
|
||||
{
|
||||
name: "sa-key",
|
||||
readOnly: true,
|
||||
mountPath: "/var/run/secrets/sa",
|
||||
},
|
||||
],
|
||||
readinessProbe: {
|
||||
httpGet: {
|
||||
path: "/healthz",
|
||||
port: 80,
|
||||
scheme: "HTTP",
|
||||
},
|
||||
periodSeconds: 5,
|
||||
timeoutSeconds: 5,
|
||||
successThreshold: 1,
|
||||
failureThreshold: 2,
|
||||
},
|
||||
},
|
||||
],
|
||||
volumes: [
|
||||
{
|
||||
name: "sa-key",
|
||||
secret: {
|
||||
secretName: params.secretName,
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
},
|
||||
}, // endpointsDeploy
|
||||
endpointsDeploy:: endpointsDeploy,
|
||||
|
||||
local endpointsCompositeController = {
|
||||
apiVersion: "metacontroller.k8s.io/v1alpha1",
|
||||
kind: "CompositeController",
|
||||
metadata: {
|
||||
name: "cloud-endpoints-controller",
|
||||
},
|
||||
spec: {
|
||||
generateSelector: true,
|
||||
resyncPeriodSeconds: 2,
|
||||
parentResource: {
|
||||
apiVersion: "ctl.isla.solutions/v1",
|
||||
resource: "cloudendpoints",
|
||||
},
|
||||
childResources: [],
|
||||
clientConfig: {
|
||||
service: {
|
||||
name: "cloud-endpoints-controller",
|
||||
namespace: params.namespace,
|
||||
caBundle: "...",
|
||||
},
|
||||
},
|
||||
hooks: {
|
||||
sync: {
|
||||
webhook: {
|
||||
url: "http://cloud-endpoints-controller." + params.namespace + "/sync",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
}, // endpointsCompositeController
|
||||
endpointsCompositeController:: endpointsCompositeController,
|
||||
|
||||
parts:: self,
|
||||
local all = [
|
||||
self.endpointsCRD,
|
||||
self.endpointsClusterRole,
|
||||
self.endpointsClusterRoleBinding,
|
||||
self.endpointsService,
|
||||
self.endpointsServiceAccount,
|
||||
self.endpointsDeploy,
|
||||
self.endpointsCompositeController,
|
||||
],
|
||||
all:: all,
|
||||
|
||||
list(obj=self.all):: k.core.v1.list.new(obj,),
|
||||
},
|
||||
}
|
||||
71
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/gcp/configure_envoy_for_iap.sh
vendored
Normal file
71
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/gcp/configure_envoy_for_iap.sh
vendored
Normal file
|
|
@ -0,0 +1,71 @@
|
|||
#!/usr/bin/env bash
|
||||
#
|
||||
# A script to modify envoy config to perform JWT validation
|
||||
# given the information for the service.
|
||||
# Script executed by the iap container to configure IAP. When finished, the envoy config is created with the JWT audience.
|
||||
|
||||
[ -z ${NAMESPACE} ] && echo Error NAMESPACE must be set && exit 1
|
||||
[ -z ${SERVICE} ] && echo Error SERVICE must be set && exit 1
|
||||
|
||||
PROJECT=$(curl -s -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/project/project-id)
|
||||
if [ -z ${PROJECT} ]; then
|
||||
echo Error unable to fetch PROJECT from compute metadata
|
||||
exit 1
|
||||
fi
|
||||
|
||||
PROJECT_NUM=$(curl -s -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/project/numeric-project-id)
|
||||
if [ -z ${PROJECT_NUM} ]; then
|
||||
echo Error unable to fetch PROJECT_NUM from compute metadata
|
||||
exit 1
|
||||
fi
|
||||
|
||||
checkIAP() {
|
||||
# created by init container.
|
||||
. /var/shared/healthz.env
|
||||
|
||||
# If node port or backend id change, so does the JWT audience.
|
||||
CURR_NODE_PORT=$(kubectl --namespace=${NAMESPACE} get svc ${SERVICE} -o jsonpath='{.spec.ports[0].nodePort}')
|
||||
CURR_BACKEND_ID=$(gcloud compute --project=${PROJECT} backend-services list --filter=name~k8s-be-${CURR_NODE_PORT}- --format='value(id)')
|
||||
[ "$BACKEND_ID" == "$CURR_BACKEND_ID" ]
|
||||
}
|
||||
|
||||
# Activate the service account
|
||||
for i in $(seq 1 10); do
|
||||
gcloud auth activate-service-account --key-file=${GOOGLE_APPLICATION_CREDENTIALS} && break || sleep 10
|
||||
done
|
||||
|
||||
# Print out the config for debugging
|
||||
gcloud config list
|
||||
|
||||
NODE_PORT=$(kubectl --namespace=${NAMESPACE} get svc ${SERVICE} -o jsonpath='{.spec.ports[0].nodePort}')
|
||||
while [[ -z ${BACKEND_ID} ]]; do
|
||||
BACKEND_ID=$(gcloud compute --project=${PROJECT} backend-services list --filter=name~k8s-be-${NODE_PORT}- --format='value(id)')
|
||||
echo "Waiting for backend id PROJECT=${PROJECT} NAMESPACE=${NAMESPACE} SERVICE=${SERVICE} filter=name~k8s-be-${NODE_PORT}-..."
|
||||
sleep 2
|
||||
done
|
||||
echo BACKEND_ID=${BACKEND_ID}
|
||||
|
||||
NODE_PORT=$(kubectl --namespace=${NAMESPACE} get svc ${SERVICE} -o jsonpath='{.spec.ports[0].nodePort}')
|
||||
BACKEND_SERVICE=$(gcloud --project=${PROJECT} compute backend-services list --filter=name~k8s-be-${NODE_PORT}- --uri)
|
||||
|
||||
JWT_AUDIENCE="/projects/${PROJECT_NUM}/global/backendServices/${BACKEND_ID}"
|
||||
|
||||
# For healthcheck compare.
|
||||
echo "JWT_AUDIENCE=${JWT_AUDIENCE}" > /var/shared/healthz.env
|
||||
echo "NODE_PORT=${NODE_PORT}" >> /var/shared/healthz.env
|
||||
echo "BACKEND_ID=${BACKEND_ID}" >> /var/shared/healthz.env
|
||||
|
||||
kubectl get configmap -n ${NAMESPACE} envoy-config -o jsonpath='{.data.envoy-config\.json}' |
|
||||
sed -e "s|{{JWT_AUDIENCE}}|${JWT_AUDIENCE}|g" >/var/shared/envoy-config.json
|
||||
|
||||
echo "Restarting envoy"
|
||||
curl -s ${ENVOY_ADMIN}/quitquitquit
|
||||
|
||||
# Verify IAP every 10 seconds.
|
||||
while true; do
|
||||
if ! checkIAP; then
|
||||
echo "$(date) WARN: IAP check failed, restarting container."
|
||||
exit 1
|
||||
fi
|
||||
sleep 10
|
||||
done
|
||||
|
|
@ -0,0 +1,115 @@
|
|||
#!/bin/bash
|
||||
|
||||
set -e
|
||||
|
||||
usage() {
|
||||
cat <<EOF
|
||||
Generate certificate suitable for use with an sidecar-injector webhook service.
|
||||
This script uses k8s' CertificateSigningRequest API to a generate a
|
||||
certificate signed by k8s CA suitable for use with sidecar-injector webhook
|
||||
services. This requires permissions to create and approve CSR. See
|
||||
https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster for
|
||||
detailed explantion and additional instructions.
|
||||
The server key/cert k8s CA cert are stored in a k8s secret.
|
||||
usage: ${0} [OPTIONS]
|
||||
The following flags are required.
|
||||
--service Service name of webhook.
|
||||
--namespace Namespace where webhook service and secret reside.
|
||||
--secret Secret name for CA certificate and server certificate/key pair.
|
||||
EOF
|
||||
exit 1
|
||||
}
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case ${1} in
|
||||
--service)
|
||||
service="$2"
|
||||
shift
|
||||
'
|
||||
--secret)
|
||||
secret="$2"
|
||||
shift
|
||||
'
|
||||
--namespace)
|
||||
namespace="$2"
|
||||
shift
|
||||
'
|
||||
*)
|
||||
usage
|
||||
'
|
||||
esac
|
||||
shift
|
||||
done
|
||||
|
||||
[ -z ${service} ] && service=gcp-cred-webhook
|
||||
[ -z ${secret} ] && secret=gcp-cred-webhook-certs
|
||||
[ -z ${namespace} ] && namespace=${NAMESPACE}
|
||||
[ -z ${namespace} ] && namespace=default
|
||||
|
||||
echo ${service}
|
||||
echo ${namespace}
|
||||
echo ${secret}
|
||||
|
||||
if [ ! -x "$(command -v openssl)" ]; then
|
||||
echo "openssl not found"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
csrName=${service}.${namespace}
|
||||
tmpdir=$(mktemp -d)
|
||||
echo "creating certs in tmpdir ${tmpdir} "
|
||||
|
||||
# x509 outputs a self signed certificate instead of certificate request, later used as self signed root CA
|
||||
openssl req -x509 -newkey rsa:2048 -keyout ${tmpdir}/self_ca.key -out ${tmpdir}/self_ca.crt -days 365 -nodes -subj /C=/ST=/L=/O=/OU=/CN=test-certificate-authority
|
||||
|
||||
cat <<EOF >> ${tmpdir}/csr.conf
|
||||
[req]
|
||||
req_extensions = v3_req
|
||||
distinguished_name = req_distinguished_name
|
||||
[req_distinguished_name]
|
||||
[ v3_req ]
|
||||
basicConstraints = CA:FALSE
|
||||
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
|
||||
extendedKeyUsage = serverAuth
|
||||
subjectAltName = @alt_names
|
||||
[alt_names]
|
||||
DNS.1 = ${service}
|
||||
DNS.2 = ${service}.${namespace}
|
||||
DNS.3 = ${service}.${namespace}.svc
|
||||
EOF
|
||||
|
||||
openssl genrsa -out ${tmpdir}/server-key.pem 2048
|
||||
openssl req -new -key ${tmpdir}/server-key.pem -subj "/CN=${service}.${namespace}.svc" -out ${tmpdir}/server.csr -config ${tmpdir}/csr.conf
|
||||
|
||||
# Self sign
|
||||
openssl x509 -req -days 365 -in ${tmpdir}/server.csr -CA ${tmpdir}/self_ca.crt -CAkey ${tmpdir}/self_ca.key -CAcreateserial -out ${tmpdir}/server-cert.pem
|
||||
|
||||
# create the secret with CA cert and server cert/key
|
||||
kubectl create secret generic ${secret} \
|
||||
--from-file=key.pem=${tmpdir}/server-key.pem \
|
||||
--from-file=cert.pem=${tmpdir}/server-cert.pem \
|
||||
--dry-run -o yaml |
|
||||
kubectl -n ${namespace} apply -f -
|
||||
|
||||
cat ${tmpdir}/self_ca.crt
|
||||
# -a means base64 encode
|
||||
caBundle=`cat ${tmpdir}/self_ca.crt | openssl enc -a -A`
|
||||
echo ${caBundle}
|
||||
|
||||
patchString='[{"op": "replace", "path": "/webhooks/0/clientConfig/caBundle", "value":"{{CA_BUNDLE}}"}]'
|
||||
patchString=`echo ${patchString} | sed "s|{{CA_BUNDLE}}|${caBundle}|g"`
|
||||
echo ${patchString}
|
||||
|
||||
checkWebhookConfig() {
|
||||
currentBundle=$(kubectl get mutatingwebhookconfigurations -n ${namespace} gcp-cred-webhook -o jsonpath='{.webhooks[0].clientConfig.caBundle}')
|
||||
[[ "$currentBundle" == "$caBundle" ]]
|
||||
}
|
||||
|
||||
while true; do
|
||||
if ! checkWebhookConfig; then
|
||||
echo "patching ca bundle for webhook configuration..."
|
||||
kubectl patch mutatingwebhookconfiguration gcp-cred-webhook \
|
||||
--type='json' -p="${patchString}"
|
||||
fi
|
||||
sleep 10
|
||||
done
|
||||
104
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/gcp/google-cloud-filestore-pv.libsonnet
vendored
Normal file
104
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/gcp/google-cloud-filestore-pv.libsonnet
vendored
Normal file
|
|
@ -0,0 +1,104 @@
|
|||
{
|
||||
local k = import "k.libsonnet",
|
||||
new(_env, _params):: {
|
||||
local params = _params + _env,
|
||||
|
||||
local persistentVolume = {
|
||||
apiVersion: "v1",
|
||||
kind: "PersistentVolume",
|
||||
metadata: {
|
||||
name: params.name,
|
||||
namespace: params.namespace,
|
||||
},
|
||||
spec: {
|
||||
capacity: {
|
||||
storage: params.storageCapacity,
|
||||
},
|
||||
accessModes: [
|
||||
"ReadWriteMany",
|
||||
],
|
||||
nfs: {
|
||||
path: params.path,
|
||||
server: params.serverIP,
|
||||
},
|
||||
},
|
||||
},
|
||||
persistentVolume:: persistentVolume,
|
||||
|
||||
local persistentVolumeClaim = {
|
||||
apiVersion: "v1",
|
||||
kind: "PersistentVolumeClaim",
|
||||
metadata: {
|
||||
name: params.name,
|
||||
namespace: params.namespace,
|
||||
},
|
||||
spec: {
|
||||
accessModes: [
|
||||
"ReadWriteMany",
|
||||
],
|
||||
storageClassName: "nfs-storage",
|
||||
volumeName: params.name,
|
||||
resources: {
|
||||
requests: {
|
||||
storage: params.storageCapacity,
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
persistentVolumeClaim:: persistentVolumeClaim,
|
||||
|
||||
// Set 777 permissions on the GCFS NFS so that non-root users
|
||||
// like jovyan can use that NFS share
|
||||
local gcfsPersmissions = {
|
||||
apiVersion: "batch/v1",
|
||||
kind: "Job",
|
||||
metadata: {
|
||||
name: "set-gcfs-permissions",
|
||||
namespace: params.namespace,
|
||||
},
|
||||
spec: {
|
||||
template: {
|
||||
spec: {
|
||||
containers: [
|
||||
{
|
||||
name: "set-gcfs-permissions",
|
||||
image: params.image,
|
||||
command: [
|
||||
"chmod",
|
||||
"777",
|
||||
"/kubeflow-gcfs",
|
||||
],
|
||||
volumeMounts: [
|
||||
{
|
||||
mountPath: "/kubeflow-gcfs",
|
||||
name: params.name,
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
restartPolicy: "OnFailure",
|
||||
volumes: [
|
||||
{
|
||||
name: params.name,
|
||||
persistentVolumeClaim: {
|
||||
claimName: params.name,
|
||||
readOnly: false,
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
gcfsPersmissions:: gcfsPersmissions,
|
||||
|
||||
parts:: self,
|
||||
all:: [
|
||||
self.persistentVolume,
|
||||
self.persistentVolumeClaim,
|
||||
self.gcfsPersmissions,
|
||||
],
|
||||
|
||||
list(obj=self.all):: k.core.v1.list.new(obj,),
|
||||
},
|
||||
}
|
||||
130
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/gcp/gpu-driver.libsonnet
vendored
Normal file
130
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/gcp/gpu-driver.libsonnet
vendored
Normal file
|
|
@ -0,0 +1,130 @@
|
|||
{
|
||||
local k = import "k.libsonnet",
|
||||
new(_env, _params):: {
|
||||
local params = _params + _env,
|
||||
|
||||
local daemonset = {
|
||||
"apiVersion": "extensions/v1beta1",
|
||||
"kind": "DaemonSet",
|
||||
"metadata": {
|
||||
"name": "nvidia-driver-installer",
|
||||
"namespace": "kube-system",
|
||||
"labels": {
|
||||
"k8s-app": "nvidia-driver-installer"
|
||||
}
|
||||
},
|
||||
"spec": {
|
||||
"template": {
|
||||
"metadata": {
|
||||
"labels": {
|
||||
"name": "nvidia-driver-installer",
|
||||
"k8s-app": "nvidia-driver-installer"
|
||||
}
|
||||
},
|
||||
"spec": {
|
||||
"affinity": {
|
||||
"nodeAffinity": {
|
||||
"requiredDuringSchedulingIgnoredDuringExecution": {
|
||||
"nodeSelectorTerms": [
|
||||
{
|
||||
"matchExpressions": [
|
||||
{
|
||||
"key": "cloud.google.com/gke-accelerator",
|
||||
"operator": "Exists"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"tolerations": [
|
||||
{
|
||||
"operator": "Exists"
|
||||
}
|
||||
],
|
||||
"hostNetwork": true,
|
||||
"hostPID": true,
|
||||
"volumes": [
|
||||
{
|
||||
"name": "dev",
|
||||
"hostPath": {
|
||||
"path": "/dev"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "nvidia-install-dir-host",
|
||||
"hostPath": {
|
||||
"path": "/home/kubernetes/bin/nvidia"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "root-mount",
|
||||
"hostPath": {
|
||||
"path": "/"
|
||||
}
|
||||
}
|
||||
],
|
||||
"initContainers": [
|
||||
{
|
||||
"image": "cos-nvidia-installer:fixed",
|
||||
"imagePullPolicy": "Never",
|
||||
"name": "nvidia-driver-installer",
|
||||
"resources": {
|
||||
"requests": {
|
||||
"cpu": 0.15
|
||||
}
|
||||
},
|
||||
"securityContext": {
|
||||
"privileged": true
|
||||
},
|
||||
"env": [
|
||||
{
|
||||
"name": "NVIDIA_INSTALL_DIR_HOST",
|
||||
"value": "/home/kubernetes/bin/nvidia"
|
||||
},
|
||||
{
|
||||
"name": "NVIDIA_INSTALL_DIR_CONTAINER",
|
||||
"value": "/usr/local/nvidia"
|
||||
},
|
||||
{
|
||||
"name": "ROOT_MOUNT_DIR",
|
||||
"value": "/root"
|
||||
}
|
||||
],
|
||||
"volumeMounts": [
|
||||
{
|
||||
"name": "nvidia-install-dir-host",
|
||||
"mountPath": "/usr/local/nvidia"
|
||||
},
|
||||
{
|
||||
"name": "dev",
|
||||
"mountPath": "/dev"
|
||||
},
|
||||
{
|
||||
"name": "root-mount",
|
||||
"mountPath": "/root"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"containers": [
|
||||
{
|
||||
"image": "gcr.io/google-containers/pause:2.0",
|
||||
"name": "pause"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
daemonset:: daemonset,
|
||||
|
||||
parts:: self,
|
||||
all:: [
|
||||
self.daemonset,
|
||||
],
|
||||
|
||||
list(obj=self.all):: k.core.v1.list.new(obj,),
|
||||
},
|
||||
}
|
||||
51
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/gcp/healthcheck_route.yaml
vendored
Normal file
51
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/gcp/healthcheck_route.yaml
vendored
Normal file
|
|
@ -0,0 +1,51 @@
|
|||
apiVersion: networking.istio.io/v1alpha3
|
||||
kind: VirtualService
|
||||
metadata:
|
||||
name: default-routes
|
||||
namespace: kubeflow
|
||||
spec:
|
||||
hosts:
|
||||
- "*"
|
||||
gateways:
|
||||
- kubeflow-gateway
|
||||
http:
|
||||
- match:
|
||||
- uri:
|
||||
exact: /healthz
|
||||
route:
|
||||
- destination:
|
||||
port:
|
||||
number: 80
|
||||
host: whoami-app.kubeflow.svc.cluster.local
|
||||
- match:
|
||||
- uri:
|
||||
exact: /whoami
|
||||
route:
|
||||
- destination:
|
||||
port:
|
||||
number: 80
|
||||
host: whoami-app.kubeflow.svc.cluster.local
|
||||
- match:
|
||||
- uri:
|
||||
prefix: /
|
||||
route:
|
||||
- destination:
|
||||
port:
|
||||
number: 80
|
||||
host: ambassador.kubeflow.svc.cluster.local
|
||||
---
|
||||
apiVersion: networking.istio.io/v1alpha3
|
||||
kind: Gateway
|
||||
metadata:
|
||||
name: kubeflow-gateway
|
||||
namespace: kubeflow
|
||||
spec:
|
||||
selector:
|
||||
istio: ingressgateway
|
||||
servers:
|
||||
- port:
|
||||
number: 80
|
||||
name: http
|
||||
protocol: HTTP
|
||||
hosts:
|
||||
- "*"
|
||||
1046
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/gcp/iap.libsonnet
vendored
Normal file
1046
pipelines/azurepipeline/code/kflow/ks_app/vendor/kubeflow/gcp/iap.libsonnet
vendored
Normal file
File diff suppressed because it is too large
Load Diff
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue