Automate release process (#7601)

* script part one

* first draft

* save progress

* update README (i)

* try script

* add more echos

* even more echos

* complete list of git add

* change git add to --all

* separate remake archive

* change --all to -A

* fix path

* add credential helper

* fix path

* build an archive of v1.6 in tori-release

* add master variable

* build an archive of v1.6 in tori-release

* update data/versions.yml and archive index page

* finish redo_archive

* add patch release

* git add

* fix go command

* complete scripts

* make linters happy

* update readme

* update readme

* add script for prepare release

* add security patch arg

* fix release note path

* fix param expansion

* update readme

* fix typo

* refine patch release script

* add git pull to make sure everything is up to date

* ensure idempotence

* add dry run option

* fix func arg

* fix brackets

* allow existing branches

* fix arg bug

* remove git config

* update readme

* update readme

* remove unnecessary note

* update readme

* update readme

* fix indentation

* fix format bug

* force mv

* fix mv command

* rm previous archive

* fix extra char

* update readme

* change 1.4 example to 1.7; remove 'the'

* change 'set -e' position

* add function doc; do git pull --ff-only

* rename target to fetch-archive

* rename to PRIVATE_PATCH; remove release note template

* update readme

* add archive build in new release dry run

* rename to build_old_archive

* build archive before edit

* set upstream

* fix language

* rm set upstream

* delete dry run branch if exists

* change wordings

* make linter happy

* patch release changes

* rm args.yml edit

* change wordings

* change heading

* change gcse in readme

* Update README.md

* rm patch release automation

* Update README.md
This commit is contained in:
Hongyi Zhang 2020-07-10 12:27:23 -04:00 committed by GitHub
parent 6711cc7bd3
commit 583d8ff0f7
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
5 changed files with 307 additions and 107 deletions

View File

@ -139,6 +139,22 @@ update_all: update_ref_docs update_examples
foo2:
hugo version
# Release related targets
export ISTIOIO_GIT_SOURCE := https://github.com/istio/istio.io.git
export MASTER := master
prepare-%:
@scripts/prepare_release.sh $@
release-%-dry-run:
@DRY_RUN=1 scripts/create_version.sh $(subst -dry-run,,$@)
release-%:
@scripts/create_version.sh $@
build-old-archive-%:
@scripts/build_old_archive.sh $@
# The init recipe was split into two recipes to solve an issue seen in prow
# where paralyzation is happening and some tasks in a recipe were occuring out
# of order. The desired behavior is for `preinit` to do the clone and set up the

136
README.md
View File

@ -6,8 +6,8 @@
## istio.io
This repository contains the source code for the [istio.io](https://istio.io),
[preliminary.istio.io](https://preliminary.istio.io), and [archive.istio.io](https://archive.istio.io) sites.
This repository contains the source code for [istio.io](https://istio.io) and
[preliminary.istio.io](https://preliminary.istio.io).
Please see the main Istio [README](https://github.com/istio/istio/blob/master/README.md)
file to learn about the overall Istio project and how to get in touch with us. To learn how you can
@ -18,7 +18,7 @@ see the Istio [contribution guidelines](https://github.com/istio/community/blob/
- [Versions and releases](#versions-and-releases)
- [How versioning works](#how-versioning-works)
- [Publishing content immediately](#publishing-content-immediately)
- [Creating a version](#creating-a-version)
- [Creating a major/minor release](#creating-a-majorminor-release)
- [Creating a patch release](#creating-a-patch-release)
- [Multi-language support](#multi-language-support)
- [Regular maintenance](#regular-maintenance)
@ -30,17 +30,15 @@ To learn how to edit and build this repo's content, please refer to
## Versions and releases
Istio maintains three variations of its public site.
Istio maintains two variations of its public site.
- [istio.io](https://istio.io) is the main site, showing documentation for the current release of the product.
- [archive.istio.io](https://archive.istio.io) contains snapshots of the documentation for previous releases of the product.
This is useful for customers still using these older releases.
[istio.io/archive](https://istio.io/archive) contains snapshots of the documentation for previous releases of the product. This is useful for customers still using these older releases.
- [preliminary.istio.io](https://preliminary.istio.io) contains the actively updated documentation for the next release of the product.
The user can trivially navigate between the different variations of the site using the gear menu in the top right
of each page. All three sites are hosted on [Netlify](https://netlify.com).
of each page. Both sites are hosted on [Netlify](https://netlify.com).
### How versioning works
@ -50,152 +48,76 @@ are automatically reflected on preliminary.istio.io.
- The content of istio.io is taken from the latest release-XXX branch. The specific branch that
is used is determined by the istio.io [Netlify](https://netlify.com) project's configuration.
- The content of archive.istio.io is taken from the older release-XXX branches. The set of branches that
are included on archive.istio.io is determined by the `TOBUILD` variable in this
[script](https://github.com/istio/istio.io/blob/master/scripts/build_archive_site.sh).
### Publishing content immediately
Checking in updates to the master branch will automatically update preliminary.istio.io, and will only be reflected on
istio.io the next time a release is created, which can be several weeks in the future. If you'd like some changes to be
immediately reflected on istio.io, you need to check your changes both to the master branch and to the
current release branch (named release-XXX such as release-1.4).
current release branch (named `release-<MAJOR>.<MINOR>` such as `release-1.7`).
This process can be taken care of automatically by our infrastructure. If you submit a PR
to the master branch and annotate the PR with the `actions/merge-to-release-branch` label,
then as soon as your PR is merged into master, it will be merged into the current release branch.
### Creating a version
### Creating a major/minor release
Here are the steps necessary to create a new documentation version. Let's assume the current
version of Istio is 1.3 and you wish to introduce 1.4 which has been under development.
version of Istio is 1.6 and you wish to introduce 1.7 which has been under development.
#### When Istio source code is branched
The documentation repo pulls content from the Istio source repos for inclusion in the published site.
When the source repos are branched in preparation for a release, a few changes are needed in the
documentation repo to track this:
1. Switch to the **master** branch of the istio/istio.io repo and make sure everything is up to date.
1. Edit the file `Makefile.core.mk` and change the `SOURCE_BRANCH_NAME` variable to the
name of the newly created source branches (in this case `release-1.4`).
1. Edit the file `data/args.yml` and set the `source_branch_name` field to the name of the newly created source
branches (in this case `release-1.4`).
1. Run `make update_all` in order to retrieve the latest material from the source repositories.
1. Commit the previous edits to your local git repo and push your **master** branch to GitHub.
Run `make prepare-1.7.0`, and that's it. This will grab the latest material from the new istio source branch.
#### On the day of the release
##### Creating the release branch
1. Run `make release-1.7.0`. This make target will change some variables in `master` and `release-1.6` as needed, and create a new branch `release-1.7` for the new version.
The day of a major Istio release, assuming you've previously done the steps from the above section, you need to:
- For a dry run before official release, run `make release-1.7.0-dry-run`, which will only create a new branch `release-1.7-dry-run`, and not touch any other branches.
1. Switch to the **master** branch of the istio/istio.io repo and make sure everything is up to date.
1. Go to the istio.io project on [Netlify](https://netlify.com) and do the following:
1. Edit the file `scripts/build_archive_site.sh` and add the new archive version
(in this case `release-1.3`) to the `TOBUILD` variable.
- Change the branch that is built from the previous release's branch to the new release branch, in this case `release-1.7` (or `release-1.7-dry-run` as appropriate).
1. Edit the file `data/versions.yml`. Set the `preliminary` field to the next Istio release
(in this case `1.5`) and the `main` field to the current release (in this case `1.4`).
- Select the option to trigger an immediate rebuild and redeployment.
1. Commit the previous edits to your local git repo and push your **master** branch to GitHub.
1. Create a new release branch off of master, named as release-**major**.**minor** (in this case `release-1.4`). There is one
such branch for every release.
1. Edit the file `data/args.yml`. Set the `preliminary` field to `false`
and the the `doc_branch_name` field to the name of the release branch (in this case `release-1.4`).
1. Commit the previous edits to your local git repo and push your **release** branch to GitHub.
#### Updating istio.io
1. Go to the istio.io project on [Netlify](https://netlify.com)
1. Change the branch that is built from the previous release's branch to the new release branch, in this case release-1.4
1. Select the option to trigger an immediate rebuild and redeployment.
1. Once deployment is done, browse istio.io and make sure everything looks good.
##### Updating archive.istio.io
- Once deployment is done, browse istio.io and make sure everything looks good.
1. Go to the [Google Custom Search Engine](https://cse.google.com) and do the following:
- Download the archive.istio.io CSE context file from the Advanced tab.
- Download the istio.io CSE context file from the Advanced tab.
- Add a new FacetItem at the top of the file containing the previous release's version number. In
this case, this would be "V1.3".
this case, this would be `V1.6`.
- Upload the updated CSE context file to the site.
- In the Setup section, add a new site that covers the previous release's archive directory. In this
case, the site URL would be archive.istio.io/v1.3/*. Set the label of this site to the name of the
facet item created above (V1.3 in this case).
1. In the **previous release's** branch (in this case `release-1.3`), edit the file `data/args.yml`. Set the
`archive` field to true and the `archive_date` field to the current date, and the `archive_search_refinement`
to the previous release version (in this case `V1.3`).
1. In the **previous release's** branch (in this case `release-1.3`), edit the file `config.toml`. Set the
`disableAliases` field to `false`.
1. Commit the previous edits to your local git repo and push the **previous release's** branch to GitHub.
1. In the **archive** branch, rebase the branch to have all changes from the current release. In this case,
all changes from the `release-1.4` branch.
1. Commit the previous edits to your local git repo and push the **archive** branch to GitHub.
1. Wait a while (~20 minutes) and browse archive.istio.io to make sure everything looks good.
##### Updating preliminary.istio.io
1. In the **master** branch, edit the file `data/args.yml`. Set the `version` and `full_version` fields to have the version
of the next Istio release, and `previous_version` to be the version of the previous release. In this case, you would set the fields to
"1.5", "1.5.0", and "1.4" respectively.
1. In the **master** branch, edit the file `data/args.yml`. Set the
`source_branch_name` and `doc_branch_name` fields to `master`.
1. In the **master** branch, edit the file `Makefile.core.mk`. Set the variable `SOURCE_BRANCH_NAKE` to
`master`.
1. Run `make update_all` in order to retrieve the latest material from the source repositories.
1. Commit the previous edits to your local git repo and push the **master** branch to GitHub.
1. Wait a while (~5 minutes) and browse preliminary.istio.io to make sure everything looks good.
case, the site URL would be `istio.io/v1.6/*`. Set the label of this site to the name of the
facet item created above (`V1.6` in this case).
### Creating a patch release
A few days before the patch release, the release managers should notify the Doc WG that the release
is built and is starting it's long running qualification test. At this time, move the doc automation
tests to use the new release to verify automated doc testing passes. To move to a new release
(make sure you are in the patch's release branch):
tests to use the new release to verify automated doc testing passes.
1. `go get istio.io/istio@1.X.Y`
To move to a new release (make sure you are in the patch's release branch):
1. `go mod tidy`
1. Run `go get istio.io/istio@A.X.Y && go mod tidy`.
1. Create a PR with the go.* changes.
1. Create a PR with the `go.*` changes.
Creating a new patch release involves modifying a few files:
1. Create the release note for the release by adding a markdown file in
`content/en/news/<YEAR>/1.X.Y/index.md`, where 1.X.Y is the name of the release. This is where
you describe the changes in the release.
1. Edit `data/args.yml` and change the `full_version` field to `"A.X.Y"`.
1. Edit the `data/args.yml` file and change the `full_version` field to the name of the release.
1. Complete the release note for the release by editing the markdown file `content/en/news/releases/A.X.x/announcing-A.X.Y/index.md`. This is where you describe the changes in the release. Please look at other existing files for example content and layout.
1. Run `make update_ref_docs` to get the latest reference docs.
For the release note file, please look at existing files in the same location for example content and
layout.
### Updating an archive
If the archived version in a newer branch (e.g., `release-1.7:archive/v1.6`) needs to be updated due to changes in the old release branch (`release-1.6` in this case), you can run `build-old-archive-1.6.0` in the `release-1.7` branch, which will re-archive `release-1.6` and substitute it for the previous archive in the current branch.
## Multi-language support

44
scripts/build_old_archive.sh Executable file
View File

@ -0,0 +1,44 @@
#!/bin/bash
# Copyright Istio Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# NOTE: this only works for v1.5+ because it needs an `archive-version`
# make target to create an archive.
set -e
[[ $1 =~ ^build-old-archive-([0-9]\.[0-9]+)\.0$ ]] ||
{ echo "Target format error: should be 'build-old-archive-x.x.0', got '$1'"; exit 1; }
VERSION="${BASH_REMATCH[1]}"
CURRENT_BRANCH=$(git rev-parse --abbrev-ref HEAD)
ARCHIVE_BRANCH="release-${VERSION}"
git checkout "${ARCHIVE_BRANCH}"
git pull --ff-only "${ISTIOIO_GIT_SOURCE}" "${ARCHIVE_BRANCH}"
echo "Making an archive for ${ARCHIVE_BRANCH}..."
make archive-version
git checkout "${CURRENT_BRANCH}"
rm -rf "archive/v${VERSION}"
mv "archived_version/v${VERSION}" archive
if [[ $(git status --porcelain) ]]; then
git add "archive/v${VERSION}"
git commit -m "build an archive of v${VERSION} in ${CURRENT_BRANCH}"
git push origin "${CURRENT_BRANCH}"
fi

181
scripts/create_version.sh Executable file
View File

@ -0,0 +1,181 @@
#!/bin/bash
# Copyright Istio Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# parse_input function parses the name of the new release, determines
# the type of the release, and runs scripts accordingly
parse_input() {
[[ $1 =~ ^release-([0-9])\.([0-9]+)\.([0-9]+)$ ]] ||
{ echo "Target format error: should be 'release-x.x.x', got '$1'"; exit 1; }
MAJOR="${BASH_REMATCH[1]}"
MINOR="${BASH_REMATCH[2]}"
PATCH="${BASH_REMATCH[3]}"
echo "Creating release for ${MAJOR}.${MINOR}.${PATCH}..."
# patch release
if [ "${PATCH}" != '0' ]; then
echo "Patch release automation is currently not supported"
exit 0
fi
# major/minor release
CURR_MINOR="${MAJOR}.${MINOR}" # current version
PREV_MINOR="${MAJOR}.$((MINOR-1))" # previous version
NEXT_MINOR="${MAJOR}.$((MINOR+1))" # next version
if [ "${DRY_RUN}" == '1' ]; then
CURR_MINOR="${CURR_MINOR}-dry-run"
git checkout "${MASTER}"
git pull --ff-only "${ISTIOIO_GIT_SOURCE}" "${MASTER}"
fi
# for a major release x.0, find the latest minor release
if [ "${MINOR}" == '0' ]; then
LAST_MINOR_OF_PREV_MAJOR=$(
git branch -a |
grep "release-$((MAJOR-1))." |
sed -r "s/^.*release-$((MAJOR-1))\.([0-9]+)$/\1/" |
sort -n |
tail -1
)
PREV_MINOR="$((MAJOR-1)).${LAST_MINOR_OF_PREV_MAJOR}"
fi
echo "Previous minor release: ${PREV_MINOR}"
echo "Upcoming minor release: ${NEXT_MINOR}"
}
# archive_old_release function checks out to the old release branch,
# creates an archive, and stores the archive in the master branch
archive_old_release() {
echo -e "\nStep 1: archive the old release branch"
build_archive() {
scripts/build_old_archive.sh "build-old-archive-${PREV_MINOR}.0"
sed -i "
s/^preliminary: .*$/preliminary: \"${NEXT_MINOR}\"/;
s/^main: .*$/main: \"${CURR_MINOR}\"/
" data/versions.yml
# add list item to index page only once
INDEX_PAGE="archive/archive/index.html"
grep -q "<a\ href=/v${PREV_MINOR}>v${PREV_MINOR}</a>" ${INDEX_PAGE} ||
sed -i "0,/<li>/s//\<li>\n\
<a href=\/v${PREV_MINOR}>v${PREV_MINOR}<\/a>\n\
<\/li>\n\
<li>/" ${INDEX_PAGE}
}
if [ "${DRY_RUN}" == '1' ]; then
echo "Archive will be added in Step 2 for dry run"
return
fi
git checkout "release-${PREV_MINOR}"
git pull --ff-only "${ISTIOIO_GIT_SOURCE}" "release-${PREV_MINOR}"
sed -i "
s/^archive: false$/archive: true/;
s/^archive_date: .*$/archive_date: $(date +'%Y-%m-%d')/;
s/^archive_search_refinement: .*$/archive_search_refinement: \"V${PREV_MINOR}\"/
" data/args.yml
sed -i "s/^disableAliases = true$/disableAliases = false/" config.toml
if [[ $(git status --porcelain) ]]; then # for idempotence
git add -u
git commit -m "mark v${PREV_MINOR} as archived"
git push origin "release-${PREV_MINOR}"
fi
# complete the archive process in master
git checkout "${MASTER}"
git pull --ff-only "${ISTIOIO_GIT_SOURCE}" "${MASTER}"
build_archive
if [[ $(git status --porcelain) ]]; then
git add -u
git commit -m "update data/versions.yml and archive index page"
git push origin "${MASTER}"
fi
}
# create_branch_for_new_release function creates a branch for the
# new release off the master branch and pushes it to origin
create_branch_for_new_release() {
NEW_RELEASE_BRANCH="release-${CURR_MINOR}"
echo -e "\nStep 2: create a new branch for ${NEW_RELEASE_BRANCH}"
# delete branch if it already exists
if [[ $(git ls-remote --heads origin "${NEW_RELEASE_BRANCH}") ]]; then
git push --delete origin "${NEW_RELEASE_BRANCH}"
fi
git checkout -B "${NEW_RELEASE_BRANCH}"
# make archive in the dry run release branch
if [ "${DRY_RUN}" == '1' ]; then
build_archive
fi
sed -i "
s/^preliminary: true$/preliminary: false/;
s/^doc_branch_name: .*$/doc_branch_name: ${NEW_RELEASE_BRANCH}/;
" data/args.yml
if [[ $(git status --porcelain) ]]; then
git add -A
git commit -m "create a new release branch for ${CURR_MINOR}"
git push origin "${NEW_RELEASE_BRANCH}"
fi
}
# advance_master_to_next_release function advances the master branch
# to the next release from which preliminary.istio.io is built
advance_master_to_next_release() {
echo -e "\nStep 3: advance master to release-${NEXT_MINOR}..."
if [ "${DRY_RUN}" == '1' ]; then
echo "Skipping step 3 in dry run"
return
fi
git checkout "${MASTER}"
sed -i "
s/^version: .*$/version: \"${NEXT_MINOR}\"/;
s/^full_version: .*$/full_version: \"${NEXT_MINOR}.0\"/;
s/^previous_version: .*$/previous_version: \"${CURR_MINOR}\"/;
s/^source_branch_name: .*$/source_branch_name: ${MASTER}/;
s/^doc_branch_name: .*$/doc_branch_name: ${MASTER}/
" data/args.yml
sed -i "s/^SOURCE_BRANCH_NAME ?=.*$/SOURCE_BRANCH_NAME ?= ${MASTER}/" Makefile.core.mk
make update_all
if [[ $(git status --porcelain) ]]; then
git add -A
git commit -m "advance master to release-${NEXT_MINOR}"
git push origin "${MASTER}"
fi
}
set -e
parse_input "$1"
archive_old_release
create_branch_for_new_release
advance_master_to_next_release
echo "[SUCCESS] New release now has been created in the branch 'release-${CURR_MINOR}'"

37
scripts/prepare_release.sh Executable file
View File

@ -0,0 +1,37 @@
#!/bin/bash
# Copyright Istio Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
set -e
[[ $1 =~ ^prepare-([0-9]\.[0-9]+)\.0$ ]] ||
{ echo "Target format error: should be 'prepare-x.x.0', got '$1'"; exit 1; }
VERSION="${BASH_REMATCH[1]}"
git checkout "${MASTER}"
git pull --ff-only "${ISTIOIO_GIT_SOURCE}" "${MASTER}"
sed -i "s/^source_branch_name: .*$/source_branch_name: release-${VERSION}/" data/args.yml
sed -i "s/^SOURCE_BRANCH_NAME ?=.*$/SOURCE_BRANCH_NAME ?= release-${VERSION}/" Makefile.core.mk
echo "Running make update_all..."
make update_all
if [[ $(git status --porcelain) ]]; then
git add -A
git commit -m "prepare for v${VERSION} as istio source is already branched"
git push origin "${MASTER}"
fi