Total overhaul of sync loop

The previous (v3) sync loop betrays my lack of understanding about git.
It tried to codify my archaic mental model (e.g. --branch and --rev
being disting things) and was ultimately a patchwork of corner-cases
evolved over a few years.

This commit is less of a "diff" and more of a "rewrite".

The new logic is simpler and more efficient.  It does not `git clone`
ever.  It does not differentiate the first sync from subsequent syncs.
It uses `git fetch` to get the exact SHA and then makes a worktree from
that.

The new `--ref` flag replaces both `--rev` and `--branch`, though it
will use those if specified.  In fact, almost all of the e2e tests
passed without change - using --ref and --branch!

I will follow this commit up with more cleanups and e2es.
This commit is contained in:
Tim Hockin 2023-02-12 18:08:42 -08:00
parent bb0128b883
commit ee6664748f
7 changed files with 439 additions and 667 deletions

View File

@ -135,10 +135,6 @@ OPTIONS
(200) and produce a series of key=value lines, including (200) and produce a series of key=value lines, including
"username=<value>" and "password=<value>". "username=<value>" and "password=<value>".
--branch <string>, $GIT_SYNC_BRANCH
The git branch to check out. If not specified, this defaults to
the default branch of --repo.
--change-permissions <int>, $GIT_SYNC_PERMISSIONS --change-permissions <int>, $GIT_SYNC_PERMISSIONS
Change permissions on the checked-out files to the specified mode. Change permissions on the checked-out files to the specified mode.
@ -165,7 +161,7 @@ OPTIONS
An optional command to be executed after syncing a new hash of the An optional command to be executed after syncing a new hash of the
remote repository. This command does not take any arguments and remote repository. This command does not take any arguments and
executes with the synced repo as its working directory. The executes with the synced repo as its working directory. The
environment variable $GITSYNC_HASH will be set to the git SHA that environment variable $GITSYNC_HASH will be set to the git hash that
was synced. The execution is subject to the overall --sync-timeout was synced. The execution is subject to the overall --sync-timeout
flag and will extend the effective period between sync attempts. flag and will extend the effective period between sync attempts.
This flag obsoletes --sync-hook-command, but if sync-hook-command This flag obsoletes --sync-hook-command, but if sync-hook-command
@ -180,14 +176,22 @@ OPTIONS
testing). This defaults to "git". testing). This defaults to "git".
--git-config <string>, $GIT_SYNC_GIT_CONFIG --git-config <string>, $GIT_SYNC_GIT_CONFIG
Additional git config options in 'key1:val1,key2:val2' format. The Additional git config options in a comma-separated 'key:val'
key parts are passed to 'git config' and must be valid syntax for format. The parsed keys and values are passed to 'git config' and
that command. The val parts can be either quoted or unquoted must be valid syntax for that command.
values. For all values the following escape sequences are
supported: '\n' => [newline], '\t' => [tab], '\"' => '"', '\,' => Both keys and values can be either quoted or unquoted strings.
',', '\\' => '\'. Within unquoted values, commas MUST be escaped. Within quoted keys and all values (quoted or not), the following
Within quoted values, commas MAY be escaped, but are not required escape sequences are supported:
to be. Any other escape sequence is an error. '\n' => [newline]
'\t' => [tab]
'\"' => '"'
'\,' => ','
'\\' => '\'
To include a colon within a key (e.g. a URL) the key must be
quoted. Within unquoted values commas must be escaped. Within
quoted values commas may be escaped, but are not required to be.
Any other escape sequence is an error.
--git-gc <string>, $GIT_SYNC_GIT_GC --git-gc <string>, $GIT_SYNC_GIT_GC
The git garbage collection behavior: one of "auto", "always", The git garbage collection behavior: one of "auto", "always",
@ -209,23 +213,27 @@ OPTIONS
The bind address (including port) for git-sync's HTTP endpoint. If The bind address (including port) for git-sync's HTTP endpoint. If
not specified, the HTTP endpoint is not enabled. not specified, the HTTP endpoint is not enabled.
Examples:
":1234": listen on any IP, port 1234
"127.0.0.1:1234": listen on localhost, port 1234
--http-metrics, $GIT_SYNC_HTTP_METRICS --http-metrics, $GIT_SYNC_HTTP_METRICS
Enable metrics on git-sync's HTTP endpoint, if it is enabled (see Enable metrics on git-sync's HTTP endpoint. Requires --http-bind
--http-bind). to be specified.
--http-pprof, $GIT_SYNC_HTTP_PPROF --http-pprof, $GIT_SYNC_HTTP_PPROF
Enable the pprof debug endpoints on git-sync's HTTP endpoint, if it Enable the pprof debug endpoints on git-sync's HTTP endpoint.
is enabled (see --http-bind). Requires --http-bind to be specified.
--link <string>, $GIT_SYNC_LINK --link <string>, $GIT_SYNC_LINK
The path to at which to create a symlink which points to the The path to at which to create a symlink which points to the
current git directory, at the currently synced SHA. This may be an current git directory, at the currently synced hash. This may be
absolute path or a relative path, in which case it is relative to an absolute path or a relative path, in which case it is relative
--root. The last path element is the name of the link and must not to --root. The last path element is the name of the link and must
start with a period. Consumers of the synced files should always not start with a period. Consumers of the synced files should
use this link - it is updated atomically and should always be always use this link - it is updated atomically and should always
valid. The basename of the target of the link is the current SHA. be valid. The basename of the target of the link is the current
If not specified, this defaults to the leaf dir of --repo. hash. If not specified, this defaults to the leaf dir of --repo.
--man --man
Print this manual and exit. Print this manual and exit.
@ -256,13 +264,14 @@ OPTIONS
will take precedence. If not specified, this defaults to 10 will take precedence. If not specified, this defaults to 10
seconds ("10s"). seconds ("10s").
--ref <string>, $GIT_SYNC_REF
The git revision (branch, tag, or hash) to check out. If not
specified, this defaults to "HEAD" (of the upstream repo's default
branch).
--repo <string>, $GIT_SYNC_REPO --repo <string>, $GIT_SYNC_REPO
The git repository to sync. This flag is required. The git repository to sync. This flag is required.
--rev <string>, $GIT_SYNC_REV
The git revision (tag or hash) to check out. If not specified,
this defaults to "HEAD".
--root <string>, $GIT_SYNC_ROOT --root <string>, $GIT_SYNC_ROOT
The root directory for git-sync operations, under which --link will The root directory for git-sync operations, under which --link will
be created. This must be a path that either a) does not exist (it be created. This must be a path that either a) does not exist (it
@ -294,6 +303,13 @@ OPTIONS
The git submodule behavior: one of "recursive", "shallow", or The git submodule behavior: one of "recursive", "shallow", or
"off". If not specified, this defaults to "recursive". "off". If not specified, this defaults to "recursive".
--sync-on-signal <string>, $GIT_SYNC_SYNC_ON_SIGNAL
Indicates that a sync attempt should occur upon receipt of the
specified signal name (e.g. SIGHUP) or number (e.g. 1). If a sync
is already in progress, another sync will be triggered as soon as
the current one completes. If not specified, signals will not
trigger syncs.
--sync-timeout <duration>, $GIT_SYNC_SYNC_TIMEOUT --sync-timeout <duration>, $GIT_SYNC_SYNC_TIMEOUT
The total time allowed for one complete sync. This must be at least The total time allowed for one complete sync. This must be at least
10ms. This flag obsoletes --timeout, but if --timeout is specified, 10ms. This flag obsoletes --timeout, but if --timeout is specified,
@ -335,14 +351,13 @@ OPTIONS
--webhook-url <string>, $GIT_SYNC_WEBHOOK_URL --webhook-url <string>, $GIT_SYNC_WEBHOOK_URL
A URL for optional webhook notifications when syncs complete. The A URL for optional webhook notifications when syncs complete. The
header 'Gitsync-Hash' will be set to the git SHA that was synced. header 'Gitsync-Hash' will be set to the git hash that was synced.
EXAMPLE USAGE EXAMPLE USAGE
git-sync \ git-sync \
--repo=https://github.com/kubernetes/git-sync \ --repo=https://github.com/kubernetes/git-sync \
--branch=main \ --ref=HEAD \
--rev=HEAD \
--period=10s \ --period=10s \
--root=/mnt/git --root=/mnt/git
@ -387,6 +402,7 @@ HOOKS
wait --exechook-backoff or --webhook-backoff (as appropriate) before wait --exechook-backoff or --webhook-backoff (as appropriate) before
re-trying the hook. re-trying the hook.
Hooks are not guaranteed to succeed on every single SHA change. For example, Hooks are not guaranteed to succeed on every single hash change. For example,
if a hook fails and a new SHA is synced during the backoff period, the if a hook fails and a new hash is synced during the backoff period, the
retried hook will fire for the newest SHA. retried hook will fire for the newest hash.
```

View File

@ -1,24 +0,0 @@
#!/bin/sh
#
# Copyright 2020 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
if [ "$1" != "clone" ]; then
git "$@"
exit $?
fi
sleep 1.1
git "$@"

View File

@ -20,5 +20,5 @@ if [ "$1" != "fetch" ]; then
exit $? exit $?
fi fi
sleep 5 sleep 2
git "$@" git "$@"

File diff suppressed because it is too large Load Diff

View File

@ -38,7 +38,6 @@ spec:
image: registry.k8s.io/git-sync/git-sync:v4.0.0 image: registry.k8s.io/git-sync/git-sync:v4.0.0
args: args:
- --repo=https://github.com/kubernetes/git-sync - --repo=https://github.com/kubernetes/git-sync
- --branch=master
- --depth=1 - --depth=1
- --period=60 - --period=60
- --link=current - --link=current

View File

@ -79,12 +79,11 @@ as user ID "65533" which is created for running git-sync as non-root.
# ... # ...
containers: containers:
- name: git-sync - name: git-sync
image: registry.k8s.io/git-sync:v3.1.5 image: registry.k8s.io/git-sync:v4.0.0
args: args:
- "--ssh" - "--ssh"
- "--repo=git@github.com:foo/bar" - "--repo=git@github.com:foo/bar"
- "--link=bar" - "--link=bar"
- "--branch=master"
volumeMounts: volumeMounts:
- name: git-secret - name: git-secret
mountPath: /etc/git-secret mountPath: /etc/git-secret
@ -138,12 +137,10 @@ spec:
defaultMode: 0400 defaultMode: 0400
containers: containers:
- name: git-sync - name: git-sync
image: registry.k8s.io/git-sync:v3.1.5 image: registry.k8s.io/git-sync:v4.0.0
args: args:
- "--ssh" - "--ssh"
- "--repo=git@github.com:torvalds/linux" - "--repo=git@github.com:torvalds/linux"
- "--link=linux"
- "--branch=master"
- "--depth=1" - "--depth=1"
securityContext: securityContext:
runAsUser: 65533 # git-sync user runAsUser: 65533 # git-sync user

View File

@ -212,7 +212,6 @@ ssh-keygen -f "$DOT_SSH/id_test" -P "" >/dev/null
cat "$DOT_SSH/id_test.pub" > "$DOT_SSH/authorized_keys" cat "$DOT_SSH/id_test.pub" > "$DOT_SSH/authorized_keys"
TEST_TOOLS="_test_tools" TEST_TOOLS="_test_tools"
SLOW_GIT_CLONE="$TEST_TOOLS/git_slow_clone.sh"
SLOW_GIT_FETCH="$TEST_TOOLS/git_slow_fetch.sh" SLOW_GIT_FETCH="$TEST_TOOLS/git_slow_fetch.sh"
ASKPASS_GIT="$TEST_TOOLS/git_askpass.sh" ASKPASS_GIT="$TEST_TOOLS/git_askpass.sh"
EXECHOOK_COMMAND="$TEST_TOOLS/exechook_command.sh" EXECHOOK_COMMAND="$TEST_TOOLS/exechook_command.sh"
@ -281,7 +280,6 @@ function e2e::init_root_doesnt_exist() {
GIT_SYNC \ GIT_SYNC \
--one-time \ --one-time \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT/subdir" \ --root="$ROOT/subdir" \
--link="link" \ --link="link" \
>> "$1" 2>&1 >> "$1" 2>&1
@ -300,7 +298,6 @@ function e2e::init_root_exists_empty() {
GIT_SYNC \ GIT_SYNC \
--one-time \ --one-time \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
>> "$1" 2>&1 >> "$1" 2>&1
@ -319,7 +316,6 @@ function e2e::init_root_flag_is_weird() {
GIT_SYNC \ GIT_SYNC \
--one-time \ --one-time \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="../../../../../$ROOT/../../../../../../$ROOT" \ --root="../../../../../$ROOT/../../../../../../$ROOT" \
--link="link" \ --link="link" \
>> "$1" 2>&1 >> "$1" 2>&1
@ -339,7 +335,6 @@ function e2e::init_root_flag_has_symlink() {
GIT_SYNC \ GIT_SYNC \
--one-time \ --one-time \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT/rootlink" \ --root="$ROOT/rootlink" \
--link="link" \ --link="link" \
>> "$1" 2>&1 >> "$1" 2>&1
@ -363,7 +358,6 @@ function e2e::init_root_is_under_another_repo() {
GIT_SYNC \ GIT_SYNC \
--one-time \ --one-time \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT/subdir/root" \ --root="$ROOT/subdir/root" \
--link="link" \ --link="link" \
>> "$1" 2>&1 >> "$1" 2>&1
@ -387,7 +381,6 @@ function e2e::init_root_fails_sanity() {
GIT_SYNC \ GIT_SYNC \
--one-time \ --one-time \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
>> "$1" 2>&1 >> "$1" 2>&1
@ -396,10 +389,6 @@ function e2e::init_root_fails_sanity() {
assert_file_eq "$ROOT"/link/file "$FUNCNAME" assert_file_eq "$ROOT"/link/file "$FUNCNAME"
} }
## FIXME: test when repo is valid git, but wrong remote
## FIXME: test when repo is valid git, but not ar ref we need
## FIXME: test when repo is valid git, and is already correct
############################################## ##############################################
# Test init with an absolute-path link # Test init with an absolute-path link
############################################## ##############################################
@ -410,8 +399,6 @@ function e2e::sync_absolute_link() {
GIT_SYNC \ GIT_SYNC \
--one-time \ --one-time \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--rev="HEAD" \
--root="$ROOT/root" \ --root="$ROOT/root" \
--link="$ROOT/other/dir/link" \ --link="$ROOT/other/dir/link" \
>> "$1" 2>&1 >> "$1" 2>&1
@ -431,8 +418,6 @@ function e2e::sync_subdir_link() {
GIT_SYNC \ GIT_SYNC \
--one-time \ --one-time \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--rev="HEAD" \
--root="$ROOT" \ --root="$ROOT" \
--link="other/dir/link" \ --link="other/dir/link" \
>> "$1" 2>&1 >> "$1" 2>&1
@ -443,9 +428,9 @@ function e2e::sync_subdir_link() {
} }
############################################## ##############################################
# Test non-zero exit with a bad branch # Test non-zero exit with a bad ref
############################################## ##############################################
function e2e::bad_branch_non_zero_exit() { function e2e::bad_ref_non_zero_exit() {
echo "$FUNCNAME" > "$REPO"/file echo "$FUNCNAME" > "$REPO"/file
git -C "$REPO" commit -qam "$FUNCNAME" git -C "$REPO" commit -qam "$FUNCNAME"
@ -454,8 +439,7 @@ function e2e::bad_branch_non_zero_exit() {
GIT_SYNC \ GIT_SYNC \
--one-time \ --one-time \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="does-not-exist" \ --ref=does-not-exist \
--rev="HEAD" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
>> "$1" 2>&1 >> "$1" 2>&1
@ -469,35 +453,9 @@ function e2e::bad_branch_non_zero_exit() {
} }
############################################## ##############################################
# Test non-zero exit with a bad rev # Test default ref syncing
############################################## ##############################################
function e2e::bad_rev_non_zero_exit() { function e2e::sync_default_ref() {
echo "$FUNCNAME" > "$REPO"/file
git -C "$REPO" commit -qam "$FUNCNAME"
(
set +o errexit
GIT_SYNC \
--one-time \
--repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--rev=does-not-exit \
--root="$ROOT" \
--link="link" \
>> "$1" 2>&1
RET=$?
if [[ "$RET" != 1 ]]; then
fail "expected exit code 1, got $RET"
fi
assert_file_absent "$ROOT"/link
assert_file_absent "$ROOT"/link/file
)
}
##############################################
# Test default-branch syncing
##############################################
function e2e::sync_default_branch() {
# First sync # First sync
echo "$FUNCNAME 1" > "$REPO"/file echo "$FUNCNAME 1" > "$REPO"/file
git -C "$REPO" commit -qam "$FUNCNAME 1" git -C "$REPO" commit -qam "$FUNCNAME 1"
@ -544,8 +502,7 @@ function e2e::sync_head() {
GIT_SYNC \ GIT_SYNC \
--period=100ms \ --period=100ms \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \ --ref=HEAD \
--rev=HEAD \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
>> "$1" 2>&1 & >> "$1" 2>&1 &
@ -583,7 +540,6 @@ function e2e::worktree_cleanup() {
GIT_SYNC \ GIT_SYNC \
--period=100ms \ --period=100ms \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
>> "$1" 2>&1 & >> "$1" 2>&1 &
@ -633,7 +589,6 @@ function e2e::readlink() {
GIT_SYNC \ GIT_SYNC \
--period=100ms \ --period=100ms \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
>> "$1" 2>&1 & >> "$1" 2>&1 &
@ -670,7 +625,7 @@ function e2e::sync_named_branch() {
GIT_SYNC \ GIT_SYNC \
--period=100ms \ --period=100ms \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$OTHER_BRANCH" \ --ref="$OTHER_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
>> "$1" 2>&1 & >> "$1" 2>&1 &
@ -713,7 +668,7 @@ function e2e::sync_branch_switch() {
GIT_SYNC \ GIT_SYNC \
--one-time \ --one-time \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \ --ref="$MAIN_BRANCH" \
--depth=1 \ --depth=1 \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
@ -730,7 +685,7 @@ function e2e::sync_branch_switch() {
GIT_SYNC \ GIT_SYNC \
--one-time \ --one-time \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch=$OTHER_BRANCH \ --ref=$OTHER_BRANCH \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
>> "$1" 2>&1 >> "$1" 2>&1
@ -753,8 +708,7 @@ function e2e::sync_tag() {
GIT_SYNC \ GIT_SYNC \
--period=100ms \ --period=100ms \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \ --ref="$TAG" \
--rev="$TAG" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
>> "$1" 2>&1 & >> "$1" 2>&1 &
@ -807,8 +761,7 @@ function e2e::sync_annotated_tag() {
GIT_SYNC \ GIT_SYNC \
--period=100ms \ --period=100ms \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \ --ref="$TAG" \
--rev="$TAG" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
>> "$1" 2>&1 & >> "$1" 2>&1 &
@ -859,8 +812,7 @@ function e2e::sync_sha() {
GIT_SYNC \ GIT_SYNC \
--period=100ms \ --period=100ms \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \ --ref="$SHA" \
--rev="$SHA" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
>> "$1" 2>&1 & >> "$1" 2>&1 &
@ -900,8 +852,7 @@ function e2e::sync_sha_once() {
GIT_SYNC \ GIT_SYNC \
--one-time \ --one-time \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \ --ref="$SHA" \
--rev="$SHA" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
>> "$1" 2>&1 >> "$1" 2>&1
@ -928,8 +879,7 @@ function e2e::sync_sha_once_sync_different_sha_known() {
GIT_SYNC \ GIT_SYNC \
--one-time \ --one-time \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \ --ref="$SHA1" \
--rev="$SHA1" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
>> "$1" 2>&1 >> "$1" 2>&1
@ -941,8 +891,7 @@ function e2e::sync_sha_once_sync_different_sha_known() {
GIT_SYNC \ GIT_SYNC \
--one-time \ --one-time \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \ --ref="$SHA2" \
--rev="$SHA2" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
>> "$1" 2>&1 >> "$1" 2>&1
@ -963,8 +912,7 @@ function e2e::sync_sha_once_sync_different_sha_unknown() {
GIT_SYNC \ GIT_SYNC \
--one-time \ --one-time \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \ --ref="$SHA1" \
--rev="$SHA1" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
>> "$1" 2>&1 >> "$1" 2>&1
@ -985,8 +933,7 @@ function e2e::sync_sha_once_sync_different_sha_unknown() {
GIT_SYNC \ GIT_SYNC \
--one-time \ --one-time \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \ --ref="$SHA2" \
--rev="$SHA2" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
>> "$1" 2>&1 >> "$1" 2>&1
@ -994,6 +941,7 @@ function e2e::sync_sha_once_sync_different_sha_unknown() {
assert_file_exists "$ROOT"/link/file assert_file_exists "$ROOT"/link/file
assert_file_eq "$ROOT"/link/file "$FUNCNAME 2" assert_file_eq "$ROOT"/link/file "$FUNCNAME 2"
} }
############################################## ##############################################
# Test syncing after a crash # Test syncing after a crash
############################################## ##############################################
@ -1005,7 +953,6 @@ function e2e::sync_crash_cleanup_retry() {
GIT_SYNC \ GIT_SYNC \
--one-time \ --one-time \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
>> "$1" 2>&1 >> "$1" 2>&1
@ -1020,7 +967,6 @@ function e2e::sync_crash_cleanup_retry() {
GIT_SYNC \ GIT_SYNC \
--one-time \ --one-time \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
>> "$1" 2>&1 >> "$1" 2>&1
@ -1040,7 +986,6 @@ function e2e::sync_repo_switch() {
# First sync # First sync
GIT_SYNC \ GIT_SYNC \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
--one-time \ --one-time \
@ -1056,7 +1001,6 @@ function e2e::sync_repo_switch() {
# Now sync the other repo # Now sync the other repo
GIT_SYNC \ GIT_SYNC \
--repo="file://$REPO2" \ --repo="file://$REPO2" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
--one-time \ --one-time \
@ -1075,11 +1019,10 @@ function e2e::error_slow_git_short_timeout() {
git -C "$REPO" commit -qam "$FUNCNAME 1" git -C "$REPO" commit -qam "$FUNCNAME 1"
GIT_SYNC \ GIT_SYNC \
--git="/$SLOW_GIT_CLONE" \ --git="/$SLOW_GIT_FETCH" \
--one-time \ --one-time \
--sync-timeout=1s \ --sync-timeout=1s \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
>> "$1" 2>&1 || true >> "$1" 2>&1 || true
@ -1098,11 +1041,10 @@ function e2e::sync_slow_git_long_timeout() {
# run with slow_git_clone but without timing out # run with slow_git_clone but without timing out
GIT_SYNC \ GIT_SYNC \
--git="/$SLOW_GIT_CLONE" \ --git="/$SLOW_GIT_FETCH" \
--period=100ms \ --period=100ms \
--sync-timeout=16s \ --sync-timeout=16s \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
>> "$1" 2>&1 & >> "$1" 2>&1 &
@ -1134,7 +1076,6 @@ function e2e::sync_on_signal_sighup() {
--period=100s \ --period=100s \
--sync-on-signal="SIGHUP" \ --sync-on-signal="SIGHUP" \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
>> "$1" 2>&1 & >> "$1" 2>&1 &
@ -1167,7 +1108,6 @@ function e2e::sync_on_signal_hup() {
--period=100s \ --period=100s \
--sync-on-signal="HUP" \ --sync-on-signal="HUP" \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
>> "$1" 2>&1 & >> "$1" 2>&1 &
@ -1200,7 +1140,6 @@ function e2e::sync_on_signal_1() {
--period=100s \ --period=100s \
--sync-on-signal=1 \ --sync-on-signal=1 \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
>> "$1" 2>&1 & >> "$1" 2>&1 &
@ -1233,7 +1172,6 @@ function e2e::sync_branch_depth_shallow() {
GIT_SYNC \ GIT_SYNC \
--period=100ms \ --period=100ms \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--depth="$expected_depth" \ --depth="$expected_depth" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
@ -1291,8 +1229,7 @@ function e2e::sync_tag_depth_shallow_out_of_range() {
GIT_SYNC \ GIT_SYNC \
--period=100ms \ --period=100ms \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \ --ref="$TAG" \
--rev="$TAG" \
--depth="$expected_depth" \ --depth="$expected_depth" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
@ -1325,49 +1262,6 @@ function e2e::sync_tag_depth_shallow_out_of_range() {
fi fi
} }
##############################################
# Test fetch skipping commit
##############################################
function e2e::sync_fetch_skip_depth_1() {
echo "$FUNCNAME" > "$REPO"/file
git -C "$REPO" commit -qam "$FUNCNAME"
GIT_SYNC \
--git="/$SLOW_GIT_FETCH" \
--period=100ms \
--depth=1 \
--repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \
--link="link" \
>> "$1" 2>&1 &
# wait for first sync which does a clone followed by an artifically slowed fetch
wait_for_sync "$((MAXWAIT * 3))"
assert_link_exists "$ROOT"/link
assert_file_exists "$ROOT"/link/file
assert_file_eq "$ROOT"/link/file "$FUNCNAME"
assert_metric_eq "${METRIC_GOOD_SYNC_COUNT}" 1
# make a second commit to trigger a sync with shallow fetch
echo "$FUNCNAME-ok" > "$REPO"/file2
git -C "$REPO" add file2
git -C "$REPO" commit -qam "$FUNCNAME new file"
# Give time for ls-remote to detect the commit and slowed fetch to start
sleep 2
# make a third commit to insert the commit between ls-remote and fetch
echo "$FUNCNAME-ok" > "$REPO"/file3
git -C "$REPO" add file3
git -C "$REPO" commit -qam "$FUNCNAME third file"
wait_for_sync "$((MAXWAIT * 3))"
assert_link_exists "$ROOT"/link
assert_file_exists "$ROOT"/link/file3
assert_file_eq "$ROOT"/link/file3 "$FUNCNAME-ok"
assert_metric_eq "${METRIC_GOOD_SYNC_COUNT}" 2
}
############################################## ##############################################
# Test password auth with the wrong password # Test password auth with the wrong password
############################################## ##############################################
@ -1382,7 +1276,6 @@ function e2e::auth_password_wrong_password() {
--password="wrong" \ --password="wrong" \
--one-time \ --one-time \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
>> "$1" 2>&1 || true >> "$1" 2>&1 || true
@ -1405,7 +1298,6 @@ function e2e::auth_password_correct_password() {
--password="my-password" \ --password="my-password" \
--period=100ms \ --period=100ms \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
>> "$1" 2>&1 & >> "$1" 2>&1 &
@ -1456,7 +1348,6 @@ function e2e::auth_askpass_url_wrong_password() {
--askpass-url="http://$IP/git_askpass" \ --askpass-url="http://$IP/git_askpass" \
--one-time \ --one-time \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
>> "$1" 2>&1 || true >> "$1" 2>&1 || true
@ -1492,7 +1383,6 @@ function e2e::auth_askpass_url_correct_password() {
--askpass-url="http://$IP/git_askpass" \ --askpass-url="http://$IP/git_askpass" \
--period=100ms \ --period=100ms \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
>> "$1" 2>&1 & >> "$1" 2>&1 &
@ -1555,7 +1445,6 @@ function e2e::auth_askpass_url_flaky() {
--max-failures=2 \ --max-failures=2 \
--period=100ms \ --period=100ms \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
>> "$1" 2>&1 & >> "$1" 2>&1 &
@ -1594,7 +1483,6 @@ function e2e::exechook_success() {
GIT_SYNC \ GIT_SYNC \
--period=100ms \ --period=100ms \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
--exechook-command="/$EXECHOOK_COMMAND" \ --exechook-command="/$EXECHOOK_COMMAND" \
@ -1636,7 +1524,6 @@ function e2e::exechook_fail_retry() {
GIT_SYNC \ GIT_SYNC \
--period=100ms \ --period=100ms \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
--exechook-command="/$EXECHOOK_COMMAND_FAIL" \ --exechook-command="/$EXECHOOK_COMMAND_FAIL" \
@ -1663,7 +1550,6 @@ function e2e::exechook_success_once() {
--period=100ms \ --period=100ms \
--one-time \ --one-time \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
--exechook-command="/$EXECHOOK_COMMAND_SLEEPY" \ --exechook-command="/$EXECHOOK_COMMAND_SLEEPY" \
@ -1696,7 +1582,6 @@ function e2e::exechook_fail_once() {
--period=100ms \ --period=100ms \
--one-time \ --one-time \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
--exechook-command="/$EXECHOOK_COMMAND_FAIL_SLEEPY" \ --exechook-command="/$EXECHOOK_COMMAND_FAIL_SLEEPY" \
@ -1740,7 +1625,6 @@ function e2e::webhook_success() {
GIT_SYNC \ GIT_SYNC \
--period=100ms \ --period=100ms \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--webhook-url="http://$IP" \ --webhook-url="http://$IP" \
--webhook-success-status=200 \ --webhook-success-status=200 \
@ -1791,7 +1675,6 @@ function e2e::webhook_fail_retry() {
GIT_SYNC \ GIT_SYNC \
--period=100ms \ --period=100ms \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--webhook-url="http://$IP" \ --webhook-url="http://$IP" \
--webhook-success-status=200 \ --webhook-success-status=200 \
@ -1847,7 +1730,6 @@ function e2e::webhook_success_once() {
--period=100ms \ --period=100ms \
--one-time \ --one-time \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--webhook-url="http://$IP" \ --webhook-url="http://$IP" \
--webhook-success-status=200 \ --webhook-success-status=200 \
@ -1886,7 +1768,6 @@ function e2e::webhook_fail_retry_once() {
--period=100ms \ --period=100ms \
--one-time \ --one-time \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--webhook-url="http://$IP" \ --webhook-url="http://$IP" \
--webhook-success-status=200 \ --webhook-success-status=200 \
@ -1931,7 +1812,6 @@ function e2e::webhook_fire_and_forget() {
GIT_SYNC \ GIT_SYNC \
--period=100ms \ --period=100ms \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--webhook-url="http://$IP" \ --webhook-url="http://$IP" \
--webhook-success-status=-1 \ --webhook-success-status=-1 \
@ -1956,10 +1836,9 @@ function e2e::expose_http() {
git -C "$REPO" commit -qam "$FUNCNAME 1" git -C "$REPO" commit -qam "$FUNCNAME 1"
GIT_SYNC \ GIT_SYNC \
--git="/$SLOW_GIT_CLONE" \ --git="/$SLOW_GIT_FETCH" \
--period=100ms \ --period=100ms \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
>> "$1" 2>&1 & >> "$1" 2>&1 &
@ -2008,7 +1887,6 @@ function e2e::expose_http_after_restart() {
GIT_SYNC \ GIT_SYNC \
--one-time \ --one-time \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
>> "$1" 2>&1 >> "$1" 2>&1
@ -2019,7 +1897,6 @@ function e2e::expose_http_after_restart() {
# Sync again and prove readiness. # Sync again and prove readiness.
GIT_SYNC \ GIT_SYNC \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
>> "$1" 2>&1 & >> "$1" 2>&1 &
@ -2076,7 +1953,6 @@ function e2e::submodule_sync_default() {
GIT_SYNC \ GIT_SYNC \
--period=100ms \ --period=100ms \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
>> "$1" 2>&1 & >> "$1" 2>&1 &
@ -2175,7 +2051,6 @@ function e2e::submodule_sync_depth() {
GIT_SYNC \ GIT_SYNC \
--period=100ms \ --period=100ms \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--depth="$expected_depth" \ --depth="$expected_depth" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
@ -2254,7 +2129,6 @@ function e2e::submodule_sync_off() {
GIT_SYNC \ GIT_SYNC \
--period=100ms \ --period=100ms \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
--submodules=off \ --submodules=off \
@ -2297,7 +2171,6 @@ function e2e::submodule_sync_shallow() {
GIT_SYNC \ GIT_SYNC \
--period=100ms \ --period=100ms \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
--submodules=shallow \ --submodules=shallow \
@ -2332,7 +2205,6 @@ function e2e::auth_ssh() {
GIT_SYNC \ GIT_SYNC \
--period=100ms \ --period=100ms \
--repo="test@$IP:/src" \ --repo="test@$IP:/src" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
--ssh \ --ssh \
@ -2380,7 +2252,6 @@ function e2e::sparse_checkout() {
GIT_SYNC \ GIT_SYNC \
--one-time \ --one-time \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
--sparse-checkout-file="$WORK/sparseconfig" \ --sparse-checkout-file="$WORK/sparseconfig" \
@ -2403,7 +2274,6 @@ function e2e::additional_git_configs() {
GIT_SYNC \ GIT_SYNC \
--one-time \ --one-time \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
--git-config='http.postBuffer:10485760,sect.k1:"a val",sect.k2:another val' \ --git-config='http.postBuffer:10485760,sect.k1:"a val",sect.k2:another val' \
@ -2424,7 +2294,7 @@ function e2e::export_error() {
set +o errexit set +o errexit
GIT_SYNC \ GIT_SYNC \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch=does-not-exit \ --ref=does-not-exit \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
--error-file="error.json" \ --error-file="error.json" \
@ -2435,14 +2305,13 @@ function e2e::export_error() {
fi fi
assert_file_absent "$ROOT"/link assert_file_absent "$ROOT"/link
assert_file_absent "$ROOT"/link/file assert_file_absent "$ROOT"/link/file
assert_file_contains "$ROOT"/error.json "Remote branch does-not-exit not found in upstream origin" assert_file_contains "$ROOT"/error.json "unknown revision"
) )
# the error.json file should be removed if sync succeeds. # the error.json file should be removed if sync succeeds.
GIT_SYNC \ GIT_SYNC \
--one-time \ --one-time \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
--error-file="error.json" \ --error-file="error.json" \
@ -2464,7 +2333,7 @@ function e2e::export_error_abs_path() {
set +o errexit set +o errexit
GIT_SYNC \ GIT_SYNC \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch=does-not-exit \ --ref=does-not-exit \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
--error-file="$ROOT/dir/error.json" \ --error-file="$ROOT/dir/error.json" \
@ -2475,7 +2344,7 @@ function e2e::export_error_abs_path() {
fi fi
assert_file_absent "$ROOT"/link assert_file_absent "$ROOT"/link
assert_file_absent "$ROOT"/link/file assert_file_absent "$ROOT"/link/file
assert_file_contains "$ROOT"/dir/error.json "Remote branch does-not-exit not found in upstream origin" assert_file_contains "$ROOT"/dir/error.json "unknown revision"
) )
} }
@ -2490,7 +2359,6 @@ function e2e::export_error_invalid_file() {
set +o errexit set +o errexit
GIT_SYNC \ GIT_SYNC \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
--error-file=".error.json" \ --error-file=".error.json" \
@ -2516,7 +2384,6 @@ function e2e::touch_file() {
GIT_SYNC \ GIT_SYNC \
--period=100ms \ --period=100ms \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
--touch-file="touch.file" \ --touch-file="touch.file" \
@ -2571,7 +2438,6 @@ function e2e::touch_file_abs_path() {
GIT_SYNC \ GIT_SYNC \
--period=100ms \ --period=100ms \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
--touch-file="$ROOT/dir/touch.file" \ --touch-file="$ROOT/dir/touch.file" \
@ -2626,7 +2492,6 @@ function e2e::touch_file_invalid_file() {
set +o errexit set +o errexit
GIT_SYNC \ GIT_SYNC \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
--touch-file=".touch.file" \ --touch-file=".touch.file" \
@ -2649,7 +2514,6 @@ function e2e::github_https() {
GIT_SYNC \ GIT_SYNC \
--one-time \ --one-time \
--repo="https://github.com/kubernetes/git-sync" \ --repo="https://github.com/kubernetes/git-sync" \
--branch=master \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
>> "$1" 2>&1 >> "$1" 2>&1
@ -2666,7 +2530,6 @@ function e2e::gc_auto() {
GIT_SYNC \ GIT_SYNC \
--one-time \ --one-time \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
--git-gc="auto" \ --git-gc="auto" \
@ -2686,7 +2549,6 @@ function e2e::gc_always() {
GIT_SYNC \ GIT_SYNC \
--one-time \ --one-time \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
--git-gc="always" \ --git-gc="always" \
@ -2706,7 +2568,6 @@ function e2e::gc_aggressive() {
GIT_SYNC \ GIT_SYNC \
--one-time \ --one-time \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
--git-gc="aggressive" \ --git-gc="aggressive" \
@ -2726,7 +2587,6 @@ function e2e::gc_off() {
GIT_SYNC \ GIT_SYNC \
--one-time \ --one-time \
--repo="file://$REPO" \ --repo="file://$REPO" \
--branch="$MAIN_BRANCH" \
--root="$ROOT" \ --root="$ROOT" \
--link="link" \ --link="link" \
--git-gc="off" \ --git-gc="off" \