This resolves a race condition with unconfigured images attempting to bring up
DBs for the first time. This does not affect fully bootstrapped images.
Currently, all jobs start at boot - this includes postgres.
Issue with the current is - postgres starts and adds the corresponding .s/.pid
files to /var/run/postgres.
Simultaneously, the unicorn job gets started, checks to see if postgres is
running (it is already at this point from boot), and runs install_postgres.
Inside the install_postgres script, we mount the shared postgres folder and
remove .s/.pid files -- after postgres has already been started. In this case,
we remove the (in-use) .s and .pid files.
Subsequent unicorn tasks fail, erroring out the service and forcing it into
a restart loop. Since postgres never restarts, it never regenerates the .s/.pid
files, and unicorn can never run successfully.
This proposal moves install_postgres into the postgres job file, eliminating the
race condition. Since they are part of the same service, install_postgres will
always run before starting postgres - it will no longer be able to remove valid
.s and .pid files.
Redis has a similar race condition with the creation of its data folder. This
isn't as disastrous as the redis service restarts until the folder exists from
unicorn run, but it provides better reasoning about the running services.
Add early exit from unicorn boot scripts to properly retry migrate as well.
Use pg_isready to check if pg is ready directly in create_db.
Merge the ready check into create_db.
Run create_db in a subshell on postgres job start, rather than in unicorn script.
remove postgres-config call
This config has several 'include' statements that serve as extension
outlets. Most of the "find & replace" instructions from templates
files have been replaced to use outlets.
This commit also deprecates IPv6 template in favor of an automatic
implementation that relies on container's connectivity.
Using `pry` as the default runtime console in the production environment
is forcing us to account for it in Discourse core's codebase. In order
to avoid the additional complexity in Discourse core's codebase, we have
decided to drop `pry` as the default runtime console and rely on IRB
which is just as good these days. We will however be keeping the
dependency on `pry` around so those that would like to use it can start
it manually.
* Removing unicorn service from phpbb template broke the build
https://meta.discourse.org/t/migrate-a-phpbb3-forum-to-discourse/30810/786?u=pfaffman
* Prevent Sidekiq from running
Usually we don't want Sidekiq to run during an import because it might mess things up while there's incomplete or inconsistent data.
---------
Co-authored-by: Gerhard Schlager <gerhard.schlager@discourse.org>
649505d869 added some new content between `listen 80;` and `gzip on;`, which broke these replacements.
This commit reorders & updates the replacements so that they should work reliably with versions of core before and after that change.
This updates the default PostgreSQL version to 15.
* image/base: update default postgres version to 15
* launcher: bump base image to include postgres 15
* postgres.template.yml: update default version to 15
* postgres.15.template.yml: current version template
Legacy version templates:
* postgres.13.template.yml
* postgres.12.template.yml
* postgres.10.template.yml
* postgres.9.5.template.yml
After 7d548ad4ae, the replacement pattern in web.socketed.template.yml no longer matches, making nginx.https.sock not exist.
As such, remove the `http2` field from the `listen` directive to match aforementioned commit.
The nginx config file used to be copied from the discourse/discourse
repository, but it has been now moved in this project, closer to the
place where it is used.
The config has several 'include' statements that implement support for
outlets that templates can then use to extend the default configuration
for various features. This is an alternative to the "find & replace"
hacks.
I noticed that use running `git reset --hard` results in ~133MB being
added to the final layer generated by `launcher bootstrap`. However, I
can not figure out why we would need to run `git reset --hard` at all.
Even if there is a reason to run `git reset --hard`, it should not be
the default. If someone for whatever reason needs to run `git reset
--hard`, they should do so using the `before_code` hook.
To replicate the problem, one can run the following steps:
1. In the `discourse_docker` repository, create a file named `containers/test.yml` with the following contents:
```
base_image: discourse/base:2.0.20241223-0016
run:
- exec: sudo -H -E -u discourse bash -c "cd /var/www/discourse && git reset --hard"
```
2. Run `./launcher bootstrap test`
3. Run `docker history local_discourse/test` and see that the new layer created by `./launcher bootstrap` is roughly 133MB.
```
IMAGE CREATED CREATED BY SIZE COMMENT
012471f3c5e4 2 minutes ago /bin/bash -c /usr/local/bin/pups --stdin 133MB
```
pnpm packages and bundle gems can add significant size to the Docker
image. Ensure that we clean up those unused files to ensure we don't
add extra diskspace to the final layer when bootstrapping.
Using e.g. `proxy_pass http://discourse` resets the Host header on the upstream
request to `discourse`.
This would break multisites, so we don't want that; the most effetive way to
ensure it's set properly is to `set_header` in the upstream block.
* Fix Warnung: the "listen ... http2" directive is deprecated
Signed-off-by: Tealk <tealk@anzah.email>
* Update templates/web.ssl.template.yml
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
* add on to http2
---------
Signed-off-by: Tealk <tealk@anzah.email>
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
install locales-all installs 245MB
Generating common and en_us locales installs 22MB
update locales for postgres templates
`locale-gen $LANG` doesn't seem to actually do anything without updates to
`/etc/locale.gen`. Update the scripts to uncomment $LANG before running `update-locale`.
Configure option does a precompile, which should forcefully disable
`PRECOMPILE_ON_BOOT` to prevent unnecessary precompile on boot when
`PRECOMPILE_ON_BOOT` env is set in templates.
This commit updates our `build` workflow to push the following manifests
which can be used to run Discourse against the stable branch.
1. discourse/base:2.0.$TIMESTAMP-stable
2. discourse/base:release-stable
Co-authored-by: David Taylor <david@taylorhq.com>
- makes decision based on current state of directory, instead of `$version`
- cleans up the correct directories
- only cleans up the contents. This is important if node_modules directores are mounted volumes (e.g. in devcontainer)
This commit updates 2 things.
1. Updates the yarn hook to replace the npm mirror before `pnpm install`.
2. Removes the `yarn.lock` patch as pnpm is now used.
After applying these modifications, I successfully installed Discourse on the Tencent Cloud China server. No more network problems.
when we have already run an initial setup, fall back to just checking for
socket, rather than outright failing if the init script has already been run.
This allows 'configure' steps to be re-run in standalone cases.
eg: `launcher2 configure app && launcher2 configure app`
current version: fails as it's missing the install_postgres file
with PR: checks for psql socket, and builds.
doing something like `launcher2 start app && launcher2 configure app` would also
print out a more correct error message, "postgres already running stop container"
This fixes a regression introduced in
bbefa1e5f3. Basically, we cannot configure
the default bundle jobs when building the image because the number of
cores used to build the image can be different from the number of cores
on the machine running the image.
A number of people have reported hitting yarn timeouts on low-spec DO droplets, which causes the build to fail. This should provide a little more leeway
This commit adds `sharedscripts` which will ensure that our `postrotate`
script is only ran once even if multiple log files in the `/shared/log/rails/`
are rotated. If `sharedscripts` is not specified, we are sending `sv 1
unicorn` once per log file rotated and this has resulted in weird
behaviours like our Sidekiq process hanging indefinitely.
Note the following from the manpage for logrotate:
```
sharedscripts
Normally, prerotate and postrotate scripts are run for each log which is rotated and the absolute path to the log file is passed as first argument to the script. That means a single script may be run multiple times for log file entries which match multiple files (such as the /var/log/news/* example). If sharedscripts is specified, the scripts are only run once, no matter how many logs match the wildcarded pattern, and whole pattern is passed to them.
```
* DEV: Updated vanilla.template.yml
* updated vanilla.template.yml to make the migration process more straight forward
* removed branch pull
* implemented suggested changes
* added suggested chantes
* added before_code hook to set remote fork
* updated with suggested changes
This commit does 2 things:
1. Added a new yarn hook to replace the npm mirror before `yarn install`.
2. Modified `web.china.template.yml` to add more mirror sources.
Below is an explanation of these modifications:
- The GitHub proxy added in `web.china.template.yml` has existed in China for many years, and its repository https://github.com/hunshcn/gh-proxy has 6k+ stars, which can ensure its security and stability.
- The NPM mirror site added in `web.china.template.yml` is maintained by Alibaba Group, one of the largest Internet companies in China.
- Modified the Gem mirror in `web.china.template.yml` to the mirror provided by Tsinghua University, one of the top universities in China.
- The reason why sed is used to replace the `yarn.lock` file is because `yarn install --frozen-lockfile` is used for installation below. If the url is not replaced, the NPM mirror will not take effect.
After applying these modifications, I successfully installed Discourse on the Tencent Cloud China server. No more network problems.
* Add tags to pups templates
The purpose here is to allow greater flexibility in how and where
docker images are built and run. It achieves this by breaking up
build steps into distinct run steps which can be saved along the way.
Customizable base images may then be prebuilt with as many batteries
included as possible, with zero environment setup so those images
can then be configured at a later stage.
Add the ability to run partial pups configuration:
`build`: build base image with no db - ember build.
`precompile`: precompile stage that requires postgres and redis.
`migrate`: run migration tasks.
`db`: start bundled postgres/redis, if included.
Adds a create_db script in postgres template for creating db on the fly.
Called below in unicorn run:
updates unicorn run command with 3 env flags:
CREATE_DB_ON_BOOT: if 1, creates base db schema, allows for deferral of creation.
MIGRATE_ON_BOOT: if 1, runs db:migrate - allows for deferral of db migration.
PRECOMPILE_ON_BOOT: if 1, precompiles assets (without ember build).
PRECOMPILE_ON_BOOT initially defaults to 1 in base builds (no tags).
During the `precompile` build step, this updates the default to be 0.
All other new flags default to 0 (off). With these three flags, we're now able
to ship and start a container from a base image, and it'll be able to bootstrap
a blank database.
Updates hook to start redis before_db_migrate as before_code hook
is not guaranteed to fire before migrate tasks if pups is filtered by tags.
The new templates/postgres.15.template.yml file allows bootstrapping
new containers using PostgreSQL version 15, or upgrading an existing
container running on older PostgreSQL versions.
The default postgres template and base image shall be bumped in a
follow-up commit.
Yarn seems to have issues 'downgrading' our set of packages from tests-passed (which are pre-installed in the base image) to the versions on the stable branch.
In the medium term we intend to switch to the pnpm package manager, which shouldn't have this issue. In the meantime, we can work around the problem by deleting the pre-installed node_modules and installing from scratch.
https://meta.discourse.org/t/270491
`devDependencies` is often used for things like ember-cli build tooling. Until now we have been moving those things to `dependencies` in our `package.json`, but it makes more sense for us to conform to the industry norms.
Jeff Wong
Alan Guo Xiang Tan
Bianca Nenciu
Jay Pfaffman
David Taylor
mwaniki-wairungu
nta~
Michael Brown
Tealk
Jeremy
Rafael dos Santos Silva
Juan David Martínez Cubillos
耗子
Mwaniki Wairungu
Andrew Schleifer
Jarek Radosz
Saj Goonatilleke