PromucFlow_constructor/deploy/docker/fs/opt/appsmith/pg-upgrade.sh

107 lines
3.6 KiB
Bash
Raw Normal View History

chore: Add script to upgrade Postgres 13 data to 14 (#34317) We're upgrading embedded Postgres from 13 to 14, and this PR includes a script to perform the upgrade of the data folder from v13 schema to v14 schema. This script temporarily installs Postgres 13, if not available, for the upgrade process, so will continue to work when and if we choose to remove `postgresql-13` from the base image. Tested this manually as well, running an Appsmith with Postgres 13, executing some workflows via webhook, getting some run data generated, then upgrading Postgres with the script in this PR, and ensuring that the workflow run history is still there and visible on the UI exactly the same. It is. No conflicts or additional changes needed on EE. [All server and Cypress tests pass on EE](https://github.com/appsmithorg/appsmith-ee/pull/4493). ![shot-2024-06-20-02-13-26](https://github.com/appsmithorg/appsmith/assets/120119/9bb60e3a-6cc9-4df9-9064-caead78729a6) **/test sanity** <!-- This is an auto-generated comment: Cypress test results --> > [!TIP] > 🟢 🟢 🟢 All cypress tests have passed! 🎉 🎉 🎉 > Workflow run: <https://github.com/appsmithorg/appsmith/actions/runs/9590240540> > Commit: 9c75da53f871ffb912015c18a7504327cba88f2c > <a href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=9590240540&attempt=1" target="_blank">Cypress dashboard</a>. > Tags: `@tag.Sanity` <!-- end of auto-generated comment: Cypress test results --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added automation script for upgrading PostgreSQL to the latest version. - Introduced testing script for PostgreSQL upgrades in Docker environments. - **Improvements** - Upgraded PostgreSQL from version 13 to 14 in Docker setup, ensuring compatibility and performance enhancements. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-06-21 14:04:08 +00:00
#!/bin/bash
set -o errexit
set -o nounset
# This script will upgrade Postgres to the "current" version of Postgres, if needed.
# Assumptions:
# 1. Postgres is currently not running. The caller of this script ensures that _no_ version of Postgres server is currently running.
# 2. The newest version of Postgres we want to use, is already installed.
# 3. Postgres is installed via apt package manager only.
# Contract:
# 1. Don't install old version of Postgres, if it's already installed.
# 2. Use absolute paths to all Postgres executables, don't rely on any of them to be coming from "\$PATH".
# 3. Be idempotent across versions.
# 4. When we can't proceed due to any exceptional scenarios, communicate clearly.
# 5. Mark old/stale/deprecated data with a date, so it can be deleted with confidence later.
# Check if any Postgres server is running
if pgrep -x "postgres" > /dev/null; then
echo "Error: A Postgres server is currently running. Please stop it before proceeding with the upgrade."
exit 1
fi
postgres_path=/usr/lib/postgresql
pg_data_dir=/appsmith-stacks/data/postgres/main
old_version=""
if [[ -f "$pg_data_dir/PG_VERSION" ]]; then
old_version="$(cat "$pg_data_dir/PG_VERSION")"
fi
if [[ -z "$old_version" ]]; then
tlog "No existing Postgres data found, not upgrading anything." >&2
exit
fi
top_available_version="$(postgres --version | grep -o '[[:digit:]]\+' | head -1)"
declare -a to_uninstall
to_uninstall=()
# 13 to 14
if [[ "$old_version" == 13 && "$top_available_version" > "$old_version" ]]; then
if [[ ! -e "$postgres_path/$old_version" ]]; then
apt-get update
apt-get install --yes "postgresql-$old_version"
to_uninstall+=("postgresql-$old_version")
fi
fix: cleanup stale postgres postmaster.pid (#35171) ## Description issue of Postgres not coming up with error` Previous Postgres was not shutdown cleanly. Please start and stop Postgres 14 properly with 'supervisorctl' only. ` https://www.notion.so/appsmith/Closed-Beta-Customer-issues-45a274a9eb8e4762a72cbff74cd3bad5?pvs=4#ecca04d205414f25a289884cebdc0f9b Fixes https://app.zenhub.com/workspaces/workflows-pod-652fff131a95920b9bf2bc7e/issues/zh/226_ ## Automation /ok-to-test tags="@tag.Sanity" ### :mag: Cypress test results <!-- This is an auto-generated comment: Cypress test results --> > [!TIP] > 🟢 🟢 🟢 All cypress tests have passed! 🎉 🎉 🎉 > Workflow run: <https://github.com/appsmithorg/appsmith/actions/runs/10107093349> > Commit: 357bf5d2ff0c1f65b4cbf46b0f3ee823ac192614 > <a href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=10107093349&attempt=1" target="_blank">Cypress dashboard</a>. > Tags: `@tag.Sanity` > Spec: > <hr>Fri, 26 Jul 2024 07:15:19 UTC <!-- end of auto-generated comment: Cypress test results --> ## Communication Should the DevRel and Marketing teams inform users about this change? - [ ] Yes - [ ] No <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Summary by CodeRabbit - **New Features** - Enhanced PostgreSQL upgrade process with improved error handling and robust management of old server instances. - **Bug Fixes** - Reinstituted logic for checking and managing the `postmaster.pid` file, ensuring proper startup and shutdown of old PostgreSQL servers. - **Refactor** - Improved formatting and readability of shell scripts without altering their functionality. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-07-26 07:22:49 +00:00
if [[ -f "$pg_data_dir/postmaster.pid" ]]; then
# Start old PostgreSQL using pg_ctl
tlog "Stale postmaster.pid found. Starting old PostgreSQL $old_version using pg_ctl to cleanup."
su postgres -c "$postgres_path/$old_version/bin/pg_ctl start -D '$pg_data_dir' "
# Wait for old PostgreSQL to be ready
until su postgres -c "$postgres_path/$old_version/bin/pg_isready"; do
tlog "Waiting for PostgreSQL $old_version to start..."
sleep 1
done
# Shut down PostgreSQL gracefully using pg_ctl
su postgres -c "$postgres_path/$old_version/bin/pg_ctl stop -D '$pg_data_dir' -m smart"
tlog "PostgreSQL $old_version has been shut down."
fi
new_version="$((old_version + 1))"
new_data_dir="$pg_data_dir-$new_version"
chore: Add script to upgrade Postgres 13 data to 14 (#34317) We're upgrading embedded Postgres from 13 to 14, and this PR includes a script to perform the upgrade of the data folder from v13 schema to v14 schema. This script temporarily installs Postgres 13, if not available, for the upgrade process, so will continue to work when and if we choose to remove `postgresql-13` from the base image. Tested this manually as well, running an Appsmith with Postgres 13, executing some workflows via webhook, getting some run data generated, then upgrading Postgres with the script in this PR, and ensuring that the workflow run history is still there and visible on the UI exactly the same. It is. No conflicts or additional changes needed on EE. [All server and Cypress tests pass on EE](https://github.com/appsmithorg/appsmith-ee/pull/4493). ![shot-2024-06-20-02-13-26](https://github.com/appsmithorg/appsmith/assets/120119/9bb60e3a-6cc9-4df9-9064-caead78729a6) **/test sanity** <!-- This is an auto-generated comment: Cypress test results --> > [!TIP] > 🟢 🟢 🟢 All cypress tests have passed! 🎉 🎉 🎉 > Workflow run: <https://github.com/appsmithorg/appsmith/actions/runs/9590240540> > Commit: 9c75da53f871ffb912015c18a7504327cba88f2c > <a href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=9590240540&attempt=1" target="_blank">Cypress dashboard</a>. > Tags: `@tag.Sanity` <!-- end of auto-generated comment: Cypress test results --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added automation script for upgrading PostgreSQL to the latest version. - Introduced testing script for PostgreSQL upgrades in Docker environments. - **Improvements** - Upgraded PostgreSQL from version 13 to 14 in Docker setup, ensuring compatibility and performance enhancements. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-06-21 14:04:08 +00:00
fix: cleanup stale postgres postmaster.pid (#35171) ## Description issue of Postgres not coming up with error` Previous Postgres was not shutdown cleanly. Please start and stop Postgres 14 properly with 'supervisorctl' only. ` https://www.notion.so/appsmith/Closed-Beta-Customer-issues-45a274a9eb8e4762a72cbff74cd3bad5?pvs=4#ecca04d205414f25a289884cebdc0f9b Fixes https://app.zenhub.com/workspaces/workflows-pod-652fff131a95920b9bf2bc7e/issues/zh/226_ ## Automation /ok-to-test tags="@tag.Sanity" ### :mag: Cypress test results <!-- This is an auto-generated comment: Cypress test results --> > [!TIP] > 🟢 🟢 🟢 All cypress tests have passed! 🎉 🎉 🎉 > Workflow run: <https://github.com/appsmithorg/appsmith/actions/runs/10107093349> > Commit: 357bf5d2ff0c1f65b4cbf46b0f3ee823ac192614 > <a href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=10107093349&attempt=1" target="_blank">Cypress dashboard</a>. > Tags: `@tag.Sanity` > Spec: > <hr>Fri, 26 Jul 2024 07:15:19 UTC <!-- end of auto-generated comment: Cypress test results --> ## Communication Should the DevRel and Marketing teams inform users about this change? - [ ] Yes - [ ] No <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Summary by CodeRabbit - **New Features** - Enhanced PostgreSQL upgrade process with improved error handling and robust management of old server instances. - **Bug Fixes** - Reinstituted logic for checking and managing the `postmaster.pid` file, ensuring proper startup and shutdown of old PostgreSQL servers. - **Refactor** - Improved formatting and readability of shell scripts without altering their functionality. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-07-26 07:22:49 +00:00
# `pg_upgrade` writes log to current folder. So change to a temp folder first.
rm -rf "$TMP/pg_upgrade" "$new_data_dir"
mkdir -p "$TMP/pg_upgrade" "$new_data_dir"
chown -R postgres "$TMP/pg_upgrade" "$new_data_dir"
cd "$TMP/pg_upgrade"
chore: Add script to upgrade Postgres 13 data to 14 (#34317) We're upgrading embedded Postgres from 13 to 14, and this PR includes a script to perform the upgrade of the data folder from v13 schema to v14 schema. This script temporarily installs Postgres 13, if not available, for the upgrade process, so will continue to work when and if we choose to remove `postgresql-13` from the base image. Tested this manually as well, running an Appsmith with Postgres 13, executing some workflows via webhook, getting some run data generated, then upgrading Postgres with the script in this PR, and ensuring that the workflow run history is still there and visible on the UI exactly the same. It is. No conflicts or additional changes needed on EE. [All server and Cypress tests pass on EE](https://github.com/appsmithorg/appsmith-ee/pull/4493). ![shot-2024-06-20-02-13-26](https://github.com/appsmithorg/appsmith/assets/120119/9bb60e3a-6cc9-4df9-9064-caead78729a6) **/test sanity** <!-- This is an auto-generated comment: Cypress test results --> > [!TIP] > 🟢 🟢 🟢 All cypress tests have passed! 🎉 🎉 🎉 > Workflow run: <https://github.com/appsmithorg/appsmith/actions/runs/9590240540> > Commit: 9c75da53f871ffb912015c18a7504327cba88f2c > <a href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=9590240540&attempt=1" target="_blank">Cypress dashboard</a>. > Tags: `@tag.Sanity` <!-- end of auto-generated comment: Cypress test results --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added automation script for upgrading PostgreSQL to the latest version. - Introduced testing script for PostgreSQL upgrades in Docker environments. - **Improvements** - Upgraded PostgreSQL from version 13 to 14 in Docker setup, ensuring compatibility and performance enhancements. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-06-21 14:04:08 +00:00
# Required by the temporary Postgres server started by `pg_upgrade`.
chown postgres /etc/ssl/private/ssl-cert-snakeoil.key
chmod 0600 /etc/ssl/private/ssl-cert-snakeoil.key
su postgres --command "
set -o errexit
set -o xtrace
'$postgres_path/$new_version/bin/initdb' --pgdata='$new_data_dir'
'$postgres_path/$new_version/bin/pg_upgrade' \
--old-datadir='$pg_data_dir' \
--new-datadir='$new_data_dir' \
--old-bindir='$postgres_path/$old_version/bin' \
--new-bindir='$postgres_path/$new_version/bin'
"
date -u '+%FT%T.%3NZ' > "$pg_data_dir/deprecated-on.txt"
mv -v "$pg_data_dir" "$pg_data_dir-$old_version"
mv -v "$new_data_dir" "$pg_data_dir"
# Dangerous generated script that deletes the now updated data folder.
rm -fv "$TMP/pg_upgrade/delete_old_cluster.sh"
fi
if [[ -n "${#to_uninstall[@]}" ]]; then
fix: cleanup stale postgres postmaster.pid (#35171) ## Description issue of Postgres not coming up with error` Previous Postgres was not shutdown cleanly. Please start and stop Postgres 14 properly with 'supervisorctl' only. ` https://www.notion.so/appsmith/Closed-Beta-Customer-issues-45a274a9eb8e4762a72cbff74cd3bad5?pvs=4#ecca04d205414f25a289884cebdc0f9b Fixes https://app.zenhub.com/workspaces/workflows-pod-652fff131a95920b9bf2bc7e/issues/zh/226_ ## Automation /ok-to-test tags="@tag.Sanity" ### :mag: Cypress test results <!-- This is an auto-generated comment: Cypress test results --> > [!TIP] > 🟢 🟢 🟢 All cypress tests have passed! 🎉 🎉 🎉 > Workflow run: <https://github.com/appsmithorg/appsmith/actions/runs/10107093349> > Commit: 357bf5d2ff0c1f65b4cbf46b0f3ee823ac192614 > <a href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=10107093349&attempt=1" target="_blank">Cypress dashboard</a>. > Tags: `@tag.Sanity` > Spec: > <hr>Fri, 26 Jul 2024 07:15:19 UTC <!-- end of auto-generated comment: Cypress test results --> ## Communication Should the DevRel and Marketing teams inform users about this change? - [ ] Yes - [ ] No <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Summary by CodeRabbit - **New Features** - Enhanced PostgreSQL upgrade process with improved error handling and robust management of old server instances. - **Bug Fixes** - Reinstituted logic for checking and managing the `postmaster.pid` file, ensuring proper startup and shutdown of old PostgreSQL servers. - **Refactor** - Improved formatting and readability of shell scripts without altering their functionality. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-07-26 07:22:49 +00:00
DEBIAN_FRONTEND=noninteractive apt-get purge --yes "${to_uninstall[@]}"
chore: Add script to upgrade Postgres 13 data to 14 (#34317) We're upgrading embedded Postgres from 13 to 14, and this PR includes a script to perform the upgrade of the data folder from v13 schema to v14 schema. This script temporarily installs Postgres 13, if not available, for the upgrade process, so will continue to work when and if we choose to remove `postgresql-13` from the base image. Tested this manually as well, running an Appsmith with Postgres 13, executing some workflows via webhook, getting some run data generated, then upgrading Postgres with the script in this PR, and ensuring that the workflow run history is still there and visible on the UI exactly the same. It is. No conflicts or additional changes needed on EE. [All server and Cypress tests pass on EE](https://github.com/appsmithorg/appsmith-ee/pull/4493). ![shot-2024-06-20-02-13-26](https://github.com/appsmithorg/appsmith/assets/120119/9bb60e3a-6cc9-4df9-9064-caead78729a6) **/test sanity** <!-- This is an auto-generated comment: Cypress test results --> > [!TIP] > 🟢 🟢 🟢 All cypress tests have passed! 🎉 🎉 🎉 > Workflow run: <https://github.com/appsmithorg/appsmith/actions/runs/9590240540> > Commit: 9c75da53f871ffb912015c18a7504327cba88f2c > <a href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=9590240540&attempt=1" target="_blank">Cypress dashboard</a>. > Tags: `@tag.Sanity` <!-- end of auto-generated comment: Cypress test results --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added automation script for upgrading PostgreSQL to the latest version. - Introduced testing script for PostgreSQL upgrades in Docker environments. - **Improvements** - Upgraded PostgreSQL from version 13 to 14 in Docker setup, ensuring compatibility and performance enhancements. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-06-21 14:04:08 +00:00
apt-get clean
fi
echo "== Fin =="