PromucFlow_constructor/deploy/docker/fs/opt/appsmith/entrypoint.sh

597 lines
20 KiB
Bash
Raw Normal View History

#!/usr/bin/env bash
# Source the helper script
source pg-utils.sh
set -e
tlog "Running as: $(id)"
stacks_path=/appsmith-stacks
chore: Create appsmith postgres DB to be used by internal processes (#36670) ## Description With the migration to postgres as Appsmith's persistent database we wanted to have central DB to be consumed by internal services. Currently all the services try to create the same DB and may end up in a race condition, so we are moving this task to `entrypoint.sh`. Steps: 1. Initialise data directory for pg 2. Run the upgrade migration if necessary 3. Start the pg server 4. Create `appsmith` db with postgres user 5. Stop the pg server /test Sanity ### :mag: Cypress test results <!-- This is an auto-generated comment: Cypress test results --> > [!TIP] > 🟢 🟢 🟢 All cypress tests have passed! 🎉 🎉 🎉 > Workflow run: <https://github.com/appsmithorg/appsmith/actions/runs/11212147481> > Commit: e3840764acb3088233bf6552d74721abb00518a6 > <a href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=11212147481&attempt=1" target="_blank">Cypress dashboard</a>. > Tags: `@tag.Sanity` > Spec: > <hr>Mon, 07 Oct 2024 09:31:39 UTC <!-- end of auto-generated comment: Cypress test results --> ## Communication Should the DevRel and Marketing teams inform users about this change? - [ ] Yes - [x] No <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Improved database initialization logic for the Appsmith application. - Automatically creates the Appsmith database if it does not already exist during startup. - **Bug Fixes** - Enhanced robustness of the database setup process with existence checks before creation. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-10-08 02:23:58 +00:00
export APPSMITH_PG_DATABASE="appsmith"
export SUPERVISORD_CONF_TARGET="$TMP/supervisor-conf.d/" # export for use in supervisord.conf
export MONGODB_TMP_KEY_PATH="$TMP/mongodb-key" # export for use in supervisor process mongodb.conf
mkdir -pv "$SUPERVISORD_CONF_TARGET" "$WWW_PATH"
setup_proxy_variables() {
export NO_PROXY="${NO_PROXY-localhost,127.0.0.1}"
# Ensure `localhost` and `127.0.0.1` are in always present in `NO_PROXY`.
local no_proxy_lines
no_proxy_lines="$(echo "$NO_PROXY" | tr , \\n)"
if ! echo "$no_proxy_lines" | grep -q '^localhost$'; then
export NO_PROXY="localhost,$NO_PROXY"
fi
if ! echo "$no_proxy_lines" | grep -q '^127.0.0.1$'; then
export NO_PROXY="127.0.0.1,$NO_PROXY"
fi
fix: Support `NO_PROXY` for RTS-Temporal connection (#37284) The Temporal connection SDK used in RTS, uses GRPC to talk to the Temporal server. The GRPC library being used only respects the lowercase proxy env variables (`http_proxy`, `https_proxy` and `no_proxy`), and not the uppercase ones. They have an [open issue asking for this since 2020](https://github.com/grpc/grpc-node/issues/1292). However, in our `entrypoint.sh`, we normalize `http_proxy` and `HTTP_PROXY`, similarly for `https_proxy` and `HTTPS_PROXY`. But we're not doing this normalization for `no_proxy` and `NO_PROXY`. This PR fixes this by doing the same normalization to `no_proxy`. /test sanity ### :mag: Cypress test results <!-- This is an auto-generated comment: Cypress test results --> > [!TIP] > 🟢 🟢 🟢 All cypress tests have passed! 🎉 🎉 🎉 > Workflow run: <https://github.com/appsmithorg/appsmith/actions/runs/11737259912> > Commit: 310fa2c85ac45b4f9a9bc5ce63b1c54839192306 > <a href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=11737259912&attempt=1" target="_blank">Cypress dashboard</a>. > Tags: `@tag.Sanity` > Spec: > <hr>Fri, 08 Nov 2024 07:00:17 UTC <!-- end of auto-generated comment: Cypress test results --> ## Communication Should the DevRel and Marketing teams inform users about this change? - [ ] Yes - [x] No <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Improved handling of proxy environment variables for enhanced configuration consistency. - **Bug Fixes** - Ensured `localhost` and `127.0.0.1` are always included in the `NO_PROXY` variable for better connectivity. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-11-08 09:24:14 +00:00
# If one of NO_PROXY or no_proxy are set, copy it to the other. If both are set, prefer NO_PROXY.
if [[ -n ${NO_PROXY-} ]]; then
export no_proxy="$NO_PROXY"
elif [[ -n ${no_proxy-} ]]; then
export NO_PROXY="$no_proxy"
fi
# If one of HTTPS_PROXY or https_proxy are set, copy it to the other. If both are set, prefer HTTPS_PROXY.
if [[ -n ${HTTPS_PROXY-} ]]; then
export https_proxy="$HTTPS_PROXY"
elif [[ -n ${https_proxy-} ]]; then
export HTTPS_PROXY="$https_proxy"
fi
# If one of HTTP_PROXY or http_proxy are set, copy it to the other. If both are set, prefer HTTP_PROXY.
if [[ -n ${HTTP_PROXY-} ]]; then
export http_proxy="$HTTP_PROXY"
elif [[ -n ${http_proxy-} ]]; then
export HTTP_PROXY="$http_proxy"
fi
}
init_env_file() {
CONF_PATH="/appsmith-stacks/configuration"
ENV_PATH="$CONF_PATH/docker.env"
TEMPLATES_PATH="/opt/appsmith/templates"
2024-06-12 12:15:19 +00:00
if [[ -n "$APPSMITH_MONGODB_URI" ]]; then
export APPSMITH_DB_URL="$APPSMITH_MONGODB_URI"
unset APPSMITH_MONGODB_URI
fi
# Build an env file with current env variables. We single-quote the values, as well as escaping any single-quote characters.
printenv | grep -E '^APPSMITH_|^MONGO_' | sed "s/'/'\\\''/g; s/=/='/; s/$/'/" > "$TMP/pre-define.env"
tlog "Initialize .env file"
if ! [[ -e "$ENV_PATH" ]]; then
# Generate new docker.env file when initializing container for first time or in Heroku which does not have persistent volume
tlog "Generating default configuration file"
mkdir -p "$CONF_PATH"
chore: create appsmith schema for postgres (#36591) ## Description The current state is default schema or public schema. This schema is accessible by default when user connects to the pg database. Hence create `appsmith` schema for Appsmith server to use. This is to avoid anyone accidentally modifying the appsmith data. ## Automation /ok-to-test tags="@tag.Sanity" ### :mag: Cypress test results <!-- This is an auto-generated comment: Cypress test results --> > [!IMPORTANT] > 🟣 🟣 🟣 Your tests are running. > Tests running at: <https://github.com/appsmithorg/appsmith/actions/runs/11111681323> > Commit: 32f91e8d7ce750e4a088996aff4abe6905aa982f > Workflow: `PR Automation test suite` > Tags: `@tag.Sanity` > Spec: `` > <hr>Mon, 30 Sep 2024 18:08:23 UTC <!-- end of auto-generated comment: Cypress test results --> ## Communication Should the DevRel and Marketing teams inform users about this change? - [ ] Yes - [ ] No <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Summary by CodeRabbit - **New Features** - Introduced a script to initialize the PostgreSQL database schema for Appsmith. - Added utilities for managing PostgreSQL database connections, including availability checks and parameter extraction. - Enhanced scripts for managing PostgreSQL connections and initialization. - Improved environment configuration for PostgreSQL database connections, including automatic password generation for local setups. - Updated JDBC URL handling to include schema parameters for PostgreSQL connections. - Added support for proxy configuration in the application setup. - **Bug Fixes** - Improved error handling and connection retry mechanisms for PostgreSQL setup. - **Documentation** - Updated comments and logging for better clarity on database operations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Abhijeet <abhi.nagarnaik@gmail.com>
2024-09-30 18:12:56 +00:00
local default_appsmith_mongodb_user="appsmith"
local generated_appsmith_mongodb_password=$(
tr -dc A-Za-z0-9 </dev/urandom | head -c 13
echo ""
)
local generated_appsmith_encryption_password=$(
tr -dc A-Za-z0-9 </dev/urandom | head -c 13
echo ""
)
local generated_appsmith_encription_salt=$(
tr -dc A-Za-z0-9 </dev/urandom | head -c 13
echo ""
)
local generated_appsmith_supervisor_password=$(
tr -dc A-Za-z0-9 </dev/urandom | head -c 13
echo ''
)
chore: create appsmith schema for postgres (#36591) ## Description The current state is default schema or public schema. This schema is accessible by default when user connects to the pg database. Hence create `appsmith` schema for Appsmith server to use. This is to avoid anyone accidentally modifying the appsmith data. ## Automation /ok-to-test tags="@tag.Sanity" ### :mag: Cypress test results <!-- This is an auto-generated comment: Cypress test results --> > [!IMPORTANT] > 🟣 🟣 🟣 Your tests are running. > Tests running at: <https://github.com/appsmithorg/appsmith/actions/runs/11111681323> > Commit: 32f91e8d7ce750e4a088996aff4abe6905aa982f > Workflow: `PR Automation test suite` > Tags: `@tag.Sanity` > Spec: `` > <hr>Mon, 30 Sep 2024 18:08:23 UTC <!-- end of auto-generated comment: Cypress test results --> ## Communication Should the DevRel and Marketing teams inform users about this change? - [ ] Yes - [ ] No <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Summary by CodeRabbit - **New Features** - Introduced a script to initialize the PostgreSQL database schema for Appsmith. - Added utilities for managing PostgreSQL database connections, including availability checks and parameter extraction. - Enhanced scripts for managing PostgreSQL connections and initialization. - Improved environment configuration for PostgreSQL database connections, including automatic password generation for local setups. - Updated JDBC URL handling to include schema parameters for PostgreSQL connections. - Added support for proxy configuration in the application setup. - **Bug Fixes** - Improved error handling and connection retry mechanisms for PostgreSQL setup. - **Documentation** - Updated comments and logging for better clarity on database operations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Abhijeet <abhi.nagarnaik@gmail.com>
2024-09-30 18:12:56 +00:00
bash "$TEMPLATES_PATH/docker.env.sh" "$default_appsmith_mongodb_user" "$generated_appsmith_mongodb_password" "$generated_appsmith_encryption_password" "$generated_appsmith_encription_salt" "$generated_appsmith_supervisor_password" > "$ENV_PATH"
fi
tlog "Load environment configuration"
2024-06-12 12:15:19 +00:00
# Load the ones in `docker.env` in the stacks folder.
set -o allexport
. "$ENV_PATH"
2024-06-12 12:15:19 +00:00
set +o allexport
if [[ -n "$APPSMITH_MONGODB_URI" ]]; then
export APPSMITH_DB_URL="$APPSMITH_MONGODB_URI"
unset APPSMITH_MONGODB_URI
fi
# Load the ones set from outside, should take precedence, and so will overwrite anything from `docker.env` above.
set -o allexport
. "$TMP/pre-define.env"
set +o allexport
}
init_env_file
setup_proxy_variables
# ip is a reserved keyword for tracking events in Mixpanel. Instead of showing the ip as is Mixpanel provides derived properties.
# As we want derived props alongwith the ip address we are sharing the ip address in separate keys
# https://help.mixpanel.com/hc/en-us/articles/360001355266-Event-Properties
if [[ -n ${APPSMITH_SEGMENT_CE_KEY-} ]]; then
ip="$(set -o pipefail; curl --connect-timeout 5 -sS https://cs.appsmith.com/api/v1/ip | grep -Eo '\d+(\.\d+){3}' || echo "unknown")"
curl \
--connect-timeout 5 \
--user "$APPSMITH_SEGMENT_CE_KEY:" \
--header 'Content-Type: application/json' \
--data '{
"userId":"'"$ip"'",
"event":"Instance Start",
"properties": {
"ip": "'"$ip"'",
"ipAddress": "'"$ip"'"
}
}' \
https://api.segment.io/v1/track \
|| true
fi
if [[ -n "${FILESTORE_IP_ADDRESS-}" ]]; then
## Trim APPSMITH_FILESTORE_IP and FILE_SHARE_NAME
FILESTORE_IP_ADDRESS="$(echo "$FILESTORE_IP_ADDRESS" | xargs)"
FILE_SHARE_NAME="$(echo "$FILE_SHARE_NAME" | xargs)"
tlog "Running appsmith for cloudRun"
tlog "creating mount point"
mkdir -p "$stacks_path"
tlog "Mounting File Sytem"
mount -t nfs -o nolock "$FILESTORE_IP_ADDRESS:/$FILE_SHARE_NAME" /appsmith-stacks
tlog "Mounted File Sytem"
tlog "Setting HOSTNAME for Cloudrun"
export HOSTNAME="cloudrun"
fi
function get_maximum_heap() {
resource=$(ulimit -u)
tlog "Resource : $resource"
if [[ "$resource" -le 256 ]]; then
maximum_heap=128
elif [[ "$resource" -le 512 ]]; then
maximum_heap=256
fi
}
function setup_backend_heap_arg() {
if [[ ! -z ${maximum_heap} ]]; then
export APPSMITH_JAVA_HEAP_ARG="-Xmx${maximum_heap}m"
fi
}
unset_unused_variables() {
# Check for enviroment vairalbes
tlog "Checking environment configuration"
if [[ -z "${APPSMITH_MAIL_ENABLED}" ]]; then
unset APPSMITH_MAIL_ENABLED # If this field is empty is might cause application crash
fi
if [[ -z "${APPSMITH_OAUTH2_GITHUB_CLIENT_ID}" ]] || [[ -z "${APPSMITH_OAUTH2_GITHUB_CLIENT_SECRET}" ]]; then
unset APPSMITH_OAUTH2_GITHUB_CLIENT_ID # If this field is empty is might cause application crash
unset APPSMITH_OAUTH2_GITHUB_CLIENT_SECRET
fi
if [[ -z "${APPSMITH_OAUTH2_GOOGLE_CLIENT_ID}" ]] || [[ -z "${APPSMITH_OAUTH2_GOOGLE_CLIENT_SECRET}" ]]; then
unset APPSMITH_OAUTH2_GOOGLE_CLIENT_ID # If this field is empty is might cause application crash
unset APPSMITH_OAUTH2_GOOGLE_CLIENT_SECRET
fi
if [[ -z "${APPSMITH_RECAPTCHA_SITE_KEY}" ]] || [[ -z "${APPSMITH_RECAPTCHA_SECRET_KEY}" ]] || [[ -z "${APPSMITH_RECAPTCHA_ENABLED}" ]]; then
unset APPSMITH_RECAPTCHA_SITE_KEY # If this field is empty is might cause application crash
unset APPSMITH_RECAPTCHA_SECRET_KEY
unset APPSMITH_RECAPTCHA_ENABLED
fi
export APPSMITH_SUPERVISOR_USER="${APPSMITH_SUPERVISOR_USER:-appsmith}"
if [[ -z "${APPSMITH_SUPERVISOR_PASSWORD-}" ]]; then
APPSMITH_SUPERVISOR_PASSWORD="$(tr -dc A-Za-z0-9 </dev/urandom | head -c 13)"
export APPSMITH_SUPERVISOR_PASSWORD
fi
}
configure_database_connection_url() {
tlog "Configuring database connection URL"
isPostgresUrl=0
isMongoUrl=0
if [[ "${APPSMITH_DB_URL}" == "postgresql:"* ]]; then
isPostgresUrl=1
elif [[ "${APPSMITH_DB_URL}" == "mongodb"* ]]; then
isMongoUrl=1
fi
}
check_db_uri() {
tlog "Checking APPSMITH_DB_URL"
isUriLocal=1
if [[ $APPSMITH_DB_URL == *"localhost"* || $APPSMITH_DB_URL == *"127.0.0.1"* ]]; then
tlog "Detected local DB"
isUriLocal=0
fi
}
init_mongodb() {
if [[ $isUriLocal -eq 0 ]]; then
tlog "Initializing local database"
MONGO_DB_PATH="$stacks_path/data/mongodb"
MONGO_LOG_PATH="$MONGO_DB_PATH/log"
MONGO_DB_KEY="$MONGO_DB_PATH/key"
mkdir -p "$MONGO_DB_PATH"
touch "$MONGO_LOG_PATH"
if [[ ! -f "$MONGO_DB_KEY" ]]; then
openssl rand -base64 756 > "$MONGO_DB_KEY"
fi
use-mongodb-key "$MONGO_DB_KEY"
./mongodb-fixer.sh &
fi
}
init_replica_set() {
tlog "Checking initialized database"
shouldPerformInitdb=1
for path in \
"$MONGO_DB_PATH/WiredTiger" \
"$MONGO_DB_PATH/journal" \
"$MONGO_DB_PATH/local.0" \
"$MONGO_DB_PATH/storage.bson"; do
if [ -e "$path" ]; then
shouldPerformInitdb=0
break
fi
done
if [[ $isUriLocal -gt 0 && -f /proc/cpuinfo ]] && ! grep --quiet avx /proc/cpuinfo; then
tlog "====================================================================================================" >&2
tlog "==" >&2
tlog "== AVX instruction not found in your CPU. Appsmith's embedded MongoDB may not start. Please use an external MongoDB instance instead." >&2
tlog "== See https://docs.appsmith.com/getting-started/setup/instance-configuration/custom-mongodb-redis#custom-mongodb for instructions." >&2
tlog "==" >&2
tlog "====================================================================================================" >&2
fi
if [[ $shouldPerformInitdb -gt 0 && $isUriLocal -eq 0 ]]; then
tlog "Initializing Replica Set for local database"
# Start installed MongoDB service - Dependencies Layer
mongod --fork --port 27017 --dbpath "$MONGO_DB_PATH" --logpath "$MONGO_LOG_PATH"
tlog "Waiting 10s for MongoDB to start"
sleep 10
tlog "Creating MongoDB user"
mongosh "127.0.0.1/appsmith" --eval "db.createUser({
user: '$APPSMITH_MONGODB_USER',
pwd: '$APPSMITH_MONGODB_PASSWORD',
roles: [{
role: 'root',
db: 'admin'
}, 'readWrite']
}
)"
tlog "Enabling Replica Set"
mongod --dbpath "$MONGO_DB_PATH" --shutdown || true
mongod --fork --port 27017 --dbpath "$MONGO_DB_PATH" --logpath "$MONGO_LOG_PATH" --replSet mr1 --keyFile "$MONGODB_TMP_KEY_PATH" --bind_ip localhost
tlog "Waiting 10s for MongoDB to start with Replica Set"
sleep 10
mongosh "$APPSMITH_DB_URL" --eval 'rs.initiate()'
mongod --dbpath "$MONGO_DB_PATH" --shutdown || true
fi
if [[ $isUriLocal -gt 0 ]]; then
tlog "Checking Replica Set of external MongoDB"
if appsmithctl check-replica-set; then
tlog "MongoDB ReplicaSet is enabled"
else
echo -e "\033[0;31m***************************************************************************************\033[0m"
echo -e "\033[0;31m* MongoDB Replica Set is not enabled *\033[0m"
echo -e "\033[0;31m* Please ensure the credentials provided for MongoDB, has 'readWrite' role. *\033[0m"
echo -e "\033[0;31m***************************************************************************************\033[0m"
exit 1
fi
fi
}
use-mongodb-key() {
# We copy the MongoDB key file to `$MONGODB_TMP_KEY_PATH`, so that we can reliably set its permissions to 600.
# Why? When the host machine of this Docker container is Windows, file permissions cannot be set on files in volumes.
# So the key file should be somewhere inside the container, and not in a volume.
mkdir -pv "$(dirname "$MONGODB_TMP_KEY_PATH")"
cp -v "$1" "$MONGODB_TMP_KEY_PATH"
chmod 600 "$MONGODB_TMP_KEY_PATH"
}
is_empty_directory() {
[[ -d $1 && -z "$(ls -A "$1")" ]]
}
check_setup_custom_ca_certificates() {
# old, deprecated, should be removed.
local stacks_ca_certs_path
stacks_ca_certs_path="$stacks_path/ca-certs"
local container_ca_certs_path
container_ca_certs_path="/usr/local/share/ca-certificates"
if [[ -d $stacks_ca_certs_path ]]; then
if [[ ! -L $container_ca_certs_path ]]; then
if is_empty_directory "$container_ca_certs_path"; then
rmdir -v "$container_ca_certs_path"
else
tlog "The 'ca-certificates' directory inside the container is not empty. Please clear it and restart to use certs from 'stacks/ca-certs' directory." >&2
return
fi
fi
ln --verbose --force --symbolic --no-target-directory "$stacks_ca_certs_path" "$container_ca_certs_path"
elif [[ ! -e $container_ca_certs_path ]]; then
rm -vf "$container_ca_certs_path" # If it exists as a broken symlink, this will be needed.
mkdir -v "$container_ca_certs_path"
fi
update-ca-certificates --fresh
}
setup-custom-ca-certificates() (
local stacks_ca_certs_path="$stacks_path/ca-certs"
local store="$TMP/cacerts"
local opts_file="$TMP/java-cacerts-opts"
rm -f "$store" "$opts_file"
if [[ -n "$(ls "$stacks_ca_certs_path"/*.pem 2>/dev/null)" ]]; then
tlog "Looks like you have some '.pem' files in your 'ca-certs' folder. Please rename them to '.crt' to be picked up automatically.".
fi
if ! [[ -d "$stacks_ca_certs_path" && "$(find "$stacks_ca_certs_path" -maxdepth 1 -type f -name '*.crt' | wc -l)" -gt 0 ]]; then
tlog "No custom CA certificates found."
return
fi
# Import the system CA certificates into the store.
keytool -importkeystore \
-srckeystore /opt/java/lib/security/cacerts \
-destkeystore "$store" \
-srcstorepass changeit \
-deststorepass changeit
# Add the custom CA certificates to the store.
find -L "$stacks_ca_certs_path" -maxdepth 1 -type f -name '*.crt' \
-print \
-exec keytool -import -alias '{}' -noprompt -keystore "$store" -file '{}' -storepass changeit ';'
{
echo "-Djavax.net.ssl.trustStore=$store"
echo "-Djavax.net.ssl.trustStorePassword=changeit"
} > "$opts_file"
)
configure_supervisord() {
local supervisord_conf_source="/opt/appsmith/templates/supervisord"
if [[ -n "$(ls -A "$SUPERVISORD_CONF_TARGET")" ]]; then
rm -f "$SUPERVISORD_CONF_TARGET"/*
fi
cp -f "$supervisord_conf_source"/application_process/*.conf "$SUPERVISORD_CONF_TARGET"
# Disable services based on configuration
if [[ -z "${DYNO}" ]]; then
feat: Make images adaptable to support both Postgres and MongoDB uris (#36424) ## Description PR to make the release tag adaptable to work with both MongoDB and PostgreSQL uris. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Introduced a new script to automate the preparation of server artifacts, improving the build process. - Added SQL files to the indentation configuration for consistent code formatting. - **Improvements** - Enhanced error handling in the Docker build process to ensure essential files are present before execution. - Updated service configuration logic to prevent misconfiguration based on the environment. - Added a new job step in the build workflow to prepare server artifacts after the build process. - Implemented conditional logic in the run script to dynamically adapt to different database configurations. - **Bug Fixes** - Adjusted the initialization process to focus on MongoDB, improving reliability in various environments. <!-- end of auto-generated comment: release notes by coderabbit.ai --> /test Sanity ### :mag: Cypress test results <!-- This is an auto-generated comment: Cypress test results --> > [!TIP] > 🟢 🟢 🟢 All cypress tests have passed! 🎉 🎉 🎉 > Workflow run: <https://github.com/appsmithorg/appsmith/actions/runs/10940528231> > Commit: 32731e8a93a25e5c9456eb89daca2d8bf327c012 > <a href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=10940528231&attempt=1" target="_blank">Cypress dashboard</a>. > Tags: `@tag.Sanity` > Spec: > <hr>Thu, 19 Sep 2024 12:21:54 UTC <!-- end of auto-generated comment: Cypress test results --> ## Communication Should the DevRel and Marketing teams inform users about this change? - [ ] Yes - [x] No
2024-09-19 16:15:46 +00:00
if [[ $isUriLocal -eq 0 && $isMongoUrl -eq 1 ]]; then
cp "$supervisord_conf_source/mongodb.conf" "$SUPERVISORD_CONF_TARGET"
fi
if [[ $APPSMITH_REDIS_URL == *"localhost"* || $APPSMITH_REDIS_URL == *"127.0.0.1"* ]]; then
cp "$supervisord_conf_source/redis.conf" "$SUPERVISORD_CONF_TARGET"
mkdir -p "$stacks_path/data/redis"
fi
feat: Embeded PostgreSQL with mockdb data for self-hosted (#21084) ## Description This PR includes changes for self-hosted instances to utilize an embedded postgres database for the Template's mockdb. After this change, by default Appsmith will always run the embedded Postgres, and connect to it instead of the mockdb cloud instance. **Solves:** Issue[#20107](https://github.com/appsmithorg/appsmith/issues/20107) **Changes:** - [x] Install Postgresql in the Docker container using Dockerfile :- _Updated Docker file to install PostgreSQL v13_ - [x] Add provision to stop/disable Postgresql from booting based on an environment variable :- _To disable embedded postgres, set the **ENV** var `APPSMITH_ENABLE_EMBEDDED_DB` to `0`; Configured supervisord to control the lifecycle of the postgres server ( Supervisor uses user postgres to start the server since root is not allowed)_ - [x] Seed data into the Postgresql DB during first container startup. This ensures that we don't have to ship the Docker container with data & bloat it unnecessarily. :- _Curretly, the SQL dump is baked into the image._ - [x] Persist the Postgresql data in /appsmith-stacks. This ensures that the data is persisted across version upgrades. :- _The postgres uses the file system at` /stacks/data/postgres/main` ; note: The file system owner is user/group postgres_ - [x] Update the mock db endpoint in the product to use the local DB by default instead of the hosted DB for self-hosted instances. The cloud instance should still use the hosted DB :- _The embedded postgres authentication is set to the type `trust`, allowing the existing template mockdb meta data to be used without any update to the existing templates mockdb endpoint and credentials_ **Note: There is an additional table `mydb` added to the mockdb for quickly verifying the db source is the embedded postgres, since the cloud mockdb does not have that table.** --------- Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
2023-03-07 09:52:41 +00:00
if [[ $runEmbeddedPostgres -eq 1 ]]; then
cp "$supervisord_conf_source/postgres.conf" "$SUPERVISORD_CONF_TARGET"
feat: Embeded PostgreSQL with mockdb data for self-hosted (#21084) ## Description This PR includes changes for self-hosted instances to utilize an embedded postgres database for the Template's mockdb. After this change, by default Appsmith will always run the embedded Postgres, and connect to it instead of the mockdb cloud instance. **Solves:** Issue[#20107](https://github.com/appsmithorg/appsmith/issues/20107) **Changes:** - [x] Install Postgresql in the Docker container using Dockerfile :- _Updated Docker file to install PostgreSQL v13_ - [x] Add provision to stop/disable Postgresql from booting based on an environment variable :- _To disable embedded postgres, set the **ENV** var `APPSMITH_ENABLE_EMBEDDED_DB` to `0`; Configured supervisord to control the lifecycle of the postgres server ( Supervisor uses user postgres to start the server since root is not allowed)_ - [x] Seed data into the Postgresql DB during first container startup. This ensures that we don't have to ship the Docker container with data & bloat it unnecessarily. :- _Curretly, the SQL dump is baked into the image._ - [x] Persist the Postgresql data in /appsmith-stacks. This ensures that the data is persisted across version upgrades. :- _The postgres uses the file system at` /stacks/data/postgres/main` ; note: The file system owner is user/group postgres_ - [x] Update the mock db endpoint in the product to use the local DB by default instead of the hosted DB for self-hosted instances. The cloud instance should still use the hosted DB :- _The embedded postgres authentication is set to the type `trust`, allowing the existing template mockdb meta data to be used without any update to the existing templates mockdb endpoint and credentials_ **Note: There is an additional table `mydb` added to the mockdb for quickly verifying the db source is the embedded postgres, since the cloud mockdb does not have that table.** --------- Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
2023-03-07 09:52:41 +00:00
fi
fi
}
# This is a workaround to get Redis working on different memory pagesize
2022-04-01 06:58:03 +00:00
# https://github.com/appsmithorg/appsmith/issues/11773
check_redis_compatible_page_size() {
local page_size
page_size="$(getconf PAGE_SIZE)"
if [[ $page_size -gt 4096 ]]; then
curl \
--connect-timeout 5 \
--silent \
--user "$APPSMITH_SEGMENT_CE_KEY:" \
--header 'Content-Type: application/json' \
--data '{ "userId": "'"$HOSTNAME"'", "event":"RedisCompile" }' \
https://api.segment.io/v1/track \
|| true
tlog "Compile Redis stable with page size of $page_size"
apt-get update
apt-get install --yes build-essential
curl --connect-timeout 5 --location https://download.redis.io/redis-stable.tar.gz | tar -xz -C /tmp
pushd /tmp/redis-stable
make
make install
popd
rm -rf /tmp/redis-stable
2022-04-01 06:58:03 +00:00
else
tlog "Redis is compatible with page size of $page_size"
2022-04-01 06:58:03 +00:00
fi
}
feat: Embeded PostgreSQL with mockdb data for self-hosted (#21084) ## Description This PR includes changes for self-hosted instances to utilize an embedded postgres database for the Template's mockdb. After this change, by default Appsmith will always run the embedded Postgres, and connect to it instead of the mockdb cloud instance. **Solves:** Issue[#20107](https://github.com/appsmithorg/appsmith/issues/20107) **Changes:** - [x] Install Postgresql in the Docker container using Dockerfile :- _Updated Docker file to install PostgreSQL v13_ - [x] Add provision to stop/disable Postgresql from booting based on an environment variable :- _To disable embedded postgres, set the **ENV** var `APPSMITH_ENABLE_EMBEDDED_DB` to `0`; Configured supervisord to control the lifecycle of the postgres server ( Supervisor uses user postgres to start the server since root is not allowed)_ - [x] Seed data into the Postgresql DB during first container startup. This ensures that we don't have to ship the Docker container with data & bloat it unnecessarily. :- _Curretly, the SQL dump is baked into the image._ - [x] Persist the Postgresql data in /appsmith-stacks. This ensures that the data is persisted across version upgrades. :- _The postgres uses the file system at` /stacks/data/postgres/main` ; note: The file system owner is user/group postgres_ - [x] Update the mock db endpoint in the product to use the local DB by default instead of the hosted DB for self-hosted instances. The cloud instance should still use the hosted DB :- _The embedded postgres authentication is set to the type `trust`, allowing the existing template mockdb meta data to be used without any update to the existing templates mockdb endpoint and credentials_ **Note: There is an additional table `mydb` added to the mockdb for quickly verifying the db source is the embedded postgres, since the cloud mockdb does not have that table.** --------- Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
2023-03-07 09:52:41 +00:00
init_postgres() {
# Initialize embedded postgres by default; set APPSMITH_ENABLE_EMBEDDED_DB to 0, to use existing cloud postgres mockdb instance
if [[ ${APPSMITH_ENABLE_EMBEDDED_DB: -1} != 0 ]]; then
tlog "Checking initialized local postgres"
feat: Embeded PostgreSQL with mockdb data for self-hosted (#21084) ## Description This PR includes changes for self-hosted instances to utilize an embedded postgres database for the Template's mockdb. After this change, by default Appsmith will always run the embedded Postgres, and connect to it instead of the mockdb cloud instance. **Solves:** Issue[#20107](https://github.com/appsmithorg/appsmith/issues/20107) **Changes:** - [x] Install Postgresql in the Docker container using Dockerfile :- _Updated Docker file to install PostgreSQL v13_ - [x] Add provision to stop/disable Postgresql from booting based on an environment variable :- _To disable embedded postgres, set the **ENV** var `APPSMITH_ENABLE_EMBEDDED_DB` to `0`; Configured supervisord to control the lifecycle of the postgres server ( Supervisor uses user postgres to start the server since root is not allowed)_ - [x] Seed data into the Postgresql DB during first container startup. This ensures that we don't have to ship the Docker container with data & bloat it unnecessarily. :- _Curretly, the SQL dump is baked into the image._ - [x] Persist the Postgresql data in /appsmith-stacks. This ensures that the data is persisted across version upgrades. :- _The postgres uses the file system at` /stacks/data/postgres/main` ; note: The file system owner is user/group postgres_ - [x] Update the mock db endpoint in the product to use the local DB by default instead of the hosted DB for self-hosted instances. The cloud instance should still use the hosted DB :- _The embedded postgres authentication is set to the type `trust`, allowing the existing template mockdb meta data to be used without any update to the existing templates mockdb endpoint and credentials_ **Note: There is an additional table `mydb` added to the mockdb for quickly verifying the db source is the embedded postgres, since the cloud mockdb does not have that table.** --------- Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
2023-03-07 09:52:41 +00:00
POSTGRES_DB_PATH="$stacks_path/data/postgres/main"
mkdir -p "$POSTGRES_DB_PATH" "$TMP/pg-runtime"
# Postgres does not allow it's server to be run with super user access, we use user postgres and the file system owner also needs to be the same user postgres
chown -R postgres:postgres "$POSTGRES_DB_PATH" "$TMP/pg-runtime"
chore: Add script to upgrade Postgres 13 data to 14 (#34317) We're upgrading embedded Postgres from 13 to 14, and this PR includes a script to perform the upgrade of the data folder from v13 schema to v14 schema. This script temporarily installs Postgres 13, if not available, for the upgrade process, so will continue to work when and if we choose to remove `postgresql-13` from the base image. Tested this manually as well, running an Appsmith with Postgres 13, executing some workflows via webhook, getting some run data generated, then upgrading Postgres with the script in this PR, and ensuring that the workflow run history is still there and visible on the UI exactly the same. It is. No conflicts or additional changes needed on EE. [All server and Cypress tests pass on EE](https://github.com/appsmithorg/appsmith-ee/pull/4493). ![shot-2024-06-20-02-13-26](https://github.com/appsmithorg/appsmith/assets/120119/9bb60e3a-6cc9-4df9-9064-caead78729a6) **/test sanity** <!-- This is an auto-generated comment: Cypress test results --> > [!TIP] > 🟢 🟢 🟢 All cypress tests have passed! 🎉 🎉 🎉 > Workflow run: <https://github.com/appsmithorg/appsmith/actions/runs/9590240540> > Commit: 9c75da53f871ffb912015c18a7504327cba88f2c > <a href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=9590240540&attempt=1" target="_blank">Cypress dashboard</a>. > Tags: `@tag.Sanity` <!-- end of auto-generated comment: Cypress test results --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added automation script for upgrading PostgreSQL to the latest version. - Introduced testing script for PostgreSQL upgrades in Docker environments. - **Improvements** - Upgraded PostgreSQL from version 13 to 14 in Docker setup, ensuring compatibility and performance enhancements. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-06-21 14:04:08 +00:00
if [[ -e "$POSTGRES_DB_PATH/PG_VERSION" ]]; then
/opt/appsmith/pg-upgrade.sh
else
tlog "Initializing local Postgres data folder"
su postgres -c "env PATH='$PATH' initdb -D $POSTGRES_DB_PATH"
feat: Embeded PostgreSQL with mockdb data for self-hosted (#21084) ## Description This PR includes changes for self-hosted instances to utilize an embedded postgres database for the Template's mockdb. After this change, by default Appsmith will always run the embedded Postgres, and connect to it instead of the mockdb cloud instance. **Solves:** Issue[#20107](https://github.com/appsmithorg/appsmith/issues/20107) **Changes:** - [x] Install Postgresql in the Docker container using Dockerfile :- _Updated Docker file to install PostgreSQL v13_ - [x] Add provision to stop/disable Postgresql from booting based on an environment variable :- _To disable embedded postgres, set the **ENV** var `APPSMITH_ENABLE_EMBEDDED_DB` to `0`; Configured supervisord to control the lifecycle of the postgres server ( Supervisor uses user postgres to start the server since root is not allowed)_ - [x] Seed data into the Postgresql DB during first container startup. This ensures that we don't have to ship the Docker container with data & bloat it unnecessarily. :- _Curretly, the SQL dump is baked into the image._ - [x] Persist the Postgresql data in /appsmith-stacks. This ensures that the data is persisted across version upgrades. :- _The postgres uses the file system at` /stacks/data/postgres/main` ; note: The file system owner is user/group postgres_ - [x] Update the mock db endpoint in the product to use the local DB by default instead of the hosted DB for self-hosted instances. The cloud instance should still use the hosted DB :- _The embedded postgres authentication is set to the type `trust`, allowing the existing template mockdb meta data to be used without any update to the existing templates mockdb endpoint and credentials_ **Note: There is an additional table `mydb` added to the mockdb for quickly verifying the db source is the embedded postgres, since the cloud mockdb does not have that table.** --------- Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
2023-03-07 09:52:41 +00:00
fi
cp /opt/appsmith/postgres/appsmith_hba.conf "$POSTGRES_DB_PATH/pg_hba.conf"
# PostgreSQL requires strict file permissions for the pg_hba.conf file. Add file permission settings after copying the configuration file.
# 600 is the recommended permission for pg_hba.conf file for read and write access to the owner only.
chown postgres:postgres "$POSTGRES_DB_PATH/pg_hba.conf"
chmod 600 "$POSTGRES_DB_PATH/pg_hba.conf"
chore: Create appsmith postgres DB to be used by internal processes (#36670) ## Description With the migration to postgres as Appsmith's persistent database we wanted to have central DB to be consumed by internal services. Currently all the services try to create the same DB and may end up in a race condition, so we are moving this task to `entrypoint.sh`. Steps: 1. Initialise data directory for pg 2. Run the upgrade migration if necessary 3. Start the pg server 4. Create `appsmith` db with postgres user 5. Stop the pg server /test Sanity ### :mag: Cypress test results <!-- This is an auto-generated comment: Cypress test results --> > [!TIP] > 🟢 🟢 🟢 All cypress tests have passed! 🎉 🎉 🎉 > Workflow run: <https://github.com/appsmithorg/appsmith/actions/runs/11212147481> > Commit: e3840764acb3088233bf6552d74721abb00518a6 > <a href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=11212147481&attempt=1" target="_blank">Cypress dashboard</a>. > Tags: `@tag.Sanity` > Spec: > <hr>Mon, 07 Oct 2024 09:31:39 UTC <!-- end of auto-generated comment: Cypress test results --> ## Communication Should the DevRel and Marketing teams inform users about this change? - [ ] Yes - [x] No <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Improved database initialization logic for the Appsmith application. - Automatically creates the Appsmith database if it does not already exist during startup. - **Bug Fixes** - Enhanced robustness of the database setup process with existence checks before creation. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-10-08 02:23:58 +00:00
create_appsmith_pg_db "$POSTGRES_DB_PATH"
feat: Embeded PostgreSQL with mockdb data for self-hosted (#21084) ## Description This PR includes changes for self-hosted instances to utilize an embedded postgres database for the Template's mockdb. After this change, by default Appsmith will always run the embedded Postgres, and connect to it instead of the mockdb cloud instance. **Solves:** Issue[#20107](https://github.com/appsmithorg/appsmith/issues/20107) **Changes:** - [x] Install Postgresql in the Docker container using Dockerfile :- _Updated Docker file to install PostgreSQL v13_ - [x] Add provision to stop/disable Postgresql from booting based on an environment variable :- _To disable embedded postgres, set the **ENV** var `APPSMITH_ENABLE_EMBEDDED_DB` to `0`; Configured supervisord to control the lifecycle of the postgres server ( Supervisor uses user postgres to start the server since root is not allowed)_ - [x] Seed data into the Postgresql DB during first container startup. This ensures that we don't have to ship the Docker container with data & bloat it unnecessarily. :- _Curretly, the SQL dump is baked into the image._ - [x] Persist the Postgresql data in /appsmith-stacks. This ensures that the data is persisted across version upgrades. :- _The postgres uses the file system at` /stacks/data/postgres/main` ; note: The file system owner is user/group postgres_ - [x] Update the mock db endpoint in the product to use the local DB by default instead of the hosted DB for self-hosted instances. The cloud instance should still use the hosted DB :- _The embedded postgres authentication is set to the type `trust`, allowing the existing template mockdb meta data to be used without any update to the existing templates mockdb endpoint and credentials_ **Note: There is an additional table `mydb` added to the mockdb for quickly verifying the db source is the embedded postgres, since the cloud mockdb does not have that table.** --------- Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
2023-03-07 09:52:41 +00:00
else
runEmbeddedPostgres=0
feat: Embeded PostgreSQL with mockdb data for self-hosted (#21084) ## Description This PR includes changes for self-hosted instances to utilize an embedded postgres database for the Template's mockdb. After this change, by default Appsmith will always run the embedded Postgres, and connect to it instead of the mockdb cloud instance. **Solves:** Issue[#20107](https://github.com/appsmithorg/appsmith/issues/20107) **Changes:** - [x] Install Postgresql in the Docker container using Dockerfile :- _Updated Docker file to install PostgreSQL v13_ - [x] Add provision to stop/disable Postgresql from booting based on an environment variable :- _To disable embedded postgres, set the **ENV** var `APPSMITH_ENABLE_EMBEDDED_DB` to `0`; Configured supervisord to control the lifecycle of the postgres server ( Supervisor uses user postgres to start the server since root is not allowed)_ - [x] Seed data into the Postgresql DB during first container startup. This ensures that we don't have to ship the Docker container with data & bloat it unnecessarily. :- _Curretly, the SQL dump is baked into the image._ - [x] Persist the Postgresql data in /appsmith-stacks. This ensures that the data is persisted across version upgrades. :- _The postgres uses the file system at` /stacks/data/postgres/main` ; note: The file system owner is user/group postgres_ - [x] Update the mock db endpoint in the product to use the local DB by default instead of the hosted DB for self-hosted instances. The cloud instance should still use the hosted DB :- _The embedded postgres authentication is set to the type `trust`, allowing the existing template mockdb meta data to be used without any update to the existing templates mockdb endpoint and credentials_ **Note: There is an additional table `mydb` added to the mockdb for quickly verifying the db source is the embedded postgres, since the cloud mockdb does not have that table.** --------- Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
2023-03-07 09:52:41 +00:00
fi
feat: Embeded PostgreSQL with mockdb data for self-hosted (#21084) ## Description This PR includes changes for self-hosted instances to utilize an embedded postgres database for the Template's mockdb. After this change, by default Appsmith will always run the embedded Postgres, and connect to it instead of the mockdb cloud instance. **Solves:** Issue[#20107](https://github.com/appsmithorg/appsmith/issues/20107) **Changes:** - [x] Install Postgresql in the Docker container using Dockerfile :- _Updated Docker file to install PostgreSQL v13_ - [x] Add provision to stop/disable Postgresql from booting based on an environment variable :- _To disable embedded postgres, set the **ENV** var `APPSMITH_ENABLE_EMBEDDED_DB` to `0`; Configured supervisord to control the lifecycle of the postgres server ( Supervisor uses user postgres to start the server since root is not allowed)_ - [x] Seed data into the Postgresql DB during first container startup. This ensures that we don't have to ship the Docker container with data & bloat it unnecessarily. :- _Curretly, the SQL dump is baked into the image._ - [x] Persist the Postgresql data in /appsmith-stacks. This ensures that the data is persisted across version upgrades. :- _The postgres uses the file system at` /stacks/data/postgres/main` ; note: The file system owner is user/group postgres_ - [x] Update the mock db endpoint in the product to use the local DB by default instead of the hosted DB for self-hosted instances. The cloud instance should still use the hosted DB :- _The embedded postgres authentication is set to the type `trust`, allowing the existing template mockdb meta data to be used without any update to the existing templates mockdb endpoint and credentials_ **Note: There is an additional table `mydb` added to the mockdb for quickly verifying the db source is the embedded postgres, since the cloud mockdb does not have that table.** --------- Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
2023-03-07 09:52:41 +00:00
}
fix: cleanup stale postgres postmaster.pid (#35171) ## Description issue of Postgres not coming up with error` Previous Postgres was not shutdown cleanly. Please start and stop Postgres 14 properly with 'supervisorctl' only. ` https://www.notion.so/appsmith/Closed-Beta-Customer-issues-45a274a9eb8e4762a72cbff74cd3bad5?pvs=4#ecca04d205414f25a289884cebdc0f9b Fixes https://app.zenhub.com/workspaces/workflows-pod-652fff131a95920b9bf2bc7e/issues/zh/226_ ## Automation /ok-to-test tags="@tag.Sanity" ### :mag: Cypress test results <!-- This is an auto-generated comment: Cypress test results --> > [!TIP] > 🟢 🟢 🟢 All cypress tests have passed! 🎉 🎉 🎉 > Workflow run: <https://github.com/appsmithorg/appsmith/actions/runs/10107093349> > Commit: 357bf5d2ff0c1f65b4cbf46b0f3ee823ac192614 > <a href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=10107093349&attempt=1" target="_blank">Cypress dashboard</a>. > Tags: `@tag.Sanity` > Spec: > <hr>Fri, 26 Jul 2024 07:15:19 UTC <!-- end of auto-generated comment: Cypress test results --> ## Communication Should the DevRel and Marketing teams inform users about this change? - [ ] Yes - [ ] No <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Summary by CodeRabbit - **New Features** - Enhanced PostgreSQL upgrade process with improved error handling and robust management of old server instances. - **Bug Fixes** - Reinstituted logic for checking and managing the `postmaster.pid` file, ensuring proper startup and shutdown of old PostgreSQL servers. - **Refactor** - Improved formatting and readability of shell scripts without altering their functionality. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-07-26 07:22:49 +00:00
safe_init_postgres() {
runEmbeddedPostgres=1
# fail safe to prevent entrypoint from exiting, and prevent postgres from starting
# when runEmbeddedPostgres=0 , postgres conf file for supervisord will not be copied
# so postgres will not be started by supervisor. Explicit message helps us to know upgrade script failed.
if init_postgres; then
tlog "init_postgres succeeded."
else
local exit_status=$?
tlog "init_postgres failed with exit status $exit_status."
runEmbeddedPostgres=0
fi
feat: Embeded PostgreSQL with mockdb data for self-hosted (#21084) ## Description This PR includes changes for self-hosted instances to utilize an embedded postgres database for the Template's mockdb. After this change, by default Appsmith will always run the embedded Postgres, and connect to it instead of the mockdb cloud instance. **Solves:** Issue[#20107](https://github.com/appsmithorg/appsmith/issues/20107) **Changes:** - [x] Install Postgresql in the Docker container using Dockerfile :- _Updated Docker file to install PostgreSQL v13_ - [x] Add provision to stop/disable Postgresql from booting based on an environment variable :- _To disable embedded postgres, set the **ENV** var `APPSMITH_ENABLE_EMBEDDED_DB` to `0`; Configured supervisord to control the lifecycle of the postgres server ( Supervisor uses user postgres to start the server since root is not allowed)_ - [x] Seed data into the Postgresql DB during first container startup. This ensures that we don't have to ship the Docker container with data & bloat it unnecessarily. :- _Curretly, the SQL dump is baked into the image._ - [x] Persist the Postgresql data in /appsmith-stacks. This ensures that the data is persisted across version upgrades. :- _The postgres uses the file system at` /stacks/data/postgres/main` ; note: The file system owner is user/group postgres_ - [x] Update the mock db endpoint in the product to use the local DB by default instead of the hosted DB for self-hosted instances. The cloud instance should still use the hosted DB :- _The embedded postgres authentication is set to the type `trust`, allowing the existing template mockdb meta data to be used without any update to the existing templates mockdb endpoint and credentials_ **Note: There is an additional table `mydb` added to the mockdb for quickly verifying the db source is the embedded postgres, since the cloud mockdb does not have that table.** --------- Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
2023-03-07 09:52:41 +00:00
}
chore: Create appsmith postgres DB to be used by internal processes (#36670) ## Description With the migration to postgres as Appsmith's persistent database we wanted to have central DB to be consumed by internal services. Currently all the services try to create the same DB and may end up in a race condition, so we are moving this task to `entrypoint.sh`. Steps: 1. Initialise data directory for pg 2. Run the upgrade migration if necessary 3. Start the pg server 4. Create `appsmith` db with postgres user 5. Stop the pg server /test Sanity ### :mag: Cypress test results <!-- This is an auto-generated comment: Cypress test results --> > [!TIP] > 🟢 🟢 🟢 All cypress tests have passed! 🎉 🎉 🎉 > Workflow run: <https://github.com/appsmithorg/appsmith/actions/runs/11212147481> > Commit: e3840764acb3088233bf6552d74721abb00518a6 > <a href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=11212147481&attempt=1" target="_blank">Cypress dashboard</a>. > Tags: `@tag.Sanity` > Spec: > <hr>Mon, 07 Oct 2024 09:31:39 UTC <!-- end of auto-generated comment: Cypress test results --> ## Communication Should the DevRel and Marketing teams inform users about this change? - [ ] Yes - [x] No <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Improved database initialization logic for the Appsmith application. - Automatically creates the Appsmith database if it does not already exist during startup. - **Bug Fixes** - Enhanced robustness of the database setup process with existence checks before creation. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-10-08 02:23:58 +00:00
# Method to create a appsmith database in the postgres
# Args:
# POSTGRES_DB_PATH (string): Path to the postgres data directory
# Returns:
# None
# Example:
# create_appsmith_pg_db "/appsmith-stacks/data/postgres/main"
create_appsmith_pg_db() {
POSTGRES_DB_PATH=$1
# Start the postgres , wait for it to be ready and create a appsmith db
su postgres -c "env PATH='$PATH' pg_ctl -D $POSTGRES_DB_PATH -l $POSTGRES_DB_PATH/logfile start"
echo "Waiting for Postgres to start"
chore: Add host explicitly for pg_isready signal at the startup (#37067) ## Description PR to add the host explicitly for the `pg_isready` check at the startup. In the current implementation we haven't provided a host which can create issues if custom unix port is configured in the `postgres.conf`. Possible issue that we have observed here: 1. Misconfigured unix socket directory: This happens if the PostgreSQL server is not using the default Unix socket directory (usually /tmp or /var/run/postgresql), and we don't specify the correct socket directory with the --host parameter, pg_isready will fail to connect. 2. `postgres` DB is manually deleted by the user by stopping the postgres procedure, but the possibility is lower. /test Sanity ### :mag: Cypress test results <!-- This is an auto-generated comment: Cypress test results --> > [!TIP] > 🟢 🟢 🟢 All cypress tests have passed! 🎉 🎉 🎉 > Workflow run: <https://github.com/appsmithorg/appsmith/actions/runs/11499341453> > Commit: ab0e1f4efead4f4571c070df8f37d2bae5a935ed > <a href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=11499341453&attempt=1" target="_blank">Cypress dashboard</a>. > Tags: `@tag.Sanity` > Spec: > <hr>Thu, 24 Oct 2024 13:04:29 UTC <!-- end of auto-generated comment: Cypress test results --> ## Communication Should the DevRel and Marketing teams inform users about this change? - [ ] Yes - [x] No <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Summary by CodeRabbit - **Bug Fixes** - Enhanced PostgreSQL readiness check for more reliable connections by extending the timeout duration to 300 seconds. - **Chores** - Minor formatting and comment updates for improved clarity in the script. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-10-25 06:08:15 +00:00
local max_attempts=300
chore: Add polling for embedded pg start to create appsmith DB (#36854) ## Description PR to add the polling for `pg_isready` signal for embedded postgres DB. This will fix the infinite loop where postgres refuse to come up. We have seen this failure earlier with `openshift` setup. Current PR will make sure when the embedded postgres fails to startup we are not copying the postgres.conf file as well so supervisor won't have to start the postgres process at all. Ref: https://theappsmith.slack.com/archives/C0341RERY4R/p1728565913269689?thread_ts=1728544650.663739&cid=C0341RERY4R /test Sanity ### :mag: Cypress test results <!-- This is an auto-generated comment: Cypress test results --> > [!TIP] > 🟢 🟢 🟢 All cypress tests have passed! 🎉 🎉 🎉 > Workflow run: <https://github.com/appsmithorg/appsmith/actions/runs/11322083108> > Commit: df827906858747a0d9d134ae8ed04cc3b901811c > <a href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=11322083108&attempt=2" target="_blank">Cypress dashboard</a>. > Tags: `@tag.Sanity` > Spec: > <hr>Mon, 14 Oct 2024 08:17:55 UTC <!-- end of auto-generated comment: Cypress test results --> ## Communication Should the DevRel and Marketing teams inform users about this change? - [ ] Yes - [x] No <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Enhanced database initialization process with a maximum attempts mechanism for waiting on PostgreSQL to start. - Improved configuration handling for `APPSMITH_DB_URL` based on database type. - **Bug Fixes** - Ensured robust error handling if PostgreSQL fails to start within the specified attempts. - **Documentation** - Added comments for improved clarity in the deployment script. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-10-14 08:39:43 +00:00
local attempt=0
local unix_socket_directory=$(get_unix_socket_directory "$POSTGRES_DB_PATH")
echo "Unix socket directory is $unix_socket_directory"
until su postgres -c "env PATH='$PATH' pg_isready -h $unix_socket_directory"; do
chore: Add polling for embedded pg start to create appsmith DB (#36854) ## Description PR to add the polling for `pg_isready` signal for embedded postgres DB. This will fix the infinite loop where postgres refuse to come up. We have seen this failure earlier with `openshift` setup. Current PR will make sure when the embedded postgres fails to startup we are not copying the postgres.conf file as well so supervisor won't have to start the postgres process at all. Ref: https://theappsmith.slack.com/archives/C0341RERY4R/p1728565913269689?thread_ts=1728544650.663739&cid=C0341RERY4R /test Sanity ### :mag: Cypress test results <!-- This is an auto-generated comment: Cypress test results --> > [!TIP] > 🟢 🟢 🟢 All cypress tests have passed! 🎉 🎉 🎉 > Workflow run: <https://github.com/appsmithorg/appsmith/actions/runs/11322083108> > Commit: df827906858747a0d9d134ae8ed04cc3b901811c > <a href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=11322083108&attempt=2" target="_blank">Cypress dashboard</a>. > Tags: `@tag.Sanity` > Spec: > <hr>Mon, 14 Oct 2024 08:17:55 UTC <!-- end of auto-generated comment: Cypress test results --> ## Communication Should the DevRel and Marketing teams inform users about this change? - [ ] Yes - [x] No <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Enhanced database initialization process with a maximum attempts mechanism for waiting on PostgreSQL to start. - Improved configuration handling for `APPSMITH_DB_URL` based on database type. - **Bug Fixes** - Ensured robust error handling if PostgreSQL fails to start within the specified attempts. - **Documentation** - Added comments for improved clarity in the deployment script. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-10-14 08:39:43 +00:00
if (( attempt >= max_attempts )); then
chore: Add host explicitly for pg_isready signal at the startup (#37067) ## Description PR to add the host explicitly for the `pg_isready` check at the startup. In the current implementation we haven't provided a host which can create issues if custom unix port is configured in the `postgres.conf`. Possible issue that we have observed here: 1. Misconfigured unix socket directory: This happens if the PostgreSQL server is not using the default Unix socket directory (usually /tmp or /var/run/postgresql), and we don't specify the correct socket directory with the --host parameter, pg_isready will fail to connect. 2. `postgres` DB is manually deleted by the user by stopping the postgres procedure, but the possibility is lower. /test Sanity ### :mag: Cypress test results <!-- This is an auto-generated comment: Cypress test results --> > [!TIP] > 🟢 🟢 🟢 All cypress tests have passed! 🎉 🎉 🎉 > Workflow run: <https://github.com/appsmithorg/appsmith/actions/runs/11499341453> > Commit: ab0e1f4efead4f4571c070df8f37d2bae5a935ed > <a href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=11499341453&attempt=1" target="_blank">Cypress dashboard</a>. > Tags: `@tag.Sanity` > Spec: > <hr>Thu, 24 Oct 2024 13:04:29 UTC <!-- end of auto-generated comment: Cypress test results --> ## Communication Should the DevRel and Marketing teams inform users about this change? - [ ] Yes - [x] No <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Summary by CodeRabbit - **Bug Fixes** - Enhanced PostgreSQL readiness check for more reliable connections by extending the timeout duration to 300 seconds. - **Chores** - Minor formatting and comment updates for improved clarity in the script. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-10-25 06:08:15 +00:00
echo "Postgres failed to start within 300 seconds."
chore: Add polling for embedded pg start to create appsmith DB (#36854) ## Description PR to add the polling for `pg_isready` signal for embedded postgres DB. This will fix the infinite loop where postgres refuse to come up. We have seen this failure earlier with `openshift` setup. Current PR will make sure when the embedded postgres fails to startup we are not copying the postgres.conf file as well so supervisor won't have to start the postgres process at all. Ref: https://theappsmith.slack.com/archives/C0341RERY4R/p1728565913269689?thread_ts=1728544650.663739&cid=C0341RERY4R /test Sanity ### :mag: Cypress test results <!-- This is an auto-generated comment: Cypress test results --> > [!TIP] > 🟢 🟢 🟢 All cypress tests have passed! 🎉 🎉 🎉 > Workflow run: <https://github.com/appsmithorg/appsmith/actions/runs/11322083108> > Commit: df827906858747a0d9d134ae8ed04cc3b901811c > <a href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=11322083108&attempt=2" target="_blank">Cypress dashboard</a>. > Tags: `@tag.Sanity` > Spec: > <hr>Mon, 14 Oct 2024 08:17:55 UTC <!-- end of auto-generated comment: Cypress test results --> ## Communication Should the DevRel and Marketing teams inform users about this change? - [ ] Yes - [x] No <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Enhanced database initialization process with a maximum attempts mechanism for waiting on PostgreSQL to start. - Improved configuration handling for `APPSMITH_DB_URL` based on database type. - **Bug Fixes** - Ensured robust error handling if PostgreSQL fails to start within the specified attempts. - **Documentation** - Added comments for improved clarity in the deployment script. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-10-14 08:39:43 +00:00
return 1
fi
tlog "Waiting for Postgres to be ready... Attempt $((++attempt))/$max_attempts"
chore: Create appsmith postgres DB to be used by internal processes (#36670) ## Description With the migration to postgres as Appsmith's persistent database we wanted to have central DB to be consumed by internal services. Currently all the services try to create the same DB and may end up in a race condition, so we are moving this task to `entrypoint.sh`. Steps: 1. Initialise data directory for pg 2. Run the upgrade migration if necessary 3. Start the pg server 4. Create `appsmith` db with postgres user 5. Stop the pg server /test Sanity ### :mag: Cypress test results <!-- This is an auto-generated comment: Cypress test results --> > [!TIP] > 🟢 🟢 🟢 All cypress tests have passed! 🎉 🎉 🎉 > Workflow run: <https://github.com/appsmithorg/appsmith/actions/runs/11212147481> > Commit: e3840764acb3088233bf6552d74721abb00518a6 > <a href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=11212147481&attempt=1" target="_blank">Cypress dashboard</a>. > Tags: `@tag.Sanity` > Spec: > <hr>Mon, 07 Oct 2024 09:31:39 UTC <!-- end of auto-generated comment: Cypress test results --> ## Communication Should the DevRel and Marketing teams inform users about this change? - [ ] Yes - [x] No <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Improved database initialization logic for the Appsmith application. - Automatically creates the Appsmith database if it does not already exist during startup. - **Bug Fixes** - Enhanced robustness of the database setup process with existence checks before creation. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-10-08 02:23:58 +00:00
sleep 1
done
# Check if the appsmith DB is present
DB_EXISTS=$(su postgres -c "env PATH='$PATH' psql -tAc \"SELECT 1 FROM pg_database WHERE datname='${APPSMITH_PG_DATABASE}'\"")
if [[ "$DB_EXISTS" != "1" ]]; then
su postgres -c "env PATH='$PATH' psql -c \"CREATE DATABASE ${APPSMITH_PG_DATABASE}\""
else
echo "Database ${APPSMITH_PG_DATABASE} already exists."
fi
su postgres -c "env PATH='$PATH' pg_ctl -D $POSTGRES_DB_PATH stop"
}
setup_caddy() {
if [[ "$APPSMITH_RATE_LIMIT" == "disabled" ]]; then
export _APPSMITH_CADDY="/opt/caddy/caddy_vanilla"
else
export _APPSMITH_CADDY="/opt/caddy/caddy"
fi
}
init_loading_pages(){
export XDG_DATA_HOME=/appsmith-stacks/data # so that caddy saves tls certs and other data under stacks/data/caddy
export XDG_CONFIG_HOME=/appsmith-stacks/configuration
mkdir -p "$XDG_DATA_HOME" "$XDG_CONFIG_HOME"
cp templates/loading.html "$WWW_PATH"
node caddy-reconfigure.mjs
"$_APPSMITH_CADDY" start --config "$TMP/Caddyfile"
}
function setup_auto_heal(){
if [[ ${APPSMITH_AUTO_HEAL-} = 1 ]]; then
# By default APPSMITH_AUTO_HEAL=0
# To enable auto heal set APPSMITH_AUTO_HEAL=1
bash /opt/appsmith/auto_heal.sh $APPSMITH_AUTO_HEAL_CURL_TIMEOUT >> "$APPSMITH_LOG_DIR"/cron/auto_heal.log 2>&1 &
fi
}
chore: JFR 24 hour script with logs file (#36041) ## Description > [!TIP] > _Add a TL;DR when the description is longer than 500 words or extremely technical (helps the content, marketing, and DevRel team)._ > > _Please also include relevant motivation and context. List any dependencies that are required for this change. Add links to Notion, Figma or any other documents that might be relevant to the PR._ Fixes #`Issue Number` _or_ Fixes `Issue URL` > [!WARNING] > _If no issue exists, please create an issue first, and check with the maintainers if the issue is valid._ ## Automation /ok-to-test tags="@tag.Sanity" ### :mag: Cypress test results <!-- This is an auto-generated comment: Cypress test results --> > [!TIP] > 🟢 🟢 🟢 All cypress tests have passed! 🎉 🎉 🎉 > Workflow run: <https://github.com/appsmithorg/appsmith/actions/runs/10807975642> > Commit: 660a59d4d52c8bf00e6126e2eb147465626a1989 > <a href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=10807975642&attempt=1" target="_blank">Cypress dashboard</a>. > Tags: `@tag.Sanity` > Spec: > <hr>Wed, 11 Sep 2024 08:50:21 UTC <!-- end of auto-generated comment: Cypress test results --> ## Communication Should the DevRel and Marketing teams inform users about this change? - [ ] Yes - [ ] No <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Summary by CodeRabbit - **New Features** - Introduced a new automated script for recording Java Flight Recorder (JFR) data over a 24-hour period, enhancing monitoring capabilities for Java applications. - The script captures thread profiles at hourly intervals, streamlining performance data collection without manual intervention. - Added a monitoring setup function that conditionally executes the JFR recording script based on environment configuration. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Goutham Pratapa <goutham@appsmith.com>
2024-09-13 07:16:54 +00:00
function setup_monitoring(){
if [[ ${APPSMITH_MONITORING-} = 1 ]]; then
# By default APPSMITH_MONITORING=0
# To enable auto heal set APPSMITH_MONITORING=1
bash /opt/appsmith/JFR-recording-24-hours.sh $APPSMITH_LOG_DIR 2>&1 &
fi
}
print_appsmith_info(){
tr '\n' ' ' < /opt/appsmith/info.json
}
function capture_infra_details(){
bash /opt/appsmith/generate-infra-details.sh || true
}
# Main Section
print_appsmith_info
setup_caddy
init_loading_pages
unset_unused_variables
configure_database_connection_url
check_db_uri
# Don't run MongoDB if running in a Heroku dyno.
if [[ -z "${DYNO}" ]]; then
if [[ $isMongoUrl -eq 1 ]]; then
# Setup MongoDB and initialize replica set
tlog "Initializing MongoDB"
init_mongodb
init_replica_set
fi
else
# These functions are used to limit heap size for Backend process when deployed on Heroku
get_maximum_heap
setup_backend_heap_arg
# set the hostname for heroku dyno
export HOSTNAME="heroku_dyno"
fi
check_setup_custom_ca_certificates
setup-custom-ca-certificates
2022-04-01 06:58:03 +00:00
check_redis_compatible_page_size
feat: Embeded PostgreSQL with mockdb data for self-hosted (#21084) ## Description This PR includes changes for self-hosted instances to utilize an embedded postgres database for the Template's mockdb. After this change, by default Appsmith will always run the embedded Postgres, and connect to it instead of the mockdb cloud instance. **Solves:** Issue[#20107](https://github.com/appsmithorg/appsmith/issues/20107) **Changes:** - [x] Install Postgresql in the Docker container using Dockerfile :- _Updated Docker file to install PostgreSQL v13_ - [x] Add provision to stop/disable Postgresql from booting based on an environment variable :- _To disable embedded postgres, set the **ENV** var `APPSMITH_ENABLE_EMBEDDED_DB` to `0`; Configured supervisord to control the lifecycle of the postgres server ( Supervisor uses user postgres to start the server since root is not allowed)_ - [x] Seed data into the Postgresql DB during first container startup. This ensures that we don't have to ship the Docker container with data & bloat it unnecessarily. :- _Curretly, the SQL dump is baked into the image._ - [x] Persist the Postgresql data in /appsmith-stacks. This ensures that the data is persisted across version upgrades. :- _The postgres uses the file system at` /stacks/data/postgres/main` ; note: The file system owner is user/group postgres_ - [x] Update the mock db endpoint in the product to use the local DB by default instead of the hosted DB for self-hosted instances. The cloud instance should still use the hosted DB :- _The embedded postgres authentication is set to the type `trust`, allowing the existing template mockdb meta data to be used without any update to the existing templates mockdb endpoint and credentials_ **Note: There is an additional table `mydb` added to the mockdb for quickly verifying the db source is the embedded postgres, since the cloud mockdb does not have that table.** --------- Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
2023-03-07 09:52:41 +00:00
safe_init_postgres
configure_supervisord
# Ensure the restore path exists in the container, so an archive can be copied to it, if need be.
mkdir -p /appsmith-stacks/data/{backup,restore} /appsmith-stacks/ssl
# Create sub-directory to store services log in the container mounting folder
export APPSMITH_LOG_DIR="${APPSMITH_LOG_DIR:-/appsmith-stacks/logs}"
mkdir -p "$APPSMITH_LOG_DIR"/{supervisor,backend,cron,editor,rts,mongodb,redis,postgres,appsmithctl}
setup_auto_heal
capture_infra_details
chore: JFR 24 hour script with logs file (#36041) ## Description > [!TIP] > _Add a TL;DR when the description is longer than 500 words or extremely technical (helps the content, marketing, and DevRel team)._ > > _Please also include relevant motivation and context. List any dependencies that are required for this change. Add links to Notion, Figma or any other documents that might be relevant to the PR._ Fixes #`Issue Number` _or_ Fixes `Issue URL` > [!WARNING] > _If no issue exists, please create an issue first, and check with the maintainers if the issue is valid._ ## Automation /ok-to-test tags="@tag.Sanity" ### :mag: Cypress test results <!-- This is an auto-generated comment: Cypress test results --> > [!TIP] > 🟢 🟢 🟢 All cypress tests have passed! 🎉 🎉 🎉 > Workflow run: <https://github.com/appsmithorg/appsmith/actions/runs/10807975642> > Commit: 660a59d4d52c8bf00e6126e2eb147465626a1989 > <a href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=10807975642&attempt=1" target="_blank">Cypress dashboard</a>. > Tags: `@tag.Sanity` > Spec: > <hr>Wed, 11 Sep 2024 08:50:21 UTC <!-- end of auto-generated comment: Cypress test results --> ## Communication Should the DevRel and Marketing teams inform users about this change? - [ ] Yes - [ ] No <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Summary by CodeRabbit - **New Features** - Introduced a new automated script for recording Java Flight Recorder (JFR) data over a 24-hour period, enhancing monitoring capabilities for Java applications. - The script captures thread profiles at hourly intervals, streamlining performance data collection without manual intervention. - Added a monitoring setup function that conditionally executes the JFR recording script based on environment configuration. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Goutham Pratapa <goutham@appsmith.com>
2024-09-13 07:16:54 +00:00
setup_monitoring || echo true
# Handle CMD command
exec "$@"