## Description
As in the past we have seen the corruption of postgres DB which is being
used for temporal we want to make sure we have a retry mechanism in
place if:
1. `APPSMITH_DB_URL` is pointing to postgres url
2. Postgres is waiting in recovery mode
As per local testing when the docker container is abruptly shutdown via
`docker rm -f {container_name}` or `docker kill {container_name}` or
even via docker desktop we end up in state where postgres goes into
recovery state.
logs:
```
2024-09-27 08:02:49 backend stdout | SQL State : 57P03
2024-09-27 08:02:49 backend stdout | Error Code : 0
2024-09-27 08:02:49 backend stdout | Message : FATAL: the database system is starting up
```
Currently we have implemented polling mechanism, but we will keep
looking for better alternative here if we can opt for.
Note:
1. With release dump this is taking ~300sec to get out of that state and
start accepting the connections.
2. With the existing implementation without retries server dies down
within 60sec.
```
INFO exited: backend (exit status 1; not expected)
INFO gave up: backend entered FATAL state, too many start retries too quickly
```
Reference doc:
https://www.notion.so/appsmith/Postgres-critical-scenarios-668f49c96aef40308e24c2a8d6b1137c
/test Sanity
### 🔍 Cypress test results
<!-- This is an auto-generated comment: Cypress test results -->
> [!TIP]
> 🟢 🟢 🟢 All cypress tests have passed! 🎉 🎉 🎉
> Workflow run:
<https://github.com/appsmithorg/appsmith/actions/runs/11100944184>
> Commit: 9dbbe4b22ba12aa82c385ad0eef3cc3d4876f217
> <a
href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=11100944184&attempt=1"
target="_blank">Cypress dashboard</a>.
> Tags: `@tag.Sanity`
> Spec:
> <hr>Mon, 30 Sep 2024 07:26:41 UTC
<!-- end of auto-generated comment: Cypress test results -->
## Communication
Should the DevRel and Marketing teams inform users about this change?
- [ ] Yes
- [x] No
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Summary by CodeRabbit
- **New Features**
- Introduced new functions for enhanced handling of PostgreSQL database
connections, including availability checks and parameter extraction.
- Added a new utility script for managing PostgreSQL connections.
- **Bug Fixes**
- Implemented a retry mechanism for PostgreSQL server availability
checks to ensure more reliable connections.
- **Tests**
- Added unit tests to validate the functionality of the PostgreSQL
parameter extraction logic.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
92 lines
3.0 KiB
Bash
Executable File
92 lines
3.0 KiB
Bash
Executable File
#!/bin/bash
|
|
|
|
waitForPostgresAvailability() {
|
|
if [ -z "$PG_DB_HOST" ]; then
|
|
tlog "PostgreSQL host name is empty. Check env variables. Error. Exiting java setup"
|
|
exit 2
|
|
else
|
|
|
|
MAX_RETRIES=50
|
|
RETRYSECONDS=10
|
|
retry_count=0
|
|
while true; do
|
|
su postgres -c "pg_isready -h '${PG_DB_HOST}' -p '${PG_DB_PORT}'"
|
|
status=$?
|
|
|
|
case $status in
|
|
0)
|
|
tlog "PostgreSQL host '$PG_DB_HOST' is ready."
|
|
break
|
|
;;
|
|
1)
|
|
tlog "PostgreSQL host '$PG_DB_HOST' is rejecting connections e.g. due to being in recovery mode or not accepting connections eg. connections maxed out."
|
|
;;
|
|
2)
|
|
tlog "PostgreSQL host '$PG_DB_HOST' is not responding or running."
|
|
;;
|
|
3)
|
|
tlog "The connection check failed e.g. due to network issues or incorrect parameters."
|
|
;;
|
|
*)
|
|
tlog "pg_isready exited with unexpected status code: $status"
|
|
break
|
|
;;
|
|
esac
|
|
|
|
retry_count=$((retry_count + 1))
|
|
if [ $retry_count -le $MAX_RETRIES ]; then
|
|
tlog "PostgreSQL connection failed. Retrying attempt $retry_count/$MAX_RETRIES in $RETRYSECONDS seconds..."
|
|
sleep $RETRYSECONDS
|
|
else
|
|
tlog "Exceeded maximum retry attempts ($MAX_RETRIES). Exiting."
|
|
# use exit code 2 to indicate that the script failed to connect to postgres and supervisor conf is set not to restart the program for 2.
|
|
exit 2
|
|
fi
|
|
|
|
done
|
|
fi
|
|
}
|
|
|
|
# for PostgreSQL, we use APPSMITH_DB_URL=postgresql://username:password@postgresserver:5432/dbname
|
|
# Args:
|
|
# conn_string (string): PostgreSQL connection string
|
|
# Returns:
|
|
# None
|
|
# Example:
|
|
# postgres syntax
|
|
# "postgresql://user:password@localhost:5432/appsmith"
|
|
# "postgresql://user:password@localhost/appsmith"
|
|
# "postgresql://user@localhost:5432/appsmith"
|
|
# "postgresql://user@localhost/appsmith"
|
|
extract_postgres_db_params() {
|
|
local conn_string=$1
|
|
|
|
# Use node to parse the URI and extract components
|
|
IFS=' ' read -r USER PASSWORD HOST PORT DB <<<"$(node -e "
|
|
const connectionString = process.argv[1];
|
|
const pgUri = connectionString.startsWith(\"postgresql://\")
|
|
? connectionString
|
|
: 'http://' + connectionString; //Prepend a fake scheme for URL parsing
|
|
const url = require('url');
|
|
const parsedUrl = new url.URL(pgUri);
|
|
|
|
// Extract the pathname and remove the leading '/'
|
|
const db = parsedUrl.pathname.substring(1);
|
|
|
|
// Default the port to 5432 if it's empty
|
|
const port = parsedUrl.port || '5432';
|
|
|
|
console.log(\`\${parsedUrl.username || '-'} \${parsedUrl.password || '-'} \${parsedUrl.hostname} \${port} \${db}\`);
|
|
" "$conn_string")"
|
|
|
|
# Now, set the environment variables
|
|
export PG_DB_USER="$USER"
|
|
export PG_DB_PASSWORD="$PASSWORD"
|
|
export PG_DB_HOST="$HOST"
|
|
export PG_DB_PORT="$PORT"
|
|
export PG_DB_NAME="$DB"
|
|
}
|
|
|
|
# Example usage of the functions
|
|
# waitForPostgresAvailability
|
|
# extract_postgres_db_params "postgresql://user:password@localhost:5432/dbname" |