## Description
issue of Postgres not coming up with error` Previous Postgres was not
shutdown cleanly. Please start and stop Postgres 14 properly with
'supervisorctl' only. `
https://www.notion.so/appsmith/Closed-Beta-Customer-issues-45a274a9eb8e4762a72cbff74cd3bad5?pvs=4#ecca04d205414f25a289884cebdc0f9b
Fixes
https://app.zenhub.com/workspaces/workflows-pod-652fff131a95920b9bf2bc7e/issues/zh/226_
## Automation
/ok-to-test tags="@tag.Sanity"
### 🔍 Cypress test results
<!-- This is an auto-generated comment: Cypress test results -->
> [!TIP]
> 🟢🟢🟢 All cypress tests have passed! 🎉🎉🎉
> Workflow run:
<https://github.com/appsmithorg/appsmith/actions/runs/10107093349>
> Commit: 357bf5d2ff0c1f65b4cbf46b0f3ee823ac192614
> <a
href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=10107093349&attempt=1"
target="_blank">Cypress dashboard</a>.
> Tags: `@tag.Sanity`
> Spec:
> <hr>Fri, 26 Jul 2024 07:15:19 UTC
<!-- end of auto-generated comment: Cypress test results -->
## Communication
Should the DevRel and Marketing teams inform users about this change?
- [ ] Yes
- [ ] No
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Summary by CodeRabbit
- **New Features**
- Enhanced PostgreSQL upgrade process with improved error handling and
robust management of old server instances.
- **Bug Fixes**
- Reinstituted logic for checking and managing the `postmaster.pid`
file, ensuring proper startup and shutdown of old PostgreSQL servers.
- **Refactor**
- Improved formatting and readability of shell scripts without altering
their functionality.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
1. This was kept for a short grace period, to be removed once we're
confident of upgrading to v14.
2. The `pg-upgrade.sh` script is capable of handling this. It will
install v13 when upgrading to v14, if it's not already available. See:
2adb12d57b/deploy/docker/fs/opt/appsmith/pg-upgrade.sh (L53-L54)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Chores**
- Updated PostgreSQL version in Docker setup from 13 to 14 for improved
performance and security.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Reason for this previously documented at
https://github.com/appsmithorg/appsmith/pull/34265#issue-2356259090.
Cypress tests don' make sense since the only diff is on
`base.dockerfile`.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Chores**
- Updated Dockerfile to streamline PostgreSQL setup. Removed creation of
a symlink to the current version of PostgreSQL.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
We're upgrading embedded Postgres from 13 to 14, and this PR includes a
script to perform the upgrade of the data folder from v13 schema to v14
schema. This script temporarily installs Postgres 13, if not available,
for the upgrade process, so will continue to work when and if we choose
to remove `postgresql-13` from the base image.
Tested this manually as well, running an Appsmith with Postgres 13,
executing some workflows via webhook, getting some run data generated,
then upgrading Postgres with the script in this PR, and ensuring that
the workflow run history is still there and visible on the UI exactly
the same. It is.
No conflicts or additional changes needed on EE. [All server and Cypress
tests pass on EE](https://github.com/appsmithorg/appsmith-ee/pull/4493).
**/test sanity**
<!-- This is an auto-generated comment: Cypress test results -->
> [!TIP]
> 🟢🟢🟢 All cypress tests have passed! 🎉🎉🎉
> Workflow run:
<https://github.com/appsmithorg/appsmith/actions/runs/9590240540>
> Commit: 9c75da53f871ffb912015c18a7504327cba88f2c
> <a
href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=9590240540&attempt=1"
target="_blank">Cypress dashboard</a>.
> Tags: `@tag.Sanity`
<!-- end of auto-generated comment: Cypress test results -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added automation script for upgrading PostgreSQL to the latest
version.
- Introduced testing script for PostgreSQL upgrades in Docker
environments.
- **Improvements**
- Upgraded PostgreSQL from version 13 to 14 in Docker setup, ensuring
compatibility and performance enhancements.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Should fix the flood of buffer overflow errors we see from Supervisor.
The one "change" this PR has, is that all the logs from the Docker
container will now be coming from `stderr`, not `stdout`. I don't expect
it to cause any problems, but it _is_ a change in behaviour nonetheless.
Why do that? Because of the way Supervisor event listening works. This
fix is based on the conversation at
https://github.com/Supervisor/supervisor/issues/1417. Essentially that
`stdout` should be reserved as a communication channel between
Supervisord and the event-listener, and for logging, we should use
`stderr`.
**/test sanity**
<!-- This is an auto-generated comment: Cypress test results -->
> [!TIP]
> 🟢🟢🟢 All cypress tests have passed! 🎉🎉🎉
> Workflow run:
<https://github.com/appsmithorg/appsmith/actions/runs/9592028116>
> Commit: 34a29eabc8dc82fb6a43b143e400d725d197a655
> <a
href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=9592028116&attempt=1"
target="_blank">Cypress dashboard</a>.
> Tags: `@tag.Sanity`
<!-- end of auto-generated comment: Cypress test results -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Refactor**
- Improved logging by switching to `print` statements with explicit
flushing for more reliable output handling.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This temporary change is no longer needed.
**/test sanity**
<!-- This is an auto-generated comment: Cypress test results -->
> [!TIP]
> 🟢🟢🟢 All cypress tests have passed! 🎉🎉🎉
> Workflow run:
<https://github.com/appsmithorg/appsmith/actions/runs/9574310795>
> Commit: abad77fb024c61b595c529aa2ed024b8dcdf5dfb
> <a
href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=9574310795&attempt=1"
target="_blank">Cypress dashboard</a>.
> Tags: `@tag.Sanity`
<!-- end of auto-generated comment: Cypress test results -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Chores**
- Improved the startup script by removing obsolete PATH modification for
PostgreSQL.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Description
> [!TIP]
> _Add a TL;DR when the description is longer than 500 words or
extremely technical (helps the content, marketing, and DevRel team)._
>
> _Please also include relevant motivation and context. List any
dependencies that are required for this change. Add links to Notion,
Figma or any other documents that might be relevant to the PR._
Fixes #`Issue Number`
_or_
Fixes `Issue URL`
> [!WARNING]
> _If no issue exists, please create an issue first, and check with the
maintainers if the issue is valid._
## Automation
/ok-to-test tags="@tag.Sanity"
### 🔍 Cypress test results
<!-- This is an auto-generated comment: Cypress test results -->
> [!TIP]
> 🟢🟢🟢 All cypress tests have passed! 🎉🎉🎉
> Workflow run:
<https://github.com/appsmithorg/appsmith/actions/runs/9547569754>
> Commit: 3516b9f76c77b499fe363d3aab583595911fa087
> <a
href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=9547569754&attempt=1"
target="_blank">Cypress dashboard</a>.
> Tags: `@tag.Sanity`
<!-- end of auto-generated comment: Cypress test results -->
## Communication
Should the DevRel and Marketing teams inform users about this change?
- [ ] Yes
- [ ] No
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Chores**
- Updated Jest dependency from version 29.1.2 to 29.7.0 for improved
testing capabilities.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The `PATH` env variable gets reset to it's original value by `su`,
because its a "secure" env variable, and is _never_ preserved across
user-switches.
Because of this, the Postgres commands like `initdb` are not found on
the `PATH`. This PR fixes this by setting the `PATH` in the command
passed to `su` itself, so that `initdb` can now be found.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved the initialization process for local Postgres data folder to
ensure compatibility with environment variables.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Instead of using absolute paths to Postgres binaries, and in doing so,
hard-coding the Postgres version number, we add the bin folder to `PATH`
so they're available directly.
This also won't need the `current` symlink we created. That symlink is
causing problems in some post-installations scripts in `apt`/`dpkg`,
since they're expecting a _number_ in that folder, and sees `current`,
it's confused and produces this error:
```
dpkg: warning: version 'current' has bad syntax: version number does not start with digit
```
After this is merged, we should be able to get rid of the `current`
symlink as well.
**/test sanity**
<!-- This is an auto-generated comment: Cypress test results -->
> [!TIP]
> 🟢🟢🟢 All cypress tests have passed! 🎉🎉🎉
> Workflow run:
<https://github.com/appsmithorg/appsmith/actions/runs/9541787920>
> Commit: 1012266e0924248b0f13a91bee464303e86b87a6
> <a
href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=9541787920&attempt=1"
target="_blank">Cypress dashboard</a>.
> Tags: `@tag.Sanity`
<!-- end of auto-generated comment: Cypress test results -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Chores**
- Updated Docker configurations to streamline PostgreSQL 13 integration.
- Simplified PostgreSQL commands in scripts by removing explicit
versioning paths.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The current fallback implementation doesn't work in the below case:
> The `APPSMITH_MONGODB_URI` is set _outside_ the container, and
`APPSMITH_DB_URL` is set in the `docker.env`.
This scenario will be showing up a lot more now that the `docker.env.sh`
that generates new `docker.env` files has `APPSMITH_DB_URL` in it.
Problem is that since we load env variables from both outside and
`docker.env` individually, we end up loading both `APPSMITH_MONGODB_URI`
and `APPSMITH_DB_URL`. And in this case, the `APPSMITH_DB_URL` will be
from the just-generated `docker.env`, so we'll end up with a localhost
URL, even though `APPSMITH_MONGODB_URI` was set to an external endpoint
outside the container.
This is the problem we were facing with our DPs recently.
This PR fixes this problem by doing the env name "rename" separately for
outside env variables, and once for `docker.env` env variables.
**/test sanity**
<!-- This is an auto-generated comment: Cypress test results -->
> [!TIP]
> 🟢🟢🟢 All cypress tests have passed! 🎉🎉🎉
> Workflow run:
<https://github.com/appsmithorg/appsmith/actions/runs/9481037167>
> Commit: c6ce2a8dda4a13d3aab64adf8c9af08abd1cea62
> Cypress dashboard url: <a
href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=9481037167&attempt=1"
target="_blank">Click here!</a>
<!-- end of auto-generated comment: Cypress test results -->
**/test sanity**
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Refactor**
- Updated logging mechanism from `echo` to `tlog` for improved log
management across various scripts.
- **Chores**
- Enhanced logging for better clarity and consistency in deployment
scripts.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Logs messages from `entrypoint.sh` and the other `run-*.sh` scripts
don't show timestamp today, and its getting hard to see the order of
things in the logs, especially between different processes.
This PR adds a command `tlog` to print logs with UTC timestamp prefixed.
We're only using it in `entrypoint.sh` now, but follow-up PR(s) will add
it to the other `run-*` scripts as well.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Chores**
- Updated Dockerfile to include `/opt/bin` in the `PATH` and modified
permissions settings for executable files.
- **Refactor**
- Enhanced logging in the entrypoint script by replacing `echo`
statements with `tlog` for better clarity and debugging.
- **New Features**
- Introduced `tlog`, a new shell script for consistent and timestamped
logging.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Description
PR to update the version references for postgres. This will help us in
keeping the diff minimum between release and pg branch as we will be
upgrading the postgres version to 15 on pg branch.
## Automation
/ok-to-test tags=""
### 🔍 Cypress test results
<!-- This is an auto-generated comment: Cypress test results -->
> [!CAUTION]
> If you modify the content in this section, you are likely to disrupt
the CI result for your PR.
<!-- end of auto-generated comment: Cypress test results -->
## Communication
Should the DevRel and Marketing teams inform users about this change?
- [ ] Yes
- [x] No
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Chores**
- Updated Docker scripts to use a generic placeholder for PostgreSQL
version, enhancing flexibility and future-proofing the setup.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Description
> [!TIP]
> _Add a TL;DR when the description is longer than 500 words or
extremely technical (helps the content, marketing, and DevRel team)._
>
> _Please also include relevant motivation and context. List any
dependencies that are required for this change. Add links to Notion,
Figma or any other documents that might be relevant to the PR._
Fixes #`Issue Number`
_or_
Fixes `Issue URL`
> [!WARNING]
> _If no issue exists, please create an issue first, and check with the
maintainers if the issue is valid._
## Automation
/ok-to-test tags=""
### 🔍 Cypress test results
<!-- This is an auto-generated comment: Cypress test results -->
> [!CAUTION]
> If you modify the content in this section, you are likely to disrupt
the CI result for your PR.
<!-- end of auto-generated comment: Cypress test results -->
## Communication
Should the DevRel and Marketing teams inform users about this change?
- [ ] Yes
- [ ] No
---------
Co-authored-by: Goutham Pratapa <goutham@appsmith.com>
The newer NodeJS version `20.13` requires a more recent version of
glibc, one which isn't available on Ubuntu 20.04 yet. We need to stick
to NodeJS v20.11 until we can update Ubuntu itself to 22.04, or the
updated glibc shows up in Ubuntu 20.04.
Pinning NodeJS version for now to unblock.
## Description
> [!TIP]
> _Add a TL;DR when the description is longer than 500 words or
extremely technical (helps the content, marketing, and DevRel team)._
>
> _Please also include relevant motivation and context. List any
dependencies that are required for this change. Add links to Notion,
Figma or any other documents that might be relevant to the PR._
Fixes #`Issue Number`
_or_
Fixes `Issue URL`
> [!WARNING]
> _If no issue exists, please create an issue first, and check with the
maintainers if the issue is valid._
## Automation
/ok-to-test tags="@tag.Sanity"
### 🔍 Cypress test results
<!-- This is an auto-generated comment: Cypress test results -->
> [!TIP]
> 🟢🟢🟢 All cypress tests have passed! 🎉🎉🎉
> Workflow run:
<https://github.com/appsmithorg/appsmith/actions/runs/9191016385>
> Commit: c74e98018a6b4bd212e922eb1b3015195a5234a0
> Cypress dashboard url: <a
href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=9191016385&attempt=1"
target="_blank">Click here!</a>
<!-- end of auto-generated comment: Cypress test results -->
## Communication
Should the DevRel and Marketing teams inform users about this change?
- [ ] Yes
- [ ] No
## Description
1. PR to enable server tests on PRs with `pg` as the base branch.
2. Replace the DB URI from `APPSMITH_MONGODB_URI` to `APPSMITH_DB_URL`
## Automation
/ok-to-test tags="@tag.Sanity, @tag.GenerateCRUD, @tag.Fork"
### 🔍 Cypress test results
<!-- This is an auto-generated comment: Cypress test results -->
> [!TIP]
> 🟢🟢🟢 All cypress tests have passed! 🎉🎉🎉
> Workflow run:
<https://github.com/appsmithorg/appsmith/actions/runs/9174148396>
> Commit: 762b4255f654946a1a47a196df5a1afae5be09f2
> Cypress dashboard url: <a
href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=9174148396&attempt=1"
target="_blank">Click here!</a>
<!-- end of auto-generated comment: Cypress test results -->
## Communication
Should the DevRel and Marketing teams inform users about this change?
- [ ] Yes
- [x] No
---------
Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
## Description
Looks like while replacing NGINX with Caddy, Heroku deployment usecase
was missed.
Updated Caddyfile generation to use `$PORT` if available.
Fixes#33555
Tested on Heroku Standard-2x Dyno.
## Description
> [!TIP]
> _Add a TL;DR when the description is longer than 500 words or
extremely technical (helps the content, marketing, and DevRel team)._
>
> _Please also include relevant motivation and context. List any
dependencies that are required for this change. Add links to Notion,
Figma or any other documents that might be relevant to the PR._
Fixes #`Issue Number`
_or_
Fixes `Issue URL`
> [!WARNING]
> _If no issue exists, please create an issue first, and check with the
maintainers if the issue is valid._
## Automation
/ok-to-test tags=""
### 🔍 Cypress test results
<!-- This is an auto-generated comment: Cypress test results -->
> [!CAUTION]
> If you modify the content in this section, you are likely to disrupt
the CI result for your PR.
<!-- end of auto-generated comment: Cypress test results -->
## Communication
Should the DevRel and Marketing teams inform users about this change?
- [ ] Yes
- [ ] No
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Chores**
- Improved proxy settings, IP retrieval with timeout, and curl request
timeouts in the deployment process.
- Enhanced Redis compatibility checks and installation steps.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
1. Add validation for the tag before building the image or doing
anything. This is strict case-sensitive validation, that will allow only
three numbers in the version, and the first number has to be a single
digit. Support for `-beta` versions is also being removed, as we just
don't use it. Thanks to removing this, we'll build the `latest` and
`v1.2.3` images together, instead of building two separate images.
2. The server and client are reading version from `version` key in
`info.json`, but RTS is reading it from `githubRef` key in `info.json`.
This discrepancy is debt, and has no reason to exist. We fix RTS to also
read the version from the `version` field.
Why are we doing this? [Slack
conversation](https://theappsmith.slack.com/archives/C0341RERY4R/p1714098736865219?thread_ts=1714066995.288859&cid=C0341RERY4R).
We're currently relying on ipify.org for this, and this PR will move to
using CS for this information. This is so that all external
communication from the core of the product's backend, is only to
cs.appsmith.com, which makes whitelisting easier for users.
Also removing the unused variables `APPSMITH_CLOUD_SERVICES_USERNAME`
and `APPSMITH_CLOUD_SERVICES_PASSWORD`.
⚠️ This will cause conflicts on sync.
/ok-to-test tags="@tag.Sanity"
<!-- This is an auto-generated comment: Cypress test results -->
> [!TIP]
> 🟢🟢🟢 All cypress tests have passed! 🎉🎉🎉
> Workflow run:
<https://github.com/appsmithorg/appsmith/actions/runs/8859016811>
> Commit: 46576ca46adcba288693c3d5aaa9cc547c1e8f57
> Cypress dashboard url: <a
href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=8859016811&attempt=1"
target="_blank">Click here!</a>
<!-- end of auto-generated comment: Cypress test results -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Refactor**
- Removed username and password fields from cloud services configuration
to enhance security.
- Updated network utilities to initialize with new cloud services
configuration, improving integration and functionality.
- **Bug Fixes**
- Adjusted the method of fetching and handling IP address data to
improve reliability and accuracy of network services.
- **Chores**
- Updated application properties and deployment scripts to align with
the new configuration and address retrieval methods.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Outputs on having the container up and running
On kubernetes:
```
root@ce32552-appsmith-66fc68d7f-97tjn:/opt/appsmith# cat /tmp/appsmith/infra.json
{"cloudProvider":"amazon","Tool":"kubernetes","EFS":"present","Hostname":"ce32552-appsmith-66fc68d7f-97tjn"}
```
On local setup:
```
root@26327db8d65a:/opt/appsmith# cat /tmp/appsmith/infra.json
{"cloudProvider":"local","Tool":"docker","EFS":"absent","Hostname":"26327db8d65a"}
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced infrastructure detection to enhance system insights,
including cloud provider, deployment tools, and host details.
- Enhanced analytics by incorporating deployment properties into event
tracking.
- **Refactor**
- Modified server configuration and initialization to integrate new
deployment properties.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Trisha Anand <trisha@appsmith.com>
This is to avoid low-impact failures from getting the whole container to
restart.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Modified health check to focus on critical services (`editor`, `rts`,
`backend`) for more efficient monitoring.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Replaces NGINX for local dev with Caddy.
The advandage here is that the script that generates `Caddyfile` at
runtime on production deployments, is also used to generate the
`Caddyfile` at local development. This reduces the local--production
gap.
/ok-to-test tags="@tag.Sanity"
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Refactor**
- Updated base Docker image and installation processes for tools within
the Docker environment.
- **Chores**
- Modified scripts to enhance file handling and environment variable
configurations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This removes the `appsmithctl migrate` command which can migrate an
Appsmith instance from on EC2 instance to another, using SSH. Why are we
removing it?
1. It's not documented on docs.appsmith.com at all.
2. The problem is better solved with a combination of `appsmithctl
backup` and `appsmithctl restore`, with much _more_ flexibility.
## Description
- update node version and appropriate git workflow
- added the path to webpack cache folder, this should speed up bundle
creation about a minute
[Test, build and push Docker
Image](https://github.com/appsmithorg/appsmith/actions/runs/8421752151)
[Build Client, Server & Run only
Cypress](https://github.com/appsmithorg/appsmith/actions/runs/8421752151)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Chores**
- Updated actions/cache and actions/setup-node to v4 across various
workflows for improved caching and Node.js setup.
- Modified the `yarn install` command to use `--immutable` flag,
enhancing dependency management.
- **Documentation**
- Updated comments within workflows to include cautionary and important
notes, ensuring better clarity.
- **Refactor**
- Adjusted caching paths and keys for more efficient caching behavior.
- Changed Node.js installation to version 20.11.1 in Dockerfile,
aligning with the latest version for better performance and security.
- **Tests**
- Modified assertion in `getCurrentLocationSaga` test to check for the
presence of a property, improving test accuracy.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Aman Agarwal <aman@appsmith.com>
Fixes: [31031](https://github.com/appsmithorg/appsmith/issues/31031)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added rate-limiting functionality to enhance security and prevent
abuse.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
Bumps [nodemailer](https://github.com/nodemailer/nodemailer) from 6.7.5
to 6.9.9.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/nodemailer/nodemailer/releases">nodemailer's
releases</a>.</em></p>
<blockquote>
<h2>v6.9.9</h2>
<h2><a
href="https://github.com/nodemailer/nodemailer/compare/v6.9.8...v6.9.9">6.9.9</a>
(2024-02-01)</h2>
<h3>Bug Fixes</h3>
<ul>
<li><strong>security:</strong> Fix issues described in
GHSA-9h6g-pr28-7cqp. Do not use eternal matching pattern if only a few
occurences are expected (<a
href="dd8f5e8a4d">dd8f5e8</a>)</li>
<li><strong>tests:</strong> Use native node test runner, added code
coverage support, removed grunt (<a
href="https://redirect.github.com/nodemailer/nodemailer/issues/1604">#1604</a>)
(<a
href="be45c1b299">be45c1b</a>)</li>
</ul>
<h2>v6.9.8</h2>
<h2><a
href="https://github.com/nodemailer/nodemailer/compare/v6.9.7...v6.9.8">6.9.8</a>
(2023-12-30)</h2>
<h3>Bug Fixes</h3>
<ul>
<li><strong>punycode:</strong> do not use native punycode module (<a
href="b4d0e0c7cc">b4d0e0c</a>)</li>
</ul>
<h2>v6.9.7</h2>
<h2><a
href="https://github.com/nodemailer/nodemailer/compare/v6.9.6...v6.9.7">6.9.7</a>
(2023-10-22)</h2>
<h3>Bug Fixes</h3>
<ul>
<li><strong>customAuth:</strong> Do not require user and pass to be set
for custom authentication schemes (fixes <a
href="https://redirect.github.com/nodemailer/nodemailer/issues/1584">#1584</a>)
(<a
href="41d482c3f0">41d482c</a>)</li>
</ul>
<h2>v6.9.6</h2>
<h2><a
href="https://github.com/nodemailer/nodemailer/compare/v6.9.5...v6.9.6">6.9.6</a>
(2023-10-09)</h2>
<h3>Bug Fixes</h3>
<ul>
<li><strong>inline:</strong> Use 'inline' as the default Content
Dispostion value for embedded images (<a
href="db32c93fef">db32c93</a>)</li>
<li><strong>tests:</strong> Removed Node v12 from test matrix as it is
not compatible with the test framework anymore (<a
href="7fe0a608ed">7fe0a60</a>)</li>
</ul>
<h2>v6.9.5</h2>
<h2><a
href="https://github.com/nodemailer/nodemailer/compare/v6.9.4...v6.9.5">6.9.5</a>
(2023-09-06)</h2>
<h3>Bug Fixes</h3>
<ul>
<li><strong>license:</strong> Updated license year (<a
href="da4744e491">da4744e</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/nodemailer/nodemailer/blob/master/CHANGELOG.md">nodemailer's
changelog</a>.</em></p>
<blockquote>
<h2><a
href="https://github.com/nodemailer/nodemailer/compare/v6.9.8...v6.9.9">6.9.9</a>
(2024-02-01)</h2>
<h3>Bug Fixes</h3>
<ul>
<li><strong>security:</strong> Fix issues described in
GHSA-9h6g-pr28-7cqp. Do not use eternal matching pattern if only a few
occurences are expected (<a
href="dd8f5e8a4d">dd8f5e8</a>)</li>
<li><strong>tests:</strong> Use native node test runner, added code
coverage support, removed grunt (<a
href="https://redirect.github.com/nodemailer/nodemailer/issues/1604">#1604</a>)
(<a
href="be45c1b299">be45c1b</a>)</li>
</ul>
<h2><a
href="https://github.com/nodemailer/nodemailer/compare/v6.9.7...v6.9.8">6.9.8</a>
(2023-12-30)</h2>
<h3>Bug Fixes</h3>
<ul>
<li><strong>punycode:</strong> do not use native punycode module (<a
href="b4d0e0c7cc">b4d0e0c</a>)</li>
</ul>
<h2><a
href="https://github.com/nodemailer/nodemailer/compare/v6.9.6...v6.9.7">6.9.7</a>
(2023-10-22)</h2>
<h3>Bug Fixes</h3>
<ul>
<li><strong>customAuth:</strong> Do not require user and pass to be set
for custom authentication schemes (fixes <a
href="https://redirect.github.com/nodemailer/nodemailer/issues/1584">#1584</a>)
(<a
href="41d482c3f0">41d482c</a>)</li>
</ul>
<h2><a
href="https://github.com/nodemailer/nodemailer/compare/v6.9.5...v6.9.6">6.9.6</a>
(2023-10-09)</h2>
<h3>Bug Fixes</h3>
<ul>
<li><strong>inline:</strong> Use 'inline' as the default Content
Dispostion value for embedded images (<a
href="db32c93fef">db32c93</a>)</li>
<li><strong>tests:</strong> Removed Node v12 from test matrix as it is
not compatible with the test framework anymore (<a
href="7fe0a608ed">7fe0a60</a>)</li>
</ul>
<h2><a
href="https://github.com/nodemailer/nodemailer/compare/v6.9.4...v6.9.5">6.9.5</a>
(2023-09-06)</h2>
<h3>Bug Fixes</h3>
<ul>
<li><strong>license:</strong> Updated license year (<a
href="da4744e491">da4744e</a>)</li>
</ul>
<h2>6.9.4 2023-07-19</h2>
<ul>
<li>Renamed SendinBlue to Brevo</li>
</ul>
<h2>6.9.3 2023-05-29</h2>
<ul>
<li>Specified license identifier (was defined as MIT, actual value
MIT-0)</li>
<li>If SMTP server disconnects with a message, process it and include as
part of the response error</li>
</ul>
<h2>6.9.2 2023-05-11</h2>
<ul>
<li>Fix uncaught exception on invalid attachment content payload</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="5a2e10f454"><code>5a2e10f</code></a>
chore(master): release 6.9.9 [skip-ci] (<a
href="https://redirect.github.com/nodemailer/nodemailer/issues/1606">#1606</a>)</li>
<li><a
href="dd8f5e8a4d"><code>dd8f5e8</code></a>
fix(security): Fix issues described in GHSA-9h6g-pr28-7cqp. Do not use
eterna...</li>
<li><a
href="2c2b46ae4c"><code>2c2b46a</code></a>
chore: do not use caret in version specifier</li>
<li><a
href="be45c1b299"><code>be45c1b</code></a>
fix(tests): Use native node test runner, added code coverage support,
removed...</li>
<li><a
href="4233f6f89e"><code>4233f6f</code></a>
chore(master): release 6.9.8 [skip-ci] (<a
href="https://redirect.github.com/nodemailer/nodemailer/issues/1605">#1605</a>)</li>
<li><a
href="09d502f979"><code>09d502f</code></a>
chore: removed double file</li>
<li><a
href="b4d0e0c7cc"><code>b4d0e0c</code></a>
fix(punycode): do not use native punycode module</li>
<li><a
href="8376c024f8"><code>8376c02</code></a>
Test new github notice syntax for README</li>
<li><a
href="bc46a3b7d4"><code>bc46a3b</code></a>
Updated stale github action</li>
<li><a
href="78bdaf8c9e"><code>78bdaf8</code></a>
chore: remove redundant AWS SDK for JavaScript v2 (<a
href="https://redirect.github.com/nodemailer/nodemailer/issues/1593">#1593</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/nodemailer/nodemailer/compare/v6.7.5...v6.9.9">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/appsmithorg/appsmith/network/alerts).
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Show the commit SHA in the version popup, instead of a snapshot version
number like `v1.11-SNAPSHOT`. But if the version number doesn't have a
`-SNAPSHOT` at the end, we show the version number as is. So if it's
`v1.12`, we show that instead of the commit SHA.
This is a fix for a user's problem. They have custom domain set, a
custom cert in the `stacks/ssl` folder, but because a different team
operates a reverse-proxy, they aren't sure which _host_ is actually used
by the reverse proxy. And the way we bind to port 443 requires that that
puzzle be solved, for very little extra value.
This change makes it so that we accept any incoming TLS connections, if
a custom domain is set, which should be much more convenient.
[Slack
Thread](https://theappsmith.slack.com/archives/C0341RERY4R/p1705700120412079).
Already deployed on users' system, and they've confirmed its working.
I've been doing this in pieces bit by bit, not to rock the boat too much
too fast, but it's taking too long, and too much effort. Instead opting
for a rip-the-bandaid style, hopefully without the pain.
Instead of downloading it from the github repo on every build (in
`base.dockerfile`), we vendor the supervisor extension so we can look to
fix the buffer overflow problem in supervisor logs.
After this is merged, after a day/two, we'll be removing `git`,
`python3-pip` and the supervisor-stdout extension from
`base.dockerfile`.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a Databricks plugin for executing queries and managing
database connections.
- Added a migration to incorporate the Databricks plugin into existing
workspaces.
- **Bug Fixes**
- Ensured robust error handling in the Databricks plugin with clear
messaging for query execution failures.
- **Tests**
- Implemented tests to validate the behavior of the Databricks plugin
under various connection scenarios.
- **Documentation**
- Included configuration properties for the Databricks plugin setup.
- **Refactor**
- Added specific error types and messages for the Databricks plugin to
improve debugging and user feedback.
- **Chores**
- Modified the Java runtime environment settings to support the new
plugin's requirements.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Arpit Mohan <arpit@appsmith.com>
Another attempt at #29550, which was reverted. Fallback is not happening
if cert provisioning fails _despite_ having the correct header. But with
the changes in this PR, since we'll listen on `:80`, fallback _will_
happen when cert provisioning fails due to incorrect domain
configuration.
We're also adding [Hurl](https://hurl.dev) based tests. They're not run
in any CI yet. That'll come in soon.
Defining custom domain as `https://example.com/` is invalid.
It should be just the domain, just `example.com`. But turns out a lot of
our users have the incorrect configuration, and our previous stack of
NGINX+Certbot was able to ignore this and serve without HTTPS. This PR
brings that behaviour back.
## Test performed
Have Appsmith running on an EC2 instance, and a domain `correct.com`
with an A-record pointed to this EC2 instance.
In the instance, we run Appsmith with `APPSMITH_CUSTOM_DOMAIN` set to
`wrong.com`. Caddy will obviously fail to provision the cert, and so we
expect it to accept connections on just HTTP.
So hitting `curl -i http://correct.com` produced a 200 with the HTML
response, and not a 308 with a redirect. Before the changes from this
PR, the same curl command produced a 308 with a redirect to
`https://correct.com`, which fails with a certificate error.
Next up, we run Appsmith with `APPSMITH_CUSTOM_DOMAIN` set to
`correct.com`. Caddy will succeed in provisioning a cert, and so we
expect HTTP URLs to be redirected to HTTPS.
So hitting `curl -i http://correct.com` produces a 308 redirect to
`http://correct.com` which then works fine, since Caddy now has the cert
for the domain.
We're setting the default value for `APPSMITH_ALLOWED_FRAME_ANCESTORS`
before we initialize env variables from `docker.env`. This make the
default value take a higher precedence over the value configured in
`docker.env`. And since the value in `docker.env` is the one configured
from Admin Settings, it feels like the value configured from the UI is
being ignored.
This fixes the problem by moving the check for this env variable to
_inside_ the reconfigure script, and so doesn't affect any env
variables.
Fixes#29114
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Performance Improvements**
- Enhanced logging capabilities to include memory footprint and context
details for better performance monitoring.
- **Configuration Updates**
- Increased the number of log file backups from 2 to 10, allowing for
more historical log retention.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Chores**
- Optimized auto-healing script by removing an unnecessary 60-second
delay.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced auto-healing functionality to automatically restart
unresponsive backend services.
- Added SSL configuration support for custom domains.
- **Chores**
- Implemented periodic backend service status checks.
- Enhanced startup scripts to support new auto-healing feature based on
environment configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
I think the route precedence in Caddy is different when using `handle`
directive, vs when directly using the `error` directive.
This is causing the file `handle {` route, which is a catch-all route is
handling `/static/*` requests that don't have a corresponding file. This
handler however, doesn't respond with 404 status, it responds with 200
status for missing files, and render the `index.html` for our SPA
behaviour.
Now, the CDN we have on release.app.appsmith.com caches responses from
upstream when the status is 200. If it is 404, it won't cache and retry
next time. This is why it's essential that we respond with 404 for files
that don't exist, irrespective of the content of the response.
When the container is starting up, Caddy doesn't have all the
information yet, and may have responded with not-found for one of the
assets. But since this went out with 200 status, our CDN cached it, and
once the file _was_ available with Caddy, the CDN wouldn't retry ever.
This fix will ensure we get 404 status code for requests to `/static/*`
that point to files that don't exist.
Fixes#29114
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced logging capabilities for better performance insights.
- **Improvements**
- Increased the number of log file backups to ensure more historical
data is preserved.
- **Documentation**
- Updated internal documentation to reflect new logging and performance
monitoring features.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This PR replaces NGINX and Certbot with Caddy.
1. Auto-HTTPS when custom domain is set, is handled by Caddy.
2. If past certs exist, that were provisioned by Certbot in older
Appsmith versions, we configure Caddy to make use of them. But this only
applies if the certs aren't already expired. If they're expired, point 1
applies.
3. If custom certs are provided in `ssl` folder, Caddy will be
configured to use them.
4. Incoming `Forwarded` header is not passed to any reverse proxies. So
redirect URL is correctly computed on Google Cloud Run.
5. All other route configurations are exactly as they are in NGINX
today.
Caddy configuration file is generated in the `caddy-reconfigure.mjs`
script, which will also reload Caddy with the new configuration.
## Description
This PR fixes the experience of Templates forking in self hosted
instances. And also for to Set up a process to keep the embedded DB up
to date with template db schemas.
We have removed the redirection of mockdb end point used in templates
App when forked in self hosted instance from localhost/internal postgres
db.
This also has a migration which is to make sure that none of existing
apps using the internal postgres does not break due to the removal of
redirection. The migration will make sure that existing self hosted
instances using the posgress db and has a datasource with mockdb end
point will be replaces with localhost.
#### PR fixes following issue(s)
Fixes https://github.com/appsmithorg/appsmith/issues/28924
#### Type of change
- Bug fix (non-breaking change which fixes an issue)
## Testing
#### How Has This Been Tested?
- [ ] Manual
#### Test Plan
> Add Testsmith test cases links that relate to this PR
>
>
#### Issues raised during DP testing
> Link issues raised during DP testing for better visiblity and tracking
(copy link from comments dropped on this PR)
>
>
>
## Checklist:
#### Dev activity
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] PR is being merged under a feature flag
#### QA activity:
- [ ] [Speedbreak
features](https://github.com/appsmithorg/TestSmith/wiki/Guidelines-for-test-plans#speedbreakers-)
have been covered
- [ ] Test plan covers all impacted features and [areas of
interest](https://github.com/appsmithorg/TestSmith/wiki/Guidelines-for-test-plans#areas-of-interest-)
- [ ] Test plan has been peer reviewed by project stakeholders and other
QA members
- [ ] Manually tested functionality on DP
- [ ] We had an implementation alignment call with stakeholders post QA
Round 2
- [ ] Cypress test cases have been added and approved by SDET/manual QA
- [ ] Added `Test Plan Approved` label after Cypress tests were reviewed
- [ ] Added `Test Plan Approved` label after JUnit tests were reviewed
---------
Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
Fix issue with alias names clashing in `keytool -import` command, when
there's more than one cert file in the `ca-certs` folder.
The fix is to explicitly set the alias for each `keytool -import` run,
to the file itself, so clashes don't happen.
Running an Appsmith as a non-root user:
```sh
docker run --name appsmith --user 70:70
```
The `70:70` figures are the UID and GID respectively. It can mostly be
any number, safe to user figures are 70 to 79, or anything above 200 and
below 65000. The important bit, is that it shouldn't change on restart
or manual updates etc.
No product functionality should be affected when running as a non-root
user.
The current latest release at
https://github.com/adoptium/temurin17-binaries/releases/tag/jdk-17.0.9%2B9.1,
doesn't include binaries for Linux, and so the Java download step fails
in our base Docker image build. This PR fixes that.
Sidenote, had we not had this base image and application image
separation, this bug would've blocked all our CI pipelines and the whole
team. 🙂
