This removes the `appsmithctl migrate` command which can migrate an
Appsmith instance from on EC2 instance to another, using SSH. Why are we
removing it?
1. It's not documented on docs.appsmith.com at all.
2. The problem is better solved with a combination of `appsmithctl
backup` and `appsmithctl restore`, with much _more_ flexibility.
Bumps [nodemailer](https://github.com/nodemailer/nodemailer) from 6.7.5
to 6.9.9.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/nodemailer/nodemailer/releases">nodemailer's
releases</a>.</em></p>
<blockquote>
<h2>v6.9.9</h2>
<h2><a
href="https://github.com/nodemailer/nodemailer/compare/v6.9.8...v6.9.9">6.9.9</a>
(2024-02-01)</h2>
<h3>Bug Fixes</h3>
<ul>
<li><strong>security:</strong> Fix issues described in
GHSA-9h6g-pr28-7cqp. Do not use eternal matching pattern if only a few
occurences are expected (<a
href="dd8f5e8a4d">dd8f5e8</a>)</li>
<li><strong>tests:</strong> Use native node test runner, added code
coverage support, removed grunt (<a
href="https://redirect.github.com/nodemailer/nodemailer/issues/1604">#1604</a>)
(<a
href="be45c1b299">be45c1b</a>)</li>
</ul>
<h2>v6.9.8</h2>
<h2><a
href="https://github.com/nodemailer/nodemailer/compare/v6.9.7...v6.9.8">6.9.8</a>
(2023-12-30)</h2>
<h3>Bug Fixes</h3>
<ul>
<li><strong>punycode:</strong> do not use native punycode module (<a
href="b4d0e0c7cc">b4d0e0c</a>)</li>
</ul>
<h2>v6.9.7</h2>
<h2><a
href="https://github.com/nodemailer/nodemailer/compare/v6.9.6...v6.9.7">6.9.7</a>
(2023-10-22)</h2>
<h3>Bug Fixes</h3>
<ul>
<li><strong>customAuth:</strong> Do not require user and pass to be set
for custom authentication schemes (fixes <a
href="https://redirect.github.com/nodemailer/nodemailer/issues/1584">#1584</a>)
(<a
href="41d482c3f0">41d482c</a>)</li>
</ul>
<h2>v6.9.6</h2>
<h2><a
href="https://github.com/nodemailer/nodemailer/compare/v6.9.5...v6.9.6">6.9.6</a>
(2023-10-09)</h2>
<h3>Bug Fixes</h3>
<ul>
<li><strong>inline:</strong> Use 'inline' as the default Content
Dispostion value for embedded images (<a
href="db32c93fef">db32c93</a>)</li>
<li><strong>tests:</strong> Removed Node v12 from test matrix as it is
not compatible with the test framework anymore (<a
href="7fe0a608ed">7fe0a60</a>)</li>
</ul>
<h2>v6.9.5</h2>
<h2><a
href="https://github.com/nodemailer/nodemailer/compare/v6.9.4...v6.9.5">6.9.5</a>
(2023-09-06)</h2>
<h3>Bug Fixes</h3>
<ul>
<li><strong>license:</strong> Updated license year (<a
href="da4744e491">da4744e</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/nodemailer/nodemailer/blob/master/CHANGELOG.md">nodemailer's
changelog</a>.</em></p>
<blockquote>
<h2><a
href="https://github.com/nodemailer/nodemailer/compare/v6.9.8...v6.9.9">6.9.9</a>
(2024-02-01)</h2>
<h3>Bug Fixes</h3>
<ul>
<li><strong>security:</strong> Fix issues described in
GHSA-9h6g-pr28-7cqp. Do not use eternal matching pattern if only a few
occurences are expected (<a
href="dd8f5e8a4d">dd8f5e8</a>)</li>
<li><strong>tests:</strong> Use native node test runner, added code
coverage support, removed grunt (<a
href="https://redirect.github.com/nodemailer/nodemailer/issues/1604">#1604</a>)
(<a
href="be45c1b299">be45c1b</a>)</li>
</ul>
<h2><a
href="https://github.com/nodemailer/nodemailer/compare/v6.9.7...v6.9.8">6.9.8</a>
(2023-12-30)</h2>
<h3>Bug Fixes</h3>
<ul>
<li><strong>punycode:</strong> do not use native punycode module (<a
href="b4d0e0c7cc">b4d0e0c</a>)</li>
</ul>
<h2><a
href="https://github.com/nodemailer/nodemailer/compare/v6.9.6...v6.9.7">6.9.7</a>
(2023-10-22)</h2>
<h3>Bug Fixes</h3>
<ul>
<li><strong>customAuth:</strong> Do not require user and pass to be set
for custom authentication schemes (fixes <a
href="https://redirect.github.com/nodemailer/nodemailer/issues/1584">#1584</a>)
(<a
href="41d482c3f0">41d482c</a>)</li>
</ul>
<h2><a
href="https://github.com/nodemailer/nodemailer/compare/v6.9.5...v6.9.6">6.9.6</a>
(2023-10-09)</h2>
<h3>Bug Fixes</h3>
<ul>
<li><strong>inline:</strong> Use 'inline' as the default Content
Dispostion value for embedded images (<a
href="db32c93fef">db32c93</a>)</li>
<li><strong>tests:</strong> Removed Node v12 from test matrix as it is
not compatible with the test framework anymore (<a
href="7fe0a608ed">7fe0a60</a>)</li>
</ul>
<h2><a
href="https://github.com/nodemailer/nodemailer/compare/v6.9.4...v6.9.5">6.9.5</a>
(2023-09-06)</h2>
<h3>Bug Fixes</h3>
<ul>
<li><strong>license:</strong> Updated license year (<a
href="da4744e491">da4744e</a>)</li>
</ul>
<h2>6.9.4 2023-07-19</h2>
<ul>
<li>Renamed SendinBlue to Brevo</li>
</ul>
<h2>6.9.3 2023-05-29</h2>
<ul>
<li>Specified license identifier (was defined as MIT, actual value
MIT-0)</li>
<li>If SMTP server disconnects with a message, process it and include as
part of the response error</li>
</ul>
<h2>6.9.2 2023-05-11</h2>
<ul>
<li>Fix uncaught exception on invalid attachment content payload</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="5a2e10f454"><code>5a2e10f</code></a>
chore(master): release 6.9.9 [skip-ci] (<a
href="https://redirect.github.com/nodemailer/nodemailer/issues/1606">#1606</a>)</li>
<li><a
href="dd8f5e8a4d"><code>dd8f5e8</code></a>
fix(security): Fix issues described in GHSA-9h6g-pr28-7cqp. Do not use
eterna...</li>
<li><a
href="2c2b46ae4c"><code>2c2b46a</code></a>
chore: do not use caret in version specifier</li>
<li><a
href="be45c1b299"><code>be45c1b</code></a>
fix(tests): Use native node test runner, added code coverage support,
removed...</li>
<li><a
href="4233f6f89e"><code>4233f6f</code></a>
chore(master): release 6.9.8 [skip-ci] (<a
href="https://redirect.github.com/nodemailer/nodemailer/issues/1605">#1605</a>)</li>
<li><a
href="09d502f979"><code>09d502f</code></a>
chore: removed double file</li>
<li><a
href="b4d0e0c7cc"><code>b4d0e0c</code></a>
fix(punycode): do not use native punycode module</li>
<li><a
href="8376c024f8"><code>8376c02</code></a>
Test new github notice syntax for README</li>
<li><a
href="bc46a3b7d4"><code>bc46a3b</code></a>
Updated stale github action</li>
<li><a
href="78bdaf8c9e"><code>78bdaf8</code></a>
chore: remove redundant AWS SDK for JavaScript v2 (<a
href="https://redirect.github.com/nodemailer/nodemailer/issues/1593">#1593</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/nodemailer/nodemailer/compare/v6.7.5...v6.9.9">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/appsmithorg/appsmith/network/alerts).
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Show the commit SHA in the version popup, instead of a snapshot version
number like `v1.11-SNAPSHOT`. But if the version number doesn't have a
`-SNAPSHOT` at the end, we show the version number as is. So if it's
`v1.12`, we show that instead of the commit SHA.
This is a fix for a user's problem. They have custom domain set, a
custom cert in the `stacks/ssl` folder, but because a different team
operates a reverse-proxy, they aren't sure which _host_ is actually used
by the reverse proxy. And the way we bind to port 443 requires that that
puzzle be solved, for very little extra value.
This change makes it so that we accept any incoming TLS connections, if
a custom domain is set, which should be much more convenient.
[Slack
Thread](https://theappsmith.slack.com/archives/C0341RERY4R/p1705700120412079).
Already deployed on users' system, and they've confirmed its working.
I've been doing this in pieces bit by bit, not to rock the boat too much
too fast, but it's taking too long, and too much effort. Instead opting
for a rip-the-bandaid style, hopefully without the pain.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a Databricks plugin for executing queries and managing
database connections.
- Added a migration to incorporate the Databricks plugin into existing
workspaces.
- **Bug Fixes**
- Ensured robust error handling in the Databricks plugin with clear
messaging for query execution failures.
- **Tests**
- Implemented tests to validate the behavior of the Databricks plugin
under various connection scenarios.
- **Documentation**
- Included configuration properties for the Databricks plugin setup.
- **Refactor**
- Added specific error types and messages for the Databricks plugin to
improve debugging and user feedback.
- **Chores**
- Modified the Java runtime environment settings to support the new
plugin's requirements.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Arpit Mohan <arpit@appsmith.com>
Another attempt at #29550, which was reverted. Fallback is not happening
if cert provisioning fails _despite_ having the correct header. But with
the changes in this PR, since we'll listen on `:80`, fallback _will_
happen when cert provisioning fails due to incorrect domain
configuration.
We're also adding [Hurl](https://hurl.dev) based tests. They're not run
in any CI yet. That'll come in soon.
Defining custom domain as `https://example.com/` is invalid.
It should be just the domain, just `example.com`. But turns out a lot of
our users have the incorrect configuration, and our previous stack of
NGINX+Certbot was able to ignore this and serve without HTTPS. This PR
brings that behaviour back.
## Test performed
Have Appsmith running on an EC2 instance, and a domain `correct.com`
with an A-record pointed to this EC2 instance.
In the instance, we run Appsmith with `APPSMITH_CUSTOM_DOMAIN` set to
`wrong.com`. Caddy will obviously fail to provision the cert, and so we
expect it to accept connections on just HTTP.
So hitting `curl -i http://correct.com` produced a 200 with the HTML
response, and not a 308 with a redirect. Before the changes from this
PR, the same curl command produced a 308 with a redirect to
`https://correct.com`, which fails with a certificate error.
Next up, we run Appsmith with `APPSMITH_CUSTOM_DOMAIN` set to
`correct.com`. Caddy will succeed in provisioning a cert, and so we
expect HTTP URLs to be redirected to HTTPS.
So hitting `curl -i http://correct.com` produces a 308 redirect to
`http://correct.com` which then works fine, since Caddy now has the cert
for the domain.
We're setting the default value for `APPSMITH_ALLOWED_FRAME_ANCESTORS`
before we initialize env variables from `docker.env`. This make the
default value take a higher precedence over the value configured in
`docker.env`. And since the value in `docker.env` is the one configured
from Admin Settings, it feels like the value configured from the UI is
being ignored.
This fixes the problem by moving the check for this env variable to
_inside_ the reconfigure script, and so doesn't affect any env
variables.
Fixes#29114
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Performance Improvements**
- Enhanced logging capabilities to include memory footprint and context
details for better performance monitoring.
- **Configuration Updates**
- Increased the number of log file backups from 2 to 10, allowing for
more historical log retention.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Chores**
- Optimized auto-healing script by removing an unnecessary 60-second
delay.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced auto-healing functionality to automatically restart
unresponsive backend services.
- Added SSL configuration support for custom domains.
- **Chores**
- Implemented periodic backend service status checks.
- Enhanced startup scripts to support new auto-healing feature based on
environment configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
I think the route precedence in Caddy is different when using `handle`
directive, vs when directly using the `error` directive.
This is causing the file `handle {` route, which is a catch-all route is
handling `/static/*` requests that don't have a corresponding file. This
handler however, doesn't respond with 404 status, it responds with 200
status for missing files, and render the `index.html` for our SPA
behaviour.
Now, the CDN we have on release.app.appsmith.com caches responses from
upstream when the status is 200. If it is 404, it won't cache and retry
next time. This is why it's essential that we respond with 404 for files
that don't exist, irrespective of the content of the response.
When the container is starting up, Caddy doesn't have all the
information yet, and may have responded with not-found for one of the
assets. But since this went out with 200 status, our CDN cached it, and
once the file _was_ available with Caddy, the CDN wouldn't retry ever.
This fix will ensure we get 404 status code for requests to `/static/*`
that point to files that don't exist.
Fixes#29114
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced logging capabilities for better performance insights.
- **Improvements**
- Increased the number of log file backups to ensure more historical
data is preserved.
- **Documentation**
- Updated internal documentation to reflect new logging and performance
monitoring features.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This PR replaces NGINX and Certbot with Caddy.
1. Auto-HTTPS when custom domain is set, is handled by Caddy.
2. If past certs exist, that were provisioned by Certbot in older
Appsmith versions, we configure Caddy to make use of them. But this only
applies if the certs aren't already expired. If they're expired, point 1
applies.
3. If custom certs are provided in `ssl` folder, Caddy will be
configured to use them.
4. Incoming `Forwarded` header is not passed to any reverse proxies. So
redirect URL is correctly computed on Google Cloud Run.
5. All other route configurations are exactly as they are in NGINX
today.
Caddy configuration file is generated in the `caddy-reconfigure.mjs`
script, which will also reload Caddy with the new configuration.
## Description
This PR fixes the experience of Templates forking in self hosted
instances. And also for to Set up a process to keep the embedded DB up
to date with template db schemas.
We have removed the redirection of mockdb end point used in templates
App when forked in self hosted instance from localhost/internal postgres
db.
This also has a migration which is to make sure that none of existing
apps using the internal postgres does not break due to the removal of
redirection. The migration will make sure that existing self hosted
instances using the posgress db and has a datasource with mockdb end
point will be replaces with localhost.
#### PR fixes following issue(s)
Fixes https://github.com/appsmithorg/appsmith/issues/28924
#### Type of change
- Bug fix (non-breaking change which fixes an issue)
## Testing
#### How Has This Been Tested?
- [ ] Manual
#### Test Plan
> Add Testsmith test cases links that relate to this PR
>
>
#### Issues raised during DP testing
> Link issues raised during DP testing for better visiblity and tracking
(copy link from comments dropped on this PR)
>
>
>
## Checklist:
#### Dev activity
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] PR is being merged under a feature flag
#### QA activity:
- [ ] [Speedbreak
features](https://github.com/appsmithorg/TestSmith/wiki/Guidelines-for-test-plans#speedbreakers-)
have been covered
- [ ] Test plan covers all impacted features and [areas of
interest](https://github.com/appsmithorg/TestSmith/wiki/Guidelines-for-test-plans#areas-of-interest-)
- [ ] Test plan has been peer reviewed by project stakeholders and other
QA members
- [ ] Manually tested functionality on DP
- [ ] We had an implementation alignment call with stakeholders post QA
Round 2
- [ ] Cypress test cases have been added and approved by SDET/manual QA
- [ ] Added `Test Plan Approved` label after Cypress tests were reviewed
- [ ] Added `Test Plan Approved` label after JUnit tests were reviewed
---------
Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
Fix issue with alias names clashing in `keytool -import` command, when
there's more than one cert file in the `ca-certs` folder.
The fix is to explicitly set the alias for each `keytool -import` run,
to the file itself, so clashes don't happen.
Running an Appsmith as a non-root user:
```sh
docker run --name appsmith --user 70:70
```
The `70:70` figures are the UID and GID respectively. It can mostly be
any number, safe to user figures are 70 to 79, or anything above 200 and
below 65000. The important bit, is that it shouldn't change on restart
or manual updates etc.
No product functionality should be affected when running as a non-root
user.
What are we solving here?
1. Installing Java in the `Dockerfile` by using Adoptium's package
repositories is fragile since they've started blocking some IP addresses
used by GitHub Actions runners. We see a message like this:
```
Failed to fetch
https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-17/temurin-17-jdk_17.0.8.1.0+1_amd64.deb
403 Forbidden [IP: 146.75.107.42 443]
```
We're seeing more and more cases of these and PRs are getting blocked.
2. Installing Java via `apt` also installs other packages like X11
libraries, that aren't really relevant to our usage of Java. Yet, these
packages are present in our Docker image, and are the source of several
CVEs to be reported by scanners on our Docker image.
3. This will give us control over trusted CA certificates, which we can
now perform under `$TMP`, which aligns with our move towards supporting
readonly root filesystem. Which is essentially not write to anything in
the Docker image at runtime, except for under `/tmp` and
`/appsmith-stacks`. This will help us move in that direction.
This removes ~70 medium/low severity CVEs reported on our Docker image,
by removing `build-essential` from being installed by default in the
Docker image.
We only need it when compiling Redis, which is needed on _some_ ARM
systems, that re configured with a page-size of greater than 4096. For
example, CentOS 8.
In the NGINX configuration we generate, we're redirecting _all_ HTTP
requests to HTTPS, when HTTPS is enabled. But the HTTP-01 challenge
works on port 80 and is getting redirected to 443.
This usually fine, as Let's Encrypt respects that redirect and completes
the challenge on port 443. But, if port 443 is blocked to outside
access, the cert renewal will fail. This PR fixes that.
Tested on a server with port 80 open and 443 closed to outside Internet.
Cert renewal fails without this PR's changes, and works with this PR's
changes.
This broke when we changed the way RTS stores version information. This
was never the right way to get the version in the `backup` command and
this PR fixes it, by getting the version from `info.json`.
Failure error:
```
Error: ENOENT: no such file or directory, open '/opt/appsmith/rts/version.js'
```
