This is to avoid low-impact failures from getting the whole container to
restart.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Modified health check to focus on critical services (`editor`, `rts`,
`backend`) for more efficient monitoring.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Updated the loop condition in the URL encoding functionality for
improved reliability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Replaces NGINX for local dev with Caddy.
The advandage here is that the script that generates `Caddyfile` at
runtime on production deployments, is also used to generate the
`Caddyfile` at local development. This reduces the local--production
gap.
/ok-to-test tags="@tag.Sanity"
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Refactor**
- Updated base Docker image and installation processes for tools within
the Docker environment.
- **Chores**
- Modified scripts to enhance file handling and environment variable
configurations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This removes the `appsmithctl migrate` command which can migrate an
Appsmith instance from on EC2 instance to another, using SSH. Why are we
removing it?
1. It's not documented on docs.appsmith.com at all.
2. The problem is better solved with a combination of `appsmithctl
backup` and `appsmithctl restore`, with much _more_ flexibility.
## Description
- update node version and appropriate git workflow
- added the path to webpack cache folder, this should speed up bundle
creation about a minute
[Test, build and push Docker
Image](https://github.com/appsmithorg/appsmith/actions/runs/8421752151)
[Build Client, Server & Run only
Cypress](https://github.com/appsmithorg/appsmith/actions/runs/8421752151)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Chores**
- Updated actions/cache and actions/setup-node to v4 across various
workflows for improved caching and Node.js setup.
- Modified the `yarn install` command to use `--immutable` flag,
enhancing dependency management.
- **Documentation**
- Updated comments within workflows to include cautionary and important
notes, ensuring better clarity.
- **Refactor**
- Adjusted caching paths and keys for more efficient caching behavior.
- Changed Node.js installation to version 20.11.1 in Dockerfile,
aligning with the latest version for better performance and security.
- **Tests**
- Modified assertion in `getCurrentLocationSaga` test to check for the
presence of a property, improving test accuracy.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Aman Agarwal <aman@appsmith.com>
Fixes: [31031](https://github.com/appsmithorg/appsmith/issues/31031)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added rate-limiting functionality to enhance security and prevent
abuse.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
Bumps [nodemailer](https://github.com/nodemailer/nodemailer) from 6.7.5
to 6.9.9.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/nodemailer/nodemailer/releases">nodemailer's
releases</a>.</em></p>
<blockquote>
<h2>v6.9.9</h2>
<h2><a
href="https://github.com/nodemailer/nodemailer/compare/v6.9.8...v6.9.9">6.9.9</a>
(2024-02-01)</h2>
<h3>Bug Fixes</h3>
<ul>
<li><strong>security:</strong> Fix issues described in
GHSA-9h6g-pr28-7cqp. Do not use eternal matching pattern if only a few
occurences are expected (<a
href="dd8f5e8a4d">dd8f5e8</a>)</li>
<li><strong>tests:</strong> Use native node test runner, added code
coverage support, removed grunt (<a
href="https://redirect.github.com/nodemailer/nodemailer/issues/1604">#1604</a>)
(<a
href="be45c1b299">be45c1b</a>)</li>
</ul>
<h2>v6.9.8</h2>
<h2><a
href="https://github.com/nodemailer/nodemailer/compare/v6.9.7...v6.9.8">6.9.8</a>
(2023-12-30)</h2>
<h3>Bug Fixes</h3>
<ul>
<li><strong>punycode:</strong> do not use native punycode module (<a
href="b4d0e0c7cc">b4d0e0c</a>)</li>
</ul>
<h2>v6.9.7</h2>
<h2><a
href="https://github.com/nodemailer/nodemailer/compare/v6.9.6...v6.9.7">6.9.7</a>
(2023-10-22)</h2>
<h3>Bug Fixes</h3>
<ul>
<li><strong>customAuth:</strong> Do not require user and pass to be set
for custom authentication schemes (fixes <a
href="https://redirect.github.com/nodemailer/nodemailer/issues/1584">#1584</a>)
(<a
href="41d482c3f0">41d482c</a>)</li>
</ul>
<h2>v6.9.6</h2>
<h2><a
href="https://github.com/nodemailer/nodemailer/compare/v6.9.5...v6.9.6">6.9.6</a>
(2023-10-09)</h2>
<h3>Bug Fixes</h3>
<ul>
<li><strong>inline:</strong> Use 'inline' as the default Content
Dispostion value for embedded images (<a
href="db32c93fef">db32c93</a>)</li>
<li><strong>tests:</strong> Removed Node v12 from test matrix as it is
not compatible with the test framework anymore (<a
href="7fe0a608ed">7fe0a60</a>)</li>
</ul>
<h2>v6.9.5</h2>
<h2><a
href="https://github.com/nodemailer/nodemailer/compare/v6.9.4...v6.9.5">6.9.5</a>
(2023-09-06)</h2>
<h3>Bug Fixes</h3>
<ul>
<li><strong>license:</strong> Updated license year (<a
href="da4744e491">da4744e</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/nodemailer/nodemailer/blob/master/CHANGELOG.md">nodemailer's
changelog</a>.</em></p>
<blockquote>
<h2><a
href="https://github.com/nodemailer/nodemailer/compare/v6.9.8...v6.9.9">6.9.9</a>
(2024-02-01)</h2>
<h3>Bug Fixes</h3>
<ul>
<li><strong>security:</strong> Fix issues described in
GHSA-9h6g-pr28-7cqp. Do not use eternal matching pattern if only a few
occurences are expected (<a
href="dd8f5e8a4d">dd8f5e8</a>)</li>
<li><strong>tests:</strong> Use native node test runner, added code
coverage support, removed grunt (<a
href="https://redirect.github.com/nodemailer/nodemailer/issues/1604">#1604</a>)
(<a
href="be45c1b299">be45c1b</a>)</li>
</ul>
<h2><a
href="https://github.com/nodemailer/nodemailer/compare/v6.9.7...v6.9.8">6.9.8</a>
(2023-12-30)</h2>
<h3>Bug Fixes</h3>
<ul>
<li><strong>punycode:</strong> do not use native punycode module (<a
href="b4d0e0c7cc">b4d0e0c</a>)</li>
</ul>
<h2><a
href="https://github.com/nodemailer/nodemailer/compare/v6.9.6...v6.9.7">6.9.7</a>
(2023-10-22)</h2>
<h3>Bug Fixes</h3>
<ul>
<li><strong>customAuth:</strong> Do not require user and pass to be set
for custom authentication schemes (fixes <a
href="https://redirect.github.com/nodemailer/nodemailer/issues/1584">#1584</a>)
(<a
href="41d482c3f0">41d482c</a>)</li>
</ul>
<h2><a
href="https://github.com/nodemailer/nodemailer/compare/v6.9.5...v6.9.6">6.9.6</a>
(2023-10-09)</h2>
<h3>Bug Fixes</h3>
<ul>
<li><strong>inline:</strong> Use 'inline' as the default Content
Dispostion value for embedded images (<a
href="db32c93fef">db32c93</a>)</li>
<li><strong>tests:</strong> Removed Node v12 from test matrix as it is
not compatible with the test framework anymore (<a
href="7fe0a608ed">7fe0a60</a>)</li>
</ul>
<h2><a
href="https://github.com/nodemailer/nodemailer/compare/v6.9.4...v6.9.5">6.9.5</a>
(2023-09-06)</h2>
<h3>Bug Fixes</h3>
<ul>
<li><strong>license:</strong> Updated license year (<a
href="da4744e491">da4744e</a>)</li>
</ul>
<h2>6.9.4 2023-07-19</h2>
<ul>
<li>Renamed SendinBlue to Brevo</li>
</ul>
<h2>6.9.3 2023-05-29</h2>
<ul>
<li>Specified license identifier (was defined as MIT, actual value
MIT-0)</li>
<li>If SMTP server disconnects with a message, process it and include as
part of the response error</li>
</ul>
<h2>6.9.2 2023-05-11</h2>
<ul>
<li>Fix uncaught exception on invalid attachment content payload</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="5a2e10f454"><code>5a2e10f</code></a>
chore(master): release 6.9.9 [skip-ci] (<a
href="https://redirect.github.com/nodemailer/nodemailer/issues/1606">#1606</a>)</li>
<li><a
href="dd8f5e8a4d"><code>dd8f5e8</code></a>
fix(security): Fix issues described in GHSA-9h6g-pr28-7cqp. Do not use
eterna...</li>
<li><a
href="2c2b46ae4c"><code>2c2b46a</code></a>
chore: do not use caret in version specifier</li>
<li><a
href="be45c1b299"><code>be45c1b</code></a>
fix(tests): Use native node test runner, added code coverage support,
removed...</li>
<li><a
href="4233f6f89e"><code>4233f6f</code></a>
chore(master): release 6.9.8 [skip-ci] (<a
href="https://redirect.github.com/nodemailer/nodemailer/issues/1605">#1605</a>)</li>
<li><a
href="09d502f979"><code>09d502f</code></a>
chore: removed double file</li>
<li><a
href="b4d0e0c7cc"><code>b4d0e0c</code></a>
fix(punycode): do not use native punycode module</li>
<li><a
href="8376c024f8"><code>8376c02</code></a>
Test new github notice syntax for README</li>
<li><a
href="bc46a3b7d4"><code>bc46a3b</code></a>
Updated stale github action</li>
<li><a
href="78bdaf8c9e"><code>78bdaf8</code></a>
chore: remove redundant AWS SDK for JavaScript v2 (<a
href="https://redirect.github.com/nodemailer/nodemailer/issues/1593">#1593</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/nodemailer/nodemailer/compare/v6.7.5...v6.9.9">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/appsmithorg/appsmith/network/alerts).
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Show the commit SHA in the version popup, instead of a snapshot version
number like `v1.11-SNAPSHOT`. But if the version number doesn't have a
`-SNAPSHOT` at the end, we show the version number as is. So if it's
`v1.12`, we show that instead of the commit SHA.
This is a fix for a user's problem. They have custom domain set, a
custom cert in the `stacks/ssl` folder, but because a different team
operates a reverse-proxy, they aren't sure which _host_ is actually used
by the reverse proxy. And the way we bind to port 443 requires that that
puzzle be solved, for very little extra value.
This change makes it so that we accept any incoming TLS connections, if
a custom domain is set, which should be much more convenient.
[Slack
Thread](https://theappsmith.slack.com/archives/C0341RERY4R/p1705700120412079).
Already deployed on users' system, and they've confirmed its working.
I've been doing this in pieces bit by bit, not to rock the boat too much
too fast, but it's taking too long, and too much effort. Instead opting
for a rip-the-bandaid style, hopefully without the pain.
Instead of downloading it from the github repo on every build (in
`base.dockerfile`), we vendor the supervisor extension so we can look to
fix the buffer overflow problem in supervisor logs.
After this is merged, after a day/two, we'll be removing `git`,
`python3-pip` and the supervisor-stdout extension from
`base.dockerfile`.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a Databricks plugin for executing queries and managing
database connections.
- Added a migration to incorporate the Databricks plugin into existing
workspaces.
- **Bug Fixes**
- Ensured robust error handling in the Databricks plugin with clear
messaging for query execution failures.
- **Tests**
- Implemented tests to validate the behavior of the Databricks plugin
under various connection scenarios.
- **Documentation**
- Included configuration properties for the Databricks plugin setup.
- **Refactor**
- Added specific error types and messages for the Databricks plugin to
improve debugging and user feedback.
- **Chores**
- Modified the Java runtime environment settings to support the new
plugin's requirements.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Arpit Mohan <arpit@appsmith.com>
Another attempt at #29550, which was reverted. Fallback is not happening
if cert provisioning fails _despite_ having the correct header. But with
the changes in this PR, since we'll listen on `:80`, fallback _will_
happen when cert provisioning fails due to incorrect domain
configuration.
We're also adding [Hurl](https://hurl.dev) based tests. They're not run
in any CI yet. That'll come in soon.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Chores**
- Updated the source URL for Docker Compose file download in the cloud
initialization script to a specific GitHub repository for improved
reliability and traceability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Defining custom domain as `https://example.com/` is invalid.
It should be just the domain, just `example.com`. But turns out a lot of
our users have the incorrect configuration, and our previous stack of
NGINX+Certbot was able to ignore this and serve without HTTPS. This PR
brings that behaviour back.
## Test performed
Have Appsmith running on an EC2 instance, and a domain `correct.com`
with an A-record pointed to this EC2 instance.
In the instance, we run Appsmith with `APPSMITH_CUSTOM_DOMAIN` set to
`wrong.com`. Caddy will obviously fail to provision the cert, and so we
expect it to accept connections on just HTTP.
So hitting `curl -i http://correct.com` produced a 200 with the HTML
response, and not a 308 with a redirect. Before the changes from this
PR, the same curl command produced a 308 with a redirect to
`https://correct.com`, which fails with a certificate error.
Next up, we run Appsmith with `APPSMITH_CUSTOM_DOMAIN` set to
`correct.com`. Caddy will succeed in provisioning a cert, and so we
expect HTTP URLs to be redirected to HTTPS.
So hitting `curl -i http://correct.com` produces a 308 redirect to
`http://correct.com` which then works fine, since Caddy now has the cert
for the domain.
We're setting the default value for `APPSMITH_ALLOWED_FRAME_ANCESTORS`
before we initialize env variables from `docker.env`. This make the
default value take a higher precedence over the value configured in
`docker.env`. And since the value in `docker.env` is the one configured
from Admin Settings, it feels like the value configured from the UI is
being ignored.
This fixes the problem by moving the check for this env variable to
_inside_ the reconfigure script, and so doesn't affect any env
variables.
Fixes#29114
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Performance Improvements**
- Enhanced logging capabilities to include memory footprint and context
details for better performance monitoring.
- **Configuration Updates**
- Increased the number of log file backups from 2 to 10, allowing for
more historical log retention.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Chores**
- Optimized auto-healing script by removing an unnecessary 60-second
delay.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced auto-healing functionality to automatically restart
unresponsive backend services.
- Added SSL configuration support for custom domains.
- **Chores**
- Implemented periodic backend service status checks.
- Enhanced startup scripts to support new auto-healing feature based on
environment configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
I think the route precedence in Caddy is different when using `handle`
directive, vs when directly using the `error` directive.
This is causing the file `handle {` route, which is a catch-all route is
handling `/static/*` requests that don't have a corresponding file. This
handler however, doesn't respond with 404 status, it responds with 200
status for missing files, and render the `index.html` for our SPA
behaviour.
Now, the CDN we have on release.app.appsmith.com caches responses from
upstream when the status is 200. If it is 404, it won't cache and retry
next time. This is why it's essential that we respond with 404 for files
that don't exist, irrespective of the content of the response.
When the container is starting up, Caddy doesn't have all the
information yet, and may have responded with not-found for one of the
assets. But since this went out with 200 status, our CDN cached it, and
once the file _was_ available with Caddy, the CDN wouldn't retry ever.
This fix will ensure we get 404 status code for requests to `/static/*`
that point to files that don't exist.
Fixes#29114
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced logging capabilities for better performance insights.
- **Improvements**
- Increased the number of log file backups to ensure more historical
data is preserved.
- **Documentation**
- Updated internal documentation to reflect new logging and performance
monitoring features.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This PR replaces NGINX and Certbot with Caddy.
1. Auto-HTTPS when custom domain is set, is handled by Caddy.
2. If past certs exist, that were provisioned by Certbot in older
Appsmith versions, we configure Caddy to make use of them. But this only
applies if the certs aren't already expired. If they're expired, point 1
applies.
3. If custom certs are provided in `ssl` folder, Caddy will be
configured to use them.
4. Incoming `Forwarded` header is not passed to any reverse proxies. So
redirect URL is correctly computed on Google Cloud Run.
5. All other route configurations are exactly as they are in NGINX
today.
Caddy configuration file is generated in the `caddy-reconfigure.mjs`
script, which will also reload Caddy with the new configuration.
## Description
This PR fixes the experience of Templates forking in self hosted
instances. And also for to Set up a process to keep the embedded DB up
to date with template db schemas.
We have removed the redirection of mockdb end point used in templates
App when forked in self hosted instance from localhost/internal postgres
db.
This also has a migration which is to make sure that none of existing
apps using the internal postgres does not break due to the removal of
redirection. The migration will make sure that existing self hosted
instances using the posgress db and has a datasource with mockdb end
point will be replaces with localhost.
#### PR fixes following issue(s)
Fixes https://github.com/appsmithorg/appsmith/issues/28924
#### Type of change
- Bug fix (non-breaking change which fixes an issue)
## Testing
#### How Has This Been Tested?
- [ ] Manual
#### Test Plan
> Add Testsmith test cases links that relate to this PR
>
>
#### Issues raised during DP testing
> Link issues raised during DP testing for better visiblity and tracking
(copy link from comments dropped on this PR)
>
>
>
## Checklist:
#### Dev activity
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] PR is being merged under a feature flag
#### QA activity:
- [ ] [Speedbreak
features](https://github.com/appsmithorg/TestSmith/wiki/Guidelines-for-test-plans#speedbreakers-)
have been covered
- [ ] Test plan covers all impacted features and [areas of
interest](https://github.com/appsmithorg/TestSmith/wiki/Guidelines-for-test-plans#areas-of-interest-)
- [ ] Test plan has been peer reviewed by project stakeholders and other
QA members
- [ ] Manually tested functionality on DP
- [ ] We had an implementation alignment call with stakeholders post QA
Round 2
- [ ] Cypress test cases have been added and approved by SDET/manual QA
- [ ] Added `Test Plan Approved` label after Cypress tests were reviewed
- [ ] Added `Test Plan Approved` label after JUnit tests were reviewed
---------
Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
Fix issue with alias names clashing in `keytool -import` command, when
there's more than one cert file in the `ca-certs` folder.
The fix is to explicitly set the alias for each `keytool -import` run,
to the file itself, so clashes don't happen.
Running an Appsmith as a non-root user:
```sh
docker run --name appsmith --user 70:70
```
The `70:70` figures are the UID and GID respectively. It can mostly be
any number, safe to user figures are 70 to 79, or anything above 200 and
below 65000. The important bit, is that it shouldn't change on restart
or manual updates etc.
No product functionality should be affected when running as a non-root
user.
The current latest release at
https://github.com/adoptium/temurin17-binaries/releases/tag/jdk-17.0.9%2B9.1,
doesn't include binaries for Linux, and so the Java download step fails
in our base Docker image build. This PR fixes that.
Sidenote, had we not had this base image and application image
separation, this bug would've blocked all our CI pipelines and the whole
team. 🙂
The layers in the Dockerfile that depend on downloading large files from
external sources, doesn't have to run every day, or at every PR. We
tried using Docker's caching configuration, but it's not as reliable as
we'd have liked.
A separate base image lends us much more control over the how long we
cache the downloaded files and how often we redo this.
This PR only _adds_ the base image. It doesn't change anything in the
build of the existing Docker image. That'll happen once we have the base
images for `release` and `master` already present on DockerHub.
What are we solving here?
1. Installing Java in the `Dockerfile` by using Adoptium's package
repositories is fragile since they've started blocking some IP addresses
used by GitHub Actions runners. We see a message like this:
```
Failed to fetch
https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-17/temurin-17-jdk_17.0.8.1.0+1_amd64.deb
403 Forbidden [IP: 146.75.107.42 443]
```
We're seeing more and more cases of these and PRs are getting blocked.
2. Installing Java via `apt` also installs other packages like X11
libraries, that aren't really relevant to our usage of Java. Yet, these
packages are present in our Docker image, and are the source of several
CVEs to be reported by scanners on our Docker image.
3. This will give us control over trusted CA certificates, which we can
now perform under `$TMP`, which aligns with our move towards supporting
readonly root filesystem. Which is essentially not write to anything in
the Docker image at runtime, except for under `/tmp` and
`/appsmith-stacks`. This will help us move in that direction.
This removes ~70 medium/low severity CVEs reported on our Docker image,
by removing `build-essential` from being installed by default in the
Docker image.
We only need it when compiling Redis, which is needed on _some_ ARM
systems, that re configured with a page-size of greater than 4096. For
example, CentOS 8.
In the NGINX configuration we generate, we're redirecting _all_ HTTP
requests to HTTPS, when HTTPS is enabled. But the HTTP-01 challenge
works on port 80 and is getting redirected to 443.
This usually fine, as Let's Encrypt respects that redirect and completes
the challenge on port 443. But, if port 443 is blocked to outside
access, the cert renewal will fail. This PR fixes that.
Tested on a server with port 80 open and 443 closed to outside Internet.
Cert renewal fails without this PR's changes, and works with this PR's
changes.
This broke when we changed the way RTS stores version information. This
was never the right way to get the version in the `backup` command and
this PR fixes it, by getting the version from `info.json`.
Failure error:
```
Error: ENOENT: no such file or directory, open '/opt/appsmith/rts/version.js'
```
This fixes RTS build to use `esbuild`.
1. This means the whole `node_modules` won't need to be copied over to
the Docker image. There's unused insignifant _test_ files in there, that
don't add any value, but are causing irrelevant CVEs to be reported on
our Docker image. See example at
https://github.com/appsmithorg/appsmith-ee/pull/2349.
2. Much faster. Not that RTS build is our slow point, but still. Perhaps
we can move client to `esbuild` too. 🙂
## Why are we doing this?
The current method of loading RTS into the Docker image means that _all_
contents of _all_ dependencies are copied over. The whole
`node_modules`. But several of these packages include _test_ files too,
that aren't needed at runtime at all. One of such test files is creating
a false alert for a CVE on our Docker image. Has absolutely no relevance
and impact, but it's there.
To fix that, I [had to `rm -rf /opt/appsmith/rts/node_modules/*/test` in
the Docker
image](https://github.com/appsmithorg/appsmith-ee/pull/2349/files). This
felt very hacky, and very dirty. It felt like we're introducing more
debt and more duct tape around the current build process.
So, `esbuild`.
## Where is `esbuild` coming from?
We're using `esbuild` v0.18.20 only, while the latest is v0.19.3. We
need to update `design-system`'s storybook dependency, I think, to get a
more recent version of `esbuild`. I'm yet to figure this out and can use
some help. 🙂
When running with a custom `PORT` env variable, NGINX server will be
listening on this port. In the backend's startup script, `run-java.sh`,
we're checking for RTS being up or not, at `localhost`. So when the port
is not 80, then this will never succeed, because it'll be looking for
NGINX at the wrong port.
Instead, the fix here will make the backend startup script hit RTS
_directly_ on RTS server's own port, instead of going via NGINX. This
means it's independent of both the `PORT` env variable and the NGINX
server, and only dependent on RTS being up, which is really what we want
here.
When PostgreSQL starts, we see the following errors in the logs:
```
mkdir:
cannot create directory ‘/tmp/appsmith/postgres-stats’
: Permission denied
```
And then this over and over again:
```
postgres stdout | 2023-09-19 15:34:34.504 UTC [1759] LOG: could not open temporary statistics file "/tmp/appsmith/postgres-stats/global.tmp": No such file or directory
```
The problem is that in `postgres.conf`, we set `user=postgres`, which
doesn't have access to create things in `/tmp`.
This PR removes this configuration and lets the default be, which will
be a temp folders _under_ the data directory.
This is part of supporting running Appsmith with readonly root FS. This
moves the supervisord configuration, and runtime files, like the unix
socket file, and the PID file, to `$TMP`.
This is moving the cleanup script to delete old log files, from being a
cron job, to being a scheduled task in the backend server. Why?
1. We want to support running Appsmith with readonly root, which is a
request from security teams at enterprise companies.
2. Cron doesn't play nice. It wants to save a `.pid` file under
`/var/run`, so it fails to start when running with readonly root. This
is not configurable.
3. Since our use of cron is minimal only, we're moving away from it,
especially since the backend server is already capable of running
scheduled tasks.
4. This moves one job, there's still another. Based on experience from
this, we'll work on the other.
Another advantage to doing this is that since this job is now running in
the backend server, which has access to MongoDB and Redis, we can
coordinate when running as a cluster, that only _one_ backend is running
the clean up job. This is for much later though.
---------
Co-authored-by: Nidhi <nidhi@appsmith.com>
This is another step towards supporting running with readonly root FS,
and only making runtime changes in the container in `/tmp` or in
`/appsmith-stacks`, and nowhere else.
Move the files that are copied into the Docker image, into an `fs`
folder, that reflects the folder structure of that in the image. This
means two things right away:
1. A single `COPY` instruction in `Dockerfile` is enough to copy all the
files to their places.
2. The structure of files in the repo reflects that in the Docker image.
This makes working with the files/folders and troubleshooting with them
much easier.
❗ Note: **There's actually only 3 files changed, rest are just moved.**
There's a few CVEs reported on the version of python-requests that's
being used in the image, which we don't need that badly anyway. This PR
removes `python-requests` altogather so we shouldn't see CVEs on it show
up on our image anymore.
Bumps [mongodb](https://github.com/mongodb/node-mongodb-native) from
5.7.0 to 5.8.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/mongodb/node-mongodb-native/releases">mongodb's
releases</a>.</em></p>
<blockquote>
<h2>v5.8.0</h2>
<h2><a
href="https://github.com/mongodb/node-mongodb-native/compare/v5.7.0...v5.8.0">5.8.0</a>
(2023-08-21)</h2>
<p>The MongoDB Node.js team is pleased to announce version 5.8.0 of the
<code>mongodb</code> package!</p>
<h2>Release Notes</h2>
<h3>The <code>AutoEncrypter</code> interface has been deprecated</h3>
<p>The <code>AutoEncrypter</code> interface was used internally but
accidentally made public in the 4.x version of the driver. It is now
deprecated and will be made internal in the next major release.</p>
<h3>Kerberos support for 1.x and 2.x</h3>
<p>Moves the kerberos dependency back to <code>^1.0.0 || ^2.0.0</code>
to indicate support for both 1.x and 2.x. Support for 1.x is removed in
6.0.</p>
<h3>Fixed accidental deprecation warning</h3>
<p>Because of internal options handling, a deprecation was emitted for
<code>tlsCertificateFile</code> when using
<code>tlsCertificateKeyFile</code>. That has been corrected.</p>
<h3>Remove credential availability on
<code>ConnectionPoolCreatedEvent</code></h3>
<p>In order to avoid mistakenly printing credentials the
<code>ConnectionPoolCreatedEvent</code> will replace the credentials
option with an empty object. The credentials are still accessble via
MongoClient options: <code>client.options.credentials</code>.</p>
<h3>Features</h3>
<ul>
<li><strong>NODE-5399:</strong> use mongodb-js/saslprep instead of
saslprep (<a
href="https://redirect.github.com/mongodb/node-mongodb-native/issues/3818">#3818</a>)
(<a
href="c0d3927379">c0d3927</a>)</li>
<li><strong>NODE-5429:</strong> deprecate the <code>AutoEncrypter</code>
interface (<a
href="https://redirect.github.com/mongodb/node-mongodb-native/issues/3764">#3764</a>)
(<a
href="9bb0d955e7">9bb0d95</a>)</li>
<li><strong>NODE-5465,NODE-5538:</strong> lower
<code>@aws-sdk/credential-providers</code> version to 3.188.0 and
<code>zstd</code> to <code>^1.0.0</code> (<a
href="https://redirect.github.com/mongodb/node-mongodb-native/issues/3821">#3821</a>)
(<a
href="39ff81d402">39ff81d</a>)</li>
</ul>
<h3>Bug Fixes</h3>
<ul>
<li><strong>NODE-5489:</strong> set kerberos compatibility to ^1.0.0 ||
^2.0.0 (<a
href="https://redirect.github.com/mongodb/node-mongodb-native/issues/3803">#3803</a>)
(<a
href="c3b35b321b">c3b35b3</a>)</li>
<li><strong>NODE-5495:</strong> do not emit deprecation warning when
tlsCertificateKeyFile is specified and tlsCertificateFile is not (<a
href="https://redirect.github.com/mongodb/node-mongodb-native/issues/3810">#3810</a>)
(<a
href="e81d4a2fc0">e81d4a2</a>)</li>
<li><strong>NODE-5537:</strong> remove credentials from
ConnectionPoolCreatedEvent options (<a
href="https://redirect.github.com/mongodb/node-mongodb-native/issues/3813">#3813</a>)
(<a
href="4cf1e96c68">4cf1e96</a>)</li>
</ul>
<h2>Documentation</h2>
<ul>
<li><a
href="https://docs.mongodb.com/drivers/node/current/">Reference</a></li>
<li><a
href="https://mongodb.github.io/node-mongodb-native/5.8/">API</a></li>
<li><a
href="https://github.com/mongodb/node-mongodb-native/blob/v5.8.0/HISTORY.md">Changelog</a></li>
</ul>
<p>We invite you to try the <code>mongodb</code> library immediately,
and report any issues to the <a
href="https://jira.mongodb.org/projects/NODE">NODE project</a>.</p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/mongodb/node-mongodb-native/blob/v5.8.0/HISTORY.md">mongodb's
changelog</a>.</em></p>
<blockquote>
<h2><a
href="https://github.com/mongodb/node-mongodb-native/compare/v5.7.0...v5.8.0">5.8.0</a>
(2023-08-21)</h2>
<h3>Features</h3>
<ul>
<li><strong>NODE-5399:</strong> use mongodb-js/saslprep instead of
saslprep (<a
href="https://redirect.github.com/mongodb/node-mongodb-native/issues/3818">#3818</a>)
(<a
href="c0d3927379">c0d3927</a>)</li>
<li><strong>NODE-5429:</strong> deprecate the <code>AutoEncrypter</code>
interface (<a
href="https://redirect.github.com/mongodb/node-mongodb-native/issues/3764">#3764</a>)
(<a
href="9bb0d955e7">9bb0d95</a>)</li>
<li><strong>NODE-5465,NODE-5538:</strong> lower
<code>@aws-sdk/credential-providers</code> version to 3.188.0 and
<code>zstd</code> to <code>^1.0.0</code> (<a
href="https://redirect.github.com/mongodb/node-mongodb-native/issues/3821">#3821</a>)
(<a
href="39ff81d402">39ff81d</a>)</li>
<li><strong>NODE-5489:</strong> update kerberos dependency (<a
href="8c25d6d0a7">8c25d6d</a>)</li>
</ul>
<h3>Bug Fixes</h3>
<ul>
<li><strong>NODE-5489:</strong> set kerberos compatibility to ^1.0.0 ||
^2.0.0 (<a
href="https://redirect.github.com/mongodb/node-mongodb-native/issues/3803">#3803</a>)
(<a
href="c3b35b321b">c3b35b3</a>)</li>
<li><strong>NODE-5495:</strong> do not emit deprecation warning when
tlsCertificateKeyFile is specified and tlsCertificateFile is not (<a
href="https://redirect.github.com/mongodb/node-mongodb-native/issues/3810">#3810</a>)
(<a
href="e81d4a2fc0">e81d4a2</a>)</li>
<li><strong>NODE-5537:</strong> remove credentials from
ConnectionPoolCreatedEvent options (<a
href="https://redirect.github.com/mongodb/node-mongodb-native/issues/3813">#3813</a>)
(<a
href="4cf1e96c68">4cf1e96</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="43673fa189"><code>43673fa</code></a>
chore(5.x): release 5.8.0 [skip-ci] (<a
href="https://redirect.github.com/mongodb/node-mongodb-native/issues/3825">#3825</a>)</li>
<li><a
href="4b2fc79525"><code>4b2fc79</code></a>
docs: fix cutoff sentence on CommandStartedEvent (<a
href="https://redirect.github.com/mongodb/node-mongodb-native/issues/3828">#3828</a>)</li>
<li><a
href="39ff81d402"><code>39ff81d</code></a>
feat(NODE-5465,NODE-5538): lower
<code>@aws-sdk/credential-providers</code> version to 3...</li>
<li><a
href="e1af3437c3"><code>e1af343</code></a>
chore: update release automation scripts 5.x (<a
href="https://redirect.github.com/mongodb/node-mongodb-native/issues/3823">#3823</a>)</li>
<li><a
href="c0d3927379"><code>c0d3927</code></a>
feat(NODE-5399): use mongodb-js/saslprep instead of saslprep (<a
href="https://redirect.github.com/mongodb/node-mongodb-native/issues/3818">#3818</a>)</li>
<li><a
href="4cf1e96c68"><code>4cf1e96</code></a>
fix(NODE-5537): remove credentials from ConnectionPoolCreatedEvent
options (#...</li>
<li><a
href="e81d4a2fc0"><code>e81d4a2</code></a>
fix(NODE-5495): do not emit deprecation warning when
tlsCertificateKeyFile is...</li>
<li><a
href="c3b35b321b"><code>c3b35b3</code></a>
fix(NODE-5489): set kerberos compatibility to ^1.0.0 || ^2.0.0 (<a
href="https://redirect.github.com/mongodb/node-mongodb-native/issues/3803">#3803</a>)</li>
<li><a
href="cc3069d836"><code>cc3069d</code></a>
Revert "feat(NODE-5489): update kerberos dependency"</li>
<li><a
href="8c25d6d0a7"><code>8c25d6d</code></a>
feat(NODE-5489): update kerberos dependency</li>
<li>Additional commits viewable in <a
href="https://github.com/mongodb/node-mongodb-native/compare/v5.7.0...v5.8.0">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/appsmithorg/appsmith/network/alerts).
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Removes this `Server` header in all responses from Appsmith.
```
curl -sSI http://localhost | grep 'server:'
```
It currently shows this:
```
server: nginx
```
Update:
- Mobed jest to` devDependenies` of package manifest.
- Updated Dockerfile to install utils with `--only=prod` arg for npm
install cmds.
Tested manually
Majority of the NGINX config is the same, for both HTTP and HTTPS.
Having two separate templates for them is making configuration changes
error-prone, where we often risk forgetting making the same change in
the other file.
This PR merges the two files into one, so the above risk isn't there. It
also makes it easier to experiment with the file while developing, since
we have to make every single change twice during development.
Note: This _will_ cause conflicts in sync, after being merged.
Why are we doing this? This will be a step towards simplifying our
`Dockerfile` with reduced layers and improved caching performance. The
image build time in CI should be faster once this is done.
So far, only calls that go to the Java backend, had the
`X-Content-Type-Options` header in the responses. This PR adds them to
all responses by
1. adding it to NGINX configuration.
2. removing it from Spring security's configuration, so we don't end up
with _two_ `X-Content-Type-Options` headers in the response.
---------
Co-authored-by: Nidhi <nidhi@appsmith.com>
When Appsmith is deployed with HTTPS, all HTTP requests are redirected
to HTTP. But this redirection response, is leaking the NGINX version in
the `Server` header. This PR fixes that.
The health endpoint in the Docker container's healthcheck command is
currently pointing to `/users/me`. But we have a designated endpoint for
this, at `/health`. This PR switches to using this endpoint instead.
This will allow us to
1. Bake different CS URLs for release and master builds.
2. Be resilient to the CS URL being set to empty string, as opposed to
not being set at all.
When `APPSMITH_CLOUD_SERVICES_BASE_URL` is an empty string, Spring's
interpolation in `application.properties` doesn't take the default
value. This PR sets up the `entrypoint.sh` script so that it'll `unset`
this env variable, when it's empty.
1. Changing the Maps API Key doesn't need restart anymore.
2. The `isRestartRequired` field in the response of updating env
settings, was being ignored. The client owns the decision of when to
restart (which is correct), so removed this from the server.
3. Write Maps API Key to the database, in the tenant configuration.
4. The Settings page for Maps Ke gets the current value from
`/tenant/current` response, and not `/admin/env`.
5. Removed `APPSMITH_GOOGLE_MAPS_API_KEY` from `/admin/env` response.
6. Tests.
DO NOT MERGE. Please only review/approve. This is expected to break EE
once it goes there, which I intend to solve alongside merging this.
Changing the Maps API Key will update it both in the tenant config in
the database, as well as in the `docker.env` file. This is predominantly
for backwards compatibility, and phased rollout. As part of a separate
PR, we'll have a migration that proactively copies the env variable
value to the database, and comment out the value in the `docker.env`
file. Then we can stop updating the `docker.env` file as well.
## New
## Old
---------
Co-authored-by: Ankita Kinger <ankita@appsmith.com>
We've stopped [building the slim
images](https://github.com/appsmithorg/appsmith/pull/25219) now and so
the instructions to run server locally, which rely on the slim server
image, need to be updated to use the newer `appsmith-ce` image.
This PR updates the instructions to use this.
## Description
- The default request timeout on nginx is set to 60s. This feat allows
users to configure custom nginx timeout value specified via the
Environment variable `APPSMITH_SERVER_TIMEOUT`
- Solves https://github.com/appsmithorg/appsmith/issues/14535
## Testing
#### How Has This Been Tested?
- [x] Manual
- [ ] Jest
- [ ] Cypress
---------
Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
This will complete:
[23522](https://github.com/appsmithorg/appsmith/issues/23522)
Currently, when we deploy appsmith on cloudrun we see this error. This
commit will fix this issue
```
2023-06-14 12:32:10.513 IST
Mounting File Sytem
2023-06-14 12:32:10.521 IST
mount.nfs: mount point /appsmith-stacks does not exist
2023-06-14 12:32:10.533 IST
Container called exit(32).
```
Fixes: [24013](https://github.com/appsmithorg/appsmith/issues/24013)
The issue started with the introduction of adding of startup-events.
Because of the ipv6 header
```
server {
listen 80 default_server;
listen [::]:80 default_server;
}
```
present in the
`/etc/nginx/sites-enabled/default` and calling nginx
[here](https://github.com/appsmithorg/appsmith/blob/release/deploy/docker/entrypoint.sh#L392).
With this change we are removing ```listen [::]:80 default_server;```
from the file before starting nginx.
---------
Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
- Removed event listener python script, removed supervisor conf for the
same.
- Added new python script to handle the starting page init.
- Python script will be called via a shell script with a failsafe
removal of the loading page.
- The above shell script will be run as a child of the run java script
Tests:
The child script exits after execution, and does not create zombie
process even when the run java script is called multiple times with
restarts.
Verified exceptions and failsafe manually by introducing syntax errors
and uninstalling requests module.
---------
Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
## Description
> Add a TL;DR when description is extra long (helps content team)
>
> Please include a summary of the changes and which issue has been
fixed. Please also include relevant motivation
> and context. List any dependencies that are required for this change
>
> Links to Notion, Figma or any other documents that might be relevant
to the PR
>
>
#### PR fixes following issue(s)
Fixes # (issue number)
> if no issue exists, please create an issue and ask the maintainers
about this first
>
>
#### Media
> A video or a GIF is preferred. when using Loom, don’t embed because it
looks like it’s a GIF. instead, just link to the video
>
>
#### Type of change
> Please delete options that are not relevant.
- Bug fix (non-breaking change which fixes an issue)
- New feature (non-breaking change which adds functionality)
- Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- Chore (housekeeping or task changes that don't impact user perception)
- This change requires a documentation update
>
>
>
## Testing
>
#### How Has This Been Tested?
> Please describe the tests that you ran to verify your changes. Also
list any relevant details for your test configuration.
> Delete anything that is not relevant
- [ ] Manual
- [ ] Jest
- [ ] Cypress
>
>
#### Test Plan
> Add Testsmith test cases links that relate to this PR
>
>
#### Issues raised during DP testing
> Link issues raised during DP testing for better visiblity and tracking
(copy link from comments dropped on this PR)
>
>
>
## Checklist:
#### Dev activity
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] PR is being merged under a feature flag
#### QA activity:
- [ ] [Speedbreak
features](https://github.com/appsmithorg/TestSmith/wiki/Test-plan-implementation#speedbreaker-features-to-consider-for-every-change)
have been covered
- [ ] Test plan covers all impacted features and [areas of
interest](https://github.com/appsmithorg/TestSmith/wiki/Guidelines-for-test-plans/_edit#areas-of-interest)
- [ ] Test plan has been peer reviewed by project stakeholders and other
QA members
- [ ] Manually tested functionality on DP
- [ ] We had an implementation alignment call with stakeholders post QA
Round 2
- [ ] Cypress test cases have been added and approved by SDET/manual QA
- [ ] Added `Test Plan Approved` label after Cypress tests were reviewed
- [ ] Added `Test Plan Approved` label after JUnit tests were reviewed
---------
Co-authored-by: Trisha Anand <trisha@appsmith.com>
Co-authored-by: Nidhi <nidhi@appsmith.com>
Co-authored-by: Arpit Mohan <mohanarpit@users.noreply.github.com>
Co-authored-by: Sumit Kumar <sumit@appsmith.com>
Co-authored-by: ChandanBalajiBP <104058110+ChandanBalajiBP@users.noreply.github.com>
Co-authored-by: Aishwarya-U-R <91450662+Aishwarya-U-R@users.noreply.github.com>
Co-authored-by: balajisoundar <balaji@appsmith.com>
Co-authored-by: Aishwarya UR <aishwarya@appsmith.com>
Co-authored-by: sidhantgoel <sidhant@hexcod.in>
Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
Co-authored-by: Satish Gandham <satish.iitg@gmail.com>
Co-authored-by: Nilesh Sarupriya <nilesh@appsmith.com>
Co-authored-by: Nilesh Sarupriya <20905988+nsarupr@users.noreply.github.com>
Co-authored-by: nilansh <nilansh@appsmith.com>
Co-authored-by: Ayangade Adeoluwa <37867493+Irongade@users.noreply.github.com>
Co-authored-by: akash-codemonk <67054171+akash-codemonk@users.noreply.github.com>
Co-authored-by: Vijetha-Kaja <119562824+Vijetha-Kaja@users.noreply.github.com>
Co-authored-by: Manish Kumar <107841575+sondermanish@users.noreply.github.com>
Co-authored-by: Sumesh Pradhan <sumesh@appsmith.com>
Co-authored-by: Rishabh Rathod <rishabh.rathod@appsmith.com>
Co-authored-by: Shrikant Sharat Kandula <shrikantsharat.k@gmail.com>
Co-authored-by: Tanvi Bhakta <tanvi@appsmith.com>
Co-authored-by: Somangshu Goswami <somangshu.goswami1508@gmail.com>
Co-authored-by: Tolulope Adetula <31691737+Tooluloope@users.noreply.github.com>
Co-authored-by: Anagh Hegde <anagh@appsmith.com>
Co-authored-by: Saroj <43822041+sarojsarab@users.noreply.github.com>
Co-authored-by: Nayan <nayan@appsmith.com>
Co-authored-by: Arsalan <arsalanyaldram0211@outlook.com>
Co-authored-by: ashit-rath <ashit@appsmith.com>
Co-authored-by: Rimil Dey <rimil@appsmith.com>
Co-authored-by: Druthi Polisetty <druthi@appsmith.com>
Co-authored-by: Sangeeth Sivan <74818788+berzerkeer@users.noreply.github.com>
Co-authored-by: subratadeypappu <subrata@appsmith.com>
Co-authored-by: Ayush Pahwa <ayush@appsmith.com>
Co-authored-by: Rajat Agrawal <rajat@appsmith.com>
Co-authored-by: Pawan Kumar <pawan.stardust@gmail.com>
Co-authored-by: Abhijeet <41686026+abhvsn@users.noreply.github.com>
Co-authored-by: Aman Agarwal <aman@appsmith.com>
Co-authored-by: Parthvi Goswami <parthvigoswami@Parthvis-MacBook-Pro.local>
Co-authored-by: Anand Srinivasan <66776129+eco-monk@users.noreply.github.com>
Co-authored-by: Anand Srinivasan <anand.a.srinivasan@outlook.com>
Co-authored-by: Satish Gandham <hello@satishgandham.com>
Co-authored-by: Goutham Pratapa <goutham@appsmith.com>
Co-authored-by: manish kumar <manish@appsmith.com>
Co-authored-by: Dipyaman Biswas <dipyaman@appsmith.com>
Co-authored-by: Ayush Pahwa <ayushpahwa96@gmail.com>
Co-authored-by: Rohit Agarwal <rohit_agarwal@live.in>
Co-authored-by: Tanvi Bhakta <tanvibhakta@gmail.com>
Co-authored-by: Ankit Srivastava <67647761+ankitsrivas14@users.noreply.github.com>
Co-authored-by: tkAppsmith <131347120+tkAppsmith@users.noreply.github.com>
We replace env variables in `index.html`, `view.html` and `edit.html`,
just before start NGINX. But the `.gz` versions of these files don't
have these changes applied to them. This PR gzips those HTML files
again, after the substitutions are applied.
From
https://github.com/appsmithorg/appsmith/pull/23539#issuecomment-1554509537.
- Added cron task to cleanup log files older than 7 days to run weekly.
- Updated supervisor log rotation config to store only 2 backups for all
processes.
## Description
We are solving 2 problems with this PR:
1. For `Installation Setup Complete` event email domain hash was
incorrect and earlier we were hashing the instanceId which means if user
with same email tries to run 2 different instances hash was different.
2. `IP` is a reserved keyword for tracking events in Mixpanel though
this is allowed in Segment. Instead of showing the ip as is, Mixpanel
provides derived property. As we want derived props alongwith the ip
address we are sharing the IP address in separate keys.
Ref:
https://help.mixpanel.com/hc/en-us/articles/360001355266-Event-Properties
#### PR fixes following issue(s)
Fixes https://github.com/appsmithorg/cloud-services/issues/675
#### Type of change
- Bug fix (non-breaking change which fixes an issue)
#### How Has This Been Tested?
- [x] Manual
## Checklist:
#### Dev activity
- [x] My code follows the style guidelines of this project
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] PR is being merged under a feature flag
#### QA activity:
- [ ] [Speedbreak
features](https://github.com/appsmithorg/TestSmith/wiki/Test-plan-implementation#speedbreaker-features-to-consider-for-every-change)
have been covered
- [ ] Test plan covers all impacted features and [areas of
interest](https://github.com/appsmithorg/TestSmith/wiki/Guidelines-for-test-plans/_edit#areas-of-interest)
- [ ] Test plan has been peer reviewed by project stakeholders and other
QA members
- [ ] Manually tested functionality on DP
- [ ] We had an implementation alignment call with stakeholders post QA
Round 2
- [ ] Cypress test cases have been added and approved by SDET/manual QA
- [ ] Added `Test Plan Approved` label after Cypress tests were reviewed
- [ ] Added `Test Plan Approved` label after JUnit tests were reviewed
Because tracing is turned on in the `entrypoint.sh`, everything we
source also log all names and values, including the `docker.env` file.
This means all env values are printed to the logs.
This PR disables the tracing to fix this. It was enabled originally for
improved logging information for some of the issues we were facing at
the time, but if we still need such logging information, we need to log
those explicitly with `echo`, instead of using `xtrace`.
Client will get supported OAuth list from the tenant API, instead of
from injected env variables like `APPSMITH_OAUTH2_GOOGLE_CLIENT_ID`.
This is a step towards moving OAuth configuration out of env variables
completely, and into the backend database, so their configuration can be
tenant-wide, instead of instance-wide.
Event Listener Python script fixes:
- Added exception handling in health check request
- Updated backend health check point to
`http://localhost:8080/api/vi/health`
**Fixes:**
- Overwriting MongoDB env data on appsmithctl restore.
- appsmithctl backup does not include the MongoDB env vars.
**Features:**
- appsmithctl backup now does not stop backend and rts service during
backup operation.
_Note:-
It will be the responsibility of the Admin to make sure that a manual
backup is really a desired snapshot when done via
the appsmithctl backup util command
To ensure this, Admins may require to stop the backend and rts processes
in the container before a manual backup( We will need to document this)
[ In future we can add an appsmithctl cmd to put appsmith to maintenance
mode with a maintenance page ]_
Fixes # [21603](https://github.com/appsmithorg/appsmith/issues/21603)
We're already checking for the AVX instruction, and producing a helpful
message about it. But, we're doing that check _after_ we invoke the
`mongod` process, which fails on such processes, and the error message
is never shown.
This PR changes this, by doing the check before that `mongod` command is
run.
If the replicaSet check fails, we see the following error:
```
readWrite command not found
```
This is because of using the command interpolation backticks in the
double-quoted string.
## Description
This PR includes changes for self-hosted instances to utilize an
embedded postgres database for the Template's mockdb.
After this change, by default Appsmith will always run the embedded
Postgres, and connect to it instead of the mockdb cloud instance.
**Solves:**
Issue[#20107](https://github.com/appsmithorg/appsmith/issues/20107)
**Changes:**
- [x] Install Postgresql in the Docker container using Dockerfile :-
_Updated Docker file to install PostgreSQL v13_
- [x] Add provision to stop/disable Postgresql from booting based on an
environment variable :- _To disable embedded postgres, set the **ENV**
var `APPSMITH_ENABLE_EMBEDDED_DB` to `0`; Configured supervisord to
control the lifecycle of the postgres server ( Supervisor uses user
postgres to start the server since root is not allowed)_
- [x] Seed data into the Postgresql DB during first container startup.
This ensures that we don't have to ship the Docker
container with data & bloat it unnecessarily. :- _Curretly, the SQL dump
is baked into the image._
- [x] Persist the Postgresql data in /appsmith-stacks. This ensures that
the data is persisted across version upgrades. :- _The postgres uses the
file system at` /stacks/data/postgres/main` ; note: The file system
owner is user/group postgres_
- [x] Update the mock db endpoint in the product to use the local DB by
default instead of the hosted DB for self-hosted instances. The cloud
instance should still use the hosted DB :- _The embedded postgres
authentication is set to the type `trust`, allowing the existing
template mockdb meta data to be used without any update to the existing
templates mockdb endpoint and credentials_
**Note: There is an additional table `mydb` added to the mockdb for
quickly verifying the db source is the embedded postgres, since the
cloud mockdb does not have that table.**
---------
Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
When services within the fat container try to communicate with each
other, like backend to RTS etc., if they use the loopback address of
IPv4, `127.0.0.1`, it works. But if they use the loopback address of
IPv6, `::1`, it fails because the NGINX inside the fat container isn't
set to bind to IPv6.
This PR fixes this.
In EE, we attempt to make connections to Keycloak without setting the
hostname on the `WebClient`. This picks up the hostname of `::1` on
systems with IPv6 as default, and so the communication between backend
and Keycloak fails.
This is affecting users on ECS Fargate, for example.
Appsmith supports running with a HTTP proxy, that can be configured with
`HTTP_PROXY` or `HTTPS_PROXY` env variables. Like this:
```sh
HTTP_PROXY=http://myproxy:8080
HTTPS_PROXY=http://myproxy:8443
```
However, this proxy support didn't support authentication. This PR
implements that. Now, proxy with authentication can be configured like
this:
```sh
HTTP_PROXY=http://user:password@myproxy:8080
HTTPS_PROXY=http://user:password@myproxy:8080
```
This is not syntax or standards invented by Appsmith. This is the
standard way proxy is usually configured.
Fixes#16330🍰
