The layers in the Dockerfile that depend on downloading large files from
external sources, doesn't have to run every day, or at every PR. We
tried using Docker's caching configuration, but it's not as reliable as
we'd have liked.
A separate base image lends us much more control over the how long we
cache the downloaded files and how often we redo this.
This PR only _adds_ the base image. It doesn't change anything in the
build of the existing Docker image. That'll happen once we have the base
images for `release` and `master` already present on DockerHub.
What are we solving here?
1. Installing Java in the `Dockerfile` by using Adoptium's package
repositories is fragile since they've started blocking some IP addresses
used by GitHub Actions runners. We see a message like this:
```
Failed to fetch
https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-17/temurin-17-jdk_17.0.8.1.0+1_amd64.deb
403 Forbidden [IP: 146.75.107.42 443]
```
We're seeing more and more cases of these and PRs are getting blocked.
2. Installing Java via `apt` also installs other packages like X11
libraries, that aren't really relevant to our usage of Java. Yet, these
packages are present in our Docker image, and are the source of several
CVEs to be reported by scanners on our Docker image.
3. This will give us control over trusted CA certificates, which we can
now perform under `$TMP`, which aligns with our move towards supporting
readonly root filesystem. Which is essentially not write to anything in
the Docker image at runtime, except for under `/tmp` and
`/appsmith-stacks`. This will help us move in that direction.
This removes ~70 medium/low severity CVEs reported on our Docker image,
by removing `build-essential` from being installed by default in the
Docker image.
We only need it when compiling Redis, which is needed on _some_ ARM
systems, that re configured with a page-size of greater than 4096. For
example, CentOS 8.
In the NGINX configuration we generate, we're redirecting _all_ HTTP
requests to HTTPS, when HTTPS is enabled. But the HTTP-01 challenge
works on port 80 and is getting redirected to 443.
This usually fine, as Let's Encrypt respects that redirect and completes
the challenge on port 443. But, if port 443 is blocked to outside
access, the cert renewal will fail. This PR fixes that.
Tested on a server with port 80 open and 443 closed to outside Internet.
Cert renewal fails without this PR's changes, and works with this PR's
changes.
This broke when we changed the way RTS stores version information. This
was never the right way to get the version in the `backup` command and
this PR fixes it, by getting the version from `info.json`.
Failure error:
```
Error: ENOENT: no such file or directory, open '/opt/appsmith/rts/version.js'
```
This fixes RTS build to use `esbuild`.
1. This means the whole `node_modules` won't need to be copied over to
the Docker image. There's unused insignifant _test_ files in there, that
don't add any value, but are causing irrelevant CVEs to be reported on
our Docker image. See example at
https://github.com/appsmithorg/appsmith-ee/pull/2349.
2. Much faster. Not that RTS build is our slow point, but still. Perhaps
we can move client to `esbuild` too. 🙂
## Why are we doing this?
The current method of loading RTS into the Docker image means that _all_
contents of _all_ dependencies are copied over. The whole
`node_modules`. But several of these packages include _test_ files too,
that aren't needed at runtime at all. One of such test files is creating
a false alert for a CVE on our Docker image. Has absolutely no relevance
and impact, but it's there.
To fix that, I [had to `rm -rf /opt/appsmith/rts/node_modules/*/test` in
the Docker
image](https://github.com/appsmithorg/appsmith-ee/pull/2349/files). This
felt very hacky, and very dirty. It felt like we're introducing more
debt and more duct tape around the current build process.
So, `esbuild`.
## Where is `esbuild` coming from?
We're using `esbuild` v0.18.20 only, while the latest is v0.19.3. We
need to update `design-system`'s storybook dependency, I think, to get a
more recent version of `esbuild`. I'm yet to figure this out and can use
some help. 🙂
When running with a custom `PORT` env variable, NGINX server will be
listening on this port. In the backend's startup script, `run-java.sh`,
we're checking for RTS being up or not, at `localhost`. So when the port
is not 80, then this will never succeed, because it'll be looking for
NGINX at the wrong port.
Instead, the fix here will make the backend startup script hit RTS
_directly_ on RTS server's own port, instead of going via NGINX. This
means it's independent of both the `PORT` env variable and the NGINX
server, and only dependent on RTS being up, which is really what we want
here.
When PostgreSQL starts, we see the following errors in the logs:
```
mkdir:
cannot create directory ‘/tmp/appsmith/postgres-stats’
: Permission denied
```
And then this over and over again:
```
postgres stdout | 2023-09-19 15:34:34.504 UTC [1759] LOG: could not open temporary statistics file "/tmp/appsmith/postgres-stats/global.tmp": No such file or directory
```
The problem is that in `postgres.conf`, we set `user=postgres`, which
doesn't have access to create things in `/tmp`.
This PR removes this configuration and lets the default be, which will
be a temp folders _under_ the data directory.
This is part of supporting running Appsmith with readonly root FS. This
moves the supervisord configuration, and runtime files, like the unix
socket file, and the PID file, to `$TMP`.
This is moving the cleanup script to delete old log files, from being a
cron job, to being a scheduled task in the backend server. Why?
1. We want to support running Appsmith with readonly root, which is a
request from security teams at enterprise companies.
2. Cron doesn't play nice. It wants to save a `.pid` file under
`/var/run`, so it fails to start when running with readonly root. This
is not configurable.
3. Since our use of cron is minimal only, we're moving away from it,
especially since the backend server is already capable of running
scheduled tasks.
4. This moves one job, there's still another. Based on experience from
this, we'll work on the other.
Another advantage to doing this is that since this job is now running in
the backend server, which has access to MongoDB and Redis, we can
coordinate when running as a cluster, that only _one_ backend is running
the clean up job. This is for much later though.
---------
Co-authored-by: Nidhi <nidhi@appsmith.com>
This is another step towards supporting running with readonly root FS,
and only making runtime changes in the container in `/tmp` or in
`/appsmith-stacks`, and nowhere else.
Move the files that are copied into the Docker image, into an `fs`
folder, that reflects the folder structure of that in the image. This
means two things right away:
1. A single `COPY` instruction in `Dockerfile` is enough to copy all the
files to their places.
2. The structure of files in the repo reflects that in the Docker image.
This makes working with the files/folders and troubleshooting with them
much easier.
❗ Note: **There's actually only 3 files changed, rest are just moved.**
There's a few CVEs reported on the version of python-requests that's
being used in the image, which we don't need that badly anyway. This PR
removes `python-requests` altogather so we shouldn't see CVEs on it show
up on our image anymore.
Bumps [mongodb](https://github.com/mongodb/node-mongodb-native) from
5.7.0 to 5.8.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/mongodb/node-mongodb-native/releases">mongodb's
releases</a>.</em></p>
<blockquote>
<h2>v5.8.0</h2>
<h2><a
href="https://github.com/mongodb/node-mongodb-native/compare/v5.7.0...v5.8.0">5.8.0</a>
(2023-08-21)</h2>
<p>The MongoDB Node.js team is pleased to announce version 5.8.0 of the
<code>mongodb</code> package!</p>
<h2>Release Notes</h2>
<h3>The <code>AutoEncrypter</code> interface has been deprecated</h3>
<p>The <code>AutoEncrypter</code> interface was used internally but
accidentally made public in the 4.x version of the driver. It is now
deprecated and will be made internal in the next major release.</p>
<h3>Kerberos support for 1.x and 2.x</h3>
<p>Moves the kerberos dependency back to <code>^1.0.0 || ^2.0.0</code>
to indicate support for both 1.x and 2.x. Support for 1.x is removed in
6.0.</p>
<h3>Fixed accidental deprecation warning</h3>
<p>Because of internal options handling, a deprecation was emitted for
<code>tlsCertificateFile</code> when using
<code>tlsCertificateKeyFile</code>. That has been corrected.</p>
<h3>Remove credential availability on
<code>ConnectionPoolCreatedEvent</code></h3>
<p>In order to avoid mistakenly printing credentials the
<code>ConnectionPoolCreatedEvent</code> will replace the credentials
option with an empty object. The credentials are still accessble via
MongoClient options: <code>client.options.credentials</code>.</p>
<h3>Features</h3>
<ul>
<li><strong>NODE-5399:</strong> use mongodb-js/saslprep instead of
saslprep (<a
href="https://redirect.github.com/mongodb/node-mongodb-native/issues/3818">#3818</a>)
(<a
href="c0d3927379">c0d3927</a>)</li>
<li><strong>NODE-5429:</strong> deprecate the <code>AutoEncrypter</code>
interface (<a
href="https://redirect.github.com/mongodb/node-mongodb-native/issues/3764">#3764</a>)
(<a
href="9bb0d955e7">9bb0d95</a>)</li>
<li><strong>NODE-5465,NODE-5538:</strong> lower
<code>@aws-sdk/credential-providers</code> version to 3.188.0 and
<code>zstd</code> to <code>^1.0.0</code> (<a
href="https://redirect.github.com/mongodb/node-mongodb-native/issues/3821">#3821</a>)
(<a
href="39ff81d402">39ff81d</a>)</li>
</ul>
<h3>Bug Fixes</h3>
<ul>
<li><strong>NODE-5489:</strong> set kerberos compatibility to ^1.0.0 ||
^2.0.0 (<a
href="https://redirect.github.com/mongodb/node-mongodb-native/issues/3803">#3803</a>)
(<a
href="c3b35b321b">c3b35b3</a>)</li>
<li><strong>NODE-5495:</strong> do not emit deprecation warning when
tlsCertificateKeyFile is specified and tlsCertificateFile is not (<a
href="https://redirect.github.com/mongodb/node-mongodb-native/issues/3810">#3810</a>)
(<a
href="e81d4a2fc0">e81d4a2</a>)</li>
<li><strong>NODE-5537:</strong> remove credentials from
ConnectionPoolCreatedEvent options (<a
href="https://redirect.github.com/mongodb/node-mongodb-native/issues/3813">#3813</a>)
(<a
href="4cf1e96c68">4cf1e96</a>)</li>
</ul>
<h2>Documentation</h2>
<ul>
<li><a
href="https://docs.mongodb.com/drivers/node/current/">Reference</a></li>
<li><a
href="https://mongodb.github.io/node-mongodb-native/5.8/">API</a></li>
<li><a
href="https://github.com/mongodb/node-mongodb-native/blob/v5.8.0/HISTORY.md">Changelog</a></li>
</ul>
<p>We invite you to try the <code>mongodb</code> library immediately,
and report any issues to the <a
href="https://jira.mongodb.org/projects/NODE">NODE project</a>.</p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/mongodb/node-mongodb-native/blob/v5.8.0/HISTORY.md">mongodb's
changelog</a>.</em></p>
<blockquote>
<h2><a
href="https://github.com/mongodb/node-mongodb-native/compare/v5.7.0...v5.8.0">5.8.0</a>
(2023-08-21)</h2>
<h3>Features</h3>
<ul>
<li><strong>NODE-5399:</strong> use mongodb-js/saslprep instead of
saslprep (<a
href="https://redirect.github.com/mongodb/node-mongodb-native/issues/3818">#3818</a>)
(<a
href="c0d3927379">c0d3927</a>)</li>
<li><strong>NODE-5429:</strong> deprecate the <code>AutoEncrypter</code>
interface (<a
href="https://redirect.github.com/mongodb/node-mongodb-native/issues/3764">#3764</a>)
(<a
href="9bb0d955e7">9bb0d95</a>)</li>
<li><strong>NODE-5465,NODE-5538:</strong> lower
<code>@aws-sdk/credential-providers</code> version to 3.188.0 and
<code>zstd</code> to <code>^1.0.0</code> (<a
href="https://redirect.github.com/mongodb/node-mongodb-native/issues/3821">#3821</a>)
(<a
href="39ff81d402">39ff81d</a>)</li>
<li><strong>NODE-5489:</strong> update kerberos dependency (<a
href="8c25d6d0a7">8c25d6d</a>)</li>
</ul>
<h3>Bug Fixes</h3>
<ul>
<li><strong>NODE-5489:</strong> set kerberos compatibility to ^1.0.0 ||
^2.0.0 (<a
href="https://redirect.github.com/mongodb/node-mongodb-native/issues/3803">#3803</a>)
(<a
href="c3b35b321b">c3b35b3</a>)</li>
<li><strong>NODE-5495:</strong> do not emit deprecation warning when
tlsCertificateKeyFile is specified and tlsCertificateFile is not (<a
href="https://redirect.github.com/mongodb/node-mongodb-native/issues/3810">#3810</a>)
(<a
href="e81d4a2fc0">e81d4a2</a>)</li>
<li><strong>NODE-5537:</strong> remove credentials from
ConnectionPoolCreatedEvent options (<a
href="https://redirect.github.com/mongodb/node-mongodb-native/issues/3813">#3813</a>)
(<a
href="4cf1e96c68">4cf1e96</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="43673fa189"><code>43673fa</code></a>
chore(5.x): release 5.8.0 [skip-ci] (<a
href="https://redirect.github.com/mongodb/node-mongodb-native/issues/3825">#3825</a>)</li>
<li><a
href="4b2fc79525"><code>4b2fc79</code></a>
docs: fix cutoff sentence on CommandStartedEvent (<a
href="https://redirect.github.com/mongodb/node-mongodb-native/issues/3828">#3828</a>)</li>
<li><a
href="39ff81d402"><code>39ff81d</code></a>
feat(NODE-5465,NODE-5538): lower
<code>@aws-sdk/credential-providers</code> version to 3...</li>
<li><a
href="e1af3437c3"><code>e1af343</code></a>
chore: update release automation scripts 5.x (<a
href="https://redirect.github.com/mongodb/node-mongodb-native/issues/3823">#3823</a>)</li>
<li><a
href="c0d3927379"><code>c0d3927</code></a>
feat(NODE-5399): use mongodb-js/saslprep instead of saslprep (<a
href="https://redirect.github.com/mongodb/node-mongodb-native/issues/3818">#3818</a>)</li>
<li><a
href="4cf1e96c68"><code>4cf1e96</code></a>
fix(NODE-5537): remove credentials from ConnectionPoolCreatedEvent
options (#...</li>
<li><a
href="e81d4a2fc0"><code>e81d4a2</code></a>
fix(NODE-5495): do not emit deprecation warning when
tlsCertificateKeyFile is...</li>
<li><a
href="c3b35b321b"><code>c3b35b3</code></a>
fix(NODE-5489): set kerberos compatibility to ^1.0.0 || ^2.0.0 (<a
href="https://redirect.github.com/mongodb/node-mongodb-native/issues/3803">#3803</a>)</li>
<li><a
href="cc3069d836"><code>cc3069d</code></a>
Revert "feat(NODE-5489): update kerberos dependency"</li>
<li><a
href="8c25d6d0a7"><code>8c25d6d</code></a>
feat(NODE-5489): update kerberos dependency</li>
<li>Additional commits viewable in <a
href="https://github.com/mongodb/node-mongodb-native/compare/v5.7.0...v5.8.0">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/appsmithorg/appsmith/network/alerts).
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Removes this `Server` header in all responses from Appsmith.
```
curl -sSI http://localhost | grep 'server:'
```
It currently shows this:
```
server: nginx
```
Update:
- Mobed jest to` devDependenies` of package manifest.
- Updated Dockerfile to install utils with `--only=prod` arg for npm
install cmds.
Tested manually
Majority of the NGINX config is the same, for both HTTP and HTTPS.
Having two separate templates for them is making configuration changes
error-prone, where we often risk forgetting making the same change in
the other file.
This PR merges the two files into one, so the above risk isn't there. It
also makes it easier to experiment with the file while developing, since
we have to make every single change twice during development.
Note: This _will_ cause conflicts in sync, after being merged.
Why are we doing this? This will be a step towards simplifying our
`Dockerfile` with reduced layers and improved caching performance. The
image build time in CI should be faster once this is done.
So far, only calls that go to the Java backend, had the
`X-Content-Type-Options` header in the responses. This PR adds them to
all responses by
1. adding it to NGINX configuration.
2. removing it from Spring security's configuration, so we don't end up
with _two_ `X-Content-Type-Options` headers in the response.
---------
Co-authored-by: Nidhi <nidhi@appsmith.com>
When Appsmith is deployed with HTTPS, all HTTP requests are redirected
to HTTP. But this redirection response, is leaking the NGINX version in
the `Server` header. This PR fixes that.
The health endpoint in the Docker container's healthcheck command is
currently pointing to `/users/me`. But we have a designated endpoint for
this, at `/health`. This PR switches to using this endpoint instead.
This will allow us to
1. Bake different CS URLs for release and master builds.
2. Be resilient to the CS URL being set to empty string, as opposed to
not being set at all.
When `APPSMITH_CLOUD_SERVICES_BASE_URL` is an empty string, Spring's
interpolation in `application.properties` doesn't take the default
value. This PR sets up the `entrypoint.sh` script so that it'll `unset`
this env variable, when it's empty.
1. Changing the Maps API Key doesn't need restart anymore.
2. The `isRestartRequired` field in the response of updating env
settings, was being ignored. The client owns the decision of when to
restart (which is correct), so removed this from the server.
3. Write Maps API Key to the database, in the tenant configuration.
4. The Settings page for Maps Ke gets the current value from
`/tenant/current` response, and not `/admin/env`.
5. Removed `APPSMITH_GOOGLE_MAPS_API_KEY` from `/admin/env` response.
6. Tests.
DO NOT MERGE. Please only review/approve. This is expected to break EE
once it goes there, which I intend to solve alongside merging this.
Changing the Maps API Key will update it both in the tenant config in
the database, as well as in the `docker.env` file. This is predominantly
for backwards compatibility, and phased rollout. As part of a separate
PR, we'll have a migration that proactively copies the env variable
value to the database, and comment out the value in the `docker.env`
file. Then we can stop updating the `docker.env` file as well.
## New
## Old
---------
Co-authored-by: Ankita Kinger <ankita@appsmith.com>
We've stopped [building the slim
images](https://github.com/appsmithorg/appsmith/pull/25219) now and so
the instructions to run server locally, which rely on the slim server
image, need to be updated to use the newer `appsmith-ce` image.
This PR updates the instructions to use this.
## Description
- The default request timeout on nginx is set to 60s. This feat allows
users to configure custom nginx timeout value specified via the
Environment variable `APPSMITH_SERVER_TIMEOUT`
- Solves https://github.com/appsmithorg/appsmith/issues/14535
## Testing
#### How Has This Been Tested?
- [x] Manual
- [ ] Jest
- [ ] Cypress
---------
Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
This will complete:
[23522](https://github.com/appsmithorg/appsmith/issues/23522)
Currently, when we deploy appsmith on cloudrun we see this error. This
commit will fix this issue
```
2023-06-14 12:32:10.513 IST
Mounting File Sytem
2023-06-14 12:32:10.521 IST
mount.nfs: mount point /appsmith-stacks does not exist
2023-06-14 12:32:10.533 IST
Container called exit(32).
```
Fixes: [24013](https://github.com/appsmithorg/appsmith/issues/24013)
The issue started with the introduction of adding of startup-events.
Because of the ipv6 header
```
server {
listen 80 default_server;
listen [::]:80 default_server;
}
```
present in the
`/etc/nginx/sites-enabled/default` and calling nginx
[here](https://github.com/appsmithorg/appsmith/blob/release/deploy/docker/entrypoint.sh#L392).
With this change we are removing ```listen [::]:80 default_server;```
from the file before starting nginx.
---------
Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
