What are we solving here?
1. Installing Java in the `Dockerfile` by using Adoptium's package
repositories is fragile since they've started blocking some IP addresses
used by GitHub Actions runners. We see a message like this:
```
Failed to fetch
https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-17/temurin-17-jdk_17.0.8.1.0+1_amd64.deb
403 Forbidden [IP: 146.75.107.42 443]
```
We're seeing more and more cases of these and PRs are getting blocked.
2. Installing Java via `apt` also installs other packages like X11
libraries, that aren't really relevant to our usage of Java. Yet, these
packages are present in our Docker image, and are the source of several
CVEs to be reported by scanners on our Docker image.
3. This will give us control over trusted CA certificates, which we can
now perform under `$TMP`, which aligns with our move towards supporting
readonly root filesystem. Which is essentially not write to anything in
the Docker image at runtime, except for under `/tmp` and
`/appsmith-stacks`. This will help us move in that direction.
This removes ~70 medium/low severity CVEs reported on our Docker image,
by removing `build-essential` from being installed by default in the
Docker image.
We only need it when compiling Redis, which is needed on _some_ ARM
systems, that re configured with a page-size of greater than 4096. For
example, CentOS 8.
In the NGINX configuration we generate, we're redirecting _all_ HTTP
requests to HTTPS, when HTTPS is enabled. But the HTTP-01 challenge
works on port 80 and is getting redirected to 443.
This usually fine, as Let's Encrypt respects that redirect and completes
the challenge on port 443. But, if port 443 is blocked to outside
access, the cert renewal will fail. This PR fixes that.
Tested on a server with port 80 open and 443 closed to outside Internet.
Cert renewal fails without this PR's changes, and works with this PR's
changes.
This broke when we changed the way RTS stores version information. This
was never the right way to get the version in the `backup` command and
this PR fixes it, by getting the version from `info.json`.
Failure error:
```
Error: ENOENT: no such file or directory, open '/opt/appsmith/rts/version.js'
```
This fixes RTS build to use `esbuild`.
1. This means the whole `node_modules` won't need to be copied over to
the Docker image. There's unused insignifant _test_ files in there, that
don't add any value, but are causing irrelevant CVEs to be reported on
our Docker image. See example at
https://github.com/appsmithorg/appsmith-ee/pull/2349.
2. Much faster. Not that RTS build is our slow point, but still. Perhaps
we can move client to `esbuild` too. 🙂
## Why are we doing this?
The current method of loading RTS into the Docker image means that _all_
contents of _all_ dependencies are copied over. The whole
`node_modules`. But several of these packages include _test_ files too,
that aren't needed at runtime at all. One of such test files is creating
a false alert for a CVE on our Docker image. Has absolutely no relevance
and impact, but it's there.
To fix that, I [had to `rm -rf /opt/appsmith/rts/node_modules/*/test` in
the Docker
image](https://github.com/appsmithorg/appsmith-ee/pull/2349/files). This
felt very hacky, and very dirty. It felt like we're introducing more
debt and more duct tape around the current build process.
So, `esbuild`.
## Where is `esbuild` coming from?
We're using `esbuild` v0.18.20 only, while the latest is v0.19.3. We
need to update `design-system`'s storybook dependency, I think, to get a
more recent version of `esbuild`. I'm yet to figure this out and can use
some help. 🙂
When running with a custom `PORT` env variable, NGINX server will be
listening on this port. In the backend's startup script, `run-java.sh`,
we're checking for RTS being up or not, at `localhost`. So when the port
is not 80, then this will never succeed, because it'll be looking for
NGINX at the wrong port.
Instead, the fix here will make the backend startup script hit RTS
_directly_ on RTS server's own port, instead of going via NGINX. This
means it's independent of both the `PORT` env variable and the NGINX
server, and only dependent on RTS being up, which is really what we want
here.
When PostgreSQL starts, we see the following errors in the logs:
```
mkdir:
cannot create directory ‘/tmp/appsmith/postgres-stats’
: Permission denied
```
And then this over and over again:
```
postgres stdout | 2023-09-19 15:34:34.504 UTC [1759] LOG: could not open temporary statistics file "/tmp/appsmith/postgres-stats/global.tmp": No such file or directory
```
The problem is that in `postgres.conf`, we set `user=postgres`, which
doesn't have access to create things in `/tmp`.
This PR removes this configuration and lets the default be, which will
be a temp folders _under_ the data directory.
This is part of supporting running Appsmith with readonly root FS. This
moves the supervisord configuration, and runtime files, like the unix
socket file, and the PID file, to `$TMP`.
This is moving the cleanup script to delete old log files, from being a
cron job, to being a scheduled task in the backend server. Why?
1. We want to support running Appsmith with readonly root, which is a
request from security teams at enterprise companies.
2. Cron doesn't play nice. It wants to save a `.pid` file under
`/var/run`, so it fails to start when running with readonly root. This
is not configurable.
3. Since our use of cron is minimal only, we're moving away from it,
especially since the backend server is already capable of running
scheduled tasks.
4. This moves one job, there's still another. Based on experience from
this, we'll work on the other.
Another advantage to doing this is that since this job is now running in
the backend server, which has access to MongoDB and Redis, we can
coordinate when running as a cluster, that only _one_ backend is running
the clean up job. This is for much later though.
---------
Co-authored-by: Nidhi <nidhi@appsmith.com>
This is another step towards supporting running with readonly root FS,
and only making runtime changes in the container in `/tmp` or in
`/appsmith-stacks`, and nowhere else.
Move the files that are copied into the Docker image, into an `fs`
folder, that reflects the folder structure of that in the image. This
means two things right away:
1. A single `COPY` instruction in `Dockerfile` is enough to copy all the
files to their places.
2. The structure of files in the repo reflects that in the Docker image.
This makes working with the files/folders and troubleshooting with them
much easier.
❗ Note: **There's actually only 3 files changed, rest are just moved.**
