## Description
> [!TIP]
> _Add a TL;DR when the description is longer than 500 words or
extremely technical (helps the content, marketing, and DevRel team)._
>
> _Please also include relevant motivation and context. List any
dependencies that are required for this change. Add links to Notion,
Figma or any other documents that might be relevant to the PR._
Fixes #`Issue Number`
_or_
Fixes `Issue URL`
> [!WARNING]
> _If no issue exists, please create an issue first, and check with the
maintainers if the issue is valid._
## Automation
/ok-to-test tags="@tag.All"
### 🔍 Cypress test results
<!-- This is an auto-generated comment: Cypress test results -->
> [!TIP]
> 🟢🟢🟢 All cypress tests have passed! 🎉🎉🎉
> Workflow run:
<https://github.com/appsmithorg/appsmith/actions/runs/16343398654>
> Commit: f8257de8135f4243309143396eca2a81bdb6f2a3
> <a
href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=16343398654&attempt=1"
target="_blank">Cypress dashboard</a>.
> Tags: `@tag.All`
> Spec:
> <hr>Thu, 17 Jul 2025 12:14:40 UTC
<!-- end of auto-generated comment: Cypress test results -->
## Communication
Should the DevRel and Marketing teams inform users about this change?
- [ ] Yes
- [x] No
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Introduced a new annotation to streamline and secure Git-related
operations in application APIs.
* Added a robust workflow for handling Git operations with enhanced
concurrency control and error handling.
* Enabled in-memory Git storage mode for improved performance in certain
environments.
* Added support for executing Git operations via shell scripts,
including branch merging and repository management.
* **Improvements**
* Enhanced configuration flexibility for Git storage and Redis
integration.
* Improved error reporting with new, descriptive Git-related error
messages.
* Broadened environment file ignore patterns for better environment
management.
* **Bug Fixes**
* Improved handling of private key formats for Git authentication.
* **Documentation**
* Added detailed documentation and flow diagrams for new Git operation
workflows.
* **Chores**
* Updated build and test configurations to align with new Git storage
paths.
* Deprecated and bypassed certain Redis operations when using in-memory
Git storage.
* **Tests**
* Removed several outdated or redundant test cases related to
auto-commit and Git serialization.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Description
> [!TIP]
> _Add a TL;DR when the description is longer than 500 words or
extremely technical (helps the content, marketing, and DevRel team)._
>
> _Please also include relevant motivation and context. List any
dependencies that are required for this change. Add links to Notion,
Figma or any other documents that might be relevant to the PR._
Fixes #`Issue Number`
_or_
Fixes `Issue URL`
> [!WARNING]
> _If no issue exists, please create an issue first, and check with the
maintainers if the issue is valid._
## Automation
/ok-to-test tags="@tag.Git"
### 🔍 Cypress test results
<!-- This is an auto-generated comment: Cypress test results -->
> [!TIP]
> 🟢🟢🟢 All cypress tests have passed! 🎉🎉🎉
> Workflow run:
<https://github.com/appsmithorg/appsmith/actions/runs/14197451933>
> Commit: a3833fe0e894bcb155455947856e7de93bbb0640
> <a
href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=14197451933&attempt=1"
target="_blank">Cypress dashboard</a>.
> Tags: `@tag.Git`
> Spec:
> <hr>Tue, 01 Apr 2025 14:54:53 UTC
<!-- end of auto-generated comment: Cypress test results -->
## Communication
Should the DevRel and Marketing teams inform users about this change?
- [ ] Yes
- [ ] No
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced an enhanced Git operation that enables reliable repository
reset through a dedicated API endpoint.
- Enabled advanced reset options controllable via a new feature flag,
improving repository state management.
- **Chores**
- Upgraded Git-related dependencies and updated the container setup to
include Git, ensuring a consistent and robust environment.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Move the `appsmithctl` code to RTS.
RTS' own build system will build `appsmithctl` as well. We're adding two
command scripts, `ctl` and `appsmithctl` to `/opt/bin`, which will be
the entrypoints for this. The `appsmithctl` is now just an alias to the
much shorter and non-redundancy-inducing `ctl`.
We aren't migrating to TypeScript in this PR so we're ignoring the new
`ctl` folder in both `tsconfig.json` and `.eslintrc`. That's temporary,
the next PR will fix that.
## Automation
/test sanity
### 🔍 Cypress test results
<!-- This is an auto-generated comment: Cypress test results -->
> [!TIP]
> 🟢🟢🟢 All cypress tests have passed! 🎉🎉🎉
> Workflow run:
<https://github.com/appsmithorg/appsmith/actions/runs/11930931528>
> Commit: 90b5f97b801ac8d4b4b0126d85edff3dccc050bd
> <a
href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=11930931528&attempt=1"
target="_blank">Cypress dashboard</a>.
> Tags: `@tag.Sanity`
> Spec:
> <hr>Wed, 20 Nov 2024 10:36:02 UTC
<!-- end of auto-generated comment: Cypress test results -->
## Communication
Should the DevRel and Marketing teams inform users about this change?
- [ ] Yes
- [x] No
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Introduced the `appsmithctl` command for easier command execution.
- Updated build process to include additional entry points.
- **Bug Fixes**
- Streamlined Docker build process, enhancing efficiency and reducing
complexity.
- **Documentation**
- Added a new section in the README for `appsmithctl` command
description.
- **Chores**
- Updated dependencies in `package.json`.
- Removed obsolete files and workflows to simplify project structure.
- **Style**
- Added a new ESLint configuration for specific project needs.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Description
The `&&` syntax for `and`-ing conditions doesn't work with `[`
**command**, it only works with `[[` **expressions**. But we can't use
`[[` expressions, since this isn't bash, it's `/bin/sh`. We can't use
bash, since doing so is throwing up a whole lot of other errors that
I've parked for another day, several months ago.
Instead of `&&`, we have to use `-a` when using the `[` command.
Currently, when building the Docker image, we see the following error:
```
#8 [3/6] RUN <<END (if ! [ -f info.json ]; then...)
#8 0.142 /bin/sh: 6: [: missing ]
#8 0.142 /bin/sh: 6: -f: not found
```
It doesn't seem to have an impact, but it _is_ an error nonetheless.
We're also refactoring to not add the executable permission twice, which
is redundant. We're only doing it once now.
### 🔍 Cypress test results
<!-- This is an auto-generated comment: Cypress test results -->
> [!TIP]
> 🟢🟢🟢 All cypress tests have passed! 🎉🎉🎉
> Workflow run:
<https://github.com/appsmithorg/appsmith/actions/runs/11718707642>
> Commit: 6c739439fa181f41245294a46321544edf61e878
> <a
href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=11718707642&attempt=2"
target="_blank">Cypress dashboard</a>.
> Tags: `@tag.Sanity`
> Spec:
> <hr>Thu, 07 Nov 2024 09:18:50 UTC
<!-- end of auto-generated comment: Cypress test results -->
## Communication
Should the DevRel and Marketing teams inform users about this change?
- [ ] Yes
- [x] No
/test sanity
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Summary by CodeRabbit
- **Chores**
- Updated the Dockerfile to enhance file existence checks and manage
executable permissions for shell scripts.
- Removed unnecessary directory creation commands for `./editor` and
`./rts`.
- Maintained existing structure, including exposed ports and health
check commands.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Arpit Mohan <mohanarpit@users.noreply.github.com>
## Description
Some scanner tools like Syft and Grype are reporting a **scary** false
positive at GHSA-2jcg-qqmg-46q6, on the following file in the Docker
image:
```
/opt/appsmith/utils/node_modules/resolve/test/resolver/multirepo/package.json
```
The advisory itself isn't applicable to Appsmith, and this above package
is not used in the product at all. This PR deletes this `test` folder so
this false positive is immediately taken out.
Nevertheless, we shouldn't even have the `node_modules` folder in the
Docker image, and we should be "building" `appsmithctl` instead. That's
part of a larger effort to improve/fix `appsmithctl` and will be coming
up in future PRs.
<details><summary><b>The SBOM entry for the package in Syft’s
proprietary format</b></summary>
<pre>
{
"id": "8686a02f6819d5a1",
"name": "monorepo-symlink-test",
"version": "0.0.0",
"type": "npm",
"foundBy": "javascript-package-cataloger",
"locations": [
{
"path":
"/opt/appsmith/utils/node_modules/resolve/test/resolver/multirepo/package.json",
"layerID":
"sha256:02e68fb671fe8bc43f75862b43445160e17e3ec2f13f09bf346a65c66cd93557",
"accessPath":
"/opt/appsmith/utils/node_modules/resolve/test/resolver/multirepo/package.json",
"annotations": {
"evidence": "primary"
}
}
],
"licenses": [
{
"value": "MIT",
"spdxExpression": "MIT",
"type": "declared",
"urls": [],
"locations": [
{
"path":
"/opt/appsmith/utils/node_modules/resolve/test/resolver/multirepo/package.json",
"layerID":
"sha256:02e68fb671fe8bc43f75862b43445160e17e3ec2f13f09bf346a65c66cd93557",
"accessPath":
"/opt/appsmith/utils/node_modules/resolve/test/resolver/multirepo/package.json",
"annotations": {
"evidence": "primary"
}
}
]
}
],
"language": "javascript",
"cpes": [
{
"cpe":
"cpe:2.3🅰️monorepo-symlink-test:monorepo-symlink-test:0.0.0:*:*:*:*:*:*:*",
"source": "syft-generated"
},
{
"cpe":
"cpe:2.3🅰️monorepo-symlink-test:monorepo_symlink_test:0.0.0:*:*:*:*:*:*:*",
"source": "syft-generated"
},
{
"cpe":
"cpe:2.3🅰️monorepo_symlink_test:monorepo-symlink-test:0.0.0:*:*:*:*:*:*:*",
"source": "syft-generated"
},
{
"cpe":
"cpe:2.3🅰️monorepo_symlink_test:monorepo_symlink_test:0.0.0:*:*:*:*:*:*:*",
"source": "syft-generated"
},
{
"cpe":
"cpe:2.3🅰️monorepo-symlink:monorepo-symlink-test:0.0.0:*:*:*:*:*:*:*",
"source": "syft-generated"
},
{
"cpe":
"cpe:2.3🅰️monorepo-symlink:monorepo_symlink_test:0.0.0:*:*:*:*:*:*:*",
"source": "syft-generated"
},
{
"cpe":
"cpe:2.3🅰️monorepo_symlink:monorepo-symlink-test:0.0.0:*:*:*:*:*:*:*",
"source": "syft-generated"
},
{
"cpe":
"cpe:2.3🅰️monorepo_symlink:monorepo_symlink_test:0.0.0:*:*:*:*:*:*:*",
"source": "syft-generated"
},
{
"cpe": "cpe:2.3🅰️monorepo:monorepo-symlink-test:0.0.0:*:*:*:*:*:*:*",
"source": "syft-generated"
},
{
"cpe": "cpe:2.3🅰️monorepo:monorepo_symlink_test:0.0.0:*:*:*:*:*:*:*",
"source": "syft-generated"
}
],
"purl": "pkg:npm/monorepo-symlink-test@0.0.0",
"metadataType": "javascript-npm-package",
"metadata": {
"name": "monorepo-symlink-test",
"version": "0.0.0",
"author": "",
"homepage": "",
"description": "",
"url": "",
"private": true
}
}
</pre>
</details>
Reported by a user.
⚠️ There will be conflicts on sync. Please do not merge unless the
author of PR is available.
/test sanity
## Communication
Should the DevRel and Marketing teams inform users about this change?
- [ ] Yes
- [x] No
<!-- This is an auto-generated comment: Cypress test results -->
> [!TIP]
> 🟢🟢🟢 All cypress tests have passed! 🎉🎉🎉
> Workflow run:
<https://github.com/appsmithorg/appsmith/actions/runs/11715737322>
> Commit: 42aa69c3de26d105a4184164f2ac9d18adce9b88
> <a
href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=11715737322&attempt=1"
target="_blank">Cypress dashboard</a>.
> Tags: `@tag.Sanity`
> Spec:
> <hr>Thu, 07 Nov 2024 03:26:39 UTC
<!-- end of auto-generated comment: Cypress test results -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Chores**
- Enhanced the Dockerfile for improved build process and error handling.
- Streamlined npm package installation and organized script execution
for better readability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Description
PR to make the release tag adaptable to work with both MongoDB and
PostgreSQL uris.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new script to automate the preparation of server
artifacts, improving the build process.
- Added SQL files to the indentation configuration for consistent code
formatting.
- **Improvements**
- Enhanced error handling in the Docker build process to ensure
essential files are present before execution.
- Updated service configuration logic to prevent misconfiguration based
on the environment.
- Added a new job step in the build workflow to prepare server artifacts
after the build process.
- Implemented conditional logic in the run script to dynamically adapt
to different database configurations.
- **Bug Fixes**
- Adjusted the initialization process to focus on MongoDB, improving
reliability in various environments.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
/test Sanity
### 🔍 Cypress test results
<!-- This is an auto-generated comment: Cypress test results -->
> [!TIP]
> 🟢🟢🟢 All cypress tests have passed! 🎉🎉🎉
> Workflow run:
<https://github.com/appsmithorg/appsmith/actions/runs/10940528231>
> Commit: 32731e8a93a25e5c9456eb89daca2d8bf327c012
> <a
href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=10940528231&attempt=1"
target="_blank">Cypress dashboard</a>.
> Tags: `@tag.Sanity`
> Spec:
> <hr>Thu, 19 Sep 2024 12:21:54 UTC
<!-- end of auto-generated comment: Cypress test results -->
## Communication
Should the DevRel and Marketing teams inform users about this change?
- [ ] Yes
- [x] No
The `tlog` command isn't predictably getting the executable permission,
so we're explicitly setting that up in `Dockerfile` here.
As well as in the setup script for Cypress, if we get a 502 even after
90 seconds, we print the logs of the `appsmith` container, so we're not
guessing the causes. This should provide us more information next time
we see such flakiness.
**/test sanity**
<!-- This is an auto-generated comment: Cypress test results -->
> [!TIP]
> 🟢🟢🟢 All cypress tests have passed! 🎉🎉🎉
> Workflow run:
<https://github.com/appsmithorg/appsmith/actions/runs/9658565348>
> Commit: 96c777dd9a1bd8c28477bf1dcd1d4748ced0022e
> <a
href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=9658565348&attempt=1"
target="_blank">Cypress dashboard</a>.
> Tags: `@tag.Sanity`
<!-- end of auto-generated comment: Cypress test results -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Chores**
- Updated Dockerfile to ensure custom command-scripts in `/opt/bin/`
have executable permissions.
- Enhanced `setup-test-ci.sh` script to output `appsmith` logs when the
server connection fails.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The microservices that run inside the Appsmith container, trust each
other, and may expose sensitive API endpoints to other internal
microservices. These sensitive APIs aren't accessible by outside the
Appsmith container, protected by Caddy's routing.
This means that the backend server's ability to make user-configured
HTTP requests, can lead to SSRFs to such sensitive API calls, if it's
allowed to call APIs on localhost.
In other words, Caddy establishes a trust boundary that protects these
internal APIs from outside the container. But we lack such a trust
boundary for the backend's plugins (API plugin, Elasticsearch plugin,
etc.). This PR solves that.
In this PR, we block both IPv4 and IPv6 loopback addresses.
No additional changes needed on EE, no conflicts, and all unit and
Cypress tests pass.
**/test all**
<!-- This is an auto-generated comment: Cypress test results -->
> [!TIP]
> 🟢🟢🟢 All cypress tests have passed! 🎉🎉🎉
> Workflow run:
<https://github.com/appsmithorg/appsmith/actions/runs/9590358198>
> Commit: 5445c70aa873942c3edae9fbfcc57a6d2554b815
> <a
href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=9590358198&attempt=2"
target="_blank">Cypress dashboard</a>.
> Tags: ``
<!-- end of auto-generated comment: Cypress test results -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Improved handling of disallowed hosts by dynamically computing based
on environment variables, offering more flexibility and control.
- **Refactor**
- Enhanced the `makeWebClient()` method to use a more efficient approach
for creating WebClient objects with custom configurations.
- **Chores**
- Added an `ENV` declaration for `IN_DOCKER` in the Dockerfile to better
manage Docker-specific configurations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Logs messages from `entrypoint.sh` and the other `run-*.sh` scripts
don't show timestamp today, and its getting hard to see the order of
things in the logs, especially between different processes.
This PR adds a command `tlog` to print logs with UTC timestamp prefixed.
We're only using it in `entrypoint.sh` now, but follow-up PR(s) will add
it to the other `run-*` scripts as well.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Chores**
- Updated Dockerfile to include `/opt/bin` in the `PATH` and modified
permissions settings for executable files.
- **Refactor**
- Enhanced logging in the entrypoint script by replacing `echo`
statements with `tlog` for better clarity and debugging.
- **New Features**
- Introduced `tlog`, a new shell script for consistent and timestamped
logging.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This PR replaces NGINX and Certbot with Caddy.
1. Auto-HTTPS when custom domain is set, is handled by Caddy.
2. If past certs exist, that were provisioned by Certbot in older
Appsmith versions, we configure Caddy to make use of them. But this only
applies if the certs aren't already expired. If they're expired, point 1
applies.
3. If custom certs are provided in `ssl` folder, Caddy will be
configured to use them.
4. Incoming `Forwarded` header is not passed to any reverse proxies. So
redirect URL is correctly computed on Google Cloud Run.
5. All other route configurations are exactly as they are in NGINX
today.
Caddy configuration file is generated in the `caddy-reconfigure.mjs`
script, which will also reload Caddy with the new configuration.
Running an Appsmith as a non-root user:
```sh
docker run --name appsmith --user 70:70
```
The `70:70` figures are the UID and GID respectively. It can mostly be
any number, safe to user figures are 70 to 79, or anything above 200 and
below 65000. The important bit, is that it shouldn't change on restart
or manual updates etc.
No product functionality should be affected when running as a non-root
user.
This will use the Appsmith base image which contains all the downloads
needed, so the image build in daily CI should be much more reliable, and
quite likely much faster.
All workflows have already been updated to use the BASE build argument
to set the base image.
Once this is merged, building the Docker image will fail if `BASE` build
argument is not passed. This is that the base is set explicitly
everywhere to not cause any confusion.
The deleted content has been moved to
8d34a2ac28/deploy/docker/base.dockerfile,
which builds the `base-*:*` images.
What are we solving here?
1. Installing Java in the `Dockerfile` by using Adoptium's package
repositories is fragile since they've started blocking some IP addresses
used by GitHub Actions runners. We see a message like this:
```
Failed to fetch
https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-17/temurin-17-jdk_17.0.8.1.0+1_amd64.deb
403 Forbidden [IP: 146.75.107.42 443]
```
We're seeing more and more cases of these and PRs are getting blocked.
2. Installing Java via `apt` also installs other packages like X11
libraries, that aren't really relevant to our usage of Java. Yet, these
packages are present in our Docker image, and are the source of several
CVEs to be reported by scanners on our Docker image.
3. This will give us control over trusted CA certificates, which we can
now perform under `$TMP`, which aligns with our move towards supporting
readonly root filesystem. Which is essentially not write to anything in
the Docker image at runtime, except for under `/tmp` and
`/appsmith-stacks`. This will help us move in that direction.
This removes ~70 medium/low severity CVEs reported on our Docker image,
by removing `build-essential` from being installed by default in the
Docker image.
We only need it when compiling Redis, which is needed on _some_ ARM
systems, that re configured with a page-size of greater than 4096. For
example, CentOS 8.
This fixes RTS build to use `esbuild`.
1. This means the whole `node_modules` won't need to be copied over to
the Docker image. There's unused insignifant _test_ files in there, that
don't add any value, but are causing irrelevant CVEs to be reported on
our Docker image. See example at
https://github.com/appsmithorg/appsmith-ee/pull/2349.
2. Much faster. Not that RTS build is our slow point, but still. Perhaps
we can move client to `esbuild` too. 🙂
## Why are we doing this?
The current method of loading RTS into the Docker image means that _all_
contents of _all_ dependencies are copied over. The whole
`node_modules`. But several of these packages include _test_ files too,
that aren't needed at runtime at all. One of such test files is creating
a false alert for a CVE on our Docker image. Has absolutely no relevance
and impact, but it's there.
To fix that, I [had to `rm -rf /opt/appsmith/rts/node_modules/*/test` in
the Docker
image](https://github.com/appsmithorg/appsmith-ee/pull/2349/files). This
felt very hacky, and very dirty. It felt like we're introducing more
debt and more duct tape around the current build process.
So, `esbuild`.
## Where is `esbuild` coming from?
We're using `esbuild` v0.18.20 only, while the latest is v0.19.3. We
need to update `design-system`'s storybook dependency, I think, to get a
more recent version of `esbuild`. I'm yet to figure this out and can use
some help. 🙂
This is another step towards supporting running with readonly root FS,
and only making runtime changes in the container in `/tmp` or in
`/appsmith-stacks`, and nowhere else.
Move the files that are copied into the Docker image, into an `fs`
folder, that reflects the folder structure of that in the image. This
means two things right away:
1. A single `COPY` instruction in `Dockerfile` is enough to copy all the
files to their places.
2. The structure of files in the repo reflects that in the Docker image.
This makes working with the files/folders and troubleshooting with them
much easier.
❗ Note: **There's actually only 3 files changed, rest are just moved.**
There's a few CVEs reported on the version of python-requests that's
being used in the image, which we don't need that badly anyway. This PR
removes `python-requests` altogather so we shouldn't see CVEs on it show
up on our image anymore.
Removes this `Server` header in all responses from Appsmith.
```
curl -sSI http://localhost | grep 'server:'
```
It currently shows this:
```
server: nginx
```
Update:
- Mobed jest to` devDependenies` of package manifest.
- Updated Dockerfile to install utils with `--only=prod` arg for npm
install cmds.
Tested manually
We're pinning MongoDB to 5.0.14, but can just as well run 5.0.19. This
PR makes sure we install the latest v5 of MongoDB, every time we build
the image.
We're currently running NodeJS v16, which is [end-of-support on
11-Sep-2023](https://endoflife.date/nodejs). This PR updates the NodeJS
used as runtime, for RTS and `appsmithctl`, in the Appsmith Docker
image.
We do not update the build-time NodeJS version, needed to build the
client code, and RTS. Depending on test results, we'll do it as part of
the same PR, or a separate one.
This adds a `/opt/appsmith/info.json` file to Docker images, with the
following sample content:
```json
{
"commitSha": "0521ba2c0d7a62cef3d4def66fc15b59cc34ceef",
"commitUrl": "0521ba2c0d",
"branch": "release",
"date": "2023-08-02T12:52:53+00:00",
"isCI": false
}
```
We're enabling this only for images built for DPs currently, and will
then extend to other workflows as well.
Notice that we copy `info.*json` instead of `info.json`. The reason is
so that the Docker build doesn't fail, even if the `info.json` file
doesn't exist. This lets us publish this to each workflow in turn,
slowly and carefully.
[Relevant Slack
conversation](https://theappsmith.slack.com/archives/C02MUD8DNUR/p1686197957141419).
This will allow us to
1. Bake different CS URLs for release and master builds.
2. Be resilient to the CS URL being set to empty string, as opposed to
not being set at all.
- Removed event listener python script, removed supervisor conf for the
same.
- Added new python script to handle the starting page init.
- Python script will be called via a shell script with a failsafe
removal of the loading page.
- The above shell script will be run as a child of the run java script
Tests:
The child script exits after execution, and does not create zombie
process even when the run java script is called multiple times with
restarts.
Verified exceptions and failsafe manually by introducing syntax errors
and uninstalling requests module.
---------
Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
## Description
1. Move everything related to client from app folder to client folder
(`.yarn`, `yarn.lock`, package.json, .gitignore)
2. Move `ast` and `rst` to client packages
3. Fix running scripts in packages
4. Add running unit tests in packages in CI
TODO: It is necessary to consider enabling the `nmHoistingLimits:
workspaces` option, since now all packages are hoisted to the root,
there may be issues with dependencies in workspaces. Also, there is a
possibility of implicit use of packages.
https://yarnpkg.com/configuration/yarnrc#nmHoistingLimits
#### PR fixes following issue(s)
Fixes#23333
#### Type of change
- Chore (housekeeping or task changes that don't impact user perception)
## Testing
#### How Has This Been Tested?
- [x] Manual
- [x] Jest
- [x] Cypress
## Checklist:
#### Dev activity
- [x] My code follows the style guidelines of this project
- [x] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] PR is being merged under a feature flag
Co-authored-by: Valera Melnikov <melnikov.vv@greendatasoft.ru>
## Description
This PR includes changes for self-hosted instances to utilize an
embedded postgres database for the Template's mockdb.
After this change, by default Appsmith will always run the embedded
Postgres, and connect to it instead of the mockdb cloud instance.
**Solves:**
Issue[#20107](https://github.com/appsmithorg/appsmith/issues/20107)
**Changes:**
- [x] Install Postgresql in the Docker container using Dockerfile :-
_Updated Docker file to install PostgreSQL v13_
- [x] Add provision to stop/disable Postgresql from booting based on an
environment variable :- _To disable embedded postgres, set the **ENV**
var `APPSMITH_ENABLE_EMBEDDED_DB` to `0`; Configured supervisord to
control the lifecycle of the postgres server ( Supervisor uses user
postgres to start the server since root is not allowed)_
- [x] Seed data into the Postgresql DB during first container startup.
This ensures that we don't have to ship the Docker
container with data & bloat it unnecessarily. :- _Curretly, the SQL dump
is baked into the image._
- [x] Persist the Postgresql data in /appsmith-stacks. This ensures that
the data is persisted across version upgrades. :- _The postgres uses the
file system at` /stacks/data/postgres/main` ; note: The file system
owner is user/group postgres_
- [x] Update the mock db endpoint in the product to use the local DB by
default instead of the hosted DB for self-hosted instances. The cloud
instance should still use the hosted DB :- _The embedded postgres
authentication is set to the type `trust`, allowing the existing
template mockdb meta data to be used without any update to the existing
templates mockdb endpoint and credentials_
**Note: There is an additional table `mydb` added to the mockdb for
quickly verifying the db source is the embedded postgres, since the
cloud mockdb does not have that table.**
---------
Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
Fixes#19969.
The previous PR that fixed this was failing with the way we were
creating the initial user on MongoDB. This PR addresses that.
Co-authored-by: Aishwarya UR <aishwarya@appsmith.com>
This upgrade takes care of our move to JDK 17, Spring Boot 3.0.1 and a
few other security upgrades along the way.
Fixes#18993
TODO:
- [x] Check CI changes for Java 17
- [x] Check vulnerability report
- [x] Mongock needs an upgrade
- [x] Add JVM args at all possible places for exposing java.time module
- [x] Add type adapters everywhere / use the same config for type
adapters everywhere
Fixes: [19207](https://github.com/appsmithorg/appsmith/issues/19207)
**By default on start-up**
```
root@ab348246fff8:/appsmith-stacks/logs# head backend/backend-ab348246fff8.log
Waiting for RTS to start ...
RTS started.
Starting Backend server...
Load environment configuration
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/appsmith/backend/server.jar!/BOOT-INF/lib/logback-classic-1.2.11.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/appsmith/backend/server.jar!/BOOT-INF/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
[2022-12-26 11:39:34,962] - Starting ServerApplication vv1.8.14.1 using Java 11.0.17 on ab348246fff8 with PID 1543 (/opt/appsmith/backend/server.jar started by root in /opt/appsmith/backend)
```
**Upon stopping RTS manually**
```
root@ab348246fff8:/appsmith-stacks/logs# supervisorctl stop rts
rts: stopped
root@ab348246fff8:/appsmith-stacks/logs# supervisorctl status
backend RUNNING pid 1497, uptime 0:01:21
cron RUNNING pid 1505, uptime 0:01:21
editor RUNNING pid 1502, uptime 0:01:21
mongodb RUNNING pid 1491, uptime 0:01:21
redis RUNNING pid 1490, uptime 0:01:21
rts STOPPED Dec 26 11:40 AM
stdout RUNNING pid 1489, uptime 0:01:21
```
**Restart Backend service to check if backend service starts**
```
root@ab348246fff8:/appsmith-stacks/logs# supervisorctl restart backend
backend: stopped
backend: started
root@ab348246fff8:/appsmith-stacks/logs# supervisorctl status
backend RUNNING pid 1802, uptime 0:00:12
cron RUNNING pid 1505, uptime 0:01:43
editor RUNNING pid 1502, uptime 0:01:43
mongodb RUNNING pid 1491, uptime 0:01:43
redis RUNNING pid 1490, uptime 0:01:43
rts STOPPED Dec 26 11:40 AM
stdout RUNNING pid 1489, uptime 0:01:43
root@ab348246fff8:/appsmith-stacks/logs# supervisorctl ^C
root@ab348246fff8:/appsmith-stacks/logs# tail -f backend/backend-ab348246fff8.log
Waiting for RTS to start ...
Waiting for RTS to start ...
Waiting for RTS to start ...
Waiting for RTS to start ...
Waiting for RTS to start ...
Waiting for RTS to start ...
Waiting for RTS to start ...
Waiting for RTS to start ...
Waiting for RTS to start ...
Waiting for RTS to start ...
^C
```
**Upon starting RTS the behaviour of backend immediately**
```
root@ab348246fff8:/appsmith-stacks/logs# supervisorctl start rts
rts: started
root@ab348246fff8:/appsmith-stacks/logs# tail -f backend/backend-ab348246fff8.log
Starting Backend server...
Load environment configuration
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/appsmith/backend/server.jar!/BOOT-INF/lib/logback-classic-1.2.11.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/appsmith/backend/server.jar!/BOOT-INF/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
[2022-12-26 11:41:33,796] - Starting ServerApplication vv1.8.14.1 using Java 11.0.17 on ab348246fff8 with PID 1968 (/opt/appsmith/backend/server.jar started by root in /opt/appsmith/backend)
[2022-12-26 11:41:33,799] - Running with Spring Boot v2.6.5, Spring v5.3.17
[2022-12-26 11:41:33,803] - No active profile set, falling back to 1 default profile: "default"
[2022-12-26 11:41:34,615] - Multiple Spring Data modules found, entering strict repository configuration mode!
[2022-12-26 11:41:34,616] - Bootstrapping Spring Data Reactive MongoDB repositories in DEFAULT mode.
[2022-12-26 11:41:34,859] - Finished Spring Data repository scanning in 239 ms. Found 55 Reactive MongoDB repository interfaces.
[2022-12-26 11:41:35,220] - Multiple Spring Data modules found, entering strict repository configuration mode!
[2022-12-26 11:41:35,221] - Bootstrapping Spring Data Redis repositories in DEFAULT mode.
```
