## Description
**Problem statement:**
In one of the instances where dynamoDB had close to 200 odd queries on
top of it, queries were timing out after a month or so, this happened 3
times with one of the users, where after a month, queries would suddenly
start timing out. One of the hunches was that since we create
dynamoDBclient object only once during datasource creation, it could be
getting stale thus resulting in query timeouts and with that hunch, we
decided to add a fix where we close the old connections regularly after
using for a day.
**Solution:**
This PR configures connection time to live property for DynamoDB plugin.
This is configured to be 1 day which means after 1 day, active/idle
connections will be closed. This is done to ensure that we don't face
issues if the connection becomes stale. Default value of connection time
to live is 0 which means the connection stays open for infinite amount
of time.
To ensure that we do not cause any issues due to introduction of this
property, we decided to add this behind a feature flag. All the feature
flagging related code exists at the `appsmith-server` module but
dynamoDb code exists in `appsmith-plugins -> dynamoDB` module, so in
order to provide feature flagging, ideally we should make feature
flagging service available at `appsmith-interface` module to ensure that
it can get consumed across different modules but that's a [known tech
debt](https://github.com/appsmithorg/appsmith/issues/39425) which we
should resolve soon, but in the meantime to expedite the fix, I have
passed in flag details to respective datasourceCreate functionality.
datasourceCreate also gets called from testDatasource, so feature
flagging has been implemented there as well.
At some point in time, based on whether this potential fix works or not,
we should plan to remove the feature flagging code, task for that added
[here](https://github.com/appsmithorg/appsmith/issues/39426)
**References:**
-
https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/http/apache/ApacheHttpClient.Builder.html#connectionTimeToLive(java.time.Duration)
-
https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/http/SdkHttpConfigurationOption.html#CONNECTION_TIME_TO_LIVE
- https://github.com/aws/aws-sdk-java/issues/2788
Fixes #[`Issue
Number`](https://github.com/appsmithorg/appsmith-ee/issues/6030)
_or_
Fixes `Issue URL`
> [!WARNING]
> _If no issue exists, please create an issue first, and check with the
maintainers if the issue is valid._
## Automation
/ok-to-test tags="@tag.Datasource"
### 🔍 Cypress test results
<!-- This is an auto-generated comment: Cypress test results -->
> [!TIP]
> 🟢🟢🟢 All cypress tests have passed! 🎉🎉🎉
> Workflow run:
<https://github.com/appsmithorg/appsmith/actions/runs/13515219987>
> Commit: 953e570b2e153087fa7b27c8c24cb50f74333159
> <a
href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=13515219987&attempt=1"
target="_blank">Cypress dashboard</a>.
> Tags: `@tag.Datasource`
> Spec:
> <hr>Tue, 25 Feb 2025 07:21:25 UTC
<!-- end of auto-generated comment: Cypress test results -->
## Communication
Should the DevRel and Marketing teams inform users about this change?
- [ ] Yes
- [x] No
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced DynamoDB plugin connection management with configurable
connection time-to-live
- Added Apache HTTP client support for improved network connection
handling
- **Dependencies**
- Updated Apache client dependency to version 2.15.3
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: “sneha122” <“sneha@appsmith.com”>
Solves a single thing in the build configurations, resulting in a few
wins.
1. Reduced number of warnings in the output.
1. In release branch:
```
mvn clean package -DskipTests | grep --fixed-strings --count '[WARNING]'
3233
```
1. In this PR's branch:
```
mvn clean package -DskipTests | grep --fixed-strings --count '[WARNING]'
172
```
2. All uber-jar files are shaded twice, currently. Once with the default
execution of `maven-shade-plugin`, and again with the `shade-plugin-jar`
execution in these `pom.xml` files. This is double-work, and is the
cause of most of the warnings we see.
1. This `shade-plugin-jar` was added to have the plugin information
included in the `/META-INF/MANIFEST.MF` file, since we can't configure
the default execution of the shade plugin (it comes to us from Spring
Boot).
2. Instead, we switch to configuring plugin information in a
`/plugin.properties` file.
3. Previously, we used `/plugin.properities` for plugin information in
dev time, and `/META-INF/MANIFEST.MF` in production. This PR will change
it so that we use `/plugin.properties` all the time. We configure PF4J
with a custom plugin manager to achieve this.
3. Moved all `plugin.properties` into `src/main/resources`, so that they
land up in the root of the final jar files. But this means, during
development, loading the plugin fails since it looks for a
`plugin.properties` at the root of the plugin module, i.e., next to the
`src` folder.
1. For this, in the custom plugin manager class, we change where we look
for the `plugin.properties` file during development mode. In this mode,
we look at the `target/classes/plugin.properties` file, which is where
maven saves this file, taken from
`src/main/resources/plugin.properties`.
2. This also solves the duplication of the plugin properties that's
currently present, between `plugin.properties` and the `<properties>`
section of `pom.xml` files.
Here's the shade plugin's default execution and configuration, from
Spring Boot:
https://github.com/spring-projects/spring-boot/blob/v3.0.1/spring-boot-project/spring-boot-starters/spring-boot-starter-parent/build.gradle#L174.
This upgrade takes care of our move to JDK 17, Spring Boot 3.0.1 and a
few other security upgrades along the way.
Fixes#18993
TODO:
- [x] Check CI changes for Java 17
- [x] Check vulnerability report
- [x] Mongock needs an upgrade
- [x] Add JVM args at all possible places for exposing java.time module
- [x] Add type adapters everywhere / use the same config for type
adapters everywhere
Upgrades vulnerable dependencies in all plugins except for MySQL. That one is still failing and I'll fix it in a separate PR. Issue #14475
Co-authored-by: Nayan <nayan@appsmith.com>
Co-authored-by: Anagh Hegde <anagh@appsmith.com>
* Added DynamoPlugin to integrate with DynamoDB
* Connection management for DynamoPlugin
* Process action from a single JSON object as input
* Moved ListTables command to separate class
* Add missing UI definition files
* Use reflection to build request objects for AWS SDK
* All DynamoDB actions are now run purely by reflection
* Serialize responses to JSON
* Remove some unused/commented code and some refactorings
* Add non-working test with generic dynamodb container
* Added working test for ListTables action
* Added test for PutItem action
* Add tests for get and update actions
* Added comments and some refactoring
* Removed unused Command class.
* Added more tests for map to SDK object conversion
* Add templates
* Add option to set an endpoint override
* Fix Dynamo plugin tests when ~/.aws is missing
* Add documentation link for DynamoDB plugin
* Fix validation checks
* Remove debug log
* Added comments on expected query structure
* Fix incorrect error condition with incorrect request format
* Add comment for converting action to method name
* Error out if region is missing
* Remove unused throws declaration
* Use rich form for action configuration
* Removed templates
* Use PNG for logo images
* Use PNG logos for all plugins
* Avoid hard-coded field names
* Change logo to PNG only for ElasticSearch for now
* Wrap errors in AppsmithPluginException
* Typo
Co-authored-by: Trisha Anand <trisha@appsmith.com>
* Fix changeset order number
Co-authored-by: Trisha Anand <trisha@appsmith.com>
* Minor refactoring and fixed error messaging
Co-authored-by: Trisha Anand <trisha@appsmith.com>
