Fixes#36073
Hi @NilanshBansal
**Issue :**
**Missing Logging Implementation :**
- Without a logging implementation (such as SLF4J Simple or Logback) in
the project's classpath, the logging statements in the plugins cannot be
executed.
- As a result, no log output is being printed to the terminal.
**Solution :**
The solution is to add a logging implementation to the plugins parent
pom. In this case, you can add the slf4j-simple dependency to your
pom.xml file. This will provide a simple logging implementation that
will output log statements to the console.
```
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.36</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<version>1.7.36</version>
</dependency>
```
**Explanation:**
- slf4j-api provides the SLF4J API, which is the interface for logging.
provides the SLF4J API, which is the interface for logging.
- slf4j-simple provides a simple implementation of the SLF4J API, which
is responsible for actually printing the log messages to the console.
**Screenshots :**
Amazon S3 Plugin and Postgres Plugin
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Summary by CodeRabbit
- **New Features**
- Enhanced logging capabilities with the integration of SLF4J API and
SLF4J Simple implementations.
- **Improvements**
- Improved log management and output formatting for better monitoring
and debugging across various plugins, transitioning from standard output
to structured logging.
- Refined logging practices in multiple plugins to support better
maintainability and performance.
- Removed method for console logging from the Stopwatch class to
streamline logging practices.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Description
This PR migrates the plugins which were old DBForm config to UQI config.
Plugins that got affected,
1. Snowflake
2. ArangoDB
3. Oracle
4. PostgreSQL
5. MySQL
6. Elasticsearch
7. DynamoDB
8. Redis
9. MSSQL
10. Redshift
Fixes#35496Fixes#35497Fixes#35500Fixes#35487Fixes#35490Fixes#35491Fixes#35499Fixes#35501Fixes#35502Fixes#35506Fixes#35890
## Automation
/ok-to-test tags="@tag.Datasource"
### 🔍 Cypress test results
<!-- This is an auto-generated comment: Cypress test results -->
> [!TIP]
> 🟢🟢🟢 All cypress tests have passed! 🎉🎉🎉
> Workflow run:
<https://github.com/appsmithorg/appsmith/actions/runs/10897087023>
> Commit: e8fe475a48aac1e2e1f3184181ae0bed154313ba
> <a
href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=10897087023&attempt=1"
target="_blank">Cypress dashboard</a>.
> Tags: `@tag.Datasource`
> Spec:
> <hr>Tue, 17 Sep 2024 06:07:27 UTC
<!-- end of auto-generated comment: Cypress test results -->
## Communication
Should the DevRel and Marketing teams inform users about this change?
- [ ] Yes
- [ ] No
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced rendering logic in the Query Editor to support additional UI
component types.
- Improved handling of form evaluations and query configurations across
various plugins, including ArangoDB, DynamoDB, Elasticsearch, and
others.
- Introduced structured layouts for editor configurations, enhancing
clarity and usability in multiple database plugins.
- **Bug Fixes**
- Adjusted control flow in sagas to ensure proper handling of UI
component types, improving user interactions.
- **Chores**
- Refined JSON configurations across multiple plugins for better
organization and maintainability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Description
This PR adds the spotless formatter and validator for the JSON files
present in the project. If there is any invalid JSON file, the formatter
fails and logs the file and the error LOC to be fixed.
Since Spotless is already added to the pre-commit hook it also makes it
necessary to fix the JSON and then commit the changes.
Screenshot of the errors displayed for Invalid JSONs
### Why is this Important?
All of our datasource forms appear on the UI from the JSON configuration
files in the plugins. If an Invalid JSON is added, it can break the
datasource usage experience for the users.
One such instance happened in the past, where due to a JSON formatting
error, the users could not use Smart Substitution feature on production.
[Reference](https://theappsmith.slack.com/archives/C040LHZN03V/p1721124893238579)
Fixes#34969
## Automation
/ok-to-test tags="@tag.Sanity"
### 🔍 Cypress test results
<!-- This is an auto-generated comment: Cypress test results -->
> [!TIP]
> 🟢🟢🟢 All cypress tests have passed! 🎉🎉🎉
> Workflow run:
<https://github.com/appsmithorg/appsmith/actions/runs/10243164292>
> Commit: 08bc87acd2e44c4a2677d3eed1bad002f991050a
> <a
href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=10243164292&attempt=1"
target="_blank">Cypress dashboard</a>.
> Tags: `@tag.Sanity`
> Spec:
> <hr>Mon, 05 Aug 2024 05:50:07 UTC
<!-- end of auto-generated comment: Cypress test results -->
## Communication
Should the DevRel and Marketing teams inform users about this change?
- [ ] Yes
- [ ] No
## Description
This PR adds JUnit test cases for rate limit functionality implemented
for datasource test and datasource connections.
#### PR fixes following issue(s)
Fixes ##27742
#### Media
> A video or a GIF is preferred. when using Loom, don’t embed because it
looks like it’s a GIF. instead, just link to the video
>
>
#### Type of change
- Chore (housekeeping or task changes that don't impact user perception)
>
>
>
## Testing
>
#### How Has This Been Tested?
> Please describe the tests that you ran to verify your changes. Also
list any relevant details for your test configuration.
> Delete anything that is not relevant
- [ ] Manual
- [x] JUnit
- [ ] Jest
- [ ] Cypress
>
>
#### Test Plan
> Add Testsmith test cases links that relate to this PR
>
>
#### Issues raised during DP testing
> Link issues raised during DP testing for better visiblity and tracking
(copy link from comments dropped on this PR)
>
>
>
## Checklist:
#### Dev activity
- [x] My code follows the style guidelines of this project
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] PR is being merged under a feature flag
#### QA activity:
- [ ] [Speedbreak
features](https://github.com/appsmithorg/TestSmith/wiki/Guidelines-for-test-plans#speedbreakers-)
have been covered
- [ ] Test plan covers all impacted features and [areas of
interest](https://github.com/appsmithorg/TestSmith/wiki/Guidelines-for-test-plans#areas-of-interest-)
- [ ] Test plan has been peer reviewed by project stakeholders and other
QA members
- [ ] Manually tested functionality on DP
- [ ] We had an implementation alignment call with stakeholders post QA
Round 2
- [ ] Cypress test cases have been added and approved by SDET/manual QA
- [ ] Added `Test Plan Approved` label after Cypress tests were reviewed
- [ ] Added `Test Plan Approved` label after JUnit tests were reviewed
Co-authored-by: “sneha122” <“sneha@appsmith.com”>
## Description
This PR adds rate limiting functionality to datasource test API. The
rate limiting configuration is added in such a way that if test API
receives more than 3 failed authentication requests for the same host
name within 5 seconds, It will block this hostname for the next 5
minutes, so that brute force attack can be stopped.
Unit test for this will be covered in another PR. Refer #27742
Currently this PR covers for postgres only, will need to extend the
implementation for all plugins. Refer #27737
#### PR fixes following issue(s)
Fixes#27736, #27739, #27744
> if no issue exists, please create an issue and ask the maintainers
about this first
>
>
#### Media
> A video or a GIF is preferred. when using Loom, don’t embed because it
looks like it’s a GIF. instead, just link to the video
>
>
#### Type of change
- Chore (housekeeping or task changes that don't impact user perception)
>
>
>
## Testing
>
#### How Has This Been Tested?
> Please describe the tests that you ran to verify your changes. Also
list any relevant details for your test configuration.
> Delete anything that is not relevant
- [x] Manual
- [ ] JUnit
- [ ] Jest
- [ ] Cypress
>
>
#### Test Plan
> Add Testsmith test cases links that relate to this PR
>
>
#### Issues raised during DP testing
> Link issues raised during DP testing for better visiblity and tracking
(copy link from comments dropped on this PR)
>
>
>
## Checklist:
#### Dev activity
- [x] My code follows the style guidelines of this project
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] PR is being merged under a feature flag
#### QA activity:
- [ ] [Speedbreak
features](https://github.com/appsmithorg/TestSmith/wiki/Guidelines-for-test-plans#speedbreakers-)
have been covered
- [ ] Test plan covers all impacted features and [areas of
interest](https://github.com/appsmithorg/TestSmith/wiki/Guidelines-for-test-plans#areas-of-interest-)
- [ ] Test plan has been peer reviewed by project stakeholders and other
QA members
- [ ] Manually tested functionality on DP
- [ ] We had an implementation alignment call with stakeholders post QA
Round 2
- [ ] Cypress test cases have been added and approved by SDET/manual QA
- [ ] Added `Test Plan Approved` label after Cypress tests were reviewed
- [ ] Added `Test Plan Approved` label after JUnit tests were reviewed
---------
Co-authored-by: “sneha122” <“sneha@appsmith.com”>
## Description
- Add changes to address the`StaleConnection` exception caused by MySQL plugin.
- Update MySQL driver version.
- Other refactor changes not related to the main issue:
- Explicit empty constructor definition is replaced with Lombok annotation for all error messages class.
- A base class is created for plugin error messages class to store all common error messages.
- Fix Indentation.
## Description
This PR fixes :
- the functionality of placeholder for the key value array control which
was earlier accessing `placeholderText` key from a wrong path for host.
- Added placeholders for different datasources as well
Solves a single thing in the build configurations, resulting in a few
wins.
1. Reduced number of warnings in the output.
1. In release branch:
```
mvn clean package -DskipTests | grep --fixed-strings --count '[WARNING]'
3233
```
1. In this PR's branch:
```
mvn clean package -DskipTests | grep --fixed-strings --count '[WARNING]'
172
```
2. All uber-jar files are shaded twice, currently. Once with the default
execution of `maven-shade-plugin`, and again with the `shade-plugin-jar`
execution in these `pom.xml` files. This is double-work, and is the
cause of most of the warnings we see.
1. This `shade-plugin-jar` was added to have the plugin information
included in the `/META-INF/MANIFEST.MF` file, since we can't configure
the default execution of the shade plugin (it comes to us from Spring
Boot).
2. Instead, we switch to configuring plugin information in a
`/plugin.properties` file.
3. Previously, we used `/plugin.properities` for plugin information in
dev time, and `/META-INF/MANIFEST.MF` in production. This PR will change
it so that we use `/plugin.properties` all the time. We configure PF4J
with a custom plugin manager to achieve this.
3. Moved all `plugin.properties` into `src/main/resources`, so that they
land up in the root of the final jar files. But this means, during
development, loading the plugin fails since it looks for a
`plugin.properties` at the root of the plugin module, i.e., next to the
`src` folder.
1. For this, in the custom plugin manager class, we change where we look
for the `plugin.properties` file during development mode. In this mode,
we look at the `target/classes/plugin.properties` file, which is where
maven saves this file, taken from
`src/main/resources/plugin.properties`.
2. This also solves the duplication of the plugin properties that's
currently present, between `plugin.properties` and the `<properties>`
section of `pom.xml` files.
Here's the shade plugin's default execution and configuration, from
Spring Boot:
https://github.com/spring-projects/spring-boot/blob/v3.0.1/spring-boot-project/spring-boot-starters/spring-boot-starter-parent/build.gradle#L174.
## Description
This PR updates the error logs
- Establishing a consistent format for all error messages.
- Revising error titles and details for improved understanding.
- Compiling internal documentation of all error categories,
subcategories, and error descriptions.
Updated Error Interface:
https://www.notion.so/appsmith/Error-Interface-for-Plugin-Execution-Error-7b3f5323ba4c40bfad281ae717ccf79b
PRD:
https://www.notion.so/appsmith/PRD-Error-Handling-Framework-4ac9747057fd4105a9d52cb8b42f4452?pvs=4#008e9c79ff3c484abf0250a5416cf052
>TL;DR
Fixes #
Media
> A video or a GIF is preferred. when using Loom, don’t embed because it
looks like it’s a GIF. instead, just link to the video
## Type of change
- New feature (non-breaking change which adds functionality)
## How Has This Been Tested?
- Manual
- Jest
- Cypress
### Test Plan
### Issues raised during DP testing
## Checklist:
### Dev activity
- [x] My code follows the style guidelines of this project
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] PR is being merged under a feature flag
### QA activity:
- [ ] Test plan has been approved by relevant developers
- [ ] Test plan has been peer reviewed by QA
- [ ] Cypress test cases have been added and approved by either SDET or
manual QA
- [ ] Organized project review call with relevant stakeholders after
Round 1/2 of QA
- [ ] Added Test Plan Approved label after reveiwing all Cypress test
---------
Co-authored-by: subrata <subrata@appsmith.com>
* Changes to testDatasource interface method and archive flow
* Tests for plugin level testDatasource implementations
* Added test for refreshing cache on deleting datasource
* Modified warnings to errors in logs
* Fixed test
* Fixed test
## Description
This PR fixes an issue where a potentially malicious user can connect to disallowed hosts from the Elasticsearch plugin within Appsmith. This is because Elasticsearch client SDK is a HTTP interface underneath the hood.
## Type of change
- Bug fix (non-breaking change which fixes an issue)
## How Has This Been Tested?
- Junits for the following:
- create datasource with disallowed host
- validate datasource with disallowed host
- test datasource with disallowed host
## Checklist:
- [x] My code follows the style guidelines of this project
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my feature works
- [x] New and existing unit tests pass locally with my changes
* This commit changes two things:
* Pom.xml --> ES- Restclient version to 7.17.5 from 7.9.2
* ElasticSearchPlugin.java ---> changed the testDatasource method to expect a 200 instead of a 200 or 404 HTTP status code
This commit fixes a high-priority bug with github issue https://github.com/appsmithorg/appsmith/issues/14909
* it can now negotiate around a 403 forbidden error as it does try to send head only to user provided URI
This commit has been tested:
* manually for CRUD at local with ES version 7.9.2
* manually for CR at user provide aws instance running 7.9.3 on docker
* |
| BugFix:
|
| This commit changes two files:
|
| * ElasticSearchPlugin.java --> added feature to distinguish between unauthorized and not found datasources.
| * ElasticSearchPluginTest.java ---> added testcases to verify the unauthorized and not found test cases
|
| This commit adds features on top of previous commit https://github.com/appsmithorg/appsmith/issues/14909
| * this commit adds the feature of more readable error messages while testing the elasticsearch datasource. i.e wrong endpoint errors and unauthorized issues
|
| This commit has been tested:
| * Junit
| * manually
* removed leftover comments from the parent commit and updated response texts for better readability
* removed wildcard imports from appsmith-plugins/elasticSearchPlugin/src/test/java/com/external/plugins/ElasticSearchPluginTest.java
- return request parameters with type for debug tab.
- request params are stored in a RequestParamDTO object.
- a configProperty -> label map is cached in pluginService, which returns the param label as per the configProperty it is mapped to via editor.json file.
- This feature currently only works with prepared statements disabled.
* WIP : Type migration from String to Object for value
* Migrating config from string to boolean for prepared statement.
Handled error for already stored actions where PS config is stored as String and not Boolean.
- add title to action execution errors to improve user experience
- all errors in AppsmithPluginErrors
- action execution related errors in AppsmithError : NO_CONFIGURATION_FOUND_IN_DATASOURCE, INVALID_ACTION, INVALID_DATASOURCE, INVALID_DATASOURCE_CONFIGURATION - as these are the likely errors in an action execution flow (excluding plugin specific flow) that might result from faulty action / datasource configuration by a user.
- title is returned as part of ActionExecutionResult.
- title is set in ActionExecutionResult on failure during action execution.
- In response to review comment - added a new BaseException class, so that both AppsmithException and AppsmithPluginException extend this base class. Also, refactored code to introduce setErrorInfo() function in ActionExecutionResult class to set its attributes.
- Some unrelated cleanup:
- catch PoolInitializationException for Postgres plugin and return AppsmithPuginException
- catch MongoTimeoutException and return AppsmithPluginException
* WIP : enrichment of analytics event for execute.
* Plugin level : Catch all exceptions and set request in the result
Server level : Adding new fields to analytics : `isSuccessfulExecution`, `statusCode`, `timeElapsed`
* Dont catch StaleConnectionException. Server handles the same.
* Removed class specification for onErrorResume in plugins since its supposed to catch all errors.
- Use dynamic input text for plugin editor forms so that the evaluated value of mustache expression is visible.
- Those fields which are meant to store Key only - i.e. they are not visible to the user and are only meant to provide key for key value pair in db have been skipped.
* Added action request in S3 plugin.
* Added request data in Dynamo plugin
* Added request in Elastic Search.
* Request in Firestore.
* Request added in Mongo
* MsSQL request added.
* Added MySQL request.
* Added Postgres request.
* Added redis request.
* Redshift doen.
* Catching AppsmithPluginExceptions at the plugin level itself to ensure that the request gets passed on as part of the result.
* Fixed failing plugin test failures
* Fixed AmazonS3 test failures.
* WIP post analytics working
* WIP : Making the request data confirm to existing analytics request data format.
* Fixed the headers in the analytics body.
* Migrations S3, Postgres to the accepted format for request in analytics
* Updated S3 action to be the query
* Migration completed for all plugins for analytics.
* Removed the old analytics event. Now sending only the new one.
* Ensuring all data is captured in S3 plugin request in case of error.
* Minor editing of firestore to ensure that the errors get caught by the plugin itself.
* Fixed test cases in Amazon S3
* Incorporated review comments.
1. Move package appsmith-interfaces/src/main/java/com/appsmith/external/pluginExceptions-> appsmith-interfaces/src/main/java/com/appsmith/external/exceptions/pluginExceptions
2. Move enum AppsmithErrorAction from appsmith-server/src/main/java/com/appsmith/server/exceptions/AppsmithError.java -> appsmith-interfaces/src/main/java/com/appsmith/external/exceptions/AppsmithErrorAction.java, so that both plugin exceptions and server exceptions could use the same enum.
3. Log exception based on the error action defined for each exception.
* Adding host:port validation checks to ES plugin & Redis plugin
Also correcting the assertions in the RedisPluginTest so that the error message on test failure is accurate.
* Removing the endpoint validation from datasourceServiceImpl
Moving the endpoint validation to the plugin implementation classes because there are databases that require complete HTTP URL in their configuration. Checking for http/https at the platform level affects the UX for the user & plugin developer when using a new integration. Hence, all plugins must implement their own client/server validations in their own implementations
* Adding tests to assert hostname validation in mysql & postgres plugins
* add elasticSearchPlugin
* Fix container startup in tests
* Add elasticsearch dependency
* Get plugin to a base working state
* Add templates and tests for all Document APIs
* Add support for bulk queries
* Add test and template for bulk operations
* Use rich form for action configuration
* Add test API for ElasticSearch
* Use rich form's values for plugin execution
* Add authorization header support
* Fix tests after config object use changes
* Add test for bulk requests with nd-json body
* Remove templates and minor refactoring
* Fix potential NPE with null body
Co-authored-by: Trisha Anand <trisha@appsmith.com>
* Add datasource validation for endpoint
* Wrap errors in AppsmithPluginException
Co-authored-by: Suman Patra <spatra@akamai.com>
Co-authored-by: Trisha Anand <trisha@appsmith.com>
