* POC : null value getting handled correctly
* Refactor code. Tested with and without quotes. This is working! Woohoo!
* Added support for mustache binding to include json objects and arrays
* Added test for smart substitution.
* Added setting to turn on/off smart substitutions in Rest API plugin
* Handling turning on smart json substitution
* Added error handling
* Added config to turn on smart substitution in JSON to make the tests pass now
* Adding a beta tag to the API setting.
* Spelling error resolved.
* Incorporated review comments.
* Catching JSON Parse exception in http call.
- Use dynamic input text for plugin editor forms so that the evaluated value of mustache expression is visible.
- Those fields which are meant to store Key only - i.e. they are not visible to the user and are only meant to provide key for key value pair in db have been skipped.
- Mongo plugin client driver does not return with exception upon first failure - instead it keeps retrying. Hence, adding timeout error to report failure before the client thread cancels due to delay in response.
- It seems that the mongdb connection string cannot be directly used for ping test, hence skipping it.
- Provide support for non Amazon S3 service providers like Upcloud, Digital Ocean, Wasabi, DreamObjects and any other S3 compliant storage service provider.
- Change plugin label from "Amazon S3" to "S3"
- A consensus has emerged that the utility of overriding endpoint is only for local development and testing. Hence, removing the fields from datasource form.
- provide a max length to keyvalue_array type. If the max length is set to 1, then the "+" button will not appear.
- front end changes are pending. A new github issue will be opened to track the same : #3315
* Added action request in S3 plugin.
* Added request data in Dynamo plugin
* Added request in Elastic Search.
* Request in Firestore.
* Request added in Mongo
* MsSQL request added.
* Added MySQL request.
* Added Postgres request.
* Added redis request.
* Redshift doen.
* Catching AppsmithPluginExceptions at the plugin level itself to ensure that the request gets passed on as part of the result.
* Fixed failing plugin test failures
* Fixed AmazonS3 test failures.
* WIP post analytics working
* WIP : Making the request data confirm to existing analytics request data format.
* Fixed the headers in the analytics body.
* Migrations S3, Postgres to the accepted format for request in analytics
* Updated S3 action to be the query
* Migration completed for all plugins for analytics.
* Removed the old analytics event. Now sending only the new one.
* Ensuring all data is captured in S3 plugin request in case of error.
* Minor editing of firestore to ensure that the errors get caught by the plugin itself.
* Fixed test cases in Amazon S3
* Incorporated review comments.
- Stop logging datasource argument error externally. The error described in the linked GitHub issue arises due to bad service account credentials provided as arg.
- Update error msg to make it clear.
- Add TC.
* check dynamodb connection validity using listTables cmd.
* add TC
* add encrypted logo to password field
* rectify the default value of region dropdown
* remove unused imports
* POC for Firestore server-side pagination
* Load pagination information values from action configuration
* Get execution information for Firestore plugin
* Working implementation of pagination for Firestore
* Add tests for next and previou page navigations
* Require ordering to be set when paginating
* Remove commented code
* Don't report error on bad configuration
* Error out on usupported operation
Co-authored-by: Trisha Anand <trisha@appsmith.com>
* Move constant indices to constant fields
* Use executeParameterized instead of execute
Co-authored-by: Trisha Anand <trisha@appsmith.com>
- add support to handle dnyamoDb Binary type
- transform raw response to show extracted response and raw response for:
GetItem
BatchGetItem
DeleteItem
PutItem
Scan
TransactGetItems
UpdateItem
- modify TC.
* Pushing minor editor form changes to ensure that prepared statement could be turned off.
* Code refactor to do variable substitution in PluginExecutor instead of action service.
* WIP : Prepared Statement handling in psql plugin
* WIP Prepared Statements.
* Working version of prepared statements
* Quote trimming added for post preparing sql statements. Now the unprepared statements and prepared statements do not require edits.
* Fixed existing test cases failing.
* Code formatting.
* Super minor code cleanup.
* Added migration for the existing postgres actions.
* Fixed failing test cases in ActionServiceTest.
* Minor change in the text for turning on and off prepared statements in the postgres query pane.
* Added test cases for prepared statement.
* Some minor comments for code readability
* Moved Prepared Statement setting from Action Configuration to Plugin Specified Templates since this setting does not make sense for all the DB plugins.
* Added function level comments
* Update app/server/appsmith-interfaces/src/main/java/com/appsmith/external/helpers/SqlStringUtils.java
Co-authored-by: Arpit Mohan <mohanarpit@users.noreply.github.com>
* Update app/server/appsmith-interfaces/src/main/java/com/appsmith/external/helpers/SqlStringUtils.java
Co-authored-by: Arpit Mohan <mohanarpit@users.noreply.github.com>
* Incorporated review comments.
* Fixed compile time error.
Co-authored-by: Arpit Mohan <mohanarpit@users.noreply.github.com>
1. Return url on file create.
2. Fix image/video upload/download.
3. Change error msg format to directly print error msg without adding any prefix sentence.
1. Move package appsmith-interfaces/src/main/java/com/appsmith/external/pluginExceptions-> appsmith-interfaces/src/main/java/com/appsmith/external/exceptions/pluginExceptions
2. Move enum AppsmithErrorAction from appsmith-server/src/main/java/com/appsmith/server/exceptions/AppsmithError.java -> appsmith-interfaces/src/main/java/com/appsmith/external/exceptions/AppsmithErrorAction.java, so that both plugin exceptions and server exceptions could use the same enum.
3. Log exception based on the error action defined for each exception.
* Increased logs to debug future connection leaks.
* Fetch a connection from the pool only if a query exists.
* Minor comment added.
* Minor rewrite
* Code formatting.
* Update app/server/appsmith-plugins/postgresPlugin/src/main/java/com/external/plugins/PostgresPlugin.java
Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
* Added hikari cp pool stats to get database structure function as well.
Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
* Increased logs to debug future connection leaks.
* Fetch a connection from the pool only if a query exists.
* Minor comment added.
* Minor rewrite
* Code formatting.
* Update app/server/appsmith-plugins/postgresPlugin/src/main/java/com/external/plugins/PostgresPlugin.java
Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
* Added hikari cp pool stats to get database structure function as well.
Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
1. Test that when a query is attempted to run on mongodb server but refused because of lack of authorization, then
also, it indicates a successful connection establishment.
1. Check if mysql plugin connection is stale. If so, then return stale connection exception which triggers a retry after
establishing a new connection.
2. Refactor execute method code to remove code duplication.
3. Add a new test case to check if stale connection gets detected.
1. Encode request body in REST API plugin when content header for URL encode is set.
2. Add a test case to test the encoding function.
3. Add a test case to check non-encoding (raw) function.
1. Add R2BC driver and remove JDBC driver usage in mysql plugin to make the plugin reactive.
2. Update dependencies in POM file.
3. Update mysql plugin to return result based on the type of sql query i.e select vs other queries. In case the queries are chained, then the last query is used for deciding the type.
4. Added a new TC for testing datasource and removed a non functional TC.
* Adding the skeleton for Firestore integration
* Adding the datasource & query editor forms
Also adding the database changelog for the firestore plugin
Commenting out the firestore.close() connection because that causes issues with multiple Firestore tenants running in the same JVM.
* Adding the code for fetching the structure of collections from Firestore
* Use single document path field for Firestore
* Fix potential NPE when datasource destroy timeouts
* Work in progress on collection level ops for Firestore
* Get documents in a collection now works
* Add collection level querying support
* Mild refactoring
* Fix NPE when some fields are missing
* Hide clientJSON as a password field for Firestore
* Make collection level querying reactive
* Make reactive
* Validate before connecting
* Add tests for all supported methods in Firestore
* Fix forms for Firestore with hidden fields
* Hide limit and order by fields when not needed
* Restore log entry deleted by mistake
* Use S3 URL for Firestore/Firebase logo
* Add comments detailing why some code is commented
* Make parsing JSON reactive and fix subscribe calls
* Fix reactive scheduler
Co-authored-by: Arpit Mohan <arpit@appsmith.com>
* Making postgres plugin blocking statements run on elastic threadpool.
* Moved the blocking calls to check for the connection validity, etc. also inside the Mono.fromCallable
1. Fix mongodb datasource cypress test failure.
2. Against certain mongodb instances, we expect to receive an exception of type "unauthorized" when testing datasource. Earlier, this case was caught and whitelisted until the recent change to replace mongodb infra with reactive mongodb infra.
Add support for reactive streams with mongodb. (fixes 1480)
1. Replace mongodb driver with reactive mongodb driver. Change APIs accordingly.
2. Use webflux + reactor framework to execute mongodb queries in event loop model.
3. Add test to test MongoPluginExecutor class' method "testDatasource".
* Adding host:port validation checks to ES plugin & Redis plugin
Also correcting the assertions in the RedisPluginTest so that the error message on test failure is accurate.
* Removing the endpoint validation from datasourceServiceImpl
Moving the endpoint validation to the plugin implementation classes because there are databases that require complete HTTP URL in their configuration. Checking for http/https at the platform level affects the UX for the user & plugin developer when using a new integration. Hence, all plugins must implement their own client/server validations in their own implementations
* Adding tests to assert hostname validation in mysql & postgres plugins
1. Fixing the build by excluding the slf4j-api from redis-plugin pom.xml
2. Adding the editor.json and form.json for the query pane & datasource pane.
3. Adding array handling in the Redis response by feeding all the output into a "result" key
* Added DynamoPlugin to integrate with DynamoDB
* Connection management for DynamoPlugin
* Process action from a single JSON object as input
* Moved ListTables command to separate class
* Add missing UI definition files
* Use reflection to build request objects for AWS SDK
* All DynamoDB actions are now run purely by reflection
* Serialize responses to JSON
* Remove some unused/commented code and some refactorings
* Add non-working test with generic dynamodb container
* Added working test for ListTables action
* Added test for PutItem action
* Add tests for get and update actions
* Added comments and some refactoring
* Removed unused Command class.
* Added more tests for map to SDK object conversion
* Add templates
* Add option to set an endpoint override
* Fix Dynamo plugin tests when ~/.aws is missing
* Add documentation link for DynamoDB plugin
* Fix validation checks
* Remove debug log
* Added comments on expected query structure
* Fix incorrect error condition with incorrect request format
* Add comment for converting action to method name
* Error out if region is missing
* Remove unused throws declaration
* Use rich form for action configuration
* Removed templates
* Use PNG for logo images
* Use PNG logos for all plugins
* Avoid hard-coded field names
* Change logo to PNG only for ElasticSearch for now
* Wrap errors in AppsmithPluginException
* Typo
Co-authored-by: Trisha Anand <trisha@appsmith.com>
* Fix changeset order number
Co-authored-by: Trisha Anand <trisha@appsmith.com>
* Minor refactoring and fixed error messaging
Co-authored-by: Trisha Anand <trisha@appsmith.com>
* add elasticSearchPlugin
* Fix container startup in tests
* Add elasticsearch dependency
* Get plugin to a base working state
* Add templates and tests for all Document APIs
* Add support for bulk queries
* Add test and template for bulk operations
* Use rich form for action configuration
* Add test API for ElasticSearch
* Use rich form's values for plugin execution
* Add authorization header support
* Fix tests after config object use changes
* Add test for bulk requests with nd-json body
* Remove templates and minor refactoring
* Fix potential NPE with null body
Co-authored-by: Trisha Anand <trisha@appsmith.com>
* Add datasource validation for endpoint
* Wrap errors in AppsmithPluginException
Co-authored-by: Suman Patra <spatra@akamai.com>
Co-authored-by: Trisha Anand <trisha@appsmith.com>
* Add option to override server timezone for MySQL datasources
* Added test for server timezone override for MySQL
* Fix serverTimezone property config
* Implement caching for datasource structure
* Avoid using final fields so spring-data can load objects
* Use type variable for getStructure method
* Initial version with base structure for MySQL plugin
* Add tests for MySQL plugin structure
* Add sorting for keys in Postgres structure as well
* Show "primary key" instead of just "primary"
* Refactor to reduce inline magic strings
* Add a connection type variable for Plugin implementations
This type variable is intended to represent the type of the
connection object, if any, that the plugin will use. This will
help make the implementations more robust by leveraging Java's
type checking instead of rudimentary type casts over the
connection objects.
* Fix missing typevar usage
* Base interface for getting datasource structure for databases
* Add keys and constraints to structure for Postgres datasources
* Use connection from datasource context for computing structure
* Refactor context retrying into a separate method
* Add base datastructures for templates in entity explorer
* Fix spring circular dependency
* Add test for postgres datasource structure
* Generate column names and sample values for INSERT query
* Add LIMIT clause to generated SELECT query
* Fix tests for generated SELECT query
* Minor refactoring
* Fix aliases not showing up for MySQL actions
* Fix date column display for MySQL actions
* Fix datetime columns for MySQL actions
* Add support for timestamp and year data types for MySQL actions
* Fix column label for MySQL plugin
* Add tests for MySQL temporal data types
* Add tests for alias columns in MySQL and Postgres
* Added information to setup domain
* Fixed incorrect var reference
* Updated documentation message
* Updated env template
* updated template
* removed debug echo
* Updated Script
* Updated Text
* Removed option to connect to external mongo for fresh installs
Exit script of docker desktop is not installed
* Updated docker installation explanation
* added a question to skip custom domain if user is installing locally
* Inverted script question and
* Removed question to determine local installation
* Updated Popup CTA to Modal
Updated query templates to contain bindings
Added a message to display on API / Query errors
* updated styles for evaluated value
updated styles for auto complete
* added spaces to questions
Co-authored-by: Nikhil Nandagopal <nikhil@appsmith.com>
Keeping the function to read plugin resource generic for any type of resource. Now the parent function can also conditionally decide which plugin would require to read which resources from the filesystem.
Co-authored-by: Arpit Mohan <arpit@appsmith.com>
* Handle stale database connection from datasources
* Fix potential secondary case of stale connection error
* Fix Postgres to MySQL
* Move validity check timeout to a constant field
* Add test for recovery when stale connection error is thrown
The plugin class loader from Pf4J looks for jars under target/lib
to use to load any classes that are missing target/classes. This
PR adds the `maven-dependency-plugin` to copy the jars to this
folder automatically so the classes are available during
development.
This commit adds the Mysql plugin to the Appsmith server. We also add a migration to ensure that this plugin is installed by default for all existing organizations. The migration also adds the plugin details into the DB.
Also adding the test cases for mysql plugin.
Co-authored-by: Arpit Mohan <arpit@appsmith.com>
Co-authored-by: Hetu Nandu <hetu@appsmith.com>
Co-authored-by: Arpit Mohan <me@arpitmohan.com>
Co-authored-by: Nupur Singhal <nupursinghal@Nupurs-MacBook-Air.local>
* Dates from Postgres queries now show up as ISO date strings
* Use constant for date column name in PostgresPlugin
* Fix formats for more date-time types on Postgres
* Add support for time and timestamp data types in Postgres plugin
* Add support for timestamptz column data type in Postgres plugin
* Add support for interval data type in PostgresPlugin
* Move application configuration to be loaded from environment variables
* Remove unused sentry.properties
* Make missing value sentinel a constant and ignore all *.env files
* Removed now-used ACL properties
* Prefix RapidAPI environment variable with APPSMITH_
* Fix application properties not being loaded into static fields
* Remove application-test.properties file
* Add required env variables for test in GitHub
* Quote URLs for MongoDB and Redis in test config
* Change RAPIDAPI to RAPID_API in environment variable names
* Source .env file in the root of repo in start script
Add server-side templates support for plugins
Two main themes in this MR:
1. Add support for saving query templates as resource files inside a plugin. This has been done for both Postgres and MongoDB plugins, based on current template contents.
2. Loading of form JSON is now reactive. The blocking call has been moved inside a `Mono.fromSupplier`.
3. The loading of from JSON as well as the templates is cached and are loaded on-demand. This means that the templates are loaded once for a plugin through the lifetime of the server process, and that they are loaded only at the first time they are needed.
4. If loading of these resources fails, we try again when they are needed the next time.
See merge request theappsmith/internal-tools-server!385
This is to simplify the query pane on the frontend client. The client doesn't need to have separate interfaces for sql and non-sql plugins. All queries will be sent to the server in the form of a String that is parsed in different formats based on the plugin.
Also adding test cases for PostgresPlugin. Used TestContainers to simulate the postgres db in Docker inside the Java test itself. Very useful.
This is because the client sends us the data for form-data in the form of a list of Property fields with keys and values. This is consistent with how the client stores & renders headers & query parameters as well.
For MediaType application/x-www-form-urlencoded we will use the bodyFormData field.
2. Add to Page changes : Documentation object has been added in Action to handle the extra Template documentation for actions that have been imported from 3p marketplace
3. Added basic structure for rapid api plugin by copy pasting the rest api plugin
2. Add to Page changes : Documentation object has been added in Action to handle the extra Template documentation for actions that have been imported from 3p marketplace
3. Added basic structure for rapid api plugin by copy pasting the rest api plugin
We are moving to a multi-module structure so that different parts of the codebase can be exposed to the public while others can remain private. Using pf4j for plugin framework.
Also adding a build script `build.sh` which compiles the code and creates the `dist` folder for distribution purposes. Now we can build the code via
```
$ ./build.sh -DskipTests
```
