The problem was that the pagination was designed & tested with only 2 pages. Firestore has a different function `limitToLast` which solves for previous page pagination commands.
* ArangoDB has two types of command - one that fetches a response and the other that updates a document. Currently, ArangoDB plugin treats all cmds like the one that returns a response, hence when update cmds are run then an empty response is returned to the client since there is nothing to fetch. This PR adds a check to fetch the response based on the type of cmd. Hence, for an update cmd the number of writes succeeded or failed would be returned as a response.
* Adding `APPSMITH_PLUGIN_MAX_RESPONSE_SIZE` which can be configured as an environment variable to increase plugin response size (currently only supported for Postgres Plugin)
* Updated error message + added the environment variable to the template docker env files for new installations
* Minor refactoring
* Partial code change to replace question mark with appsmith placeholder
* Working version
* Removing unnecessary code
* Added test case to assert that when evaluated value contains a "?", the replacements are still correct
* Added test case in Mongo Plugin as well for the same scenario
* Minor change in the language of the comment
* Added smart defaults for query in Count, Distinct & Find commands
* Added test cases for smart inputs for query and limit for find, and query for count and distinct
* This method removes the outermost quotes - single or double quotes - so that end users don't have to do it via javascript inside widget fields where they are meant to be bound.
* The input box label in the Firestore plugin action editor reads Document / Collection Path which is misleading since a document is nested inside a collection and not the other way round. Hence changing it to Collection / Document Path
* A fix had been introduced earlier (I had introduced it a few months back) to remove data type info keys from the raw response. However, the transformation was not getting applied to the Query action as the transformation was explicitly limited to few actions. This PR removes that check and allows the transformation for all actions as there does not seem to be any reason to withhold this transformation for other actions.
* The earlier fix also restricted this transformation to results that had specific keys. This PR removes this check as well as it does not seem to be required and further may lead to similar issues if any particular key gets missed.
* Some other redundant checks have been removed from the flow as the transformation function itself takes care of it as edge/base case.
* Earlier split method was used to segregate cmd and args which failed when multiple words inside a quoted string formed one argument. e.g. set key "my value" would produce set, key, "my, value" as the tokens when split method is used, which is not correct. This change introduces a regex that would create the following tokens: set, key, "my value"
* Added field to autogenerate values in DB column
* Added comment and refactor
* Autogenerate field update according to names instead of type for ArangoDB
* 1. For binary data in the response, base64 encoding the same before sending it out on the wire (instead of default conversion to string)
2. Added a new header which sets the data type of the response body.
* Minor code reformatting
* Fixed failing test cases
* The new header created should send the values in an array instead of a single value
* API to run DB query using plugin specified templates
* Included get spreadsheet metadata in get info method
* Added TCs for checking Spreadsheet info response
* Added error message for invalid datasources
* Authentication check for datasource modified to AuthenticationStatus field
* Adding support for explicit data type casting in postgres prepared statement
* Added text and int psql data types for support in explicit typecasting
* Documenting the code
The default charset is determined during virtual-machine startup and typically depends upon the locale and charset of the underlying operating system (#5788)
Co-authored-by: 王昆 <quincy@cloudtogo.cn>
* add support in Redis plugin to select database when creating db connection.
* refactor code to use URI string instead of constructor call, because of lack of appropriate constructor.
* add TCs.
* fix validate datasource.
* return hint message to user if the expected content type is json but the api response is not a valid json.
Co-authored-by: Apeksha Bhosale <7846888+ApekshaBhosale@users.noreply.github.com>
* Raw is now a command option instead of Form vs Raw
* Added database migration for migration raw input type to be of raw command type
* Fixed test case failure
* Minor cleanup
* add api key based and bearer token based authentication options to REST API datasource
* this change is ported to release from Feature: datasource changes for api key auth, bearer token auth, combine config. #4683
* Update commands have been merged into a single update command for both single and multi updates
* Added migrations to migrate any update one mongo form command to new update command
* Incorporated review comments
* POC for snowflake integration with Appsmith
This is just a rough version to see if the functionality works. Needs to be cleaned up & enhanced for production usage.
* Plugin with connection, monos, ds testing, validation, templates, structure and forms
* Better comments
* Default schema to public
* Changed logging destination, populated request in result and used thread group for datasource creation
Co-authored-by: Arpit Mohan <arpit@appsmith.com>
* Reduced verbosity of the command names for Insert, Find and Delete
* WIP : Smart substitution untested changes for mongo form
* Tested code for smart substitution working for mongo form fields
* Added test case for assert for smart substitution
* Correcting text for Mongo Form Insert command
* Getting Prepared Statement for Postgres, MySQL, and MS-SQL and Smart Substitution for Rest API plugins out of beta
* Fixed failing test case
* Fixed Ms-sql plugin test failure
* fix socket error
* fix default port
* remove invalid check for missing port, since default port would be supplied if user leaves the port field empty.
* Handling multipart form data in the CURL import flow
* Adding Cypress test for curl import with multipart/form-data
* Also fixing minor bug where the form values need not always be double-quoted
Because of the Java version not being explicitly set
in this plugin, the build of the whole project fails
sporadically in IntelliJ since it's trying to compile the
code as Java 8, which will obviously fail.
- return request parameters with type for debug tab.
- request params are stored in a RequestParamDTO object.
- a configProperty -> label map is cached in pluginService, which returns the param label as per the configProperty it is mapped to via editor.json file.
- This feature currently only works with prepared statements disabled.
* Initial scaffolding for comments CRUD APIs
* add actions
* add assets
* state management for existing comments and creating new
* add ui components
* add overlay comments wrapper to baseWidget
* add toggle comment mode button at editor header
* trigger tests
* Disallow commenting as someone else
* Add applicationId for comments
* lint
* Add overlay blacklist to prevent component interaction while adding comments
* Comment thread style updates
* Placeholder comment context menu
* Controlled comment thread visibility for making new comments visible by default
* Update comment type description
* Reset input on save
* Resolve comment thread button ui
* fix close on esc key, dont create new comment on outside click
* Submit on enter
* add emoji picker
* Attempt at adding a websocket server in Java
* CRUD APIs for comment threads
* Add API for getting all threads in application
* Move types to a separate file
* Initial commit for real time server (RTS)
* Add script to start RTS
* Fix position property
* Use create comment thread API
* Use add comment to thread API
* Add custom cursor
* Dispatch logout init on 401 errors
* Allow CORS for real time connection
* Add more logs to RTS
* Fix construction of MongoClient
* WIP: Real time comments
* Enable comments
* Minor updates
* Read backend API base URL from environment
* Escape to reset comments mode
* Set popover position as auto and boundary as scroll parent
* Disable warning
* Added permissions for comment threads
* Add resolved API for comment threads
* Migration to set commenting permission on existing apps
* Fix updates bringing the RTS down
* Show view latest button, scroll to bottom on creating a new comment
* Cleanup comment reducer
* Move to typescript for RTS
* Add missing server.ts and tsconfig files
* Resolve / unresolve comment
* Scaffold app comments
* Minor fixes: comment on top of all widgets, add toggle button at viewer header
* Reconnect socket on creating a new app, set connected status in store
* Retry socket connection flow
* Integration tests for comments with api mocks using msw
* Fix circular depependency
* rm file
* Minor cleanup and comments
* Minor refactors: move isScrolledToBottom to common hooks, decouple prevent interactions overlay from comments wrapper
* Use policies when pushing updates in RTS
* ENV var to set if comments are enabled
* Fix: check if editor/viewer is initialised before waiting for init action
* Add tests for comments reducer
* Revert "ENV var to set if comments are enabled"
This reverts commit 988efeaa69d378d943a387e1e73510334958adc5.
* Enable comments for users with appsmith email
* lint
* fix
* Try running a socket.io server inside backend
* Update comment reducer tests
* Init mentions within comments
* Fix comment thread updates with email rooms
* Minor fixes
* Refactors / review suggestions
* lint
* increase cache limit for builds
* Comment out tests for feature that's under development
* Add Dockerfile for RTS
* Fix policies missing for first comment in threads
* Use draftJS for comments input with mentions support
* fix fixtures
* Use thread's policies when querying for threads
* Update socket.io to v4
* Add support for richer body with mentions
* Update comment body type to RawDraftContentState
* fix stale method
* Fix mentions search
* Minor cleanups
* Comment context menu and thread UI updates
* revert: Scaffold app comments
* Yarn dependencies
* Delete comment using id api added
* Init app comments
* Add test for creating thread
* Api for delete comment with id
* Test comment creation response and policies
* Copy comment links
* Fix reset editor state
* Delete valid comment testcase added
* Delete comment TC : code refactor
* Don't allow creating comments with an empty body
* Pin comments WIP[]
* Ignore dependency-reduced-pom.xml files from VCS
* Cleanup of some dev-only files, for review
* Delete comment
* Update socket.io to v4 in RTS
* Pin and resolve comment thread object added in commentThread
* Pin and resolve comment thread object added in commentThread
* Update comment thread API
* Added creationTime and updationTime in comment thread response
* Added creationTime and updationTime in comment thread response
* Added human readable id to comment threads, fallback to username for null name in user document
* Refactor
* lint
* fix test, rm duplicate selector
* comment out saga used for dev
* CommentThread viewed status, username fallback for getName=null, username field added in pin & resolve status
* lint
* trigger tests
Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
Co-authored-by: Abhijeet <abhi.nagarnaik@gmail.com>
* Only consider headings till value grid size, but consider all values till this size
* Switched from null to blank for consistency
* Whoops, we don't do null row objects anymore
- return helpful error message when get structure fails for mongo db due to lack of read permission. The message returned is: Appsmith has failed to get database structure. Please provide read permission on the database to fix this.
- any get structure error is also generically being caught by the getStrucuture method in DatasourceStructureSolution.java
- error msg and status code is now returned as part of DatasourceStructure object.
- ErrorDTO is used to store error info.
- ErrorDTO definition is moved to appsmith-interfaces package so that it can be used all packages.
- Other changes:
- stop logging PLUGIN_GET_STRUCTURE_ERROR externally, as this error is mostly triggered by credentials error or -lack of read permission.
- start logging PLUGIN_GET_STRUCTURE_TIMEOUT_ERROR externally, because this is not a user error.
* WIP : Migration of data type for smart substitution configuration
* Fixed the migration code. Also reverted editor.jsons back to equality check after migrations of existing actions
* Only fetching actions for plugins which belong to supported plugins.
* Setting a default value of "false" for all the actions which don't have a valid configuration for smart substitution.
* Minor code formatting
* Minor language change in comment
* Removed debug logs
* cherry pick -make new
* revert to enable fix from release
* attempt to hook into existing datasource editor
* gSheets plugin skeleton from Rest API
* Changes for database migration
* fix for auth code
* separate it out
* action page loads!
* add to explorer
* create action from datasource
* Editor JSON WIP
* working query form
* Editor JSON WIP
* import to
* fix toast message
* redirect from datasource and editor pages
* fix onboarding
* fix imports and constants
* refactor form out
* refactor queryForm
* Merge branch 'release' into feature/google-sheets
* Merge branch 'release' into feature/google-sheets
* initial values from settings, editor and form
* Check
* remove dangling code around lightTheme
* Safety net
* remove class
* try mouseover solve
* force click
* changes from review
* fix action form name on import
* Merge branch 'release' into feature/google-sheets
* minor cleanup
* Merge branch 'release' into feature/google-sheets
* WIP
* Google sheets changes
* Merge conflicts
* Merging and fixes, needs refactoring
* Check
* Merge branch 'release' into feature/google-sheets
* Fixed tests
* Add cloud services env variable
* Clean up saga
* Clean up
* Refactoring
* Deleted svg file
* Minor fixes
* Modified design to allow behaviour in google sheets methods (#3486)
* Modified design to allow behaviour in google sheets methods
* Review changes
* Removed sysout
* Added handling of edge cases with table data
* Merge branch 'release' into feature/google-sheets
* Fixes
* Fixes
* Added validations
* Improved tests
* Removed extraneous injected bean
* Review changes
* Fixed bug with method
* Changes to Google sheets plugin's request and response structures (#3692)
* Method changes
* Removed logging
* Renaming options
* Reverting pom version
* Modified type of collection variables, fixed errors
* Converted row offset field to one that supports dynamic bindings
* Review changes
* List SAAS plugin type actions under lightning menu apis (#3820)
* list saas plugin type actions under lightning menu apis
* combine saas plugin type actions in the other sub menu of lightning menu
Co-authored-by: Hetu Nandu <hetunandu@gmail.com>
* Fix merge issues
* Prettified query editor and a few fixes w/ ux
* Test fixes
* Reformatting request
* code for REST added (#3876)
Co-authored-by: hetunandu <hetu@appsmith.com>
* Renamed body to row object
* Renamed placeholder for range
* Renamed range heading
* Modifications to handle range semantics
* Use spreadsheet Url instead of id
* Ordering of methods
* Removed logging
* Add tests for Dynamic text controls
* Add tests for url helpers
* Fix coverage config
* Nevermind
* Interface changes
* There is no body here
* Yay to hints
* Delete row field is separately handled as row index
* placeholder support (#4001)
* Fixed tests, typos and creating new sheets with random rows
* Switched to using 'rowIndex' throughout
* binding path added for query input field (#4016)
* - Fixed QA bugs (#4032)
- Split delete sheet into two
- Removed dynamic query input types from hidden keys
* Proper exceptions
* Removed extra logging
* Throw exception if update method does not match any of the columns
* Same for bulk update
* Zero-indexed delete row
* I'm a space bound rocket ship
* Logic to register installations with cs (#4062)
* Logic to register installations with cs
* Clean up
* Casting to string
* Checking to see if this makes the test pass
* Added an extra null check
Co-authored-by: Piyush <piyush@codeitout.com>
Co-authored-by: hetunandu <hetu@appsmith.com>
Co-authored-by: Hetu Nandu <hetunandu@gmail.com>
Co-authored-by: Apeksha Bhosale <7846888+ApekshaBhosale@users.noreply.github.com>
* WIP : Type migration from String to Object for value
* Migrating config from string to boolean for prepared statement.
Handled error for already stored actions where PS config is stored as String and not Boolean.
* Added dependencies to be added to dependencyMap on client to Postgres, MSSQL, MySQL and Mongo plugins
* Added Dependency config for API
* Fixed the test case
Maven 3.8 now blocks HTTP repositories and only allows HTTPS. However, the version redshift jdbc dependency we are using connects to a separate repository that runs on HTTP. The latest version has switched to using HTTPS instead.
So this PR upgrades redshift dependency from version 2.0.0.1 to 2.0.0.4.
Ref: https://maven.apache.org/docs/3.8.1/release-notes.html#cve-2021-26291
- Return data type list along with action execution result so that the data can be displayed in the correct format.
- Handles these three data types for now : table, json, raw.
- If a plugin has already assigned the data type, then common handler is not used.
- Mongo plugin assigns the data types in the plugin specific flow i.e. does not use the common data type parse / assign method.
- To address review comments:
- added a new enum ActionResultDataType for data types. It is consumed by ParsedDataType. A list of ParsedDataType is returned by ActionExecutionResult
- parsed data in parallel using streams.parallel().
- add title to action execution errors to improve user experience
- all errors in AppsmithPluginErrors
- action execution related errors in AppsmithError : NO_CONFIGURATION_FOUND_IN_DATASOURCE, INVALID_ACTION, INVALID_DATASOURCE, INVALID_DATASOURCE_CONFIGURATION - as these are the likely errors in an action execution flow (excluding plugin specific flow) that might result from faulty action / datasource configuration by a user.
- title is returned as part of ActionExecutionResult.
- title is set in ActionExecutionResult on failure during action execution.
- In response to review comment - added a new BaseException class, so that both AppsmithException and AppsmithPluginException extend this base class. Also, refactored code to introduce setErrorInfo() function in ActionExecutionResult class to set its attributes.
- Some unrelated cleanup:
- catch PoolInitializationException for Postgres plugin and return AppsmithPuginException
- catch MongoTimeoutException and return AppsmithPluginException
* Feature : JSON Smart Substitution in Mongo Plugin
* Added BSON data type for Mongo substitution. Added test case
* Minor comment added
* Minor variable re-naming and correcting failing test cases.
* REST API : Escaping special characters in string before smart substitution
* Incorporated review comment
- Allow users to define key path for FieldValue.delete() and FieldValue.serverTimestamp() values.
- delete() value is only valid for update operation, as Firestore does not seem to support it for any other ops.
- serverTimestamp() is valid for all operations excluding get and delete operations.
* added empty byte array for request body to fix aws signed url signature match problem as detailed here - https://github.com/aws/aws-sdk-java/issues/2205
Co-authored-by: Bernard Worthy <>
* Smart Substitution Debugging - Adding appsmith data type to each parameter in response's request object for execute
* Incorporated review comments.
* Fixed failing test cases.
* Fixed MS-SQL test failures.
* WIP : Adding timestamp datatype in Prepared Statement
* Added simple tests for commonly used data types' identification
* Added plugin level test cases for timestamp with Prepared Statement to Postgres
- Return hint message if identical column names are found in SQL query for postgres, MySQL, mssql, redshift plugin.
- Add a PluginUtils class to hold general utility functions for plugins.
- Provide non client certificate based SSL support for Mysql, Mongo, Postgres plugins.
- Added a new option default, apart from the the SSL mode types supported by the driver. Default means that go with whatever default configuration driver provides.
* WIP : enrichment of analytics event for execute.
* Plugin level : Catch all exceptions and set request in the result
Server level : Adding new fields to analytics : `isSuccessfulExecution`, `statusCode`, `timeElapsed`
* Dont catch StaleConnectionException. Server handles the same.
* Removed class specification for onErrorResume in plugins since its supposed to catch all errors.
* POC : null value getting handled correctly
* Refactor code. Tested with and without quotes. This is working! Woohoo!
* Added support for mustache binding to include json objects and arrays
* Added test for smart substitution.
* Added setting to turn on/off smart substitutions in Rest API plugin
* Handling turning on smart json substitution
* Added error handling
* Added config to turn on smart substitution in JSON to make the tests pass now
* Adding a beta tag to the API setting.
* Spelling error resolved.
* Incorporated review comments.
* Catching JSON Parse exception in http call.
- Use dynamic input text for plugin editor forms so that the evaluated value of mustache expression is visible.
- Those fields which are meant to store Key only - i.e. they are not visible to the user and are only meant to provide key for key value pair in db have been skipped.
- Mongo plugin client driver does not return with exception upon first failure - instead it keeps retrying. Hence, adding timeout error to report failure before the client thread cancels due to delay in response.
- It seems that the mongdb connection string cannot be directly used for ping test, hence skipping it.
- Provide support for non Amazon S3 service providers like Upcloud, Digital Ocean, Wasabi, DreamObjects and any other S3 compliant storage service provider.
- Change plugin label from "Amazon S3" to "S3"
- A consensus has emerged that the utility of overriding endpoint is only for local development and testing. Hence, removing the fields from datasource form.
- provide a max length to keyvalue_array type. If the max length is set to 1, then the "+" button will not appear.
- front end changes are pending. A new github issue will be opened to track the same : #3315
* Added action request in S3 plugin.
* Added request data in Dynamo plugin
* Added request in Elastic Search.
* Request in Firestore.
* Request added in Mongo
* MsSQL request added.
* Added MySQL request.
* Added Postgres request.
* Added redis request.
* Redshift doen.
* Catching AppsmithPluginExceptions at the plugin level itself to ensure that the request gets passed on as part of the result.
* Fixed failing plugin test failures
* Fixed AmazonS3 test failures.
* WIP post analytics working
* WIP : Making the request data confirm to existing analytics request data format.
* Fixed the headers in the analytics body.
* Migrations S3, Postgres to the accepted format for request in analytics
* Updated S3 action to be the query
* Migration completed for all plugins for analytics.
* Removed the old analytics event. Now sending only the new one.
* Ensuring all data is captured in S3 plugin request in case of error.
* Minor editing of firestore to ensure that the errors get caught by the plugin itself.
* Fixed test cases in Amazon S3
* Incorporated review comments.
- Stop logging datasource argument error externally. The error described in the linked GitHub issue arises due to bad service account credentials provided as arg.
- Update error msg to make it clear.
- Add TC.
* check dynamodb connection validity using listTables cmd.
* add TC
* add encrypted logo to password field
* rectify the default value of region dropdown
* remove unused imports
* POC for Firestore server-side pagination
* Load pagination information values from action configuration
* Get execution information for Firestore plugin
* Working implementation of pagination for Firestore
* Add tests for next and previou page navigations
* Require ordering to be set when paginating
* Remove commented code
* Don't report error on bad configuration
* Error out on usupported operation
Co-authored-by: Trisha Anand <trisha@appsmith.com>
* Move constant indices to constant fields
* Use executeParameterized instead of execute
Co-authored-by: Trisha Anand <trisha@appsmith.com>
- add support to handle dnyamoDb Binary type
- transform raw response to show extracted response and raw response for:
GetItem
BatchGetItem
DeleteItem
PutItem
Scan
TransactGetItems
UpdateItem
- modify TC.
* Pushing minor editor form changes to ensure that prepared statement could be turned off.
* Code refactor to do variable substitution in PluginExecutor instead of action service.
* WIP : Prepared Statement handling in psql plugin
* WIP Prepared Statements.
* Working version of prepared statements
* Quote trimming added for post preparing sql statements. Now the unprepared statements and prepared statements do not require edits.
* Fixed existing test cases failing.
* Code formatting.
* Super minor code cleanup.
* Added migration for the existing postgres actions.
* Fixed failing test cases in ActionServiceTest.
* Minor change in the text for turning on and off prepared statements in the postgres query pane.
* Added test cases for prepared statement.
* Some minor comments for code readability
* Moved Prepared Statement setting from Action Configuration to Plugin Specified Templates since this setting does not make sense for all the DB plugins.
* Added function level comments
* Update app/server/appsmith-interfaces/src/main/java/com/appsmith/external/helpers/SqlStringUtils.java
Co-authored-by: Arpit Mohan <mohanarpit@users.noreply.github.com>
* Update app/server/appsmith-interfaces/src/main/java/com/appsmith/external/helpers/SqlStringUtils.java
Co-authored-by: Arpit Mohan <mohanarpit@users.noreply.github.com>
* Incorporated review comments.
* Fixed compile time error.
Co-authored-by: Arpit Mohan <mohanarpit@users.noreply.github.com>
1. Return url on file create.
2. Fix image/video upload/download.
3. Change error msg format to directly print error msg without adding any prefix sentence.
1. Move package appsmith-interfaces/src/main/java/com/appsmith/external/pluginExceptions-> appsmith-interfaces/src/main/java/com/appsmith/external/exceptions/pluginExceptions
2. Move enum AppsmithErrorAction from appsmith-server/src/main/java/com/appsmith/server/exceptions/AppsmithError.java -> appsmith-interfaces/src/main/java/com/appsmith/external/exceptions/AppsmithErrorAction.java, so that both plugin exceptions and server exceptions could use the same enum.
3. Log exception based on the error action defined for each exception.
* Increased logs to debug future connection leaks.
* Fetch a connection from the pool only if a query exists.
* Minor comment added.
* Minor rewrite
* Code formatting.
* Update app/server/appsmith-plugins/postgresPlugin/src/main/java/com/external/plugins/PostgresPlugin.java
Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
* Added hikari cp pool stats to get database structure function as well.
Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
* Increased logs to debug future connection leaks.
* Fetch a connection from the pool only if a query exists.
* Minor comment added.
* Minor rewrite
* Code formatting.
* Update app/server/appsmith-plugins/postgresPlugin/src/main/java/com/external/plugins/PostgresPlugin.java
Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
* Added hikari cp pool stats to get database structure function as well.
Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
1. Test that when a query is attempted to run on mongodb server but refused because of lack of authorization, then
also, it indicates a successful connection establishment.
1. Check if mysql plugin connection is stale. If so, then return stale connection exception which triggers a retry after
establishing a new connection.
2. Refactor execute method code to remove code duplication.
3. Add a new test case to check if stale connection gets detected.
1. Encode request body in REST API plugin when content header for URL encode is set.
2. Add a test case to test the encoding function.
3. Add a test case to check non-encoding (raw) function.
1. Add R2BC driver and remove JDBC driver usage in mysql plugin to make the plugin reactive.
2. Update dependencies in POM file.
3. Update mysql plugin to return result based on the type of sql query i.e select vs other queries. In case the queries are chained, then the last query is used for deciding the type.
4. Added a new TC for testing datasource and removed a non functional TC.
* Adding the skeleton for Firestore integration
* Adding the datasource & query editor forms
Also adding the database changelog for the firestore plugin
Commenting out the firestore.close() connection because that causes issues with multiple Firestore tenants running in the same JVM.
* Adding the code for fetching the structure of collections from Firestore
* Use single document path field for Firestore
* Fix potential NPE when datasource destroy timeouts
* Work in progress on collection level ops for Firestore
* Get documents in a collection now works
* Add collection level querying support
* Mild refactoring
* Fix NPE when some fields are missing
* Hide clientJSON as a password field for Firestore
* Make collection level querying reactive
* Make reactive
* Validate before connecting
* Add tests for all supported methods in Firestore
* Fix forms for Firestore with hidden fields
* Hide limit and order by fields when not needed
* Restore log entry deleted by mistake
* Use S3 URL for Firestore/Firebase logo
* Add comments detailing why some code is commented
* Make parsing JSON reactive and fix subscribe calls
* Fix reactive scheduler
Co-authored-by: Arpit Mohan <arpit@appsmith.com>
* Making postgres plugin blocking statements run on elastic threadpool.
* Moved the blocking calls to check for the connection validity, etc. also inside the Mono.fromCallable
1. Fix mongodb datasource cypress test failure.
2. Against certain mongodb instances, we expect to receive an exception of type "unauthorized" when testing datasource. Earlier, this case was caught and whitelisted until the recent change to replace mongodb infra with reactive mongodb infra.
Add support for reactive streams with mongodb. (fixes 1480)
1. Replace mongodb driver with reactive mongodb driver. Change APIs accordingly.
2. Use webflux + reactor framework to execute mongodb queries in event loop model.
3. Add test to test MongoPluginExecutor class' method "testDatasource".
* Adding host:port validation checks to ES plugin & Redis plugin
Also correcting the assertions in the RedisPluginTest so that the error message on test failure is accurate.
* Removing the endpoint validation from datasourceServiceImpl
Moving the endpoint validation to the plugin implementation classes because there are databases that require complete HTTP URL in their configuration. Checking for http/https at the platform level affects the UX for the user & plugin developer when using a new integration. Hence, all plugins must implement their own client/server validations in their own implementations
* Adding tests to assert hostname validation in mysql & postgres plugins
1. Fixing the build by excluding the slf4j-api from redis-plugin pom.xml
2. Adding the editor.json and form.json for the query pane & datasource pane.
3. Adding array handling in the Redis response by feeding all the output into a "result" key
* Added DynamoPlugin to integrate with DynamoDB
* Connection management for DynamoPlugin
* Process action from a single JSON object as input
* Moved ListTables command to separate class
* Add missing UI definition files
* Use reflection to build request objects for AWS SDK
* All DynamoDB actions are now run purely by reflection
* Serialize responses to JSON
* Remove some unused/commented code and some refactorings
* Add non-working test with generic dynamodb container
* Added working test for ListTables action
* Added test for PutItem action
* Add tests for get and update actions
* Added comments and some refactoring
* Removed unused Command class.
* Added more tests for map to SDK object conversion
* Add templates
* Add option to set an endpoint override
* Fix Dynamo plugin tests when ~/.aws is missing
* Add documentation link for DynamoDB plugin
* Fix validation checks
* Remove debug log
* Added comments on expected query structure
* Fix incorrect error condition with incorrect request format
* Add comment for converting action to method name
* Error out if region is missing
* Remove unused throws declaration
* Use rich form for action configuration
* Removed templates
* Use PNG for logo images
* Use PNG logos for all plugins
* Avoid hard-coded field names
* Change logo to PNG only for ElasticSearch for now
* Wrap errors in AppsmithPluginException
* Typo
Co-authored-by: Trisha Anand <trisha@appsmith.com>
* Fix changeset order number
Co-authored-by: Trisha Anand <trisha@appsmith.com>
* Minor refactoring and fixed error messaging
Co-authored-by: Trisha Anand <trisha@appsmith.com>
* add elasticSearchPlugin
* Fix container startup in tests
* Add elasticsearch dependency
* Get plugin to a base working state
* Add templates and tests for all Document APIs
* Add support for bulk queries
* Add test and template for bulk operations
* Use rich form for action configuration
* Add test API for ElasticSearch
* Use rich form's values for plugin execution
* Add authorization header support
* Fix tests after config object use changes
* Add test for bulk requests with nd-json body
* Remove templates and minor refactoring
* Fix potential NPE with null body
Co-authored-by: Trisha Anand <trisha@appsmith.com>
* Add datasource validation for endpoint
* Wrap errors in AppsmithPluginException
Co-authored-by: Suman Patra <spatra@akamai.com>
Co-authored-by: Trisha Anand <trisha@appsmith.com>
* Add option to override server timezone for MySQL datasources
* Added test for server timezone override for MySQL
* Fix serverTimezone property config
* Implement caching for datasource structure
* Avoid using final fields so spring-data can load objects
* Use type variable for getStructure method
* Initial version with base structure for MySQL plugin
* Add tests for MySQL plugin structure
* Add sorting for keys in Postgres structure as well
* Show "primary key" instead of just "primary"
* Refactor to reduce inline magic strings
* Add a connection type variable for Plugin implementations
This type variable is intended to represent the type of the
connection object, if any, that the plugin will use. This will
help make the implementations more robust by leveraging Java's
type checking instead of rudimentary type casts over the
connection objects.
* Fix missing typevar usage
* Base interface for getting datasource structure for databases
* Add keys and constraints to structure for Postgres datasources
* Use connection from datasource context for computing structure
* Refactor context retrying into a separate method
* Add base datastructures for templates in entity explorer
* Fix spring circular dependency
* Add test for postgres datasource structure
* Generate column names and sample values for INSERT query
* Add LIMIT clause to generated SELECT query
* Fix tests for generated SELECT query
* Minor refactoring
* Fix aliases not showing up for MySQL actions
* Fix date column display for MySQL actions
* Fix datetime columns for MySQL actions
* Add support for timestamp and year data types for MySQL actions
* Fix column label for MySQL plugin
* Add tests for MySQL temporal data types
* Add tests for alias columns in MySQL and Postgres
* Added information to setup domain
* Fixed incorrect var reference
* Updated documentation message
* Updated env template
* updated template
* removed debug echo
* Updated Script
* Updated Text
* Removed option to connect to external mongo for fresh installs
Exit script of docker desktop is not installed
* Updated docker installation explanation
* added a question to skip custom domain if user is installing locally
* Inverted script question and
* Removed question to determine local installation
* Updated Popup CTA to Modal
Updated query templates to contain bindings
Added a message to display on API / Query errors
* updated styles for evaluated value
updated styles for auto complete
* added spaces to questions
Co-authored-by: Nikhil Nandagopal <nikhil@appsmith.com>
Keeping the function to read plugin resource generic for any type of resource. Now the parent function can also conditionally decide which plugin would require to read which resources from the filesystem.
Co-authored-by: Arpit Mohan <arpit@appsmith.com>
* Handle stale database connection from datasources
* Fix potential secondary case of stale connection error
* Fix Postgres to MySQL
* Move validity check timeout to a constant field
* Add test for recovery when stale connection error is thrown
The plugin class loader from Pf4J looks for jars under target/lib
to use to load any classes that are missing target/classes. This
PR adds the `maven-dependency-plugin` to copy the jars to this
folder automatically so the classes are available during
development.
This commit adds the Mysql plugin to the Appsmith server. We also add a migration to ensure that this plugin is installed by default for all existing organizations. The migration also adds the plugin details into the DB.
Also adding the test cases for mysql plugin.
Co-authored-by: Arpit Mohan <arpit@appsmith.com>
Co-authored-by: Hetu Nandu <hetu@appsmith.com>
Co-authored-by: Arpit Mohan <me@arpitmohan.com>
Co-authored-by: Nupur Singhal <nupursinghal@Nupurs-MacBook-Air.local>
* Dates from Postgres queries now show up as ISO date strings
* Use constant for date column name in PostgresPlugin
* Fix formats for more date-time types on Postgres
* Add support for time and timestamp data types in Postgres plugin
* Add support for timestamptz column data type in Postgres plugin
* Add support for interval data type in PostgresPlugin
* Move application configuration to be loaded from environment variables
* Remove unused sentry.properties
* Make missing value sentinel a constant and ignore all *.env files
* Removed now-used ACL properties
* Prefix RapidAPI environment variable with APPSMITH_
* Fix application properties not being loaded into static fields
* Remove application-test.properties file
* Add required env variables for test in GitHub
* Quote URLs for MongoDB and Redis in test config
* Change RAPIDAPI to RAPID_API in environment variable names
* Source .env file in the root of repo in start script
Add server-side templates support for plugins
Two main themes in this MR:
1. Add support for saving query templates as resource files inside a plugin. This has been done for both Postgres and MongoDB plugins, based on current template contents.
2. Loading of form JSON is now reactive. The blocking call has been moved inside a `Mono.fromSupplier`.
3. The loading of from JSON as well as the templates is cached and are loaded on-demand. This means that the templates are loaded once for a plugin through the lifetime of the server process, and that they are loaded only at the first time they are needed.
4. If loading of these resources fails, we try again when they are needed the next time.
See merge request theappsmith/internal-tools-server!385
This is to simplify the query pane on the frontend client. The client doesn't need to have separate interfaces for sql and non-sql plugins. All queries will be sent to the server in the form of a String that is parsed in different formats based on the plugin.
Also adding test cases for PostgresPlugin. Used TestContainers to simulate the postgres db in Docker inside the Java test itself. Very useful.
This is because the client sends us the data for form-data in the form of a list of Property fields with keys and values. This is consistent with how the client stores & renders headers & query parameters as well.
For MediaType application/x-www-form-urlencoded we will use the bodyFormData field.
