Commit Graph

541 Commits

Author SHA1 Message Date
Shrikant Sharat Kandula
c7f330c4c7 Missing isEncrypted should mean a true value 2020-12-23 14:23:03 +05:30
Shrikant Sharat Kandula
cfcd7aa0bb
Fix Firestore datasources not encrypting secret key JSON when editing (#2296)
* Fix encrypted field not being ignored in JSON

* Don't mask error in decrypting

* Revert masking of error in decrypting
2020-12-21 17:43:15 +05:30
Nidhi
e3cf285d1e
Fixed encryption related bugs for empty/new datasource (#2287) 2020-12-21 10:41:57 +05:30
Nidhi
5a36d17f7a
Fixed encryption related bugs for empty/new datasource (#2272) 2020-12-18 16:13:01 +05:30
Nidhi
a00c144e85
Subclassing authentication (#2215)
* Sublcassing authentication

* Removed TODO

* Review changes
2020-12-17 21:38:52 +05:30
Trisha Anand
2062c60ca3
Moved all the scheduling of the plugin executions to bounded elastic instead of elastic. (#1931) 2020-11-26 17:50:43 +05:30
Shrikant Sharat Kandula
74cd362057
Firestore Integration (#1799)
* Adding the skeleton for Firestore integration

* Adding the datasource & query editor forms

Also adding the database changelog for the firestore plugin

Commenting out the firestore.close() connection because that causes issues with multiple Firestore tenants running in the same JVM.

* Adding the code for fetching the structure of collections from Firestore

* Use single document path field for Firestore

* Fix potential NPE when datasource destroy timeouts

* Work in progress on collection level ops for Firestore

* Get documents in a collection now works

* Add collection level querying support

* Mild refactoring

* Fix NPE when some fields are missing

* Hide clientJSON as a password field for Firestore

* Make collection level querying reactive

* Make reactive

* Validate before connecting

* Add tests for all supported methods in Firestore

* Fix forms for Firestore with hidden fields

* Hide limit and order by fields when not needed

* Restore log entry deleted by mistake

* Use S3 URL for Firestore/Firebase logo

* Add comments detailing why some code is commented

* Make parsing JSON reactive and fix subscribe calls

* Fix reactive scheduler

Co-authored-by: Arpit Mohan <arpit@appsmith.com>
2020-11-23 18:42:33 +05:30
Nidhi
c7efe55c9c
Fixed timeout error message, added new type of error for plugin timeout (#1519)
* Fixed timeout error message, added new type of error for plugin timeouts

* Switched error code and added test for a secondary StaleConnectionException
2020-11-04 12:56:52 +05:30
Prashant Chaubey
bf69c7d66b
Adding basic structure for plugin integrating with Redis (#1085)
Co-authored-by: nitesh261193 <nitesh261193@gmail.com>
2020-10-22 13:26:58 +05:30
dependabot[bot]
668c8e9c1c
Bump junit from 4.11 to 4.13.1 in /app/server/appsmith-interfaces (#1161)
Bumps [junit](https://github.com/junit-team/junit4) from 4.11 to 4.13.1.
- [Release notes](https://github.com/junit-team/junit4/releases)
- [Changelog](https://github.com/junit-team/junit4/blob/main/doc/ReleaseNotes4.11.md)
- [Commits](https://github.com/junit-team/junit4/compare/r4.11...r4.13.1)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2020-10-13 14:33:50 +05:30
Shrikant Sharat Kandula
13ddcc0cc9
Upgrade pf4j & test containers and fix plugin tests (#1156) 2020-10-13 13:42:14 +05:30
Shrikant Sharat Kandula
4f3de3ab9f
Datasource structure support for MongoDB (#641)
* Added structure for MongoDB datasources

* Fixed tests for MongoDB template queries
2020-10-01 10:41:29 +05:30
Shrikant Sharat Kandula
75ba90928e
Add datasource structure caching and MySQL structure support (#553)
* Implement caching for datasource structure

* Avoid using final fields so spring-data can load objects

* Use type variable for getStructure method

* Initial version with base structure for MySQL plugin

* Add tests for MySQL plugin structure

* Add sorting for keys in Postgres structure as well

* Show "primary key" instead of just "primary"

* Refactor to reduce inline magic strings
2020-09-17 13:59:46 +05:30
Shrikant Sharat Kandula
6c7e45d010
Add a connection type variable for Plugin implementations (#531)
* Add a connection type variable for Plugin implementations

This type variable is intended to represent the type of the
connection object, if any, that the plugin will use. This will
help make the implementations more robust by leveraging Java's
type checking instead of rudimentary type casts over the
connection objects.

* Fix missing typevar usage
2020-09-14 20:36:47 +05:30
Shrikant Sharat Kandula
3f5930e52e
Add datasource structure support for DB plugins (#523)
* Base interface for getting datasource structure for databases

* Add keys and constraints to structure for Postgres datasources

* Use connection from datasource context for computing structure

* Refactor context retrying into a separate method

* Add base datastructures for templates in entity explorer

* Fix spring circular dependency

* Add test for postgres datasource structure

* Generate column names and sample values for INSERT query

* Add LIMIT clause to generated SELECT query

* Fix tests for generated SELECT query

* Minor refactoring
2020-09-14 18:59:11 +05:30
Shrikant Sharat Kandula
b511c3ada9
Change return type of plugin execution to be more specific (#247) 2020-08-10 14:41:32 +05:30
Shrikant Sharat Kandula
dfcabab4cc
Handle stale database connection from datasources (#151)
* Handle stale database connection from datasources

* Fix potential secondary case of stale connection error

* Fix Postgres to MySQL

* Move validity check timeout to a constant field

* Add test for recovery when stale connection error is thrown
2020-07-24 12:18:25 +05:30
Trisha Anand
fd0f23b9cc
Database credentials encryption in MongoDB (#80)
* Encrypting the password stored in AuthenticationDTO for every db.

* Adding comment to the properties file to denote that adding encryption salt and password are mandatory to the server coming up.

* Added the encryption salt and password to server.yml to allow the github actions to succeed.

* Adding database migration to encrypt the existing passwords for authentication object (used for storing db connection username/password)

Changes to the installation script install.sh:

1. Instead of overwriting the existing encryption password or salt, giving the user an option to conserve the previous encryption credentials to ensure that the developer users do not lose access to their database configurations (passwords).
2. Added another file for writing encryption credentials (encryption.env) to ensure that we dont delete the encryption password and salt by mistake.
2020-07-14 14:45:08 +05:30
Trisha Anand
8734067cad
Resolving the dependabot vulnerabilities. (#69) 2020-07-09 21:12:16 +05:30
Shrikant Kandula
3217daab14 Fix Mustache rendering to work on object fields directly 2020-06-25 09:58:20 +00:00
Trisha Anand
0603da29ee User test - Add request URL and http method to the execution result 2020-06-19 16:24:58 +00:00
Trisha Anand
73757c3425 Merge branch 'release' into feature/acl-spring-object
# Conflicts:
#	.gitignore
#	appsmith-plugins/mongoPlugin/plugin.properties
#	appsmith-plugins/postgresPlugin/plugin.properties
#	appsmith-plugins/rapidApiPlugin/plugin.properties
#	appsmith-plugins/restApiPlugin/plugin.properties
#	appsmith-plugins/restApiPlugin/src/main/java/com/external/plugins/RestApiPlugin.java
#	appsmith-server/src/main/java/com/appsmith/server/constants/FieldName.java
#	appsmith-server/src/main/java/com/appsmith/server/repositories/ActionRepository.java
#	appsmith-server/src/main/java/com/appsmith/server/services/ActionServiceImpl.java
#	appsmith-server/src/main/java/com/appsmith/server/services/DatasourceContextServiceImpl.java
#	appsmith-server/src/main/java/com/appsmith/server/services/DatasourceServiceImpl.java
#	appsmith-server/src/test/java/com/appsmith/server/services/ActionServiceTest.java
#	appsmith-server/src/test/java/com/appsmith/server/services/ApplicationServiceTest.java
#	appsmith-server/src/test/java/com/appsmith/server/services/LayoutServiceTest.java
#	appsmith-server/src/test/java/com/appsmith/server/services/PageServiceTest.java
#	build.sh
2020-06-12 19:14:31 +05:30
Arpit Mohan
bbd33c2968 Removing the query field in ActionConfiguration and using String body as the field to store the query
This is to simplify the query pane on the frontend client. The client doesn't need to have separate interfaces for sql and non-sql plugins. All queries will be sent to the server in the form of a String that is parsed in different formats based on the plugin.

Also adding test cases for PostgresPlugin. Used TestContainers to simulate the postgres db in Docker inside the Java test itself. Very useful.
2020-06-09 12:12:27 +00:00
Trisha Anand
6475be63d1 Baseline code added for cascading the org level permissions to applications, pages and eventually actions. 2020-05-27 14:36:57 +00:00
Arpit Mohan
fb9e9fbf4c Request body and header in Action Execution Response 2020-05-14 02:08:46 +00:00
Trisha Anand
5829a92998 Merge branch 'release' into feature/acl-spring-object
# Conflicts:
#	appsmith-server/src/main/java/com/appsmith/server/services/ActionServiceImpl.java
2020-05-13 23:37:31 +05:30
Shrikant Kandula
9c48f42990 Rename ActionExecutionResult.shouldCacheResponse -> isExecutionSuccess. 2020-05-12 18:25:49 +05:30
Trisha Anand
957116409d Resolved the review comments. Changed the structure of the DTO returned. Added the userPermissions as part of the base domain leading to this field always computed before returning any object to the user. 2020-05-07 10:49:48 +00:00
Trisha Anand
2d1fe9d8b4 Merge branch 'release' into feature/acl-spring-object
# Conflicts:
#	appsmith-interfaces/src/main/java/com/appsmith/external/models/SSHPrivateKey.java
#	appsmith-server/src/main/java/com/appsmith/server/filters/AclFilter.java
#	appsmith-server/src/main/java/com/appsmith/server/services/CurlImporterService.java
#	appsmith-server/src/test/java/com/appsmith/server/services/CurlImporterServiceTest.java
2020-04-28 18:13:06 +05:30
Shrikant Kandula
23bd4e1722 Add NO_SSL option to SSL authType to turn off SSL explicitly. 2020-04-28 12:31:34 +00:00
Shrikant Kandula
89d04f11e7 Merge branch 'bug/show-password-in-datasource-form' into 'release'
Show passwords in datasource forms.

See merge request theappsmith/internal-tools-server!308
2020-04-28 09:34:46 +00:00
Shrikant Kandula
1dbb98e1e8 Fix cURL import bugs with new cURL command parser 2020-04-28 09:09:03 +00:00
Shrikant Kandula
0a05036f9f Show passwords in datasource forms. 2020-04-28 11:12:03 +05:30
Shrikant Kandula
e0880c7d31 Support separate default database for MongoPlugin, besides authDatabase. 2020-04-23 11:09:02 +00:00
Trisha Anand
a8f32d8b2a Merge branch 'release' into feature/acl-spring-object
# Conflicts:
#	appsmith-server/src/main/java/com/appsmith/server/services/MarketplaceService.java
2020-04-23 14:45:40 +05:30
Trisha Anand
4cde1413fc Provider credential steps is not a write only property. It must be readable. 2020-04-21 13:45:28 +05:30
Trisha Anand
373c9ac738 Code formatting corrected. 2020-04-20 18:17:25 +05:30
Trisha Anand
6be0b7ae23 Merge branch 'release' into feature/acl-spring-object
# Conflicts:
#	appsmith-server/pom.xml
#	appsmith-server/src/main/java/com/appsmith/server/migrations/DatabaseChangelog.java
#	appsmith-server/src/main/java/com/appsmith/server/repositories/BaseRepositoryImpl.java
#	appsmith-server/src/main/java/com/appsmith/server/services/LayoutActionServiceImpl.java
#	appsmith-server/src/test/java/com/appsmith/server/services/OrganizationServiceTest.java
2020-04-20 18:13:55 +05:30
Shrikant Kandula
08b7f51d5d Datasource API cleanup and better error reporting 2020-04-17 07:27:56 +00:00
Shrikant Kandula
2dbf9d1c6b Test API for data sources 2020-04-15 10:02:09 +00:00
Shrikant Kandula
781eec0d24 Add a boolean field for enabling/disabling SSH Proxy on datasources. 2020-04-13 07:22:29 +00:00
Shrikant Kandula
abd9235360 Richer plugin validations 2020-04-10 09:59:50 +00:00
Shrikant Kandula
42d0060ecd Upgrade pf4j-spring to 0.6.0 so we can consistently use pf4j 3.2.0. 2020-04-09 19:55:58 +05:30
Shrikant Kandula
91dd8bf5c3 Fix old enums being used in Postgres plugin. Build failed because of this. 2020-04-08 12:45:18 +00:00
Shrikant Kandula
edf4043430 Merge branch 'bug/datasource-enum-mismatches' into 'release'
Fix Datasource data structure to be inline with UI

See merge request theappsmith/internal-tools-server!268
2020-04-08 12:32:57 +00:00
Shrikant Kandula
1b75be0fe3 Fix Datasource data structure to be inline with UI 2020-04-08 12:32:56 +00:00
Shrikant Kandula
0ad15b4cb6 Fix: Duplicate key error when reusing the name of a deleted application 2020-04-08 12:09:41 +00:00
Shrikant Kandula
ff6b0ce6bc Increase restapi client buffer size from 256KB to 10MB. 2020-04-07 11:20:46 +00:00
Trisha Anand
5911234276 To ensure that special characters like "\n", "\t", etc are preserved in the request json body, convert the json string into object. 2020-04-03 14:27:10 +00:00
Arpit Mohan
b7b7328c3f Merge branch 'release' into feature/acl-spring-object 2020-04-02 13:53:31 +05:30
Trisha Anand
f882ffcdf3 Fetching the providers from the Marketplace using webclient. Introduced a temporary end point to support the new source for Providers 2020-04-01 18:53:39 +00:00
Shrikant Kandula
1f524827b9 Datasource CRUD APIs 2020-04-01 08:50:36 +00:00
Arpit Mohan
995bfd0bbd * Adding CustomRepository interfaces and implementations to all the domain objects in the classpath
* Adding generic get(Multivalue<String, String> params) implementation to the BaseService. Now all the domain obejcts can simply leverage this base implementation for get queries out of the box for all the fields.

* IMP: For some reason, the query by example has stopped working across the board. Even the super implementation in SimpleReactiveMongoRepository doesn't work. No idea why this is happening. For the time being, have moved to Criteria queries for the get request.
2020-04-01 10:00:18 +05:30
Arpit Mohan
4725421cba Merge branch 'release' into feature/acl-spring-object 2020-03-19 11:54:36 +05:30
Trisha Anand
0b7b4716db Credential steps should only be written but not read. 2020-03-18 14:40:02 +05:30
Trisha Anand
fad606a156 Updated Provider to include fields to help with display and sort order 2020-03-18 14:21:38 +05:30
Arpit Mohan
1f35bd6a07 Adding the policy hierarchy graph and the lateral policy graph
These graphs help us map policies that are inherited from the parent and also lateral policies that are assigned to the users given that the user has a particular permission. Currently, the hierarchy has been defined for org & application. Need to cascade it to more documents such as pages & actions.
2020-03-16 10:46:28 +05:30
Arpit Mohan
a892ee90b5 Merge release branch 2020-03-13 12:47:16 +05:30
Arpit Mohan
2da4727cbd Removing document version from all the Mongo documents.
Also adding Origin header to the user invite flow so that we can send the correct links to the user
2020-03-11 18:18:10 +05:30
Arpit Mohan
1475d9124a Fixing all the test cases for OrganizationService
We still need to add a lot more test cases to ensure that no functionality has broken. But this is a start in the right direction.
2020-03-05 19:35:24 +05:30
Arpit Mohan
c813678f49 Adding the AclPermission enum to track all possible permissions globally
Also moving the argument AclPermission to the custom Repository interface level. This is to ensure that all the service functions can invoke the same repository function with different permissions based on their requirements.
2020-03-05 15:33:41 +05:30
Arpit Mohan
c5e8f3bef4 Removing unnecessary and extra code in AclFilter, MongoAspect and CustomWebExpressionHandler 2020-03-04 19:12:34 +05:30
Arpit Mohan
0914acdca6 Adding default principal for anonymous Users in SecurityConfig
Also, hard-coding the document fields in BaseRepositoryImpl criteria queries.
2020-03-04 18:57:46 +05:30
Arpit Mohan
7019bae082 Adding policy to the document object instead of the user
The login has broken and all users are being treated as un-authenticated users
2020-03-04 18:57:46 +05:30
Arpit Mohan
3a1673dfca Adding QueryDsl to the appsmith-server and appsmith-interfaces modules
This will generate the annotated classes so that we can refer to the fields easily and scalably.
2020-03-04 18:57:46 +05:30
Arpit Mohan
249bbbde39 Adding the field shouldCacheResponse to the ActionExecutionResult to make the response caching generic across various plugins 2020-02-25 11:36:02 +00:00
Trisha Anand
6109bd4b6c Added valueOptions to store the permitted values for a key (which is being sent by rapidApi today) 2020-02-21 11:16:07 +05:30
Trisha Anand
977bac62dc Added hashValue and scraperid to Template 2020-02-20 19:18:48 +05:30
Trisha Anand
1795be3c76 Added route parameters to action configuration to handle the scenario of rapid api putting variables in the path 2020-02-19 17:47:26 +05:30
Trisha Anand
634a625906 1. Adding provider plan subscribed to and list of all the plans to Provider
2. Curl Importer should call create action instead of saving it directly so that all the defaults can be set up properly.
2020-02-17 10:08:35 +00:00
Trisha Anand
96cd15e9ce Extended the ActionConfiguration to accept rest api body in the form of key-value for form-data input. Also extended Property to include fields which would help the user in configuring the API (like description, mandatory, etc.) 2020-02-14 13:18:26 +05:30
Trisha Anand
2e11a5c2d5 Added datasource configuration to the provider to support the provider level configurations in rapid api 2020-02-14 06:32:34 +00:00
Trisha Anand
8661f71a80 Mock apis for 3P frontend work 2020-02-07 09:05:14 +00:00
Arpit Mohan
ffc7328e13 Renaming isPaginated to PaginationType to support different types of pagination on the frontend
If the API is paginated via the table's page number, then the next and prev URLs in the action don't play any role. Those fields are only required when the paginationType is of type URL. In all other scenarios, the action is executed normally.
2020-02-05 14:41:53 +05:30
Trisha Anand
17f4a6f2e8 Handle server side pagination in action 2020-01-28 14:40:59 +00:00
Trisha Anand
670ca4cb7e Added log for execute action 2020-01-15 09:13:27 +00:00
Trisha Anand
f4a2a73abe In case of non json results, set the result body as string, otherwise parse the json and then store it as jsonNode 2020-01-14 09:48:43 +00:00
Trisha Anand
b16909de61 JS functions can now be stored to an action. 2019-12-05 09:19:45 +00:00
Arpit Mohan
ca4ff6a28b Fixing action execution timeout bug which was defaulting the execution timeout to 0 instead of 10.
Now if we provide a null / negative value to action execution timeout, it'll default to 10s
2019-11-26 18:19:12 +05:30
Trisha Anand
0fd6351a76 Added timeout on the plugin execution.
Next TODO : Make the timeout duration configurable
2019-11-26 11:34:27 +00:00
Trisha Anand
31cf94f08e 1. Added datasource validator function in the plugin interface which simply returns true for all the current plugins.
2. Added static method getPluginExecutor as a util function which removes duplicate code from across different files.
2019-11-14 08:50:02 +00:00
Arpit Mohan
bcda3f0feb Renaming all the references of resource to datasource
This renaming is reflective of the entity and it's potential values in the future.
2019-11-06 16:25:08 +05:30
Arpit Mohan
3622d8b0d3 Converting actionConfiguration to a string to extract mustache keys
Also making jsonPathKeys a read-only field. Cannot be written via the API.
2019-11-05 10:40:09 +00:00
Trisha Anand
84837fe57f Mongo Plugin works here. Barring the scenarios where the query json contains special characters in keys ($, .) it works properly. 2019-10-18 08:27:19 +00:00
Arpit Mohan
3a9f344a51 Removing plugins from appsmith-interfaces module 2019-10-17 09:25:58 +05:30
Trisha Anand
e1d59ca3a6 Multi data source support for plugins using Resource Context 2019-10-11 10:32:18 +00:00
Trisha Anand
8658df95a9 All action executions now return object of type ActionExecutionResult. RestApiPlugin returns the same object which contains statusCode, headers and body. 2019-09-30 18:17:35 +00:00
Trisha Anand
566a245451 Segment & Rollbar Integration 2019-09-25 16:20:51 +00:00
Trisha Anand
bec8b2e0a1 Adding the docker-compose up command to the build
This should ideally run the mongo-seed container to restore the DB before each run
2019-09-24 12:14:41 +00:00
Arpit Mohan
0edc0f6363 Adding the executeAction API in the ActionController.
Also changed the interface for the PluginExecutor to include the resourceConfig and ActionConfig as well.
2019-09-17 12:18:23 +00:00
Arpit Mohan
b3c987a9f9 Convert the project into a multi-module structure
We are moving to a multi-module structure so that different parts of the codebase can be exposed to the public while others can remain private. Using pf4j for plugin framework.

Also adding a build script `build.sh` which compiles the code and creates the `dist` folder for distribution purposes. Now we can build the code via

```
$ ./build.sh -DskipTests
```
2019-09-16 11:21:57 +00:00