* Adding the skeleton for Firestore integration
* Adding the datasource & query editor forms
Also adding the database changelog for the firestore plugin
Commenting out the firestore.close() connection because that causes issues with multiple Firestore tenants running in the same JVM.
* Adding the code for fetching the structure of collections from Firestore
* Use single document path field for Firestore
* Fix potential NPE when datasource destroy timeouts
* Work in progress on collection level ops for Firestore
* Get documents in a collection now works
* Add collection level querying support
* Mild refactoring
* Fix NPE when some fields are missing
* Hide clientJSON as a password field for Firestore
* Make collection level querying reactive
* Make reactive
* Validate before connecting
* Add tests for all supported methods in Firestore
* Fix forms for Firestore with hidden fields
* Hide limit and order by fields when not needed
* Restore log entry deleted by mistake
* Use S3 URL for Firestore/Firebase logo
* Add comments detailing why some code is commented
* Make parsing JSON reactive and fix subscribe calls
* Fix reactive scheduler
Co-authored-by: Arpit Mohan <arpit@appsmith.com>
* Making postgres plugin blocking statements run on elastic threadpool.
* Moved the blocking calls to check for the connection validity, etc. also inside the Mono.fromCallable
Instead of using compound index, using single indices to ensure that both fetch actions by app id (during load of actions in edit/view mode) and fetch actions by page id (during update layout) are indexed queries.
* Doing a deep copy during clone application instead of updating the original application which may have been causing a concurrency bug.
* Ensuring that once the clone application flow is triggered, the flow completes eventually even if the client cancels the request before completion.
* Cloned application would not be public.
* Added parametrized Application constructor
* Removed lombok all args constructor
* Optimized import
* Incorporated review comments :
1. Updated the constructor for creating the application
2. Added a test case to assert that if during cloning of an application the flow gets cancelled, the cloning would still complete and ensure that the application created is sane.
Also adding the APPSMITH_SEGMENT_CE_KEY as a build arg to the server Dockerfile. This will be used to set the environment variable in the Docker image itself.
* Experimenting with spring metrics, prometheus
* Added prometheus config in application.properties.
* Added a few more configurations for spring metrics to improve performance tooling.
* Added basic authentication to allow prometheus to scrape our monitoring data in an authenticated fashion.
* Only exposing prometheus end point at this junction instead of other metric endpoints as well.
* Fixed test case which failed because of actuator instantiating bean.
* Report selective errors to rollbar.
1. Assign an action attribute to each AppsmithError enum item. Take action based on the action attribute.
* Fixing the condition for organization name being empty
* Report selective errors to rollbar.
1. Assign an action attribute to each AppsmithError enum item. Take action based on the action attribute.
Co-authored-by: Sumit Kumar <sumitsum@Sumits-MacBook-Pro.local>
Co-authored-by: Arpit Mohan <arpit@appsmith.com>
* Fixed timeout error message, added new type of error for plugin timeouts
* Switched error code and added test for a secondary StaleConnectionException
* [Do Not Merge]Brought down cloud migration time from 122 minutes to 9.5 minutes
* Removed commented code.
TODO : Figure out if we want to change the migration order numbers once the release environment is reverted back to pre original migration data.
* Added plugin id for the migrated action.
* Renamed the createIndex functions for new action and new page and added a drop collection command before creating the index. This ensures that if any developer has pulled code in the last two days, this migration does not negatively impact them
* Adding host:port validation checks to ES plugin & Redis plugin
Also correcting the assertions in the RedisPluginTest so that the error message on test failure is accurate.
* Removing the endpoint validation from datasourceServiceImpl
Moving the endpoint validation to the plugin implementation classes because there are databases that require complete HTTP URL in their configuration. Checking for http/https at the platform level affects the UX for the user & plugin developer when using a new integration. Hence, all plugins must implement their own client/server validations in their own implementations
* Adding tests to assert hostname validation in mysql & postgres plugins
* Bug Fix : Reading the datasource configuration incorrectly from actionDTO instead of datasource object.
* Added test case to assert that datasource configuration and action configurations are not null when action execute is called.
* Introduced new page which stores the published and unpublished pages as separate.
* Mid level commit to save the state.
* Parity of new page repository with old page repository (custom functions)
* WIP : Delete a page. This requires changes across application structure as well.
* Added publishedPages construct inside application to store the pages in the deployed view as well as isDefault so that the same changes (delete or isDefault) in unpublished view doesn't alter these fields for the published application
* Parity reached with PageService.
* Minor ActionService refactor to remove unnecessary code.
ApplicationPageService, LayoutActionService, LayoutService use the new page service to fetch the pages
Minor corrections in fetching the page from new page service in tests
* New save function which sets the PageDTO for unpublished page and then saves the new page into repository.
* Migration of page service functions to new page service functions across other services/tests/controller
* Finished migrating all the page service functions to the new page service functions
* Application Service Tests have been fixed.
* All the existing test cases are working now.
* Publish application implemented to store published pages as well. Added a basic test case to check that published pages is being set and that page's publishedPageDTO is being set accordingly.
* Minor TODOs added to add test cases for published application.
* A few tests to ascertain that published application page fields (deleted, isDefault) does not get changed when these statuses are changed for a page in edit mode.
* Added a new controller end point to fetch application in view mode.
* Added new endpoint for fetching an application in view mode on the client.
* Bug fix where get application in view mode API was not getting called.
* Fixed the get page names by application & archive pages which have been deleted in edit mode during publishing of application.
* During delete page, if a page was never published and it was deleted during edit, delete the entire page instead of just deleting the unpublished PageDTO
* Minor formatting.
* Non working client side code to fetch page list using view mode.
* revert unnecassary changes and streamlined view and edit actions
* Fix missed import
* Fixed a bug where if a page is not published, it should not be returned in view mode in list of page names api.
* Fixed update for a page which was not working in integration test.
* ActionDTO added.
* Solidified the new action structure.
* Migration added for NewAction index creation and NewAction per Action insertion in the database.
* Basic file structure added the new repository, custom repository, service, etc.
* Delete OldPage.java
* Repo functions added - TODO : Haven;t handled the published/edited views
* Helper functions added to convert Action to NewAction and vice-versa. Removed unused currentUserMono usage.
* Create & update action functionality added.
* Execute Action refactored. Removed dry run specific code.
* Repository migrated to handle new data structure. Execute action refactored to no longer support dry runs of actions.
* TODO added for special handling of change view of application to handle edge cases of pages/actions which either exist in published mode but don't exist in unpublished mode or vice versa.
* Migrated finding on load actions from spring repository to custom repository.
* In view mode, now actions are being fetched by application id directly instead of first fetching application and then using the page ids, fetching the actions. This reduces the db calls from 2 to 1 per fetch actions in view mode api call.
* Delete action and get all actions (used in edit mode on the client side) implemented.
* Updated CollectionService and ActionCollectionService to use the new action service instead of the old one.
* LayoutActionService refactored to now use the new service functions.
* ActionController now no longer used ActionService. The remaining service functions have been migrated to the new action service.
* Refactor across ACL code for addition/removal of policies during addition/removal of users to organization, making app public, refactor for services like policy utils, item service, etc.
* Removed the last of action repository and action service and replaced with new action repo and new action service.
* Compile and run time issues fixed. The server is coming up without any spring dependency errors.
* WIP in fixing fetching actions by page id.
* Finally!!! Fixed the fetch actions (both published and unpublished actions) by page id repository function.
* Fixed create action bug where null datasource in published actiondto (inside newly created action) leads to error.
* Fixed the execute action issues :
1. Removed the dry runs from the tests
2. Fixed the null pointer error in variable substituted action and datasource configurations.
* 1. Fixed the custom action repository field names.
2. Fixed the data structures used in ExamplesOrganizationClonerTests
* Fixed countByDatasourceId repository function which was querying the actions incorrectly.
* Fixed the clone example organization error where the id of the action was not getting updated in the page correctly. Yay!
* Fixed post merge compilation failure.
* Fixed more compilation time failures in ActionServiceTest
* Fixed failing test case for fetching actions in view mode.
* Minor changes to resolve merge changes and incorporate in the new refactored code.
* 1. Fixed compile time errors on Client code.
2. Fixed fetching of actions in view mode by application id. The repository function did not need name parameter. Removed the same.
* [Integration Testing Error Fix] : Added a new test case for refactor action name.
* Instead of fetching actions in the page, mistakenly used the base service which was fetching all the actions in the repository, barring none which led to the name refactor being blocked even though no action in the current page exists with the new proposed name,
* Added delete functionality to action service.
* Minor code cleanup
* Adding viewMode to action execution
* Replacing action with actionId.
* 1. Bug fix for deletion of unpublished action. In case of never published action, the entire action should be deleted. In case an action was published, only the unpublished action should be deleted.
2. In case of DB actions (external datasources), only the bare minimum fields should be stored inside the action (datasource id and datasource plugin id). The other fields should not be duplicated across code.
* Fixed yarn build compilation issues.
* Update app/server/appsmith-server/src/main/java/com/appsmith/server/controllers/ActionController.java
Co-authored-by: Arpit Mohan <mohanarpit@users.noreply.github.com>
* Changed the API path for GET applications in view mode. Some minor code formatting.
* Incorporated review comments.
* Some more unnecessary code removed.
* Instead of returning Page, now the interface object between client and server for Pages is PageDTO
* Migrated Page and Action to PageDTO and ActionDTO
Fixed the compilation issues.
TODO : Fix the test compilation issues.
* Fixed compilation time issues with all the tests by migrating Page and Action to PageDTO and ActionDTO respectively
* Action Controller and Page Controller no longer extend Base Controller. All the required functions have now been implemented and no base line API end points are being re-used from the base.
* Test case fixes.
* Bug Fix : Updating an action was not updating execute on load. Fixed the data flow leading to the error.
* Deprecating Page and Action domain objects. This is to ensure no new code is written with these till we remove this old code.
* Cloned example applications are now published before returning. This is to ensure that the applications are in ready to view mode when the new user signs up.
* Added a function comment to expand on the usage of new param introduced.
* When cloning a page, new actions were not being stored. Added that. Also updated the clonePage test to assert that the actions are also cloned when the pages are cloned.
* Updated a Api call
* removed extra slash
Co-authored-by: Hetu Nandu <hetunandu@gmail.com>
Co-authored-by: Satbir Singh <satbir121@gmail.com>
Co-authored-by: Arpit Mohan <mohanarpit@users.noreply.github.com>
Co-authored-by: nandan.anantharamu <nandan@thinkify.io>
1. Fixing the build by excluding the slf4j-api from redis-plugin pom.xml
2. Adding the editor.json and form.json for the query pane & datasource pane.
3. Adding array handling in the Redis response by feeding all the output into a "result" key
* Added DynamoPlugin to integrate with DynamoDB
* Connection management for DynamoPlugin
* Process action from a single JSON object as input
* Moved ListTables command to separate class
* Add missing UI definition files
* Use reflection to build request objects for AWS SDK
* All DynamoDB actions are now run purely by reflection
* Serialize responses to JSON
* Remove some unused/commented code and some refactorings
* Add non-working test with generic dynamodb container
* Added working test for ListTables action
* Added test for PutItem action
* Add tests for get and update actions
* Added comments and some refactoring
* Removed unused Command class.
* Added more tests for map to SDK object conversion
* Add templates
* Add option to set an endpoint override
* Fix Dynamo plugin tests when ~/.aws is missing
* Add documentation link for DynamoDB plugin
* Fix validation checks
* Remove debug log
* Added comments on expected query structure
* Fix incorrect error condition with incorrect request format
* Add comment for converting action to method name
* Error out if region is missing
* Remove unused throws declaration
* Use rich form for action configuration
* Removed templates
* Use PNG for logo images
* Use PNG logos for all plugins
* Avoid hard-coded field names
* Change logo to PNG only for ElasticSearch for now
* Wrap errors in AppsmithPluginException
* Typo
Co-authored-by: Trisha Anand <trisha@appsmith.com>
* Fix changeset order number
Co-authored-by: Trisha Anand <trisha@appsmith.com>
* Minor refactoring and fixed error messaging
Co-authored-by: Trisha Anand <trisha@appsmith.com>
* add elasticSearchPlugin
* Fix container startup in tests
* Add elasticsearch dependency
* Get plugin to a base working state
* Add templates and tests for all Document APIs
* Add support for bulk queries
* Add test and template for bulk operations
* Use rich form for action configuration
* Add test API for ElasticSearch
* Use rich form's values for plugin execution
* Add authorization header support
* Fix tests after config object use changes
* Add test for bulk requests with nd-json body
* Remove templates and minor refactoring
* Fix potential NPE with null body
Co-authored-by: Trisha Anand <trisha@appsmith.com>
* Add datasource validation for endpoint
* Wrap errors in AppsmithPluginException
Co-authored-by: Suman Patra <spatra@akamai.com>
Co-authored-by: Trisha Anand <trisha@appsmith.com>
* Returning validation failures while creating organization
* Added unit tests
* Changing the name of test security config
Co-authored-by: nitesh261193 <nitesh261193@gmail.com>
This is because this `cacheResponse` field is only used for auto-complete on the UI. The user can still run the action manually and get the auto-complete to work. The downside to sending the `cacheResponse` field in the JSON response is that for large responses, the client times out as our response times increase to 20 secs. Hence removing this for now.
In future, we'll extract the JSON schema structure from the response body and send that to the client for populating auto-complete.
* Implement caching for datasource structure
* Avoid using final fields so spring-data can load objects
* Use type variable for getStructure method
* Initial version with base structure for MySQL plugin
* Add tests for MySQL plugin structure
* Add sorting for keys in Postgres structure as well
* Show "primary key" instead of just "primary"
* Refactor to reduce inline magic strings
* Add a connection type variable for Plugin implementations
This type variable is intended to represent the type of the
connection object, if any, that the plugin will use. This will
help make the implementations more robust by leveraging Java's
type checking instead of rudimentary type casts over the
connection objects.
* Fix missing typevar usage
* Base interface for getting datasource structure for databases
* Add keys and constraints to structure for Postgres datasources
* Use connection from datasource context for computing structure
* Refactor context retrying into a separate method
* Add base datastructures for templates in entity explorer
* Fix spring circular dependency
* Add test for postgres datasource structure
* Generate column names and sample values for INSERT query
* Add LIMIT clause to generated SELECT query
* Fix tests for generated SELECT query
* Minor refactoring
* Add migration to fix incorrect action IDs for onLoad
* Remove redundant comments
* Fix migration to work with published onLoad actions as well
* Only update the action ID when correcting onLoad action IDs
* Fix migration name for correcting action IDs
Co-authored-by: Trisha Anand <trisha@appsmith.com>
* Fix migration id for correcting action IDs
* Reformat code
Co-authored-by: Trisha Anand <trisha@appsmith.com>
* Get list of application IDs to be cloned as examples
* Fix tests so that apps to be cloned are explicit
* Add migration to set applicationIds for example cloning
* Fix example org cloning tests
* Add a node.js script to dump examples organization
This dump file is used to setup initial examples organization
in a self-hosted setup.
* Remove unused variable
* Add npm run shortcut for dump script
* Start with new controller for assets
* Progress on uploading logo image
* Saving and loading organization logo assets now works
* Remove existing logo asset before saving a new one
* Fix 500 when uploading logo for the first time
* Fix URL in response for uploading logo image
* Add test for uploading logo for ogranization
* Mild refactoring in tests
* GET API actions recognized as page load actions should also be updated as executeOnLoad actions.
* Update action during updateLayout only if executeOnLoad is not set or is false.
* WIP : First draft of clone applications
* Cloning of application now works correctly.
* Fixed the failing test case for page cloning.
* Added test case to assert the clone of application successfully.
* Minor function renaming to avoid confusion.
* Minor code refactoring to remove duplicate code.
1. Fixed the bug where policies are being set to empty during organization update.
2. Added email field in organization.
* Updated the test case for update organization to assert the organization policies being present post the update.
We are currently getting the *list* of all pages, updating the
`isDefault` fields inside, and then saving the whole *list* of
all pages. If a new page got added to that list in the DB during
this process, that page would be lost. This commit fixes this
problem.
This race condition was causing tests for cloning applications
to fail *sometimes*.
* Working version of cloning page given page id. The clone is created inside the same application and is in unpublished state.
* Added a test case for Clone Page feature
* Incorporated review comments.
* Don't always automatically create a blank personal org
It will still be created, only if a example template organization
has not been configured in the system.
* Name examples organization same as personal organization
* Minor fixes in tests
* Remove unused userService in clone tests
* During add role to an organziation, the application was only inheriting from subset of the organization permissions. Generalized this code to ensure that this doesnt happen again in the future when more permissions are introduced. Refactored some code as well.
* Introduced new permissions for making app public and inviting users.
* WIP : Returning a subset of roles depending on the current user's role
* Added code and tests for generating hierarchical children given a role name
* Get all invite roles now returns roles depending on the current users permissions (aka - admin can invite users for all roles, developers can invite users for roles develoeprs and below)
* When invite api is called, check if the user has correct permissions ( invite permission) and check if the role for which the users are bieng invited is allowed (aka the current user has the permission to invite for the said role)
* Introduced a new permission at application level which is inherited from manage:organizations. This new permission gives the administrator of the organization to make any application public .
* Incorporated review comments.
* Minor code changes.
* Added migration script to add the new policies introduced as part of this new share modal changes.
* Null pointer exception handled due to improper data in the database.
* Integration testing bug fixes.
* Correcting the Docker image for Appsmith server for Github actions test
* Also fixing the bug where the datasource config is not returned properly for restapi-plugin
Co-authored-by: Arpit Mohan <arpit@appsmith.com>
* WIP: Not working bulk invite.
* WIP : Non compilable.
* Current test cases running.
TODO : Code Cleanup and write new test cases for bulk invite.
* Code cleanup
* Added a test case for bulk invite of users to an organization.
* Incorporated review comment.
* Merged the latest release and incorporated review comments.
* Corrected the error thrown when usernames is empty or null for invite user API.
* Auto-login on signup works!
* Support form-encoded data body for signup requests
* Remove debug log entry when getting session
* Refactoring and add some docs to UserSignup solution
* Move user object construction to UserSignup solution
* Redirect with error message on signup errors
Keeping the function to read plugin resource generic for any type of resource. Now the parent function can also conditionally decide which plugin would require to read which resources from the filesystem.
Co-authored-by: Arpit Mohan <arpit@appsmith.com>
* Overwriting the base service's getById for OrganizationService to ensure that the response contains userPermissions (by using custom repository instead of default mongo repository classes)
* Fixed the failing test cases.
* Minor code formatting.
The client will have to send a query parameter redirectUrl or a header X-Redirect-Url in order for the server to redirect the client to the appropriate URL post login. If neither of these parameters are present, the client is redirected to /applications by default.
Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
* During signup, if the user already exists, instead of throwing a generic duplicate key exception, request the user to sign in.
* Minor change in the error message.
* Fix cloning fails in some cases for organizations
Cloning currently fails in cases like the following:
- Application with no pages
- Pages with no actions
- Pages with more than one action
* Remove debug naming of cloned datasources
* Add test for organization cloning
* Add more tests for organization cloning
* Fix potential race condition in adding pages to an application
* Move db update call to add page to application, into repository
* Use `getIdCriteria` to query for document's _id
* Reuse layout that's automatically created with page
* Use correct API for updating a layout
* Commenting out organization cloning test.
* Removed organization cloning test
* Adding a dummy commit to run the pipeline.
Co-authored-by: Arpit Mohan <arpit@appsmith.com>
* Fix layouts not be updated when an action is updated
* Add test for updating onLoadActions when action updated
* Split layout and action test to separate class
* Remove duplicated test in ActionServiceTest
The policies of `User` objects is an empty set by default. So, when using a new `User` object to call `update`, it also updates existing policies to be an empty list.
* Fixing test cases failing due to non unique names.
* Adding execute action permissions policy to match read actions policy in existing actions.
* Only save the action if there are read actions permission present (and hence could have been updated)
* Datasource test should always test the latest object given by the client (instead of fetching the same from the db). If it is a saved datasource, then we should decrypt and set the password in the datasource (unless the password is one of the changes that is being tested).
* Added function level comment to clarify that this function can no longer be used by just sending the datasource.id field set.
* Handle stale database connection from datasources
* Fix potential secondary case of stale connection error
* Fix Postgres to MySQL
* Move validity check timeout to a constant field
* Add test for recovery when stale connection error is thrown
The client can send the redirect URL in X-Redirect-Url header. The server will honour the value set and redirect the client browser to this particular url. By default the server will redirect to /applications if the custom header is missing. The redirect url can either be absolute or relative.
Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
1. Bug fix for when actions for a public application use an external datasource (aka db queries). The db queries were not getting executed because the permission for execute datasource was not getting set for anonymousUser.
2. When using the Test endpoint for datasource, if the datasource is being read from the db and there are encrypted fields, first decrypt the field and then send to the plugin for testing the co
This commit adds the Mysql plugin to the Appsmith server. We also add a migration to ensure that this plugin is installed by default for all existing organizations. The migration also adds the plugin details into the DB.
Also adding the test cases for mysql plugin.
Co-authored-by: Arpit Mohan <arpit@appsmith.com>
Co-authored-by: Hetu Nandu <hetu@appsmith.com>
Co-authored-by: Arpit Mohan <me@arpitmohan.com>
Co-authored-by: Nupur Singhal <nupursinghal@Nupurs-MacBook-Air.local>
* When cloning examples organization, clone only public applications
* Create template organization within the test
* Cleaned up test for cloning of examples organization
* Fix Mono chaning
Co-authored-by: Trisha Anand <trisha@appsmith.com>
* Create test apps and config simultaneously
Co-authored-by: Trisha Anand <trisha@appsmith.com>
* Datasource authentication object fields should only be encrypted during create or if the update object contains authentication object. Added a test case to check that update to any other field doesnt update the encrypted fields.
* Incorporated review comment
* Trying to clone the examples repo on new user signup
* Working implementation of cloning examples organization on user sign up
* Fix personal org not being created when template org is missing
* A working version of cloning of examples organization on first-login
* Add docs for methods in ExamplesOrganizationCloner
* Refactor computing user's first name into a method
* Add some menial tests for examples organization cloning
* Use explicit permissions when fetching applications and datasources
* Fix template organization config name hard-coding
* Refactor implicit permissions in service methods as function arguments
* Revert an unintended change that got committed
* Fix permission variable name
Co-authored-by: Trisha Anand <trisha@appsmith.com>
Co-authored-by: Trisha Anand <trisha@appsmith.com>
