* Zipping flux with mono leads to mono not getting repeated and only the first emitted value of flux is processed further. This is leading to only a single page getting emitted in a multi page application.
* Added a test case to assert this for a multi page application
* All newly created applications now have a default published state.
* Added a test case to assert a new application comes up with default published states for both application and the default page.
* Function name changed.
* When trying to fetch a published version of a page when the page has not been published, throw a 404.
* WIP : Not working
* Doing a deep copy during clone application instead of updating the original application which may have been causing a concurrency bug.
* Ensuring that once the clone application flow is triggered, the flow completes eventually even if the client cancels the request before completion.
* Cloned application would not be public.
* Added parametrized Application constructor
* Removed lombok all args constructor
* Optimized import
* Incorporated review comments :
1. Updated the constructor for creating the application
2. Added a test case to assert that if during cloning of an application the flow gets cancelled, the cloning would still complete and ensure that the application created is sane.
* Experimenting with spring metrics, prometheus
* Added prometheus config in application.properties.
* Added a few more configurations for spring metrics to improve performance tooling.
* Added basic authentication to allow prometheus to scrape our monitoring data in an authenticated fashion.
* Only exposing prometheus end point at this junction instead of other metric endpoints as well.
* Fixed test case which failed because of actuator instantiating bean.
* Fixed timeout error message, added new type of error for plugin timeouts
* Switched error code and added test for a secondary StaleConnectionException
* Adding host:port validation checks to ES plugin & Redis plugin
Also correcting the assertions in the RedisPluginTest so that the error message on test failure is accurate.
* Removing the endpoint validation from datasourceServiceImpl
Moving the endpoint validation to the plugin implementation classes because there are databases that require complete HTTP URL in their configuration. Checking for http/https at the platform level affects the UX for the user & plugin developer when using a new integration. Hence, all plugins must implement their own client/server validations in their own implementations
* Adding tests to assert hostname validation in mysql & postgres plugins
* Bug Fix : Reading the datasource configuration incorrectly from actionDTO instead of datasource object.
* Added test case to assert that datasource configuration and action configurations are not null when action execute is called.
* Introduced new page which stores the published and unpublished pages as separate.
* Mid level commit to save the state.
* Parity of new page repository with old page repository (custom functions)
* WIP : Delete a page. This requires changes across application structure as well.
* Added publishedPages construct inside application to store the pages in the deployed view as well as isDefault so that the same changes (delete or isDefault) in unpublished view doesn't alter these fields for the published application
* Parity reached with PageService.
* Minor ActionService refactor to remove unnecessary code.
ApplicationPageService, LayoutActionService, LayoutService use the new page service to fetch the pages
Minor corrections in fetching the page from new page service in tests
* New save function which sets the PageDTO for unpublished page and then saves the new page into repository.
* Migration of page service functions to new page service functions across other services/tests/controller
* Finished migrating all the page service functions to the new page service functions
* Application Service Tests have been fixed.
* All the existing test cases are working now.
* Publish application implemented to store published pages as well. Added a basic test case to check that published pages is being set and that page's publishedPageDTO is being set accordingly.
* Minor TODOs added to add test cases for published application.
* A few tests to ascertain that published application page fields (deleted, isDefault) does not get changed when these statuses are changed for a page in edit mode.
* Added a new controller end point to fetch application in view mode.
* Added new endpoint for fetching an application in view mode on the client.
* Bug fix where get application in view mode API was not getting called.
* Fixed the get page names by application & archive pages which have been deleted in edit mode during publishing of application.
* During delete page, if a page was never published and it was deleted during edit, delete the entire page instead of just deleting the unpublished PageDTO
* Minor formatting.
* Non working client side code to fetch page list using view mode.
* revert unnecassary changes and streamlined view and edit actions
* Fix missed import
* Fixed a bug where if a page is not published, it should not be returned in view mode in list of page names api.
* Fixed update for a page which was not working in integration test.
* ActionDTO added.
* Solidified the new action structure.
* Migration added for NewAction index creation and NewAction per Action insertion in the database.
* Basic file structure added the new repository, custom repository, service, etc.
* Delete OldPage.java
* Repo functions added - TODO : Haven;t handled the published/edited views
* Helper functions added to convert Action to NewAction and vice-versa. Removed unused currentUserMono usage.
* Create & update action functionality added.
* Execute Action refactored. Removed dry run specific code.
* Repository migrated to handle new data structure. Execute action refactored to no longer support dry runs of actions.
* TODO added for special handling of change view of application to handle edge cases of pages/actions which either exist in published mode but don't exist in unpublished mode or vice versa.
* Migrated finding on load actions from spring repository to custom repository.
* In view mode, now actions are being fetched by application id directly instead of first fetching application and then using the page ids, fetching the actions. This reduces the db calls from 2 to 1 per fetch actions in view mode api call.
* Delete action and get all actions (used in edit mode on the client side) implemented.
* Updated CollectionService and ActionCollectionService to use the new action service instead of the old one.
* LayoutActionService refactored to now use the new service functions.
* ActionController now no longer used ActionService. The remaining service functions have been migrated to the new action service.
* Refactor across ACL code for addition/removal of policies during addition/removal of users to organization, making app public, refactor for services like policy utils, item service, etc.
* Removed the last of action repository and action service and replaced with new action repo and new action service.
* Compile and run time issues fixed. The server is coming up without any spring dependency errors.
* WIP in fixing fetching actions by page id.
* Finally!!! Fixed the fetch actions (both published and unpublished actions) by page id repository function.
* Fixed create action bug where null datasource in published actiondto (inside newly created action) leads to error.
* Fixed the execute action issues :
1. Removed the dry runs from the tests
2. Fixed the null pointer error in variable substituted action and datasource configurations.
* 1. Fixed the custom action repository field names.
2. Fixed the data structures used in ExamplesOrganizationClonerTests
* Fixed countByDatasourceId repository function which was querying the actions incorrectly.
* Fixed the clone example organization error where the id of the action was not getting updated in the page correctly. Yay!
* Fixed post merge compilation failure.
* Fixed more compilation time failures in ActionServiceTest
* Fixed failing test case for fetching actions in view mode.
* Minor changes to resolve merge changes and incorporate in the new refactored code.
* 1. Fixed compile time errors on Client code.
2. Fixed fetching of actions in view mode by application id. The repository function did not need name parameter. Removed the same.
* [Integration Testing Error Fix] : Added a new test case for refactor action name.
* Instead of fetching actions in the page, mistakenly used the base service which was fetching all the actions in the repository, barring none which led to the name refactor being blocked even though no action in the current page exists with the new proposed name,
* Added delete functionality to action service.
* Minor code cleanup
* Adding viewMode to action execution
* Replacing action with actionId.
* 1. Bug fix for deletion of unpublished action. In case of never published action, the entire action should be deleted. In case an action was published, only the unpublished action should be deleted.
2. In case of DB actions (external datasources), only the bare minimum fields should be stored inside the action (datasource id and datasource plugin id). The other fields should not be duplicated across code.
* Fixed yarn build compilation issues.
* Update app/server/appsmith-server/src/main/java/com/appsmith/server/controllers/ActionController.java
Co-authored-by: Arpit Mohan <mohanarpit@users.noreply.github.com>
* Changed the API path for GET applications in view mode. Some minor code formatting.
* Incorporated review comments.
* Some more unnecessary code removed.
* Instead of returning Page, now the interface object between client and server for Pages is PageDTO
* Migrated Page and Action to PageDTO and ActionDTO
Fixed the compilation issues.
TODO : Fix the test compilation issues.
* Fixed compilation time issues with all the tests by migrating Page and Action to PageDTO and ActionDTO respectively
* Action Controller and Page Controller no longer extend Base Controller. All the required functions have now been implemented and no base line API end points are being re-used from the base.
* Test case fixes.
* Bug Fix : Updating an action was not updating execute on load. Fixed the data flow leading to the error.
* Deprecating Page and Action domain objects. This is to ensure no new code is written with these till we remove this old code.
* Cloned example applications are now published before returning. This is to ensure that the applications are in ready to view mode when the new user signs up.
* Added a function comment to expand on the usage of new param introduced.
* When cloning a page, new actions were not being stored. Added that. Also updated the clonePage test to assert that the actions are also cloned when the pages are cloned.
* Updated a Api call
* removed extra slash
Co-authored-by: Hetu Nandu <hetunandu@gmail.com>
Co-authored-by: Satbir Singh <satbir121@gmail.com>
Co-authored-by: Arpit Mohan <mohanarpit@users.noreply.github.com>
Co-authored-by: nandan.anantharamu <nandan@thinkify.io>
* Returning validation failures while creating organization
* Added unit tests
* Changing the name of test security config
Co-authored-by: nitesh261193 <nitesh261193@gmail.com>
* Get list of application IDs to be cloned as examples
* Fix tests so that apps to be cloned are explicit
* Add migration to set applicationIds for example cloning
* Fix example org cloning tests
* Start with new controller for assets
* Progress on uploading logo image
* Saving and loading organization logo assets now works
* Remove existing logo asset before saving a new one
* Fix 500 when uploading logo for the first time
* Fix URL in response for uploading logo image
* Add test for uploading logo for ogranization
* Mild refactoring in tests
* WIP : First draft of clone applications
* Cloning of application now works correctly.
* Fixed the failing test case for page cloning.
* Added test case to assert the clone of application successfully.
* Minor function renaming to avoid confusion.
* Minor code refactoring to remove duplicate code.
1. Fixed the bug where policies are being set to empty during organization update.
2. Added email field in organization.
* Updated the test case for update organization to assert the organization policies being present post the update.
* Working version of cloning page given page id. The clone is created inside the same application and is in unpublished state.
* Added a test case for Clone Page feature
* Incorporated review comments.
* Don't always automatically create a blank personal org
It will still be created, only if a example template organization
has not been configured in the system.
* Name examples organization same as personal organization
* Minor fixes in tests
* Remove unused userService in clone tests
* Introduced new permissions for making app public and inviting users.
* WIP : Returning a subset of roles depending on the current user's role
* Added code and tests for generating hierarchical children given a role name
* Get all invite roles now returns roles depending on the current users permissions (aka - admin can invite users for all roles, developers can invite users for roles develoeprs and below)
* When invite api is called, check if the user has correct permissions ( invite permission) and check if the role for which the users are bieng invited is allowed (aka the current user has the permission to invite for the said role)
* Introduced a new permission at application level which is inherited from manage:organizations. This new permission gives the administrator of the organization to make any application public .
* Incorporated review comments.
* Minor code changes.
* Added migration script to add the new policies introduced as part of this new share modal changes.
* Null pointer exception handled due to improper data in the database.
* Integration testing bug fixes.
* Correcting the Docker image for Appsmith server for Github actions test
* Also fixing the bug where the datasource config is not returned properly for restapi-plugin
Co-authored-by: Arpit Mohan <arpit@appsmith.com>
* WIP: Not working bulk invite.
* WIP : Non compilable.
* Current test cases running.
TODO : Code Cleanup and write new test cases for bulk invite.
* Code cleanup
* Added a test case for bulk invite of users to an organization.
* Incorporated review comment.
* Merged the latest release and incorporated review comments.
* Corrected the error thrown when usernames is empty or null for invite user API.
* Overwriting the base service's getById for OrganizationService to ensure that the response contains userPermissions (by using custom repository instead of default mongo repository classes)
* Fixed the failing test cases.
* Minor code formatting.
* Fix cloning fails in some cases for organizations
Cloning currently fails in cases like the following:
- Application with no pages
- Pages with no actions
- Pages with more than one action
* Remove debug naming of cloned datasources
* Add test for organization cloning
* Add more tests for organization cloning
* Fix potential race condition in adding pages to an application
* Move db update call to add page to application, into repository
* Use `getIdCriteria` to query for document's _id
* Reuse layout that's automatically created with page
* Use correct API for updating a layout
* Commenting out organization cloning test.
* Removed organization cloning test
* Adding a dummy commit to run the pipeline.
Co-authored-by: Arpit Mohan <arpit@appsmith.com>
* Fix layouts not be updated when an action is updated
* Add test for updating onLoadActions when action updated
* Split layout and action test to separate class
* Remove duplicated test in ActionServiceTest
* Handle stale database connection from datasources
* Fix potential secondary case of stale connection error
* Fix Postgres to MySQL
* Move validity check timeout to a constant field
* Add test for recovery when stale connection error is thrown
1. Bug fix for when actions for a public application use an external datasource (aka db queries). The db queries were not getting executed because the permission for execute datasource was not getting set for anonymousUser.
2. When using the Test endpoint for datasource, if the datasource is being read from the db and there are encrypted fields, first decrypt the field and then send to the plugin for testing the co
* When cloning examples organization, clone only public applications
* Create template organization within the test
* Cleaned up test for cloning of examples organization
* Fix Mono chaning
Co-authored-by: Trisha Anand <trisha@appsmith.com>
* Create test apps and config simultaneously
Co-authored-by: Trisha Anand <trisha@appsmith.com>
* Datasource authentication object fields should only be encrypted during create or if the update object contains authentication object. Added a test case to check that update to any other field doesnt update the encrypted fields.
* Incorporated review comment
* Trying to clone the examples repo on new user signup
* Working implementation of cloning examples organization on user sign up
* Fix personal org not being created when template org is missing
* A working version of cloning of examples organization on first-login
* Add docs for methods in ExamplesOrganizationCloner
* Refactor computing user's first name into a method
* Add some menial tests for examples organization cloning
* Use explicit permissions when fetching applications and datasources
* Fix template organization config name hard-coding
* Refactor implicit permissions in service methods as function arguments
* Revert an unintended change that got committed
* Fix permission variable name
Co-authored-by: Trisha Anand <trisha@appsmith.com>
Co-authored-by: Trisha Anand <trisha@appsmith.com>
* Disable the direct `create` methods for pages and applications
* Create differently named versions of create methods to avoid ambiguity
* Removed unused pageService in DatasourceServiceTest
* Rename createPlain method to createDefault to better convey intent
* Added pageId in the ActionViewDTO and NPE check for setting the json path keys.
* Checking for both json path keys being null and not empty before copying the json path keys into action view dto.
* Encrypting the password stored in AuthenticationDTO for every db.
* Adding comment to the properties file to denote that adding encryption salt and password are mandatory to the server coming up.
* Added the encryption salt and password to server.yml to allow the github actions to succeed.
* Adding database migration to encrypt the existing passwords for authentication object (used for storing db connection username/password)
Changes to the installation script install.sh:
1. Instead of overwriting the existing encryption password or salt, giving the user an option to conserve the previous encryption credentials to ensure that the developer users do not lose access to their database configurations (passwords).
2. Added another file for writing encryption credentials (encryption.env) to ensure that we dont delete the encryption password and salt by mistake.
* On setting an application to public view, correct permissions are assigned to the application and its pages & actions.
* If anonymous user is allowed a certain permission, the all users (anonymous/logged in) should be allowed the certain permission.
* Bug fix is working. Added a test case which is not working.
* Code cleanup. Test case fixed. Now, when we set the user to enabled, we also store the password that has been passed to the flatMap.
* Minor code cleanup.
* 404 error added for a few API calls which would be specially handled on the frontend to display an ACL 404 error.
* Putting everything except GET actions, GET pages, and GET applications behind authenticated. This ensures that in the future public applications (view only) would not lead to 401 but any other page would.
* Code formatted.
* Move application configuration to be loaded from environment variables
* Remove unused sentry.properties
* Make missing value sentinel a constant and ignore all *.env files
* Removed now-used ACL properties
* Prefix RapidAPI environment variable with APPSMITH_
* Fix application properties not being loaded into static fields
* Remove application-test.properties file
* Add required env variables for test in GitHub
* Quote URLs for MongoDB and Redis in test config
* Change RAPIDAPI to RAPID_API in environment variable names
* Source .env file in the root of repo in start script
* Adding the Github action workflow for the server code
* Modifying the redis endpoint in application-test.properties to point to localhost because the Docker service exposes ports to the host
* Fixing the move action API by removing invocations to subscribe
Calling subscribe() inside function calls is an anti-pattern and we shouldn't be doing it.
The reactiveContext is not called if the subscribe() function is called in the middle of execution flows. This breaks DB queries.
* Added test case for move action.
Co-authored-by: Trisha Anand <trisha@appsmith.com>
When organizationId is null in a datasource, fail with the right error message
The organization id is subsequently used to find a matching plugin (checking for installation essentially), but when organization id is missing, this query never returns. But the error message reads as if the plugin is not installed, whereas it should've been that organization id is missing.
See merge request theappsmith/internal-tools-server!406
Adding the widget names set to the default layout for a page
During action name refactor, we require the widget names to exist in the layout. Hence all default layouts must have the widgetName set.
See merge request theappsmith/internal-tools-server!387
This is to simplify the query pane on the frontend client. The client doesn't need to have separate interfaces for sql and non-sql plugins. All queries will be sent to the server in the form of a String that is parsed in different formats based on the plugin.
Also adding test cases for PostgresPlugin. Used TestContainers to simulate the postgres db in Docker inside the Java test itself. Very useful.
Fix : In getAllApplications, instead of iterating over collections of applications which could be empty, we iterate over organizations where we are guaranteed to have atleast one organization.
In the request from frontend for datasource creation, if the data
doesn't have a `name` value, we now set it to an automatically numbered
value and save with that.
Now we have a lot of flexibility around the SecurityContext during testing. We can instantiate any type of mock users bearing different roles & permissions.
* Adding generic get(Multivalue<String, String> params) implementation to the BaseService. Now all the domain obejcts can simply leverage this base implementation for get queries out of the box for all the fields.
* IMP: For some reason, the query by example has stopped working across the board. Even the super implementation in SimpleReactiveMongoRepository doesn't work. No idea why this is happening. For the time being, have moved to Criteria queries for the get request.
Mongock is a fork of Mongobee with a better integration with
Spring. It also does not suffer from trying to access the
`system.indexes` collection.
See https://github.com/mongobee/mongobee/pull/87.
Also fixing the SeedMongoData file with the seed data to correspond to the changes made in the OrganizationRepository. We will default to using the mongoTemplate in the SeedMongoData file in the future so that we don't have to deal with ACL when we are simply populating the DB.
Also adding test case for get applications. Now we will fetch all applications that the user has read permissions to by default. It's not dependent on the organization that they are a part of.
These graphs help us map policies that are inherited from the parent and also lateral policies that are assigned to the users given that the user has a particular permission. Currently, the hierarchy has been defined for org & application. Need to cascade it to more documents such as pages & actions.
In order to create an application with valid permissions, we copy the relevant permissions from the organization of the user. This involves changes to the organizationService and OrganizationRepository as well.
Also moving the argument AclPermission to the custom Repository interface level. This is to ensure that all the service functions can invoke the same repository function with different permissions based on their requirements.
2. Only dependencies of actions over other actions are counted. If the action is dependent on widgets, thats is not counted as a dependency and is left for the frontend to handle.
Note that we have to delete the Mongo volume for the inidtb script to take effect. Else, it will not execute. To be used only for testing right now. Will figure out a more robust solution later.
Also using Google's JIB Maven plugin to reduce our Docker image footprint. Will make container upload and download much faster
We are moving to a multi-module structure so that different parts of the codebase can be exposed to the public while others can remain private. Using pf4j for plugin framework.
Also adding a build script `build.sh` which compiles the code and creates the `dist` folder for distribution purposes. Now we can build the code via
```
$ ./build.sh -DskipTests
```
