* During add role to an organziation, the application was only inheriting from subset of the organization permissions. Generalized this code to ensure that this doesnt happen again in the future when more permissions are introduced. Refactored some code as well.
This is a hack to get around the fact that Github Actions doesn't support conditional status checks for monorepo PRs. Hence, we create similar jobs in the both server & client builds. In the server build, those jobs are dummy jobs that do nothing but satisfy the all-encompassing green tick so that PRs can be merged without using Admin privileges.
* Introduced new permissions for making app public and inviting users.
* WIP : Returning a subset of roles depending on the current user's role
* Added code and tests for generating hierarchical children given a role name
* Get all invite roles now returns roles depending on the current users permissions (aka - admin can invite users for all roles, developers can invite users for roles develoeprs and below)
* When invite api is called, check if the user has correct permissions ( invite permission) and check if the role for which the users are bieng invited is allowed (aka the current user has the permission to invite for the said role)
* Introduced a new permission at application level which is inherited from manage:organizations. This new permission gives the administrator of the organization to make any application public .
* Incorporated review comments.
* Minor code changes.
* Added migration script to add the new policies introduced as part of this new share modal changes.
* Null pointer exception handled due to improper data in the database.
* Integration testing bug fixes.
* Correcting the Docker image for Appsmith server for Github actions test
* Also fixing the bug where the datasource config is not returned properly for restapi-plugin
Co-authored-by: Arpit Mohan <arpit@appsmith.com>
* WIP: Not working bulk invite.
* WIP : Non compilable.
* Current test cases running.
TODO : Code Cleanup and write new test cases for bulk invite.
* Code cleanup
* Added a test case for bulk invite of users to an organization.
* Incorporated review comment.
* Merged the latest release and incorporated review comments.
* Corrected the error thrown when usernames is empty or null for invite user API.
* Auto-login on signup works!
* Support form-encoded data body for signup requests
* Remove debug log entry when getting session
* Refactoring and add some docs to UserSignup solution
* Move user object construction to UserSignup solution
* Redirect with error message on signup errors
* Added information to setup domain
* Fixed incorrect var reference
* Updated documentation message
* Updated env template
* updated template
* removed debug echo
* Updated Script
* Updated Text
* Removed option to connect to external mongo for fresh installs
Exit script of docker desktop is not installed
* Updated docker installation explanation
* added a question to skip custom domain if user is installing locally
* Inverted script question and
* Removed question to determine local installation
* Updated Popup CTA to Modal
Updated query templates to contain bindings
Added a message to display on API / Query errors
* updated styles for evaluated value
updated styles for auto complete
* added spaces to questions
Co-authored-by: Nikhil Nandagopal <nikhil@appsmith.com>
Keeping the function to read plugin resource generic for any type of resource. Now the parent function can also conditionally decide which plugin would require to read which resources from the filesystem.
Co-authored-by: Arpit Mohan <arpit@appsmith.com>
* Overwriting the base service's getById for OrganizationService to ensure that the response contains userPermissions (by using custom repository instead of default mongo repository classes)
* Fixed the failing test cases.
* Minor code formatting.
The client will have to send a query parameter redirectUrl or a header X-Redirect-Url in order for the server to redirect the client to the appropriate URL post login. If neither of these parameters are present, the client is redirected to /applications by default.
Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
* During signup, if the user already exists, instead of throwing a generic duplicate key exception, request the user to sign in.
* Minor change in the error message.
* Fix cloning fails in some cases for organizations
Cloning currently fails in cases like the following:
- Application with no pages
- Pages with no actions
- Pages with more than one action
* Remove debug naming of cloned datasources
* Add test for organization cloning
* Add more tests for organization cloning
* Fix potential race condition in adding pages to an application
* Move db update call to add page to application, into repository
* Use `getIdCriteria` to query for document's _id
* Reuse layout that's automatically created with page
* Use correct API for updating a layout
* Commenting out organization cloning test.
* Removed organization cloning test
* Adding a dummy commit to run the pipeline.
Co-authored-by: Arpit Mohan <arpit@appsmith.com>
* Fix layouts not be updated when an action is updated
* Add test for updating onLoadActions when action updated
* Split layout and action test to separate class
* Remove duplicated test in ActionServiceTest
The policies of `User` objects is an empty set by default. So, when using a new `User` object to call `update`, it also updates existing policies to be an empty list.
* Fixing test cases failing due to non unique names.
* Adding execute action permissions policy to match read actions policy in existing actions.
* Only save the action if there are read actions permission present (and hence could have been updated)
* Datasource test should always test the latest object given by the client (instead of fetching the same from the db). If it is a saved datasource, then we should decrypt and set the password in the datasource (unless the password is one of the changes that is being tested).
* Added function level comment to clarify that this function can no longer be used by just sending the datasource.id field set.
* Handle stale database connection from datasources
* Fix potential secondary case of stale connection error
* Fix Postgres to MySQL
* Move validity check timeout to a constant field
* Add test for recovery when stale connection error is thrown
The plugin class loader from Pf4J looks for jars under target/lib
to use to load any classes that are missing target/classes. This
PR adds the `maven-dependency-plugin` to copy the jars to this
folder automatically so the classes are available during
development.
The client can send the redirect URL in X-Redirect-Url header. The server will honour the value set and redirect the client browser to this particular url. By default the server will redirect to /applications if the custom header is missing. The redirect url can either be absolute or relative.
Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
1. Bug fix for when actions for a public application use an external datasource (aka db queries). The db queries were not getting executed because the permission for execute datasource was not getting set for anonymousUser.
2. When using the Test endpoint for datasource, if the datasource is being read from the db and there are encrypted fields, first decrypt the field and then send to the plugin for testing the co
This commit adds the Mysql plugin to the Appsmith server. We also add a migration to ensure that this plugin is installed by default for all existing organizations. The migration also adds the plugin details into the DB.
Also adding the test cases for mysql plugin.
Co-authored-by: Arpit Mohan <arpit@appsmith.com>
Co-authored-by: Hetu Nandu <hetu@appsmith.com>
Co-authored-by: Arpit Mohan <me@arpitmohan.com>
Co-authored-by: Nupur Singhal <nupursinghal@Nupurs-MacBook-Air.local>
* When cloning examples organization, clone only public applications
* Create template organization within the test
* Cleaned up test for cloning of examples organization
* Fix Mono chaning
Co-authored-by: Trisha Anand <trisha@appsmith.com>
* Create test apps and config simultaneously
Co-authored-by: Trisha Anand <trisha@appsmith.com>
* Datasource authentication object fields should only be encrypted during create or if the update object contains authentication object. Added a test case to check that update to any other field doesnt update the encrypted fields.
* Incorporated review comment
* Trying to clone the examples repo on new user signup
* Working implementation of cloning examples organization on user sign up
* Fix personal org not being created when template org is missing
* A working version of cloning of examples organization on first-login
* Add docs for methods in ExamplesOrganizationCloner
* Refactor computing user's first name into a method
* Add some menial tests for examples organization cloning
* Use explicit permissions when fetching applications and datasources
* Fix template organization config name hard-coding
* Refactor implicit permissions in service methods as function arguments
* Revert an unintended change that got committed
* Fix permission variable name
Co-authored-by: Trisha Anand <trisha@appsmith.com>
Co-authored-by: Trisha Anand <trisha@appsmith.com>
* Disable the direct `create` methods for pages and applications
* Create differently named versions of create methods to avoid ambiguity
* Removed unused pageService in DatasourceServiceTest
* Rename createPlain method to createDefault to better convey intent
* Bug fixes :
1. Get actions in view mode should be open for unauthorized requests. Fixed that.
2. set transient fields in action is called post fetching all the actions (after taking care of all the permissions). At this point, we shouldnt be using permissions to fetch datasource.
* Added pageId in the ActionViewDTO and NPE check for setting the json path keys.
* Checking for both json path keys being null and not empty before copying the json path keys into action view dto.
The MAIL_FROM & REPLY_TO addresses default to appsmith@localhost. This should be changed in the generated docker.env file during deployment to actual values.
Most email providers will not send emails unless they originate from a valid sender ID.
* Encrypting the password stored in AuthenticationDTO for every db.
* Adding comment to the properties file to denote that adding encryption salt and password are mandatory to the server coming up.
* Added the encryption salt and password to server.yml to allow the github actions to succeed.
* Adding database migration to encrypt the existing passwords for authentication object (used for storing db connection username/password)
Changes to the installation script install.sh:
1. Instead of overwriting the existing encryption password or salt, giving the user an option to conserve the previous encryption credentials to ensure that the developer users do not lose access to their database configurations (passwords).
2. Added another file for writing encryption credentials (encryption.env) to ensure that we dont delete the encryption password and salt by mistake.
* On setting an application to public view, correct permissions are assigned to the application and its pages & actions.
* If anonymous user is allowed a certain permission, the all users (anonymous/logged in) should be allowed the certain permission.
* Bug fix is working. Added a test case which is not working.
* Code cleanup. Test case fixed. Now, when we set the user to enabled, we also store the password that has been passed to the flatMap.
* Minor code cleanup.
* 404 error added for a few API calls which would be specially handled on the frontend to display an ACL 404 error.
* Putting everything except GET actions, GET pages, and GET applications behind authenticated. This ensures that in the future public applications (view only) would not lead to 401 but any other page would.
* Code formatted.
* Dates from Postgres queries now show up as ISO date strings
* Use constant for date column name in PostgresPlugin
* Fix formats for more date-time types on Postgres
* Add support for time and timestamp data types in Postgres plugin
* Add support for timestamptz column data type in Postgres plugin
* Add support for interval data type in PostgresPlugin
* Move application configuration to be loaded from environment variables
* Remove unused sentry.properties
* Make missing value sentinel a constant and ignore all *.env files
* Removed now-used ACL properties
* Prefix RapidAPI environment variable with APPSMITH_
* Fix application properties not being loaded into static fields
* Remove application-test.properties file
* Add required env variables for test in GitHub
* Quote URLs for MongoDB and Redis in test config
* Change RAPIDAPI to RAPID_API in environment variable names
* Source .env file in the root of repo in start script
* Adding the Github action workflow for the server code
* Modifying the redis endpoint in application-test.properties to point to localhost because the Docker service exposes ports to the host
* Fixing the move action API by removing invocations to subscribe
Calling subscribe() inside function calls is an anti-pattern and we shouldn't be doing it.
The reactiveContext is not called if the subscribe() function is called in the middle of execution flows. This breaks DB queries.
* Added test case for move action.
Co-authored-by: Trisha Anand <trisha@appsmith.com>
When organizationId is null in a datasource, fail with the right error message
The organization id is subsequently used to find a matching plugin (checking for installation essentially), but when organization id is missing, this query never returns. But the error message reads as if the plugin is not installed, whereas it should've been that organization id is missing.
See merge request theappsmith/internal-tools-server!406
Race condition bug caught in remove user from organization flow where multiple applicaitons/pages/actions exist for an organization.
See merge request theappsmith/internal-tools-server!407
Bug fix : Create Organization is failing because the user is read from the session and not from the repository. Now reading the user at appropriate places.
See merge request theappsmith/internal-tools-server!405
Add server-side templates support for plugins
Two main themes in this MR:
1. Add support for saving query templates as resource files inside a plugin. This has been done for both Postgres and MongoDB plugins, based on current template contents.
2. Loading of form JSON is now reactive. The blocking call has been moved inside a `Mono.fromSupplier`.
3. The loading of from JSON as well as the templates is cached and are loaded on-demand. This means that the templates are loaded once for a plugin through the lifetime of the server process, and that they are loaded only at the first time they are needed.
4. If loading of these resources fails, we try again when they are needed the next time.
See merge request theappsmith/internal-tools-server!385
Bug Fix : Application and page can only be deleted if you have manage application/page permissions respectively.
See merge request theappsmith/internal-tools-server!395
Adding the widget names set to the default layout for a page
During action name refactor, we require the widget names to exist in the layout. Hence all default layouts must have the widgetName set.
See merge request theappsmith/internal-tools-server!387
This is to simplify the query pane on the frontend client. The client doesn't need to have separate interfaces for sql and non-sql plugins. All queries will be sent to the server in the form of a String that is parsed in different formats based on the plugin.
Also adding test cases for PostgresPlugin. Used TestContainers to simulate the postgres db in Docker inside the Java test itself. Very useful.
This is because the client sends us the data for form-data in the form of a list of Property fields with keys and values. This is consistent with how the client stores & renders headers & query parameters as well.
For MediaType application/x-www-form-urlencoded we will use the bodyFormData field.
Fix : In getAllApplications, instead of iterating over collections of applications which could be empty, we iterate over organizations where we are guaranteed to have atleast one organization.
Adding the ForwardedHeaderTransform bean to enable spring to parse...
Adding the ForwardedHeaderTransform bean to enable spring to parse X-Forwarded-* headers from Nginx proxy
See merge request theappsmith/internal-tools-server!344
Automatically use a unique name for creating datasources without name.
In the request from frontend for datasource creation, if the data
doesn't have a `name` value, we now set it to an automatically numbered
value and save with that.
See merge request theappsmith/internal-tools-server!333
In the request from frontend for datasource creation, if the data
doesn't have a `name` value, we now set it to an automatically numbered
value and save with that.
Also Mono.zipping the update Action and update Layout instead of this.then because in some optimization done by Spring, one mono doesnt end up emitting.
Now we have a lot of flexibility around the SecurityContext during testing. We can instantiate any type of mock users bearing different roles & permissions.
* Adding generic get(Multivalue<String, String> params) implementation to the BaseService. Now all the domain obejcts can simply leverage this base implementation for get queries out of the box for all the fields.
* IMP: For some reason, the query by example has stopped working across the board. Even the super implementation in SimpleReactiveMongoRepository doesn't work. No idea why this is happening. For the time being, have moved to Criteria queries for the get request.
Mongock is a fork of Mongobee with a better integration with
Spring. It also does not suffer from trying to access the
`system.indexes` collection.
See https://github.com/mongobee/mongobee/pull/87.
This ensures that the client can change the config at will without depending on the platform team to add the field to the POJO.
Also deleting redundant classes and controllers for propertyPane and Widget. These are no longer required and can be removed from the codebase.
Also fixing the SeedMongoData file with the seed data to correspond to the changes made in the OrganizationRepository. We will default to using the mongoTemplate in the SeedMongoData file in the future so that we don't have to deal with ACL when we are simply populating the DB.
Also adding test case for get applications. Now we will fetch all applications that the user has read permissions to by default. It's not dependent on the organization that they are a part of.
2. Add to Page changes : Documentation object has been added in Action to handle the extra Template documentation for actions that have been imported from 3p marketplace
3. Added basic structure for rapid api plugin by copy pasting the rest api plugin
These graphs help us map policies that are inherited from the parent and also lateral policies that are assigned to the users given that the user has a particular permission. Currently, the hierarchy has been defined for org & application. Need to cascade it to more documents such as pages & actions.
Correcting the writeKey for Segment in Staging
Also adding Mono.cache in the user invite flow to ensure that the same Mono is not executed multiple times.
See merge request theappsmith/internal-tools-server!224
2. Add to Page changes : Documentation object has been added in Action to handle the extra Template documentation for actions that have been imported from 3p marketplace
3. Added basic structure for rapid api plugin by copy pasting the rest api plugin
2. Add to Page changes : Documentation object has been added in Action to handle the extra Template documentation for actions that have been imported from 3p marketplace
3. Added basic structure for rapid api plugin by copy pasting the rest api plugin
Also adding the Origin header to the BaseController create function. This is required by the user creation flow in order to customize the links in the email. For most of the controllers overriding the BaseController, the request header parameter is non-mandatory and can be skipped for testing or otherwise.
In order to create an application with valid permissions, we copy the relevant permissions from the organization of the user. This involves changes to the organizationService and OrganizationRepository as well.
Also moving the argument AclPermission to the custom Repository interface level. This is to ensure that all the service functions can invoke the same repository function with different permissions based on their requirements.
This is present in the BaseRepositoryImpl. In this manner, we can add default conditions to all the basic JPA queries. Now we only need to figure out how to intercept the custom queries to add user details into.
This is to ensure that setting the createdAt nad updatedAt properties by spring-data will still succeed even if the client doesn't provide us with the datasource details.
Creating an embedded datasource for the action
Now the user can create an embedded datasource as well as a saved datasource for the action.
This MR also fixes the import for a curl command by setting defaults when an action is created via the curl command. For now, the curl command only accepts short flags and not the long form flags.
See merge request theappsmith/internal-tools-server!204
We now create another list from the value provided in the properties file. All checks in the codebase are performed against this list. This ensures that there are no NPE and exceptions when the property oauth2.allowed-domains is removed from the properties file.
Correcting the error response returned by the AclFilter to match the ErrorDTO...
Correcting the error response returned by the AclFilter to match the ErrorDTO returned from controller functions
This ensures consistent responses from different parts of our application. The client can then rely on the server to provide a constistent response structure.
See merge request theappsmith/internal-tools-server!194
This ensures consistent responses from different parts of our application. The client can then rely on the server to provide a constistent response structure.
Renaming isPaginated to PaginationType to support different types of pagination on the frontend
If the API is paginated via the table's page number, then the next and prev URLs in the action don't play any role. Those fields are only required when the paginationType is of type URL. In all other scenarios, the action is executed normally.
See merge request theappsmith/internal-tools-server!192
If the API is paginated via the table's page number, then the next and prev URLs in the action don't play any role. Those fields are only required when the paginationType is of type URL. In all other scenarios, the action is executed normally.
Now, regardless of the route with which an organization is created (invitation, signup or self-create by user), the correct groups and permissions are assigned to the user.
We override the SimpleReactiveMongoRepository with our custom implementation to add criteria for filtering soft deleted records.
Also, adding a new function to archive record instead of a hard delete.
/actions should return all the actions of the current organization only.
Added organizationId to Action. This gets set only during create operation. Whenever get is performed on actions, it is filtered using organizationId.
See merge request theappsmith/internal-tools-server!164
At the moment, reactive Spring does not support AuditorAware implementation which also inserts the username of the user making the changes. Reference: https://jira.spring.io/browse/DATACMNS-1231.
This is to ensure that when we deploy in production, we can add redirect rules on Nginx. Now /api and /oauth2 urls will proxy to the API server while all other URLs will proxy to the frontend server.
Cleaning up the Github OAuth2 login flow by adding userNameAttribute to properties
This property helps define for Spring security which field in the OAuth2 user info to read in order to determine the username of the user. This is because this field is non-standard across different OAuth2 implementations. For each new OAuth2 provider that we support, this field will be required. Else the default name field will be picked up by Spring security (which is usually the id of the user).
See merge request theappsmith/internal-tools-server!156
This property helps define for Spring security which field in the OAuth2 user info to read in order to determine the username of the user. This is because this field is non-standard across different OAuth2 implementations. For each new OAuth2 provider that we support, this field will be required. Else the default name field will be picked up by Spring security (which is usually the id of the user).
The domain restriction has been done by adding parameter `hd` in the function CustomServerOAuth2AuthorizationRequestResolver#authorizationRequest. We still verify if the OAuth2 response has the parameter `hd` to ensure that no client side manipulation has been performed.
This property helps the spring security library derive the host name, protocol and port accurately even while running behind a Nginx load balancer. This is because nginx adds X-Forward-* headers that are parsed by Spring security library.
