This is because this `cacheResponse` field is only used for auto-complete on the UI. The user can still run the action manually and get the auto-complete to work. The downside to sending the `cacheResponse` field in the JSON response is that for large responses, the client times out as our response times increase to 20 secs. Hence removing this for now.
In future, we'll extract the JSON schema structure from the response body and send that to the client for populating auto-complete.
* Implement caching for datasource structure
* Avoid using final fields so spring-data can load objects
* Use type variable for getStructure method
* Initial version with base structure for MySQL plugin
* Add tests for MySQL plugin structure
* Add sorting for keys in Postgres structure as well
* Show "primary key" instead of just "primary"
* Refactor to reduce inline magic strings
* Add a connection type variable for Plugin implementations
This type variable is intended to represent the type of the
connection object, if any, that the plugin will use. This will
help make the implementations more robust by leveraging Java's
type checking instead of rudimentary type casts over the
connection objects.
* Fix missing typevar usage
* Base interface for getting datasource structure for databases
* Add keys and constraints to structure for Postgres datasources
* Use connection from datasource context for computing structure
* Refactor context retrying into a separate method
* Add base datastructures for templates in entity explorer
* Fix spring circular dependency
* Add test for postgres datasource structure
* Generate column names and sample values for INSERT query
* Add LIMIT clause to generated SELECT query
* Fix tests for generated SELECT query
* Minor refactoring
* Add migration to fix incorrect action IDs for onLoad
* Remove redundant comments
* Fix migration to work with published onLoad actions as well
* Only update the action ID when correcting onLoad action IDs
* Fix migration name for correcting action IDs
Co-authored-by: Trisha Anand <trisha@appsmith.com>
* Fix migration id for correcting action IDs
* Reformat code
Co-authored-by: Trisha Anand <trisha@appsmith.com>
* Get list of application IDs to be cloned as examples
* Fix tests so that apps to be cloned are explicit
* Add migration to set applicationIds for example cloning
* Fix example org cloning tests
* Fix aliases not showing up for MySQL actions
* Fix date column display for MySQL actions
* Fix datetime columns for MySQL actions
* Add support for timestamp and year data types for MySQL actions
* Fix column label for MySQL plugin
* Add tests for MySQL temporal data types
* Add tests for alias columns in MySQL and Postgres
* Add a node.js script to dump examples organization
This dump file is used to setup initial examples organization
in a self-hosted setup.
* Remove unused variable
* Add npm run shortcut for dump script
* Start with new controller for assets
* Progress on uploading logo image
* Saving and loading organization logo assets now works
* Remove existing logo asset before saving a new one
* Fix 500 when uploading logo for the first time
* Fix URL in response for uploading logo image
* Add test for uploading logo for ogranization
* Mild refactoring in tests
* GET API actions recognized as page load actions should also be updated as executeOnLoad actions.
* Update action during updateLayout only if executeOnLoad is not set or is false.
* WIP : First draft of clone applications
* Cloning of application now works correctly.
* Fixed the failing test case for page cloning.
* Added test case to assert the clone of application successfully.
* Minor function renaming to avoid confusion.
* Minor code refactoring to remove duplicate code.
1. Fixed the bug where policies are being set to empty during organization update.
2. Added email field in organization.
* Updated the test case for update organization to assert the organization policies being present post the update.
We are currently getting the *list* of all pages, updating the
`isDefault` fields inside, and then saving the whole *list* of
all pages. If a new page got added to that list in the DB during
this process, that page would be lost. This commit fixes this
problem.
This race condition was causing tests for cloning applications
to fail *sometimes*.
* Working version of cloning page given page id. The clone is created inside the same application and is in unpublished state.
* Added a test case for Clone Page feature
* Incorporated review comments.
* Don't always automatically create a blank personal org
It will still be created, only if a example template organization
has not been configured in the system.
* Name examples organization same as personal organization
* Minor fixes in tests
* Remove unused userService in clone tests
* During add role to an organziation, the application was only inheriting from subset of the organization permissions. Generalized this code to ensure that this doesnt happen again in the future when more permissions are introduced. Refactored some code as well.
This is a hack to get around the fact that Github Actions doesn't support conditional status checks for monorepo PRs. Hence, we create similar jobs in the both server & client builds. In the server build, those jobs are dummy jobs that do nothing but satisfy the all-encompassing green tick so that PRs can be merged without using Admin privileges.
* Introduced new permissions for making app public and inviting users.
* WIP : Returning a subset of roles depending on the current user's role
* Added code and tests for generating hierarchical children given a role name
* Get all invite roles now returns roles depending on the current users permissions (aka - admin can invite users for all roles, developers can invite users for roles develoeprs and below)
* When invite api is called, check if the user has correct permissions ( invite permission) and check if the role for which the users are bieng invited is allowed (aka the current user has the permission to invite for the said role)
* Introduced a new permission at application level which is inherited from manage:organizations. This new permission gives the administrator of the organization to make any application public .
* Incorporated review comments.
* Minor code changes.
* Added migration script to add the new policies introduced as part of this new share modal changes.
* Null pointer exception handled due to improper data in the database.
* Integration testing bug fixes.
* Correcting the Docker image for Appsmith server for Github actions test
* Also fixing the bug where the datasource config is not returned properly for restapi-plugin
Co-authored-by: Arpit Mohan <arpit@appsmith.com>
* WIP: Not working bulk invite.
* WIP : Non compilable.
* Current test cases running.
TODO : Code Cleanup and write new test cases for bulk invite.
* Code cleanup
* Added a test case for bulk invite of users to an organization.
* Incorporated review comment.
* Merged the latest release and incorporated review comments.
* Corrected the error thrown when usernames is empty or null for invite user API.
* Auto-login on signup works!
* Support form-encoded data body for signup requests
* Remove debug log entry when getting session
* Refactoring and add some docs to UserSignup solution
* Move user object construction to UserSignup solution
* Redirect with error message on signup errors
* Added information to setup domain
* Fixed incorrect var reference
* Updated documentation message
* Updated env template
* updated template
* removed debug echo
* Updated Script
* Updated Text
* Removed option to connect to external mongo for fresh installs
Exit script of docker desktop is not installed
* Updated docker installation explanation
* added a question to skip custom domain if user is installing locally
* Inverted script question and
* Removed question to determine local installation
* Updated Popup CTA to Modal
Updated query templates to contain bindings
Added a message to display on API / Query errors
* updated styles for evaluated value
updated styles for auto complete
* added spaces to questions
Co-authored-by: Nikhil Nandagopal <nikhil@appsmith.com>
Keeping the function to read plugin resource generic for any type of resource. Now the parent function can also conditionally decide which plugin would require to read which resources from the filesystem.
Co-authored-by: Arpit Mohan <arpit@appsmith.com>
* Overwriting the base service's getById for OrganizationService to ensure that the response contains userPermissions (by using custom repository instead of default mongo repository classes)
* Fixed the failing test cases.
* Minor code formatting.
The client will have to send a query parameter redirectUrl or a header X-Redirect-Url in order for the server to redirect the client to the appropriate URL post login. If neither of these parameters are present, the client is redirected to /applications by default.
Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
* During signup, if the user already exists, instead of throwing a generic duplicate key exception, request the user to sign in.
* Minor change in the error message.
* Fix cloning fails in some cases for organizations
Cloning currently fails in cases like the following:
- Application with no pages
- Pages with no actions
- Pages with more than one action
* Remove debug naming of cloned datasources
* Add test for organization cloning
* Add more tests for organization cloning
* Fix potential race condition in adding pages to an application
* Move db update call to add page to application, into repository
* Use `getIdCriteria` to query for document's _id
* Reuse layout that's automatically created with page
* Use correct API for updating a layout
* Commenting out organization cloning test.
* Removed organization cloning test
* Adding a dummy commit to run the pipeline.
Co-authored-by: Arpit Mohan <arpit@appsmith.com>
* Fix layouts not be updated when an action is updated
* Add test for updating onLoadActions when action updated
* Split layout and action test to separate class
* Remove duplicated test in ActionServiceTest
The policies of `User` objects is an empty set by default. So, when using a new `User` object to call `update`, it also updates existing policies to be an empty list.
* Fixing test cases failing due to non unique names.
* Adding execute action permissions policy to match read actions policy in existing actions.
* Only save the action if there are read actions permission present (and hence could have been updated)
* Datasource test should always test the latest object given by the client (instead of fetching the same from the db). If it is a saved datasource, then we should decrypt and set the password in the datasource (unless the password is one of the changes that is being tested).
* Added function level comment to clarify that this function can no longer be used by just sending the datasource.id field set.
* Handle stale database connection from datasources
* Fix potential secondary case of stale connection error
* Fix Postgres to MySQL
* Move validity check timeout to a constant field
* Add test for recovery when stale connection error is thrown
The plugin class loader from Pf4J looks for jars under target/lib
to use to load any classes that are missing target/classes. This
PR adds the `maven-dependency-plugin` to copy the jars to this
folder automatically so the classes are available during
development.
The client can send the redirect URL in X-Redirect-Url header. The server will honour the value set and redirect the client browser to this particular url. By default the server will redirect to /applications if the custom header is missing. The redirect url can either be absolute or relative.
Co-authored-by: Shrikant Sharat Kandula <shrikant@appsmith.com>
1. Bug fix for when actions for a public application use an external datasource (aka db queries). The db queries were not getting executed because the permission for execute datasource was not getting set for anonymousUser.
2. When using the Test endpoint for datasource, if the datasource is being read from the db and there are encrypted fields, first decrypt the field and then send to the plugin for testing the co
This commit adds the Mysql plugin to the Appsmith server. We also add a migration to ensure that this plugin is installed by default for all existing organizations. The migration also adds the plugin details into the DB.
Also adding the test cases for mysql plugin.
Co-authored-by: Arpit Mohan <arpit@appsmith.com>
Co-authored-by: Hetu Nandu <hetu@appsmith.com>
Co-authored-by: Arpit Mohan <me@arpitmohan.com>
Co-authored-by: Nupur Singhal <nupursinghal@Nupurs-MacBook-Air.local>
* When cloning examples organization, clone only public applications
* Create template organization within the test
* Cleaned up test for cloning of examples organization
* Fix Mono chaning
Co-authored-by: Trisha Anand <trisha@appsmith.com>
* Create test apps and config simultaneously
Co-authored-by: Trisha Anand <trisha@appsmith.com>
* Datasource authentication object fields should only be encrypted during create or if the update object contains authentication object. Added a test case to check that update to any other field doesnt update the encrypted fields.
* Incorporated review comment
* Trying to clone the examples repo on new user signup
* Working implementation of cloning examples organization on user sign up
* Fix personal org not being created when template org is missing
* A working version of cloning of examples organization on first-login
* Add docs for methods in ExamplesOrganizationCloner
* Refactor computing user's first name into a method
* Add some menial tests for examples organization cloning
* Use explicit permissions when fetching applications and datasources
* Fix template organization config name hard-coding
* Refactor implicit permissions in service methods as function arguments
* Revert an unintended change that got committed
* Fix permission variable name
Co-authored-by: Trisha Anand <trisha@appsmith.com>
Co-authored-by: Trisha Anand <trisha@appsmith.com>
* Disable the direct `create` methods for pages and applications
* Create differently named versions of create methods to avoid ambiguity
* Removed unused pageService in DatasourceServiceTest
* Rename createPlain method to createDefault to better convey intent
* Bug fixes :
1. Get actions in view mode should be open for unauthorized requests. Fixed that.
2. set transient fields in action is called post fetching all the actions (after taking care of all the permissions). At this point, we shouldnt be using permissions to fetch datasource.
* Added pageId in the ActionViewDTO and NPE check for setting the json path keys.
* Checking for both json path keys being null and not empty before copying the json path keys into action view dto.
The MAIL_FROM & REPLY_TO addresses default to appsmith@localhost. This should be changed in the generated docker.env file during deployment to actual values.
Most email providers will not send emails unless they originate from a valid sender ID.
* Encrypting the password stored in AuthenticationDTO for every db.
* Adding comment to the properties file to denote that adding encryption salt and password are mandatory to the server coming up.
* Added the encryption salt and password to server.yml to allow the github actions to succeed.
* Adding database migration to encrypt the existing passwords for authentication object (used for storing db connection username/password)
Changes to the installation script install.sh:
1. Instead of overwriting the existing encryption password or salt, giving the user an option to conserve the previous encryption credentials to ensure that the developer users do not lose access to their database configurations (passwords).
2. Added another file for writing encryption credentials (encryption.env) to ensure that we dont delete the encryption password and salt by mistake.
* On setting an application to public view, correct permissions are assigned to the application and its pages & actions.
* If anonymous user is allowed a certain permission, the all users (anonymous/logged in) should be allowed the certain permission.
* Bug fix is working. Added a test case which is not working.
* Code cleanup. Test case fixed. Now, when we set the user to enabled, we also store the password that has been passed to the flatMap.
* Minor code cleanup.
* 404 error added for a few API calls which would be specially handled on the frontend to display an ACL 404 error.
* Putting everything except GET actions, GET pages, and GET applications behind authenticated. This ensures that in the future public applications (view only) would not lead to 401 but any other page would.
* Code formatted.
* Dates from Postgres queries now show up as ISO date strings
* Use constant for date column name in PostgresPlugin
* Fix formats for more date-time types on Postgres
* Add support for time and timestamp data types in Postgres plugin
* Add support for timestamptz column data type in Postgres plugin
* Add support for interval data type in PostgresPlugin
* Move application configuration to be loaded from environment variables
* Remove unused sentry.properties
* Make missing value sentinel a constant and ignore all *.env files
* Removed now-used ACL properties
* Prefix RapidAPI environment variable with APPSMITH_
* Fix application properties not being loaded into static fields
* Remove application-test.properties file
* Add required env variables for test in GitHub
* Quote URLs for MongoDB and Redis in test config
* Change RAPIDAPI to RAPID_API in environment variable names
* Source .env file in the root of repo in start script
* Adding the Github action workflow for the server code
* Modifying the redis endpoint in application-test.properties to point to localhost because the Docker service exposes ports to the host
* Fixing the move action API by removing invocations to subscribe
Calling subscribe() inside function calls is an anti-pattern and we shouldn't be doing it.
The reactiveContext is not called if the subscribe() function is called in the middle of execution flows. This breaks DB queries.
* Added test case for move action.
Co-authored-by: Trisha Anand <trisha@appsmith.com>
When organizationId is null in a datasource, fail with the right error message
The organization id is subsequently used to find a matching plugin (checking for installation essentially), but when organization id is missing, this query never returns. But the error message reads as if the plugin is not installed, whereas it should've been that organization id is missing.
See merge request theappsmith/internal-tools-server!406
Race condition bug caught in remove user from organization flow where multiple applicaitons/pages/actions exist for an organization.
See merge request theappsmith/internal-tools-server!407
Bug fix : Create Organization is failing because the user is read from the session and not from the repository. Now reading the user at appropriate places.
See merge request theappsmith/internal-tools-server!405
Add server-side templates support for plugins
Two main themes in this MR:
1. Add support for saving query templates as resource files inside a plugin. This has been done for both Postgres and MongoDB plugins, based on current template contents.
2. Loading of form JSON is now reactive. The blocking call has been moved inside a `Mono.fromSupplier`.
3. The loading of from JSON as well as the templates is cached and are loaded on-demand. This means that the templates are loaded once for a plugin through the lifetime of the server process, and that they are loaded only at the first time they are needed.
4. If loading of these resources fails, we try again when they are needed the next time.
See merge request theappsmith/internal-tools-server!385
Bug Fix : Application and page can only be deleted if you have manage application/page permissions respectively.
See merge request theappsmith/internal-tools-server!395
Adding the widget names set to the default layout for a page
During action name refactor, we require the widget names to exist in the layout. Hence all default layouts must have the widgetName set.
See merge request theappsmith/internal-tools-server!387
This is to simplify the query pane on the frontend client. The client doesn't need to have separate interfaces for sql and non-sql plugins. All queries will be sent to the server in the form of a String that is parsed in different formats based on the plugin.
Also adding test cases for PostgresPlugin. Used TestContainers to simulate the postgres db in Docker inside the Java test itself. Very useful.
This is because the client sends us the data for form-data in the form of a list of Property fields with keys and values. This is consistent with how the client stores & renders headers & query parameters as well.
For MediaType application/x-www-form-urlencoded we will use the bodyFormData field.
Fix : In getAllApplications, instead of iterating over collections of applications which could be empty, we iterate over organizations where we are guaranteed to have atleast one organization.
Adding the ForwardedHeaderTransform bean to enable spring to parse...
Adding the ForwardedHeaderTransform bean to enable spring to parse X-Forwarded-* headers from Nginx proxy
See merge request theappsmith/internal-tools-server!344
Automatically use a unique name for creating datasources without name.
In the request from frontend for datasource creation, if the data
doesn't have a `name` value, we now set it to an automatically numbered
value and save with that.
See merge request theappsmith/internal-tools-server!333
In the request from frontend for datasource creation, if the data
doesn't have a `name` value, we now set it to an automatically numbered
value and save with that.
Also Mono.zipping the update Action and update Layout instead of this.then because in some optimization done by Spring, one mono doesnt end up emitting.
Now we have a lot of flexibility around the SecurityContext during testing. We can instantiate any type of mock users bearing different roles & permissions.
* Adding generic get(Multivalue<String, String> params) implementation to the BaseService. Now all the domain obejcts can simply leverage this base implementation for get queries out of the box for all the fields.
* IMP: For some reason, the query by example has stopped working across the board. Even the super implementation in SimpleReactiveMongoRepository doesn't work. No idea why this is happening. For the time being, have moved to Criteria queries for the get request.
Mongock is a fork of Mongobee with a better integration with
Spring. It also does not suffer from trying to access the
`system.indexes` collection.
See https://github.com/mongobee/mongobee/pull/87.
This ensures that the client can change the config at will without depending on the platform team to add the field to the POJO.
Also deleting redundant classes and controllers for propertyPane and Widget. These are no longer required and can be removed from the codebase.
Also fixing the SeedMongoData file with the seed data to correspond to the changes made in the OrganizationRepository. We will default to using the mongoTemplate in the SeedMongoData file in the future so that we don't have to deal with ACL when we are simply populating the DB.
Also adding test case for get applications. Now we will fetch all applications that the user has read permissions to by default. It's not dependent on the organization that they are a part of.
2. Add to Page changes : Documentation object has been added in Action to handle the extra Template documentation for actions that have been imported from 3p marketplace
3. Added basic structure for rapid api plugin by copy pasting the rest api plugin
These graphs help us map policies that are inherited from the parent and also lateral policies that are assigned to the users given that the user has a particular permission. Currently, the hierarchy has been defined for org & application. Need to cascade it to more documents such as pages & actions.
Correcting the writeKey for Segment in Staging
Also adding Mono.cache in the user invite flow to ensure that the same Mono is not executed multiple times.
See merge request theappsmith/internal-tools-server!224
2. Add to Page changes : Documentation object has been added in Action to handle the extra Template documentation for actions that have been imported from 3p marketplace
3. Added basic structure for rapid api plugin by copy pasting the rest api plugin
2. Add to Page changes : Documentation object has been added in Action to handle the extra Template documentation for actions that have been imported from 3p marketplace
3. Added basic structure for rapid api plugin by copy pasting the rest api plugin
Also adding the Origin header to the BaseController create function. This is required by the user creation flow in order to customize the links in the email. For most of the controllers overriding the BaseController, the request header parameter is non-mandatory and can be skipped for testing or otherwise.
