AWS AMI configuration script (#1471)

* Add Aws AMI deployment guideline, fix configure-ssl.sh script

Co-authored-by: Minh Hieu <hieu.dv@geekup.io>

deploy/aws_ami/README.md

@@ -0,0 +1,154 @@
+# Introduction
+In this tutorial, I’ll walk you, step by step, through the process of using the AWS Console to create and provision a new AWS cloud server. And since AWS offers a Free Tier valid for 12 months, you’ll have plenty of time to experiment with your server and Appsmith image without worrying about being billed for usage.
+
+# Overview
+In this tutorial, I’ll launch a cloud server with the Appsmith Stack, which gives you a pre-configured Appsmith web application that you can start using right way.
+
+Here are the steps you’ll follow in this tutorial:
+- Register with Amazon Web Servces (AWS) 
+- Generate a SSH key pair
+- Create an AWS Security Group
+- Deploy Appsmith on an AWS cloud server
+- Log in and start using Appsmith
+- Configure custom domain for your app
+
+The next sections will walk you through these steps in detail.
+
+# Step 1: Register With Amazon Web Services (AWS)
+```
+At the end of this step, you will have signed up for the Amazon Web Services free tier. If you already have an Amazon Web Services account, you may skip this step.
+```
+You will need an existing Amazon account to log in and sign up. To create it, follow these steps:
+  - Browse to [http://aws.amazon.com](http://aws.amazon.com) and click the “Create an AWS account” button at the top of the page.
+  - In the resulting page, enter an email address, a password, and an AWS account name. Then, click “Continue” to start the registration process.
+  ![Aws Register Page](./images/aws-account-info.png)
+  - Once you’ve signed in to Amazon, sign up for AWS by selecting the account type and providing some basic contact information and your mobile phone number. 
+  ![Aws Register Page](./images/aws-account-info-2.png)
+  - Once that’s done, proceed to the next stage by entering your credit card information. Click the “Secure Submit” button to continue with the account creation.
+  ![Aws Payment](./images/aws-payment.png)
+  - Amazon will now verify your identity, by making an automated call to your mobile phone number and prompting you to enter the PIN number displayed on the screen.
+  - Once your identity is verified, choose the “Basic” support plan (also free) and confirm your account.
+  - The AWS account registration machine will churn away for a minute or so, and you will then be redirected to a welcome page, which includes a link to the AWS management console. You should also receive an account confirmation email, which tells you that your account is good to go. 
+ 
+
+
+
+
+# Step 2: Generate a SSH key pair
+```
+At the end of this step, you will have generated an SSH key pair to access your EC2 instances. If you already have an SSH key pair for the AWS region you are operating in, you can skip this step.
+```
+
+To generate an SSH key pair, which you will need to log in to your EC2 instances, follow the steps below:
+
+- Log in to the AWS Console.
+
+- Select the EC2 service from Amazon Web Services menu.
+    ![AWS Services](./images/aws-services.png)
+
+- If required, use the region selector in the top right corner to switch to the region where your instance will be launched.
+
+- From the Amazon EC2 dashboard, select the “Key Pairs” option in the “Network & Security” menu.
+    ![Key pairs](./images/key-pairs.png)
+
+- Click the “Create Key Pair” button. Then enter a name for the new key pair in the open dialog and click the “Create” button.
+    ![Create key pairs](./images/create-key-pairs.png)
+
+- A new key pair, consisting of an SSH public and private key, will be generated. You will be prompted to download the private SSH key to your computer.
+
+# Step 3: Create an AWS Security Group
+```
+At the end of this step, you will have created an AWS security group for your cloud server.
+```
+
+By default, AWS cloud servers have their ports closed to secure them against external attacks. Since Appsmith is a Web application, it is necessary to open ports 80 and 443 for HTTP access, and port 22 for SSH access. To do this:
+
+- From the Amazon Web Services menu, select the EC2 service.
+  ![AWS Services](./images/aws-services.png)
+- From the Amazon EC2 dashboard, select the “Security Groups” option in the “Network & Security” menu.<image>
+  ![AWS Security Group](./images/security-group.png)
+- Click the “Create Security Group” button.
+- In the Create Security Group page, enter a name and description for the for the new security group.
+  ![AWS Security Group Page](./images/security-group-page.png)
+- Click the “Add Rule” button in "Inbound Rule" section and add new rules for HTTP, HTTPS and SSH access using the following guidelines:
+  - Type: Use the pre-defined types “HTTP”, “HTTPS” and “SSH”.
+  - Source: Use “Anywhere” to allow access from anywhere, or use “Custom IP” and specify an IP address range.
+  ![Inbound Rules](./images/inbound-rules.png)
+- Click the “Create” button to save your changes.
+
+# Step 4: Deploy Appsmith On An AWS Cloud Server
+```
+At the end of this step, your Appsmith blog will be running on an AWS cloud server.
+```
+
+The next step is to launch a cloud server with the Appsmith Amazon Machine Image (AMI) running on it. The AWS Console lets you do this in just a couple of clicks. Follow these steps:
+
+- From the Amazon EC2 dashboard, select the “AMIs” option in the “Images” menu.  
+  ![AMI](./images/AMI.png)
+- Search for the Appsmith Stack by entering the search term “appsmith” in the search bar at the top.
+  ![Search AMI](./images/search-ami.png)
+
+- Select the image in the list of search results and click the “Launch” button.
+
+- On the resulting detail page, review the available server sizes. Select the server size you wish to use and click “Review and Launch” to proceed.
+  ![Instance Preview](./images/preview.png)
+
+- On the review page, click the “Edit security groups” link
+
+- On the “Configure Security Group” page, choose the option to “Select an existing security group”. Find the security group you created in Step 3 and select it. Click the “Review and Launch” button to proceed.
+
+- Verify that the correct key pair (created in Step 2) will be used for the server.
+
+- Confirm your selection by hitting the “Launch Instances” button.
+
+The AWS Console will now begin spinning up the new server.
+![Launch](./images/launch.png)
+
+The process usually takes a few minutes, use EC2 Dashboard to check the status of the server. Once the server has launched, you will be able to obtain its public IP address from the EC2 Dashboard, as shown below: 
+
+![EC2 Detail](./images/ec2-detail.png)
+
+At this point, you should be able to browse to the cloud server, by entering the cloud server IP address or DNS name directly into your browser’s address bar. You should now see your webapp home page as shown below:
+![Login Page](./images/login-page.png)
+
+
+## Step 5: Login & start using Appsmith
+```
+At the end of this step, you will have logged in to Appsmith and build your internal tools
+```
+
+To log in to the Appsmith dashboard, follow these steps:
+- You will need to register an account to login
+  - Go to Sign Up page, fill in your user's email & password, then click Sign Up
+  ![Appsmith Signup Page](./images/appsmith_signup.png)
+
+Once you completed the registration, you will be moved to Personal Organization page. Now you can begin to create your custom app. 
+![Appsmith Dashboard Page](./images/appsmith_dashboard.png)
+
+For more tutorials on how to create your custom app by Appsmith, please take a look at [https://docs.appsmith.com/](https://docs.appsmith.com/)
+
+
+## Step 6: Configure custom domain for your app
+```
+At the end of this step, you should be able to access to your web app by your custom domain
+```
+
+The next step is to configure your app so that it can be access by your custom domain. Follow these steps:
+
+- Once your instance is ready, connect to that instance (via SSH) using your key pair (Create in step 2) and the public ip of your instance (Create in step 4) by terminal or any SSH Client that you have
+- Move to `/home/ubuntu/appsmith/scripts`
+
+- Run `configure-ssl.sh` script
+  ```
+  ./configure-ssl.sh
+  ```
+
+- You will be asked to your input domain to proceed the configuration(Please make sure that you have map your domain with EC2 Instance's public ip)
+
+    ![Custom Domain](./images/custom-domain.png)
+
+- There will be option for you to configure SSL for your domain
+  ![SSL](./images/ssl.png)
+
+At this point, you should be able to browse to the cloud server, by entering the your custom domain directly into your browser’s address bar. You should be able to see your webapp home page now
+![Login Page](./images/login-page.png)

deploy/aws_ami/base-install.sh

@@ -0,0 +1,59 @@
+#!/bin/bash
+
+set -o errexit
+
+if [[ $EUID > 0 ]]; then
+    echo "Please run with sudo." >&2
+    exit 1
+fi
+
+install_package() {
+    sudo apt-get -y update --quiet 
+    sudo apt-get install -y ntp bc python3-pip --quiet
+    pip3 install boto3
+    sudo apt-get install -y apt-transport-https ca-certificates curl software-properties-common virtualenv python3-setuptools --quiet
+
+    # Installing docker
+    sudo apt-get  -y --quiet install gnupg-agent
+    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
+    sudo add-apt-repository \
+    "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
+    $(lsb_release -cs) \
+    stable"
+
+    sudo apt-get -y update --quiet
+    sudo apt-get -y install docker-ce docker-ce-cli containerd.io --quiet
+    sudo usermod -aG docker $USER
+
+    # Installing docker compose
+    if [ ! -f /usr/bin/docker-compose ];then
+         echo "Installing docker-compose"
+         sudo curl -L "https://github.com/docker/compose/releases/download/1.26.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
+         sudo chmod +x /usr/local/bin/docker-compose
+         sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
+     fi
+
+}
+
+install_package
+
+#Download boot.sh and schedule at boot time.
+app_path="/home/ubuntu/appsmith"
+script_path="script"
+boot_script_path=$app_path/$script_path
+boot_file_name="boot.sh"
+first_time_setup_file_name="first-time-setup.sh"
+config_ssl_file_name="configure-ssl.sh"
+mkdir -p $boot_script_path
+sudo chown -R ubuntu:ubuntu $app_path
+cd $boot_script_path
+
+sudo curl -O https://raw.githubusercontent.com/appsmithorg/appsmith/release/deploy/aws_ami/boot.sh
+sudo curl -O https://raw.githubusercontent.com/appsmithorg/appsmith/release/deploy/aws_ami/first-time-setup.sh
+sudo chown ubuntu:ubuntu $boot_script_path/$boot_file_name && sudo chmod +x $boot_script_path/$boot_file_name
+sudo chown ubuntu:ubuntu $boot_script_path/$first_time_setup_file_name && sudo chmod +x $boot_script_path/$first_time_setup_file_name
+
+USER="ubuntu"
+CRON_FILE="/etc/cron.d/appsmith"
+echo "@reboot /bin/bash $boot_script_path/$boot_file_name" > $CRON_FILE
+sudo chmod 0600 $CRON_FILE

deploy/aws_ami/boot.sh

@@ -0,0 +1,30 @@
+#!/bin/bash
+set -o errexit
+# Check if Lock File exists, if not create it and set trap on exit
+if { set -C; 2>/dev/null >/home/ubuntu/.appsmith.lock; }; then
+    trap "rm -f /home/ubuntu/.appsmith.lock" EXIT
+else
+    exit
+fi
+
+start_docker() {
+    if [ `sudo systemctl is-active docker.service` == "inactive" ];then
+        echo "Starting docker"
+        sudo systemctl start docker.service
+    fi
+}
+start_docker
+
+install_dir="/home/ubuntu/appsmith"
+# Check if Apsmith setup, if not create run setup up script
+if [ ! -f $install_dir/docker-compose.yml ]; then
+    echo ""
+    echo "Setting Appsmith"
+    first_time_setup_script=$install_dir/scripts/first-time-setup.sh 
+    chmod +x $first_time_setup_script;
+    /bin/bash $first_time_setup_script
+else
+    echo "Booting appsmith"
+    cd $install_dir
+    docker-compose up --detach --remove-orphans
+fi

deploy/aws_ami/configure-ssl.sh

@@ -0,0 +1,66 @@
+#!/bin/bash
+set -o errexit
+
+install_dir="/home/ubuntu/appsmith"
+config_ssl_pwd=$(pwd)
+
+confirm() {
+    local default="$1"  # Should be `y` or `n`.
+    local prompt="$2"
+
+    local options="y/N"
+    if [[ $default == y || $default == Y ]]; then
+        options="Y/n"
+    fi
+
+    local answer
+    read -rp "$prompt [$options] " answer
+    if [[ -z $answer ]]; then
+        # No answer given, the user just hit the Enter key. Take the default value as the answer.
+        answer="$default"
+    else
+        # An answer was given. This means the user didn't get to hit Enter so the cursor on the same line. Do an empty
+        # echo so the cursor moves to a new line.
+        echo
+    fi
+
+    [[ yY =~ $answer ]]
+}
+
+read -rp 'Enter the domain or subdomain on which you want to host appsmith (example.com / app.example.com): ' custom_domain
+echo "Would you like to provision an SSL certificate for your custom domain / subdomain?"
+if confirm y '(Your DNS records must be updated for us to proceed)'; then
+    ssl_enable="true"
+fi
+
+if [[ -z ssl_enable ]]; then
+    NGINX_SSL_CMNT="#"
+fi
+
+templates_dir="$(mktemp -d)"
+mkdir -p "$templates_dir"
+
+cd "$templates_dir"
+curl --remote-name-all --silent --show-error \
+    https://raw.githubusercontent.com/appsmithorg/appsmith/master/deploy/template/nginx_app.conf.sh \
+
+bash "$templates_dir/nginx_app.conf.sh" "$NGINX_SSL_CMNT" "$custom_domain" > "$install_dir/data/nginx/app.conf.template"
+
+current_dir=$(pwd)
+init_letsencrypt_file="$config_ssl_pwd/init-letsencrypt.sh"
+
+if ! [[ -z $ssl_enable ]]; then
+    sudo chown ubuntu:ubuntu "$init_letsencrypt_file" && sudo chmod +x "$init_letsencrypt_file"
+    /bin/bash "$init_letsencrypt_file" "$custom_domain"
+fi
+
+echo "+++++++++++ SUCCESS ++++++++++++++++++++++++++++++"
+echo "Your installation is complete!"
+echo ""
+
+if [[ -z $ssl_enable ]]; then
+    echo "Your application is running on 'http://$custom_domain'."
+else
+    echo "Your application is running on 'https://$custom_domain'."
+fi
+

deploy/aws_ami/first-time-setup.sh

@@ -0,0 +1,82 @@
+#!/bin/bash
+
+# helpers function
+move_file() {
+    local relative_path="$1"
+    local template_file="$2"
+    local full_path="$install_dir/$relative_path"
+    mv -f "$template_file" "$full_path"
+}
+
+generate_random_string() {
+    # Picked up the following method of generation from : https://gist.github.com/earthgecko/3089509
+    LC_CTYPE=C tr -dc 'a-zA-Z0-9' < /dev/urandom | fold -w 13 | head -n 1
+}
+
+urlencode() {
+    # urlencode <string>
+    local old_lc_collate="$LC_COLLATE"
+    LC_COLLATE=C
+
+    local length="${#1}"
+    for (( i = 0; i < length; i++ )); do
+        local c="${1:i:1}"
+        case $c in
+            [a-zA-Z0-9.~_-]) printf "$c" ;;
+            *) printf '%%%02X' "'$c" ;;
+        esac
+    done
+
+    LC_COLLATE="$old_lc_collate"
+}
+
+## Default configuration
+install_dir="/home/ubuntu/appsmith"
+mongo_host="mongo"
+mongo_database="appsmith"
+mongo_root_user=$(generate_random_string)
+mongo_root_password=$(generate_random_string)
+user_encryption_password=$(generate_random_string)
+user_encryption_salt=$(generate_random_string)
+NGINX_SSL_CMNT="#"
+
+# Step 1: Download the templates
+echo "Downloading the configuration templates..."
+templates_dir="$(mktemp -d)"
+mkdir -p "$templates_dir"
+(
+    cd "$templates_dir"
+    curl --remote-name-all --silent --show-error \
+        https://raw.githubusercontent.com/appsmithorg/appsmith/master/deploy/template/docker-compose.yml.sh \
+        https://raw.githubusercontent.com/appsmithorg/appsmith/master/deploy/template/mongo-init.js.sh \
+        https://raw.githubusercontent.com/appsmithorg/appsmith/master/deploy/template/docker.env.sh \
+        https://raw.githubusercontent.com/appsmithorg/appsmith/master/deploy/template/nginx_app.conf.sh \
+        https://raw.githubusercontent.com/appsmithorg/appsmith/master/deploy/template/encryption.env.sh
+)
+# Step 2: Generate config from template
+mkdir -p "$install_dir/data/"{nginx,mongo/db}
+echo "Generating the configuration files from the templates"
+bash "$templates_dir/nginx_app.conf.sh" "$NGINX_SSL_CMNT" "$custom_domain" > nginx_app.conf
+bash "$templates_dir/docker-compose.yml.sh" "$mongo_root_user" "$mongo_root_password" "$mongo_database" > docker-compose.yml
+bash "$templates_dir/mongo-init.js.sh" "$mongo_root_user" "$mongo_root_password" > mongo-init.js
+bash "$templates_dir/docker.env.sh" "$mongo_root_user" "$mongo_root_password" "$mongo_host" > docker.env
+bash "$templates_dir/encryption.env.sh" "$user_encryption_password" "$user_encryption_salt" > encryption.env
+
+move_file "data/nginx/app.conf.template" "nginx_app.conf"
+move_file "docker-compose.yml" "docker-compose.yml"
+move_file "data/mongo/init.js" "mongo-init.js"
+move_file "docker.env" "docker.env"
+move_file "encryption.env" "encryption.env"
+rm -rf "$templates_dir"
+
+# Step 3: Pulling the latest container images
+cd $install_dir;
+echo ""
+echo "Pulling the latest container images"
+docker-compose pull
+echo ""
+# Step 4: Starting the Appsmith containers
+echo "Starting the Appsmith containers"
+docker-compose up --detach --remove-orphans
+
+echo -e "\nPeace out \U1F596\n"

deploy/aws_ami/init-letsencrypt.sh

@@ -0,0 +1,103 @@
+certbot_cmd() {
+    sudo docker-compose run --rm --entrypoint "$1" certbot
+}
+
+confirm() {
+    local default="$1"  # Should be `y` or `n`.
+    local prompt="$2"
+
+    local options="y/N"
+    if [[ $default == y || $default == Y ]]; then
+        options="Y/n"
+    fi
+
+    local answer
+    read -rp "$prompt [$options] " answer
+    if [[ -z $answer ]]; then
+        # No answer given, the user just hit the Enter key. Take the default value as the answer.
+        answer="$default"
+    else
+        # An answer was given. This means the user didn't get to hit Enter so the cursor on the same line. Do an empty
+        # echo so the cursor moves to a new line.
+        echo
+    fi
+
+    [[ yY =~ $answer ]]
+}
+
+install_dir="/home/ubuntu/appsmith"
+domain="$1"
+echo "Creating certificate for '$domain'."
+
+rsa_key_size=4096
+data_path="$install_dir/data/certbot"
+
+sudo chown -R ubuntu:ubuntu "$data_path"
+
+if [ -d "$data_path/conf/keys" ] && [ -d "$data_path/conf/live" ]; then
+    if ! confirm n "Existing certificate data found at '$data_path'. Continue and replace existing certificate?"; then
+        exit 0
+    fi
+fi
+
+mkdir -p "$data_path"/{conf,www}
+
+if ! [[ -e "$data_path/conf/options-ssl-nginx.conf" && -e "$data_path/conf/ssl-dhparams.pem" ]]; then
+    echo "### Downloading recommended TLS parameters..."
+    curl -s https://raw.githubusercontent.com/certbot/certbot/master/certbot-nginx/certbot_nginx/_internal/tls_configs/options-ssl-nginx.conf > "$data_path/conf/options-ssl-nginx.conf"
+    curl -s https://raw.githubusercontent.com/certbot/certbot/master/certbot/certbot/ssl-dhparams.pem > "$data_path/conf/ssl-dhparams.pem"
+    echo
+fi
+
+echo "### Requesting Let's Encrypt certificate for '$domain'..."
+
+read -rp 'Enter email address to create SSL certificate: (Optional, but strongly recommended): ' email
+if [[ -z $email ]]; then
+    email_arg="--register-unsafely-without-email"
+else
+    email_arg="--email $email --no-eff-email"
+fi
+
+if confirm n 'Do you want to create certificate in staging mode (which is used for dev purposes and is not subject to rate limits)?'; then
+    staging_arg="--staging"
+else
+    staging_arg=""
+fi
+
+echo "### Generating OpenSSL key for '$domain'..."
+live_path="/etc/letsencrypt/live/$domain"
+
+cd "$install_dir"
+
+certbot_cmd \
+    "sh -c \"mkdir -p '$live_path' && openssl req -x509 -nodes -newkey rsa:1024 -days 1 \
+        -keyout '$live_path/privkey.pem' \
+        -out '$live_path/fullchain.pem' \
+        -subj '/CN=localhost' \
+        \""
+echo
+
+echo "### Starting nginx..."
+sudo docker-compose up --force-recreate --detach nginx
+echo
+
+echo "### Removing key now that validation is done for $domain..."
+certbot_cmd \
+    "rm -Rfv /etc/letsencrypt/live/$domain /etc/letsencrypt/archive/$domain /etc/letsencrypt/renewal/$domain.conf"
+echo
+
+# The following command exits with a non-zero status code even if the certificate was generated, but some checks failed.
+# So we explicitly ignore such failure with a `|| true` in the end, to avoid bash quitting on us because this looks like
+# a failed command.
+certbot_cmd "certbot certonly --webroot --webroot-path=/var/www/certbot \
+        $staging_arg \
+        $email_arg \
+        --domains $domain \
+        --rsa-key-size $rsa_key_size \
+        --agree-tos \
+        --force-renewal" \
+    || true
+echo
+
+echo "### Reloading nginx..."
+sudo docker-compose exec nginx nginx -s reload