Protei/PromucFlow_constructor
2
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

chore: New schema change for vulnerabilities test (#37204)

Browse Source 
## Description
optimise the view for large number of records 


Fixes #`37036`  

## Automation

/ok-to-test tags="@tag.IDE"

### 🔍 Cypress test results
<!-- This is an auto-generated comment: Cypress test results  -->
> [!TIP]
> 🟢 🟢 🟢 All cypress tests have passed! 🎉 🎉 🎉
> Workflow run:
<https://github.com/appsmithorg/appsmith/actions/runs/11677745846>
> Commit: 083266c8bdd82c3ff2aa0d58c1acb57d974cd46d
> <a
href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=11677745846&attempt=1"
target="_blank">Cypress dashboard</a>.
> Tags: `@tag.IDE`
> Spec:
> <hr>Tue, 05 Nov 2024 05:12:55 UTC
<!-- end of auto-generated comment: Cypress test results  -->


## Communication
Should the DevRel and Marketing teams inform users about this change?
- [ ] Yes
- [x] No


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- Enhanced error handling and functionality in vulnerability data
scripts.
- Introduced a new function for inserting vulnerabilities into the
database, improving data management.
  
- **Bug Fixes**
- Improved reliability of database insertion and handling of existing
records.
- Streamlined installation processes for Docker Scout and Trivy with
increased retry attempts.

- **Refactor**
- Simplified scripts by removing old vulnerability comparisons and
unnecessary comments.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
Sagar Khalasi 2024-11-05 12:11:51 +05:30 committed by GitHub
parent 6dc61946d4
commit 6660bd9917
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
3 changed files with 130 additions and 272 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

79
.github/workflows/test-vulnerabilities-data.yml vendored
View File

@ -29,85 +29,6 @@ jobs:
- name: Install pg - name: Install pg
run: npm install pg run: npm install pg
- name: Fetch vulnerability data
id: vulnerability_data
env:
DB_HOST: ${{ secrets.CYPRESS_DB_HOST }}
DB_NAME: ${{ secrets.CYPRESS_DB_NAME }}
DB_USER: ${{ secrets.CYPRESS_DB_USER }}
DB_PWD: ${{ secrets.CYPRESS_DB_PWD }}
uses: actions/github-script@v7
with:
script: |
const { Pool } = require("pg");
const fs = require('fs');
const path = require('path');
const { DB_HOST, DB_NAME, DB_USER, DB_PWD } = process.env;
const pool = new Pool({
user: DB_USER,
host: DB_HOST,
database: DB_NAME,
password: DB_PWD,
port: 5432,
connectionTimeoutMillis: 60000,
});
(async () => {
const client = await pool.connect();
try {
// Fetch vurn_id, product, scanner_tool, and priority from the database
const result = await client.query(`SELECT vurn_id, product, scanner_tool, priority FROM vulnerability_tracking`);
console.log('Vulnerability Data:', result.rows);
// Extract relevant fields from the result
const extractedData = result.rows.map(({ vurn_id, product, scanner_tool, priority }) => ({
vurn_id,
product,
scanner_tool,
priority
}));
console.log('Extracted Vulnerability Data:', extractedData);
// Prepare CSV content
const csvContent = [
['vurn_id', 'product', 'scanner_tool', 'priority'], // Add priority column header
...extractedData.map(row => [row.vurn_id, row.product, row.scanner_tool, row.priority])
]
.map(e => e.join(',')) // Join columns
.join('\n'); // Join rows
// Write to CSV file in workspace
const csvFilePath = path.join(process.env.GITHUB_WORKSPACE, 'vulnerability_base_data.csv');
fs.writeFileSync(csvFilePath, csvContent);
console.log(`Data successfully written to ${csvFilePath}`);
// Prepare TXT content
const txtContent = extractedData
.map(row => `vurn_id: ${row.vurn_id}, product: ${row.product}, scanner_tool: ${row.scanner_tool}, priority: ${row.priority}`)
.join('\n'); // Join rows
// Write to TXT file in workspace
const txtFilePath = path.join(process.env.GITHUB_WORKSPACE, 'vulnerability_base_data.txt');
fs.writeFileSync(txtFilePath, txtContent);
console.log(`Data successfully written to ${txtFilePath}`);
client.release();
return extractedData; // Return the extracted data
} catch (err) {
console.error('Error fetching vulnerability data:', err);
client.release();
}
})();
- name: Upload Vulnerability Data
uses: actions/upload-artifact@v3
with:
name: vulnerability-data
path: |
vulnerability_base_data.csv
vulnerability_base_data.txt
# Run Scout vulnerability data script # Run Scout vulnerability data script
- name: Run Scout vulnerability data script - name: Run Scout vulnerability data script
if: always() if: always()

103
scripts/scout_vulnerabilities_data.sh
View File

@ -1,6 +1,6 @@
#!/bin/bash #!/bin/bash
#Check required environment variables # Check required environment variables
required_vars=("DB_HOST" "DB_NAME" "DB_USER" "DB_PWD") required_vars=("DB_HOST" "DB_NAME" "DB_USER" "DB_PWD")
for var in "${required_vars[@]}"; do for var in "${required_vars[@]}"; do
if [ -z "${!var}" ] || [[ "${!var}" == "your_${var,,}" ]]; then if [ -z "${!var}" ] || [[ "${!var}" == "your_${var,,}" ]]; then
@ -25,7 +25,7 @@ OLD_VULN_FILE="${5:-vulnerability_base_data.csv}"
install_docker_scout() { install_docker_scout() {
echo "Installing Docker Scout..." echo "Installing Docker Scout..."
local attempts=0 local attempts=0
while [ $attempts -lt 3 ]; do while [ $attempts -lt 5 ]; do
echo "Attempt $((attempts + 1))..." echo "Attempt $((attempts + 1))..."
curl -fsSL https://raw.githubusercontent.com/docker/scout-cli/main/install.sh -o install-scout.sh curl -fsSL https://raw.githubusercontent.com/docker/scout-cli/main/install.sh -o install-scout.sh
sh install-scout.sh &> install_scout_log.txt sh install-scout.sh &> install_scout_log.txt
@ -75,7 +75,7 @@ docker scout cves "$IMAGE" | grep -E "✗ |CVE-" | awk -v product_name="$product
{ {
# Check for valid vulnerability data and format it correctly # Check for valid vulnerability data and format it correctly
if ($2 != "" && $3 ~ /^CVE-/) { if ($2 != "" && $3 ~ /^CVE-/) {
# Extract severity level, CVE ID and format output correctly # Extract severity level, CVE ID, and format output correctly
print $3","product_name",""SCOUT"","$2 print $3","product_name",""SCOUT"","$2
} }
}' | sort -u > "$CSV_OUTPUT_FILE" }' | sort -u > "$CSV_OUTPUT_FILE"
@ -83,90 +83,75 @@ docker scout cves "$IMAGE" | grep -E "✗ |CVE-" | awk -v product_name="$product
# Check if the CSV output file is empty # Check if the CSV output file is empty
[ -s "$CSV_OUTPUT_FILE" ] || echo "No vulnerabilities found for image: $IMAGE" > "$CSV_OUTPUT_FILE" [ -s "$CSV_OUTPUT_FILE" ] || echo "No vulnerabilities found for image: $IMAGE" > "$CSV_OUTPUT_FILE"
# Compare new vulnerabilities against old vulnerabilities
echo "Comparing new vulnerabilities with existing vulnerabilities in $OLD_VULN_FILE..."
if [ -s "$OLD_VULN_FILE" ]; then
awk -F, 'NR==FNR {seen[$1","$2","$3","$4]; next} !($1","$2","$3","$4 in seen)' "$OLD_VULN_FILE" "$CSV_OUTPUT_FILE" > "scout_vulnerabilities_diff.csv"
else
echo "$OLD_VULN_FILE is empty. All new vulnerabilities will be included."
cp "$CSV_OUTPUT_FILE" "scout_vulnerabilities_diff.csv"
fi
# Output for verification
echo "Fetching passed data..."
cat "$OLD_VULN_FILE"
echo ""
echo "Fetching new data..."
cat "$CSV_OUTPUT_FILE"
echo ""
echo "Fetching diff..."
cat "scout_vulnerabilities_diff.csv"
echo ""
# Insert new vulnerabilities into the PostgreSQL database using psql # Insert new vulnerabilities into the PostgreSQL database using psql
insert_vulns_into_db() { insert_vulns_into_db() {
local count=0
local query_file="insert_vulns.sql" local query_file="insert_vulns.sql"
echo "BEGIN;" > "$query_file" # Start the transaction echo "BEGIN;" > "$query_file"
# Create an associative array to hold existing entries from the database
declare -A existing_entries
# Fetch existing vulnerabilities from the database to avoid duplicates
psql -t -c "SELECT vurn_id, product, scanner_tool, priority FROM vulnerability_tracking WHERE scanner_tool = 'SCOUT'" "postgresql://$DB_USER:$DB_PWD@$DB_HOST/$DB_NAME" | while IFS='|' read -r db_vurn_id db_product db_scanner_tool db_priority; do
existing_entries["$db_product,$db_scanner_tool,$db_vurn_id"]="$db_priority"
done
while IFS=, read -r vurn_id product scanner_tool priority; do while IFS=, read -r vurn_id product scanner_tool priority; do
# Skip empty lines
if [[ -z "$vurn_id" || -z "$priority" || -z "$product" || -z "$scanner_tool" ]]; then if [[ -z "$vurn_id" || -z "$priority" || -z "$product" || -z "$scanner_tool" ]]; then
echo "Skipping empty vulnerability entry" echo "Skipping empty vulnerability entry"
continue continue
fi fi
# Check if the entry already exists local pr_id="${GITHUB_PR_ID:-}"
if [[ -n "${existing_entries["$product,$scanner_tool,$vurn_id"]}" ]]; then local pr_link="${GITHUB_PR_LINK:-}"
echo "Entry for $vurn_id already exists in the database. Skipping."
continue
fi
local pr_id="$GITHUB_PR_ID"
local pr_link="$GITHUB_PR_LINK"
local created_date=$(date -u +"%Y-%m-%dT%H:%M:%SZ") local created_date=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
local update_date="$created_date"
local comments="Initial vulnerability report" local comments="Initial vulnerability report"
local owner="John Doe" local owner="John Doe"
local pod="Security" local pod="Security"
# Escape single quotes in vulnerability ID, product, and priority # Clean up input values
vurn_id=$(echo "$vurn_id" | sed "s/'/''/g") vurn_id=$(echo "$vurn_id" | sed "s/'/''/g")
priority=$(echo "$priority" | sed "s/'/''/g") priority=$(echo "$priority" | sed "s/'/''/g")
product=$(echo "$product" | sed "s/'/''/g") product=$(echo "$product" | sed "s/'/''/g" | tr -d '[:space:]' | sed 's/[|]//g' | sed 's/,$//')
scanner_tool=$(echo "$scanner_tool" | sed "s/'/''/g") scanner_tool=$(echo "$scanner_tool" | sed "s/'/''/g" | tr -d '[:space:]' | sed 's/[|]//g' | sed 's/,$//')
# Write each insert query to the SQL file # Fetch existing values for this vulnerability ID
echo "INSERT INTO vulnerability_tracking (product, scanner_tool, vurn_id, priority, pr_id, pr_link, github_run_id, created_date, update_date, comments, owner, pod) VALUES ('$product', '$scanner_tool', '$vurn_id', '$priority', '$pr_id', '$pr_link', '$GITHUB_RUN_ID', '$created_date', '$update_date', '$comments', '$owner', '$pod');" >> "$query_file" existing_entry=$(psql -t -c "SELECT product, scanner_tool FROM vulnerability_tracking WHERE vurn_id = '$vurn_id'" "postgresql://$DB_USER:$DB_PWD@$DB_HOST/$DB_NAME" 2>/dev/null)
((count++)) # Process fetched data
done < "scout_vulnerabilities_diff.csv" if [[ -z "$existing_entry" ]]; then
combined_products="$product"
combined_scanner_tools="$scanner_tool"
else
IFS='|' read -r existing_product existing_scanner_tool <<< "$existing_entry"
combined_products=$(echo "$existing_product,$product" | tr ',' '\n' | sed '/^$/d' | sort -u | tr '\n' ',' | sed 's/^,//; s/,$//')
combined_scanner_tools=$(echo "$existing_scanner_tool,$scanner_tool" | tr ',' '\n' | sed '/^$/d' | sort -u | tr '\n' ',' | sed 's/^,//; s/,$//')
fi
echo "COMMIT;" >> "$query_file" # End the transaction # Write the insert query to the SQL file
echo "INSERT INTO vulnerability_tracking (product, scanner_tool, vurn_id, priority, pr_id, pr_link, github_run_id, created_date, update_date, comments, owner, pod)
VALUES ('$combined_products', '$combined_scanner_tools', '$vurn_id', '$priority', '$pr_id', '$pr_link', '$GITHUB_RUN_ID', '$created_date', '$created_date', '$comments', '$owner', '$pod')
ON CONFLICT (vurn_id)
DO UPDATE SET
product = '$combined_products',
scanner_tool = '$combined_scanner_tools',
priority = EXCLUDED.priority,
pr_id = EXCLUDED.pr_id,
pr_link = EXCLUDED.pr_link,
github_run_id = EXCLUDED.github_run_id,
update_date = EXCLUDED.update_date,
comments = EXCLUDED.comments,
owner = EXCLUDED.owner,
pod = EXCLUDED.pod;" >> "$query_file"
done < "$CSV_OUTPUT_FILE"
echo "COMMIT;" >> "$query_file"
echo "Queries written to $query_file." echo "Queries written to $query_file."
# Execute the SQL file # Execute the SQL file and rollback on failure
psql -e "postgresql://$DB_USER:$DB_PWD@$DB_HOST/$DB_NAME" -f "$query_file" if psql -e "postgresql://$DB_USER:$DB_PWD@$DB_HOST/$DB_NAME" -f "$query_file"; then
# Check if the execution was successful
if [ $? -eq 0 ]; then
echo "Vulnerabilities successfully inserted into the database." echo "Vulnerabilities successfully inserted into the database."
else else
echo "Error: Failed to insert vulnerabilities. Please check the database connection or query." echo "Error: Failed to insert vulnerabilities. Performing rollback."
echo "ROLLBACK;" | psql "postgresql://$DB_USER:$DB_PWD@$DB_HOST/$DB_NAME"
exit 1 exit 1
fi fi
} }
# Call the function to generate the insert queries and execute them if [ -s "$CSV_OUTPUT_FILE" ]; then
if [ -s "scout_vulnerabilities_diff.csv" ]; then
insert_vulns_into_db insert_vulns_into_db
else else
echo "No new vulnerabilities to insert." echo "No new vulnerabilities to insert."

220
scripts/trivy_vulnerabilities_data.sh
View File

@ -1,6 +1,6 @@
#!/bin/bash #!/bin/bash
#Check required environment variables # Check required environment variables
required_vars=("DB_HOST" "DB_NAME" "DB_USER" "DB_PWD") required_vars=("DB_HOST" "DB_NAME" "DB_USER" "DB_PWD")
for var in "${required_vars[@]}"; do for var in "${required_vars[@]}"; do
if [ -z "${!var}" ] || [[ "${!var}" == "your_${var,,}" ]]; then if [ -z "${!var}" ] || [[ "${!var}" == "your_${var,,}" ]]; then
@ -14,16 +14,13 @@ DB_NAME="${DB_NAME}"
DB_USER="${DB_USER}" DB_USER="${DB_USER}"
DB_PWD="${DB_PWD}" DB_PWD="${DB_PWD}"
# Assign the parameters from the workflow # Assign parameters from the workflow
IMAGE="$1" IMAGE="$1"
GITHUB_PR_ID="$2" GITHUB_PR_ID="$2"
GITHUB_PR_LINK="$3" GITHUB_PR_LINK="$3"
GITHUB_RUN_ID="$4" GITHUB_RUN_ID="$4"
OLD_VULN_FILE="${5:-vulnerability_base_data.csv}"
MAX_RETRIES=5
# Define the maximum number of retries
MAX_RETRIES=3
# Function to install Trivy with retry logic # Function to install Trivy with retry logic
install_trivy_with_retry() { install_trivy_with_retry() {
@ -33,23 +30,14 @@ install_trivy_with_retry() {
while [[ $count -lt $MAX_RETRIES ]]; do while [[ $count -lt $MAX_RETRIES ]]; do
echo "Attempting to install Trivy (attempt $((count + 1)))..." echo "Attempting to install Trivy (attempt $((count + 1)))..."
# Fetch the latest release dynamically instead of hardcoding
TRIVY_VERSION=$(curl -s https://api.github.com/repos/aquasecurity/trivy/releases/latest | grep '"tag_name"' | sed -E 's/.*"v([^"]+)".*/\1/') TRIVY_VERSION=$(curl -s https://api.github.com/repos/aquasecurity/trivy/releases/latest | grep '"tag_name"' | sed -E 's/.*"v([^"]+)".*/\1/')
TRIVY_URL="https://github.com/aquasecurity/trivy/releases/download/v$TRIVY_VERSION/trivy_"$TRIVY_VERSION"_Linux-64bit.tar.gz" TRIVY_URL="https://github.com/aquasecurity/trivy/releases/download/v$TRIVY_VERSION/trivy_${TRIVY_VERSION}_Linux-64bit.tar.gz"
echo "Attempting to install $TRIVY_VERSION from $TRIVY_URL"
# Download and extract Trivy
curl -sfL "$TRIVY_URL" | tar -xzf - trivy
# Check if extraction was successful curl -sfL "$TRIVY_URL" | tar -xzf - trivy
if [[ $? -eq 0 ]]; then if [[ $? -eq 0 ]]; then
# Create a local bin directory if it doesn't exist
mkdir -p "$HOME/bin" mkdir -p "$HOME/bin"
# Move Trivy to the local bin directory
mv trivy "$HOME/bin/" mv trivy "$HOME/bin/"
# Manually add the bin directory to PATH for this session
export PATH="$HOME/bin:$PATH" export PATH="$HOME/bin:$PATH"
# Check if Trivy is successfully installed
if command -v trivy &> /dev/null; then if command -v trivy &> /dev/null; then
success=true success=true
break break
@ -64,22 +52,18 @@ install_trivy_with_retry() {
echo "Error: Trivy installation failed after $MAX_RETRIES attempts." echo "Error: Trivy installation failed after $MAX_RETRIES attempts."
exit 1 exit 1
fi fi
echo "Trivy installed successfully." echo "Trivy installed successfully."
} }
# Check if Trivy is installed, if not, install it with retry logic # Check if Trivy is installed, if not, install it
if ! command -v trivy &> /dev/null; then if ! command -v trivy &> /dev/null; then
install_trivy_with_retry install_trivy_with_retry
fi fi
NEW_VULN_FILE="trivy_vulnerabilities_new.csv" NEW_VULN_FILE="trivy_vulnerabilities_new.csv"
DIFF_OUTPUT_FILE="trivy_vulnerabilities_diff.csv" rm -f "$NEW_VULN_FILE"
rm -f "$NEW_VULN_FILE" "$DIFF_OUTPUT_FILE" # Determine product name based on the image name
touch "$OLD_VULN_FILE"
# Extract the product name from the image name
case "$IMAGE" in case "$IMAGE" in
*appsmith/appsmith-ce:*) product_name="CE" ;; *appsmith/appsmith-ce:*) product_name="CE" ;;
*appsmith/appsmith-ee:*) product_name="EE" ;; *appsmith/appsmith-ee:*) product_name="EE" ;;
@ -87,132 +71,100 @@ case "$IMAGE" in
*) product_name="UNKNOWN" ;; *) product_name="UNKNOWN" ;;
esac esac
# Function to run Trivy scan # Run Trivy scan
run_trivy_scan() { echo "Running Trivy scan for image: $IMAGE..."
echo "Cleaning up Trivy data..." trivy image --db-repository public.ecr.aws/aquasecurity/trivy-db --java-db-repository public.ecr.aws/aquasecurity/trivy-java-db --insecure --format json "$IMAGE" > "trivy_vulnerabilities.json" || {
trivy clean --all echo "Error: Trivy scan failed for image: $IMAGE"
exit 1
echo "Running Trivy scan for image: $IMAGE..."
if ! trivy image \
--db-repository public.ecr.aws/aquasecurity/trivy-db \
--java-db-repository public.ecr.aws/aquasecurity/trivy-java-db \
--insecure \
--format json \
"$IMAGE" > "trivy_vulnerabilities.json"; then
echo "Error: Trivy is not available or the image does not exist."
exit 1
fi
} }
# Call the function to run the scan
run_trivy_scan
# Process vulnerabilities and generate the desired CSV format # Process vulnerabilities and generate CSV
if jq -e '.Results | length > 0' "trivy_vulnerabilities.json" > /dev/null; then if jq -e '.Results | length > 0' "trivy_vulnerabilities.json" > /dev/null; then
jq -r --arg product "$product_name" '.Results[].Vulnerabilities[] | "\(.VulnerabilityID),\($product),TRIVY,\(.Severity)"' "trivy_vulnerabilities.json" | sed 's/^\s*//;s/\s*$//' | sort -u > "$NEW_VULN_FILE" jq -r --arg product "$product_name" '.Results[]? | .Vulnerabilities[]? | "\(.VulnerabilityID),\($product),TRIVY,\(.Severity)"' "trivy_vulnerabilities.json" | sort -u > "$NEW_VULN_FILE"
echo "Vulnerabilities saved to $NEW_VULN_FILE" echo "Vulnerabilities saved to $NEW_VULN_FILE"
else else
echo "No vulnerabilities found for image: $IMAGE" echo "No vulnerabilities found for image: $IMAGE"
echo "No vulnerabilities found." > "$NEW_VULN_FILE" echo "No vulnerabilities found." > "$NEW_VULN_FILE"
fi fi
# Compare new vulnerabilities with the old file
if [ -s "$NEW_VULN_FILE" ]; then
sort "$OLD_VULN_FILE" -o "$OLD_VULN_FILE" # Sort the old vulnerabilities file
sort "$NEW_VULN_FILE" -o "$NEW_VULN_FILE" # Sort the new vulnerabilities file
# Get the difference between new and old vulnerabilities
comm -13 "$OLD_VULN_FILE" "$NEW_VULN_FILE" > "$DIFF_OUTPUT_FILE"
if [ -s "$DIFF_OUTPUT_FILE" ]; then # Insert new vulnerabilities into PostgreSQL
echo "New vulnerabilities found and recorded in $DIFF_OUTPUT_FILE."
else
echo "No new vulnerabilities found for image: $IMAGE."
fi
else
echo "No new vulnerabilities found for image: $IMAGE."
fi
# Cleanup JSON file
rm -f "trivy_vulnerabilities.json"
# Output for verification
echo "Fetching passed data..."
cat "$OLD_VULN_FILE"
echo ""
echo "Fetching new data..."
cat "$NEW_VULN_FILE"
echo ""
echo "Fetching diff..."
cat $DIFF_OUTPUT_FILE
echo ""
# Insert new vulnerabilities into the PostgreSQL database using psql
insert_vulns_into_db() { insert_vulns_into_db() {
local count=0 local query_file="insert_vulns.sql"
local query_file="insert_vulns.sql" echo "BEGIN;" > "$query_file"
echo "BEGIN;" > "$query_file" # Start the transaction
# Create an associative array to hold existing entries from the database while IFS=, read -r vurn_id product scanner_tool priority; do
declare -A existing_entries if [[ -z "$vurn_id" || -z "$priority" || -z "$product" || -z "$scanner_tool" ]]; then
continue
fi
# Fetch existing vulnerabilities from the database to avoid duplicates local pr_id="${GITHUB_PR_ID:-}"
psql -t -c "SELECT vurn_id, product, scanner_tool, priority FROM vulnerability_tracking WHERE scanner_tool = 'TRIVY'" "postgresql://$DB_USER:$DB_PWD@$DB_HOST/$DB_NAME" | while IFS='|' read -r db_vurn_id db_product db_scanner_tool db_priority; do local pr_link="${GITHUB_PR_LINK:-}"
existing_entries["$db_product,$db_scanner_tool,$db_vurn_id"]="$db_priority" local created_date=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
done local comments="Initial vulnerability report"
local owner="John Doe"
local pod="Security"
while IFS=, read -r vurn_id product scanner_tool priority; do # Remove spaces and redundant commas, and escape single quotes for SQL
# Skip empty lines vurn_id=$(echo "$vurn_id" | sed "s/'/''/g")
if [[ -z "$vurn_id" || -z "$priority" || -z "$product" || -z "$scanner_tool" ]]; then priority=$(echo "$priority" | sed "s/'/''/g")
echo "Skipping empty vulnerability entry" product=$(echo "$product" | sed "s/'/''/g" | tr -d ' ' | sed 's/,*$//')
continue scanner_tool=$(echo "$scanner_tool" | sed "s/'/''/g" | tr -d ' ' | sed 's/,*$//')
# Fetch existing product and scanner_tool values for the vulnerability
existing_entry=$(psql -t -c "SELECT product, scanner_tool FROM vulnerability_tracking WHERE vurn_id = '$vurn_id'" "postgresql://$DB_USER:$DB_PWD@$DB_HOST/$DB_NAME" 2>/dev/null)
if [ -n "$existing_entry" ]; then
# Parse existing products and tools
existing_product=$(echo "$existing_entry" | cut -d '|' -f 1 | tr -d ' ')
existing_scanner_tool=$(echo "$existing_entry" | cut -d '|' -f 2 | tr -d ' ')
# Merge with new values, ensuring uniqueness
combined_products="$existing_product,$product"
unique_products=$(echo "$combined_products" | tr ',' '\n' | sed '/^$/d' | sort -u | tr '\n' ',' | sed 's/^,//; s/,$//')
combined_scanner_tools="$existing_scanner_tool,$scanner_tool"
unique_scanner_tools=$(echo "$combined_scanner_tools" | tr ',' '\n' | sed '/^$/d' | sort -u | tr '\n' ',' | sed 's/^,//; s/,$//')
else
unique_products="$product"
unique_scanner_tools="$scanner_tool"
fi
# Write the insert query to the SQL file
echo "INSERT INTO vulnerability_tracking (product, scanner_tool, vurn_id, priority, pr_id, pr_link, github_run_id, created_date, update_date, comments, owner, pod)
VALUES ('$unique_products', '$unique_scanner_tools', '$vurn_id', '$priority', '$pr_id', '$pr_link', '$GITHUB_RUN_ID', '$created_date', '$created_date', '$comments', '$owner', '$pod')
ON CONFLICT (vurn_id)
DO UPDATE SET
product = '$unique_products',
scanner_tool = '$unique_scanner_tools',
priority = EXCLUDED.priority,
pr_id = EXCLUDED.pr_id,
pr_link = EXCLUDED.pr_link,
github_run_id = EXCLUDED.github_run_id,
update_date = EXCLUDED.update_date,
comments = EXCLUDED.comments,
owner = EXCLUDED.owner,
pod = EXCLUDED.pod;" >> "$query_file"
done < "$NEW_VULN_FILE"
echo "COMMIT;" >> "$query_file"
# Execute the SQL file
if psql -e "postgresql://$DB_USER:$DB_PWD@$DB_HOST/$DB_NAME" -f "$query_file"; then
echo "Vulnerabilities successfully inserted into the database."
else
echo "Error: Failed to insert vulnerabilities. Check logs for details."
exit 1
fi fi
# Check if the entry already exists
if [[ -n "${existing_entries["$product,$scanner_tool,$vurn_id"]}" ]]; then
echo "Entry for $vurn_id already exists in the database. Skipping."
continue
fi
local pr_id="$GITHUB_PR_ID"
local pr_link="$GITHUB_PR_LINK"
local created_date=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
local update_date="$created_date"
local comments="Initial vulnerability report"
local owner="John Doe"
local pod="Security"
# Escape single quotes in vulnerability ID, product, and priority
vurn_id=$(echo "$vurn_id" | sed "s/'/''/g")
priority=$(echo "$priority" | sed "s/'/''/g")
product=$(echo "$product" | sed "s/'/''/g")
scanner_tool=$(echo "$scanner_tool" | sed "s/'/''/g")
# Write each insert query to the SQL file
echo "INSERT INTO vulnerability_tracking (product, scanner_tool, vurn_id, priority, pr_id, pr_link, github_run_id, created_date, update_date, comments, owner, pod) VALUES ('$product', '$scanner_tool', '$vurn_id', '$priority', '$pr_id', '$pr_link', '$GITHUB_RUN_ID', '$created_date', '$update_date', '$comments', '$owner', '$pod');" >> "$query_file"
((count++))
done < $DIFF_OUTPUT_FILE
echo "COMMIT;" >> "$query_file" # End the transaction
echo "Queries written to $query_file."
# Execute the SQL file
psql -e "postgresql://$DB_USER:$DB_PWD@$DB_HOST/$DB_NAME" -f "$query_file"
# Check if the execution was successful
if [ $? -eq 0 ]; then
echo "Vulnerabilities successfully inserted into the database."
else
echo "Error: Failed to insert vulnerabilities. Please check the database connection or query."
exit 1
fi
} }
# Call the function to generate the insert queries and execute them # Run insertion if vulnerabilities are found
if [ -s $DIFF_OUTPUT_FILE ]; then if [ -s "$NEW_VULN_FILE" ]; then
insert_vulns_into_db insert_vulns_into_db
else else
echo "No new vulnerabilities to insert." echo "No vulnerabilities to insert."
fi fi
# Cleanup
rm -f "trivy_vulnerabilities.json"