Day 11: Linux - Shell Scripting use cases in DevOps : Part 2

Listing some of the best shell scripts in the DevOps industry showcase the power of automation in simplifying complex tasks, ensuring consistency, and enabling rapid and reliable software delivery. By leveraging shell scripting for these use cases, DevOps teams can streamline their workflows, increase efficiency, and focus more on delivering value to their customers.

I. Automated Server Configuration

Automating the setup and configuration of servers is a common task in DevOps, and it can be achieved using shell scripting. In this example, I'll provide a simple shell script that automates the setup of an Ubuntu server. This script assumes you have basic knowledge of shell scripting and system administration.

1. Let's call this script server_setup.sh:

   Before running this script, ensure the following:
   https://github.com/techwithankush/shell-scripting/blob/12bd7454b03e39b1e7180cdceca21bba3759bc05/1.Automated_Server_Configuration.sh

   1. Customize the package1, package2, and package3 variables with the names of the packages you need to install.

   2. Replace IP_ADDRESS, GATEWAY, and DNS_SERVERS with appropriate values for your network configuration.

   3. Replace USERNAME with the desired username for your new user account.

Please note that this is a basic example, and in a real DevOps project, you may need to add more sophisticated error handling, additional configurations, or even make use of configuration management tools like Ansible, Puppet, or Chef for more complex setups.

Once you have customized the script, make it executable using the following command:

    chmod +x server_setup.sh

Then, run the script with superuser privileges:

    sudo ./server_setup.sh

This script will automate the server setup and configuration, making it more efficient and less prone to human error in a DevOps context. Always remember to double-check the configurations and test the script in a non-production environment before applying it to production servers.

II. Continuous Integration and Deployment (CI/CD) Pipeline

Automating the Continuous Integration and Deployment (CI/CD) process is essential for ensuring frequent and reliable releases in a DevOps project. In this example, I'll provide a simple shell script to demonstrate a basic CI/CD pipeline for a web application. This pipeline will involve building the application, running tests, and deploying it to a web server.

Let's assume you have a Node.js web application that is version-controlled using Git. We'll use Git hooks to trigger the CI/CD pipeline on a push to the repository. The script will be executed on the deployment server.

1. Create a shell script named ci_cd_pipeline.sh on the deployment server:
   https://github.com/techwithankush/shell-scripting/blob/12bd7454b03e39b1e7180cdceca21bba3759bc05/2.CI_CD.sh

2. Make the script executable:

    chmod +x ci_cd_pipeline.sh

1. Next, set up a Git hook to automatically trigger the CI/CD pipeline on a push. Create a post-receive hook in the Git repository on the deployment server:

    #!/bin/bash

    # Path to the deployment script
    DEPLOY_SCRIPT="/path/to/ci_cd_pipeline.sh"

    # Run the deployment script
    $DEPLOY_SCRIPT

1. Make the post-receive hook executable:

    chmod +x /path/to/your/repo.git/hooks/post-receive

With this setup, whenever you push changes to the Git repository, the post-receive hook will be triggered, which in turn executes the ci_cd_pipeline.sh script. The script will clone the latest changes from the repository, install dependencies, run tests, build the application, and deploy it to the specified deployment directory.

Please note that this example is simplified and assumes a basic web application using Node.js. In a real-world scenario, you may need to adjust the script and Git hook based on your specific application's requirements, build tools, testing frameworks, and deployment process.

Additionally, for more complex CI/CD pipelines, consider using dedicated CI/CD tools like Jenkins, GitLab CI/CD, Travis CI, CircleCI, or others, as they offer more robust features, integrations, and scalability for larger projects.

III. Log Analysis and Monitoring

Automating infrastructure provisioning is a crucial part of the DevOps process. In this example, I'll provide a shell script to automate the provisioning of an Amazon EC2 instance on AWS. To use this script, you'll need to have the AWS Command Line Interface (CLI) installed and configured with appropriate credentials.

Let's call this script aws_ec2_provision.sh:
https://github.com/techwithankush/shell-scripting/blob/8dfc763602e7196f59e749e2d80b837fcc11f290/3.Infrastructure_Provisioning.sh

Before running this script, ensure the following:

1. Replace the placeholder values in the REGION, INSTANCE_TYPE, KEY_PAIR_NAME, SECURITY_GROUP_NAME, and AMI_ID variables with appropriate values for your AWS environment.

2. Ensure that you have the AWS CLI installed and configured with the necessary credentials using aws configure.

3. Customize the "Optional" section to include additional setup and configuration steps as needed for your specific use case.

Once you have customized the script, make it executable using the following command:

    chmod +x aws_ec2_provision.sh

Then, run the script:

    ./aws_ec2_provision.sh

This script will use the AWS CLI to create a new EC2 instance with the specified configurations, wait until the instance is running, and then display its public IP address. It is essential to ensure that you have the appropriate IAM permissions to perform the EC2-related actions in your AWS account.

Remember that this example is tailored specifically for AWS EC2 provisioning. If you want to provision virtual machines or containers in other cloud environments like Azure or Docker, you will need to use their respective APIs and CLI tools to accomplish the task. Each cloud provider may have its own set of CLI commands and parameters for provisioning resources.

For more complex infrastructure provisioning and management, consider using Infrastructure as Code (IaC) tools like Terraform, Ansible, or cloud-specific tools like AWS CloudFormation or Azure Resource Manager templates. IaC tools allow you to define your infrastructure in code, making it easier to manage, version, and automate infrastructure changes.

IV. Environment Configuration Management:

Automating environment configuration management is a critical aspect of DevOps to ensure consistency and reproducibility across different environments. In this example, I'll provide a shell script that automates the setup and configuration of a simple Node.js application for three different environments: development, staging, and production.

Let's call this script environment_setup.sh:
https://github.com/techwithankush/shell-scripting/blob/dabdf8fa26ca30928feb4422cf6afb45bd592348/4.Environment_Configuration_Management.sh

Before running this script, ensure the following:

1. Customize the environment-specific settings inside the configure_environment() function to match your specific application's requirements. In this example, we are setting different API base URLs and log levels for each environment.

2. Replace /path/to/your/application with the actual path to your Node.js application's directory.

3. Ensure that your Node.js application uses the config.js file to load environment-specific settings. For example, if your application requires configurations, use const config = require('./config.js'); to load the configurations.

4. Make the script executable using the following command:

    chmod +x environment_setup.sh

Then, run the script with the desired environment name (development, staging, or production):

    ./environment_setup.sh development

The script will create or update the config.js file with the environment-specific settings and variables for the chosen environment. The Node.js application can then load these settings at runtime based on the environment it is running in.

It's worth noting that this example assumes a simple configuration approach.

In real-world projects, you might need more sophisticated configuration management solutions like using environment variables, configuration files in different formats (YAML, JSON, etc.), or even using specialized configuration management tools like Consul, etcd, or AWS Parameter Store.

For more complex setups, consider using infrastructure automation tools like Ansible, Chef, or Puppet, which can manage not only application configurations but also the provisioning and setup of entire server environments. These tools allow you to define the desired state of your infrastructure and automatically apply configurations across different environments.

V. Containerization:

Automating the creation and management of Docker containers is a common practice in DevOps to ensure consistent deployments across various environments. In this example, I'll provide a shell script that automates the build, deployment, and management of a Docker container for a simple Node.js application.

Let's call this script docker_deploy.sh:
https://github.com/techwithankush/shell-scripting/blob/dabdf8fa26ca30928feb4422cf6afb45bd592348/5.Containerization.sh

Before running this script, ensure the following:

1. Replace /path/to/your/application with the actual path to your Node.js application's directory.

2. Customize the APP_NAME, DOCKER_IMAGE_TAG, DOCKER_CONTAINER_NAME, PORT, and ENVIRONMENT variables to match your application's requirements.

3. Make sure you have Docker installed and running on the machine where you're executing the script.

Make the script executable using the following command:

    chmod +x docker_deploy.sh

Then, run the script:

    ./docker_deploy.sh

The script will build the Docker image for your Node.js application using the Dockerfile in your application directory. It will then stop and remove any existing container with the same name (if running) and start a new container with the specified name, exposed port, and environment variable settings.

This simple example demonstrates how to automate the process of building and deploying a Docker container for your Node.js application. In a real-world project, you might need to extend this script to include additional configurations, manage multiple containers, handle environment-specific settings, and integrate with container orchestration tools like Kubernetes or Docker Compose for more complex deployments.

For more advanced Docker container management and orchestration, consider using tools like Kubernetes, Docker Compose, or Docker Swarm. These tools enable you to define complex container architectures, manage scaling, handle networking, and provide high availability for your applications. Additionally, you can use configuration management tools like Ansible, Chef, or Puppet to automate the setup and management of Docker environments and deployments more comprehensively.

VI. Deployment Orchestration

To demonstrate deployment orchestration using Ansible, let's create a simple shell script that automates the deployment of a Node.js application to remote servers. We'll assume you have Ansible installed on the machine where you'll run the script, and you have SSH access to the target servers.

1. First, make sure you have your Node.js application code and necessary files ready for deployment. Your application directory structure might look like this:

    goCopy codemy-node-app/
    ├── app.js
    ├── package.json
    └── ...

1. Next, create a shell script named deploy_app.sh:

   https://github.com/techwithankush/shell-scripting/blob/dabdf8fa26ca30928feb4422cf6afb45bd592348/6.Deployment_Orchestration.sh

Before running this script, ensure the following:

1. Replace /path/to/your/application with the actual path to your Node.js application's directory.

2. Customize APP_NAME with your application name.

3. Replace server1.example.com and server2.example.com with the IP addresses or hostnames of your target servers. You can add more servers to the TARGET_SERVERS array as needed.

4. Replace your_remote_user with the remote user that has SSH access to the target servers.

5. Modify /path/to/your/ssh/key.pem to the path of your SSH private key for authentication.

Make the script executable using the following command:

    chmod +x deploy_app.sh

Then, run the script:

    ./deploy_app.sh

The script will create a temporary deployment directory, copy the application files to it, generate an Ansible playbook to copy the application files to the target servers, and finally execute the Ansible playbook to deploy the application to all specified servers.

Please note that this example is a simple demonstration. In a real-world project, you might need to include additional tasks in the Ansible playbook, handle environment-specific configurations, install Node.js dependencies, set up a reverse proxy (like Nginx) for the Node.js application, and more, depending on your specific requirements.

Using Ansible (or other configuration management tools like Puppet or Chef) allows you to automate the deployment process and ensure consistent and reliable application deployments across multiple servers or cloud instances. These tools can be particularly valuable when managing complex infrastructures or when deploying to cloud environments where automated provisioning and configuration are crucial for maintaining efficiency and scalability.

VII. Log Analysis and Monitoring

Analyzing log files and monitoring them for specific conditions or patterns can be essential for identifying issues and ensuring the smooth operation of your applications. In this example, I'll provide a shell script that uses grep and awk to analyze a sample log file, extract meaningful information, and generate alerts based on certain patterns.

Let's assume you have a log file named app.log, and you want to monitor it for specific error messages and generate an alert when such errors occur.

Create a shell script named log_monitor.sh:
https://github.com/techwithankush/shell-scripting/blob/dabdf8fa26ca30928feb4422cf6afb45bd592348/7.Log_Analysis_and_Monitoring.sh

Before running this script, ensure the following:

1. Replace /path/to/your/log/file/app.log with the actual path to your log file.

2. Customize the send_alert() function to include your alerting mechanism, such as sending an email, triggering a webhook, or sending notifications to a monitoring system like Nagios, Grafana, or Datadog.

Make the script executable using the following command:

    chmod +x log_monitor.sh

Then, run the script:

    ./log_monitor.sh

The script will analyze the log file, count the occurrences of the "ERROR" message, and display the latest error if any. If errors are found, it will generate an alert by calling the send_alert() function. For demonstration purposes, the alert is logged to a file named alerts.log. You can modify the send_alert() function to integrate with your preferred alerting mechanism.

Please note that this example is a simplified demonstration of log analysis and monitoring. In real-world projects, you might need to enhance the script to analyze different log files, extract more detailed information, apply more sophisticated patterns, and integrate with comprehensive log monitoring and analysis tools for more advanced alerts and insights.

For more complex log analysis and monitoring, consider using specialized log analysis tools like ELK Stack (Elasticsearch, Logstash, and Kibana), Splunk, Graylog, or other log management solutions. These tools offer powerful features for centralizing logs, analyzing them, setting up alerts, and visualizing log data in real-time. They provide a more robust and scalable approach to log monitoring and analysis in a DevOps environment.

VIII. Backup and Recovery

Automating the backup and recovery process is crucial for ensuring data integrity and minimizing downtime in case of failures. In this example, I'll provide a shell script that automates the backup of critical data and applications and demonstrates a simple recovery process.

Let's assume you have a web application with a MySQL database that needs to be backed up. We'll create a shell script to perform the backup and another script to simulate the recovery process.

1. Create a shell script named backup_data.sh for performing the backup:

Before running this script, ensure the following:

https://github.com/techwithankush/shell-scripting/blob/dabdf8fa26ca30928feb4422cf6afb45bd592348/8.Backup_and_Recovery.sh

1. Replace /path/to/your/backup/directory with the actual path to your backup directory where you want to store the backups.

2. Replace /path/to/your/application with the actual path to your application directory.

3. Replace your_database_user, your_database_password, and your_database_name with your MySQL database credentials.

Make the script executable using the following command:

    chmod +x backup_data.sh

Then, run the script:

    ./backup_data.sh

The script will create a backup of the MySQL database and application files and store them in the specified backup directory. The backup files will be timestamped for easy identification.

1. Create another shell script named restore_data.sh to simulate the recovery process:
   https://github.com/techwithankush/shell-scripting/blob/dabdf8fa26ca30928feb4422cf6afb45bd592348/8.Backup_and_Recovery.sh

Before running this script, ensure the following:

1. Replace /path/to/your/backup/directory with the actual path to your backup directory where the backups are stored.

Make the script executable using the following command:

    chmod +x restore_data.sh

Then, run the script:

    ./restore_data.sh

The script will list the available backups in the backup directory and prompt the user to enter the timestamp of the backup they want to restore. After the user selects the backup, the script will restore the MySQL database and application files from the chosen backup.

Please note that this example is a simplified demonstration of backup and recovery. In real-world projects, you might need to enhance the scripts to handle more complex recovery scenarios, automate backup scheduling, manage multiple backups, and integrate with cloud storage services or external backup solutions for improved redundancy and reliability.

Additionally, it's crucial to test your backup and recovery processes regularly to ensure they work as expected in case of real failures. Automated backup and recovery are essential components of a robust disaster recovery plan in a DevOps environment.

IX. Performance Monitoring and Optimization

Performance monitoring and optimization are vital in maintaining the health and efficiency of your systems. In this example, I'll provide a shell script that collects system performance metrics using basic Unix commands and performs some optimizations based on those metrics.

Let's create a shell script named performance_monitoring.sh:

https://github.com/techwithankush/shell-scripting/blob/dabdf8fa26ca30928feb4422cf6afb45bd592348/9.Performance_Monitoring_and_Optimization.sh

Before running this script, ensure the following:

1. Customize the optimize_resources() function based on your application's memory requirements and acceptable usage limits. In this example, we identify and kill the top 3 memory-hungry processes if memory usage exceeds 50% of the total available memory.

2. You may add more metrics and optimizations depending on your specific use case and requirements.

Make the script executable using the following command:

chmod +x performance_monitoring.sh

Then, run the script:

./performance_monitoring.sh

The script will collect system performance metrics, including CPU utilization, memory utilization, disk usage, and the number of TCP connections. It will then perform memory optimization by identifying and killing memory-hungry processes if the memory usage exceeds the defined threshold.

Please note that this example provides a basic demonstration of performance monitoring and optimization. In real-world projects, you may need to use specialized monitoring tools like Prometheus, Grafana, or Nagios for more extensive monitoring, alerting, and optimization.

Additionally, for more sophisticated performance optimization, consider analyzing the application's code and architecture, profiling resource-intensive functions, and optimizing database queries. Adopting performance-oriented programming practices and utilizing caching mechanisms can also contribute to significant improvements in system performance.

The effectiveness of performance monitoring and optimization varies based on your specific application, infrastructure, and user workload. Continuous monitoring, analysis, and iterative optimization are essential components of maintaining peak performance in a DevOps environment.

X. Automated Testing

Automating the execution of various tests as part of the CI/CD pipeline is crucial for ensuring the quality and reliability of your applications. In this example, I'll provide a shell script that automates the execution of unit tests, integration tests, and end-to-end tests for a Node.js application.

Assumptions:

1. You have a Node.js application with unit tests written using a test framework like Jest.

2. You have integration tests that require the application to be running on a specific port.

3. You have end-to-end tests written using a testing framework like Cypress.

Let's create a shell script named run_tests.sh:
https://github.com/techwithankush/shell-scripting/blob/dabdf8fa26ca30928feb4422cf6afb45bd592348/10.Automated_Testing.sh

Before running this script, ensure the following:

1. Replace /path/to/your/application with the actual path to your Node.js application's directory.

2. Customize the paths in UNIT_TEST_DIR, INTEGRATION_TEST_DIR, and E2E_TEST_DIR to match your application's test directories.

3. Replace 3000 in APP_PORT with the port number your application listens on during testing.

Make the script executable using the following command:

chmod +x run_tests.sh

Then, run the script:

./run_tests.sh

The script will execute unit tests, integration tests, and end-to-end tests for your Node.js application. It will start the application on the specified port before running integration and end-to-end tests to ensure the tests can interact with the running application.

Please note that this example assumes you have set up your test frameworks and scripts accordingly. In real-world projects, you might need to customize the script based on the specific testing frameworks, configurations, and setup required for your application's testing needs.

For more complex projects, consider using dedicated testing frameworks like Mocha, Chai, or Jasmine for different types of tests, and incorporate tools like TestCafe or Selenium for end-to-end testing. Additionally, you may integrate this script into your CI/CD pipeline, version control system, and other automation tools to ensure automated testing as part of your development workflow.