# Part 5: Making Jenkins Automation Scalable

### Jenkins CI/CD Series

| Part | Article |
| --- | --- |
| 1 | [Install Jenkins with Docker](https://ask-abhi.com/part-1-install-jenkins-with-docker) |
| 2 | [Creating Your First Jenkins Pipeline](https://ask-abhi.com/part-2-creating-your-first-jenkins-pipeline) |
| 3 | [Jenkins SSH Remote Execution](https://ask-abhi.com/part-3-jenkins-ssh-remote-execution) |
| 4 | [Automating MySQL Backup to AWS S3](https://ask-abhi.com/part-4-automating-mysql-backup-to-aws-s3) |
| 5 | [Making Jenkins Automation Scalable](https://ask-abhi.com/part-5-making-jenkins-automation-scalable) |
| 6 | [Running Ansible from Jenkins](https://ask-abhi.com/part-6-running-ansible-from-jenkins) |
| 7 | [Jenkins Pipeline from GitHub](https://ask-abhi.com/part-7-control-jenkins-pipeline-through-github) |
| 8 | [Multibranch Pipelines](https://ask-abhi.com/part-8-multibranch-pipelines) |
| 9 | [Parameterized Pipelines](https://ask-abhi.com/part-9-parameterized-pipelines) |
| 10 | [Environment Variables in Pipelines](https://ask-abhi.com/part-10-environment-variables-in-pipelines) |

* * *

### Goal

The goal of this project is to transform the existing Jenkins automation into a **scalable and reusable solution** by externalizing scripts and ensuring persistence across container restarts.

In this exercise, we will move the backup script outside the container and mount it using Docker volumes, allowing it to persist even if the container is recreated. Additionally, we will extend the setup to support **multiple databases and multiple S3 buckets**, making the automation flexible for real-world use cases.

By the end of this project, you will have a Jenkins-driven automation setup that is **resilient, reusable, and scalable across multiple environments**.

* * *

### Purpose

The purpose of this exercise is to demonstrate how to design Jenkins automation in a way that is **production-ready and maintainable**.

In earlier parts, the automation worked, but it was tightly coupled to the container lifecycle. In real-world DevOps environments, automation must be:

*   **Persistent** – survives container restarts or failures
    
*   **Reusable** – can be applied across multiple use cases
    
*   **Scalable** – supports multiple applications, databases, or environments
    

By externalizing the script and using Docker volume mapping, we ensure that critical automation logic is not lost when containers are recreated.

Extending the solution to handle multiple databases and S3 buckets introduces flexibility, which is essential for managing **enterprise-scale infrastructure and data workflows**.

This approach reflects real DevOps practices where pipelines are designed to be **modular, reusable, and environment-agnostic**.

* * *

### Preresuisite

*   Ready to use `Host` and the `directory` structure to run `Dockerfiles` and `docker-compose.yml` (Refer to [Part 1](https://ask-abhi.com/part-1-install-jenkins-with-docker))
    

* * *

### Step-by-step implementation

*   **How to create this script permanently on a container?**
    
*   **Go to the home folder and create an** [**aws-s3.sh**](http://aws-s3.sh) **and copy the content of the** [**script.sh**](http://script.sh)
    

```plaintext
/home/jenkins/jenkins_home$vi aws-s3.sh
```

*   **Go to docker-compose.yml and update the** `volumes` **block for** `remote_host` which will copy this script to the remote host in case of a container crash
    

```yaml
services:
  jenkins:
    container_name: jenkins
    image: jenkins/jenkins:lts
    ports:
      - "8080:8080"
    volumes:
      - $PWD/jenkins_home:/var/jenkins_home
    networks:
      - net

  remote_host:
    container_name: remote-host
    image: remote-host
    build:
      context: centos7
    volumes:
      - "$PWD/aws-s3.sh:/tmp/script.sh"
    networks:
      - net

  db_host:
    container_name: db
    image: mysql:8.0
    environment:
      MYSQL_ROOT_PASSWORD: 1234
    volumes:
      - $PWD/db_data:/var/lib/mysql
    networks:
      - net

networks:
  net:
```

*   **Test it by deleting the remote-host**
    

![](https://cdn.hashnode.com/uploads/covers/6997142f587b14d25b5231b7/c5730e33-93dd-410f-93c7-2d2adf0b5166.png align="center")

*   **What if we have multiple databases and the backup has to be uploaded to different S3 buckets?**
    
*   **Create another S3** `"jenkins-mysql-backup-mylab-2"`
    

![](https://cdn.hashnode.com/uploads/covers/6997142f587b14d25b5231b7/238389d3-d0e9-4484-9ec3-89512a0f8999.png align="center")

*   **Create another database** `testdb2`
    

![](https://cdn.hashnode.com/uploads/covers/6997142f587b14d25b5231b7/8ad40b7a-3630-4f5a-b1d3-b16020bca5d9.png align="center")

*   **Now, let's try to take this** `testdb2` **backup**
    

![](https://cdn.hashnode.com/uploads/covers/6997142f587b14d25b5231b7/65f30c71-3934-4cfe-b872-35060c000c99.png align="center")

*   **Check logs**
    

![](https://cdn.hashnode.com/uploads/covers/6997142f587b14d25b5231b7/c29592cf-3217-4721-a87e-ad4abb5970f0.png align="center")

*   **Verify upload**
    

![](https://cdn.hashnode.com/uploads/covers/6997142f587b14d25b5231b7/445f83d5-97e4-4cbc-b17b-f464cfe44a00.png align="center")

### **Done!!!**

* * *

### Conclusion

In this part of the series, we enhanced our Jenkins automation to make it **scalable, reusable, and production-ready**.

We achieved this by:

*   ***Externalizing the backup script to ensure persistence***
    
*   ***Using Docker volumes to maintain scripts across container restarts***
    
*   ***Testing resilience by recreating containers***
    
*   ***Extending the solution to support multiple databases and S3 buckets***
    

This transformation moves our setup from a simple working automation to a **robust DevOps solution capable of handling real-world scenarios**.

Designing scalable automation is a key skill for DevOps engineers, as it ensures systems can grow and adapt without requiring constant reconfiguration.

* * *

### 🔗 Continue the Series

⬅️ **Previous Article:** [Part 4 **Automating MySQL Backup to AWS S3**](https://ask-abhi.com/part-4-automating-mysql-backup-to-aws-s3)  
➡️ **Next Article:** [Part 6 Running Ansible from Jenkins](https://ask-abhi.com/part-6-running-ansible-from-jenkins)

* * *

⭐ If you found this article useful, follow **https://ask-abhi.com** for more DevOps tutorials.

* * *