Project 1: Linux Servers on AWS
In this project an AWS cloud infrastructure is deployed using the AWS CLI command line tool in bash shell scripts.The AWS cloud environment will have a VPC, internet gateway, public subnet, public route table, and EC2 instances.
- Category: AWS/Automation tools
- Script: Terraform & Ansible
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
- Create an AWS VPC. Create an internet gateway and associate it to the VPC.
- Create one private subnet and one public subnet in the VPC.
- Create a public route table routing 0.0.0.0/0 to the internet gateway.
- Create security groups for the subnets.
- Database server.
- API server.
- Autoscaling group.
- Application load balancer.
Project 2: Infrastructure Provisioning Automation
Set up a multi-tiered application infrastructure architecture using Terraform and ansible.
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
- Create a Web App: A simple web app should be created or leverage an existing one, such as those available on GitHub.
- Dockerize the App: Dockerize the web app by creating a Dockerfile with instructions to build a Docker image for the app.
- Create a DockerHub Account: If not already done, create a DockerHub account to host the Docker image.
- Build and Push Docker Image: Build the Docker image in both x86_64 and arm64 formats using Docker Buildx tool, and push the images to DockerHub registry.
- Deploy the App on AWS: Deploy the app container on AWS by creating and launching an EC2 instance to serve as the container server.
- Install Docker Engine:
Project 3: Web App Deployment on AWS Using Docker
This project a simple web app will be deployed as a Docker container on the AWS cloud, allowing users to access the app via the public internet.
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Project 4: Multi-Container Web App Deployment Project
In this project, students will work in a team to deploy a web app that leverages multiple Docker images. They will orchestrate this multi-container setup using Docker Compose. The web app containers environment will be hosted on the AWS cloud, and users will be able to access the web app via the public internet.
Requirements
1. Create or Leverage a Web App:
Develop or use a web app that is publicly available, such as on GitHub.
2. Multiple Microservices:
The web app environment should contain 2 or more microservices that communicate with each other. For instance, it could include an API app that makes calls to a database.
3. Dockerize the App(s) and/or Database(s):
Create Dockerfiles with instructions to create the respective Docker images for the app(s), microservice(s), and database(s).
4. DockerHub Account:
Create a DockerHub account if you don’t have one already and push the Docker images to your DockerHub registry.
5. Build Docker Images:
Build the Docker images in both x86_64 and arm64 formats, for example, using Docker Buildx tool.
6. Deploy with Docker Compose on AWS:
Deploy and orchestrate the containers with Docker Compose on AWS.
7. EC2 Instance:
Create and launch an EC2 instance to serve as the container server to host your app containers.
8. Install Docker Engine and Docker Compose:
On the EC2 server, download and install Docker Engine and Docker Compose.
9. Docker Compose Configuration:
Create a docker-compose.yaml file containing the configurations necessary to run your multi-container setup with Docker Compose.
10. Launch and Run Containers:
Launch and run your containers on the server with Docker Compose and ensure they are accessible via the public internet.
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Project 5: API’s and Databases
API’s and Databases
In this project, an API microservice connected to a database will be deployed. This API will have exposed endpoints that users can send HTTP requests to and get a response. Data will be returned to users as JSON payloads. AWS cloud infrastructure will be provisioned using any of AWS CLI, boto3, docker. Specifically, the AWS cloud environment will have a VPC, internet gateway, public subnet, public route table, EC2 instances. These scripts will be stored in GitHub.
Requirements
Database Creation: Create a database to store data (e.g., NHL player stats) using a database server (MySQL). API Server:Create an API server using any programming language and framework (e.g., FastAPI, Node.js + Express, Java Spring Boot).
Endpoints: Create GET endpoints for retrieving data from the database (e.g., /players, /toronto, /points).
Deployment: Deploy the database server and API microservice on AWS cloud, ensuring accessibility on the public internet.
Documentation
GitHub Repository:
https://github.com/auzadeh7049/WeCloudData-Projects/tree/main
Architectural Diagrams:
Architectural diagram depicting the AWS cloud infrastructure setup
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
- Source:
- Build:
- Test:
- Push:
- Deploy:
Project 6: CI/CD
A CI/CD pipeline will be deployed with Jenkins with build, test and deploy stages for a web app. The web app is also deployed as a container in a environment Docker Compose and AWS EKS. The app will be updated on the developer’s local machine, and these changes to the app will be commited in Git and pushed to the main branch on a remote GitHub repo. This triggers a Jenkins CI pipeline to build the app as a new Docker image, test the new build and push the Docker image to Docker Hub. Finally, the Jenkins pipeline will then deploy the updated app as a container into a the container environment of choice with appropriate Docker compose, Kubernetes manifest and configuration files.
Requirements
Deploy a Jenkins server on AWS: 1. Provision a Jenkins server on AWS using any method you prefer. 2. Ensure you are able to connect to the Jenkins server and log in to the Jenkins dashboard. Deploy Container infrastructure:1. Provision a container environment - (Create an AWS EKS cluster to automate the infrastructure provisioning.
2. Ensure the container environment is and ready to deploy containerized apps.
App CI/CD with Jenkins pipeline: 1. Create the container and Kubernetes manifest and configuration files with deployment configuration information for your app(s).
2. Create a Jenkinsfile containing a Jenkins pipeline with the following stages
This stage will execute a Git checkout of your GitHub repo containing your app source code, Dockerfile and Kubernetes manifest configuration files.
This stage will build the app’s Docker image locally on the Jenkins server.
This stage will execute tests for your app and stop the remaining Jenkins pipeline if the tests fail or move on to the next stages if the tests pass.
If the tests pass, this stage will then push the newly built Docker image to your Docker Hub repo.
This stage gets the Jenkins server to deploy the app into the container environment by Docker, Docker compose, AWS EKSc. Using the corresponding Docker and/or Kubernetes manifest files.
Project 7: Kubernetes
In this project, a web app will be deployed onto a cloud production Kubernetes cluster that can be consumed by users on the public internet. Because this is deployed in a Kubernetes environment the orchestration of the app containers is done using the Kubernetes tech stack. For this specific project AWS's managed Kubernetes service will be leveraged for their cloud production Kubernetes cluster.
Requirements
1. Create a web app or leverage a web app that is already publicly available on GitHub.
2. Dockerize the app.
a. Create Dockerfile with instructions to create the respective Docker image for this app.
b. Create a DockerHub account
c. Build the Docker images and push the Docker images to your DockerHub registry
3. Deploy and orchestrate the app containers with the AWS managed Kubernetes service AWS EKS.
4. Deploy and orchestrate the app containers with the AWS managed Kubernetes service AWS EKS.
a. Launch and run your app containers in the configured AWS EKS Kubernetes cluster(s) and ensure your app is accessible by users from the public internet.
b. The app should have a valid IP address or DNS endpoint where users can connect to from their browser.
Project 8 - Observability Systems
In this project, a web app will deployed onto a cloud production Kubernetes cluster that is also actively monitored with observability systems. Fluentd and Elasticsearch are used to provide observability systems. They may do the scraping metrics for system's logs and visualization.
Requirements
1. Create a web app or leverage a web app that is already publicly available on GitHub.
2. Dockerize the app.
a. Create Dockerfile with instructions to create the respective Docker image for this app.
b. Build the Docker images in both x86_64 and arm64 formats and push the Docker images to the DockerHub registry.
3. Deploy and orchestrate the app containers with the AWS managed Kubernetes service AWS EKS.
4. Install and activate observability systems.
a. Add Fluentd
i. You can do this by installing Fluentd as a daemonset on the Kubernetes cluster.
ii. Fluentd will be running as an agent in a pod in each of the Kubernetes nodes.
iii. For log collection the Fluentd tail plugin is used.
iv. Set the Fluentd configuration accordingly to collect the logs for the setup and allow for Prometheus to scrape metrics.
b. add Prometheus.
i. Install Prometheus.
ii. Scrape the logs generated by Fluentd and store as metrics for Grafana to query and display on its visual dashboards.
c. add Grafana.
i. Install Grafana.
ii. Query Prometheus metrics and display the metrics as visuals on Grafana dashboards.