EECS 485 Lab

EECS 485 Lab 11: Scaling Server-Side Dynamic Pages with PaaS 1

Due: 8:00pm EST November 22, 2020.

Goals

By the end of this lab, you should:

Prerequisites

Restarting this tutorial

To restart this tutorial, you will need to revert your code changes, delete your ECR Repository, and remove your local Docker images.

Revert code changes

WARNING This will undo ALL changes in the current branch!

$ pwd
/Users/awdeorio/src/eecs485/p3-insta485-clientside/
$ git checkout aws-uniqname
M       ...
Already on 'aws-uniqname'
Your branch is up to date with 'origin/aws-uniqname'.
$ git branch # Verify that you are on the aws-uniqname branch
* aws-uniqname
  master
$ git reset --hard # Reverts all code changes back to your previous commit
HEAD is now at ... [previous commit message]

Delete ECR Repository

To delete your created repository in AWS ECR, you can either delete from the console or using the CLI:

$ aws ecr delete-repository --repository-name insta485 --force
{
    "repository": {
        "repositoryArn": "arn:aws:ecr:us-east-2:943896243526:repository/insta485",
        "registryId": "943896243526",
        "repositoryName": "insta485",
        "repositoryUri": "943896243526.dkr.ecr.us-east-2.amazonaws.com/insta485",
        "createdAt": "2020-08-17T16:19:56-04:00"
    }
}

Remove Local Created Docker Images

The only local images that we created are your local insta485, python, and your AWS insta485. We will remove these with the docker rmi command

$ docker images
REPOSITORY                                              TAG                 IMAGE ID            CREATED             SIZE
943896243526.dkr.ecr.us-east-2.amazonaws.com/insta485   latest              213bd50edc70        17 hours ago        364MB
insta485                                                latest              213bd50edc70        17 hours ago        364MB
python                                                  3.8.1-slim-buster   b99890b7a7dc        6 months ago        193MB
hello-world                                             latest              bf756fb1ae65        7 months ago        13.3kB
$ docker rmi -f 943896243526.dkr.ecr.us-east-2.amazonaws.com/insta485 insta485 python
Untagged: 943896243526.dkr.ecr.us-east-2.amazonaws.com/insta485:latest
Untagged: insta485:latest
Untagged: python:3.8.1-slim-buster

Dev and Prod Configuration

A development environment is for code development and debug. It usually runs on one computer, like the developer’s laptop. For example, flask run is a development server. A production environment is for end users, and usually involves many specialized and scalable programs running on many computers in a data center.

In the context of Project 3, here are some things we need to separate for dev and prod:

Before we get started on any code, make sure you are on the aws-uniqname branch.

$ git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean
$ git checkout aws-uniqname
...
$ git status
On branch aws-uniqname
nothing to commit, working tree clean

Separating Flask Configurations

As mentioned previously, we want to eventually separate out the secret keys for encrypting cookies for our environments. However, there are some commonalities between our environments such as the app root directory, database directory, upload folder, etc.. Since it is probably not the best idea to duplicate code, we will create 3 Flask config files: 1 common config that will be used regardless of the environment, 1 dev config, and 1 prod config.

In your project 3 insta485/ directory, create a new file called your config_common.py and add the following code. All the configuration settings in this file are common between the prod and dev environments.

"""
Insta485 common configuration.
Andrew DeOrio <awdeorio@umich.edu>
"""
import pathlib


# Root of this application, useful if it doesn't occupy an entire domain
APPLICATION_ROOT = '/'

# Cookies
SESSION_COOKIE_NAME = 'login'

# File upload limitations
ALLOWED_EXTENSIONS = set(['png', 'jpg', 'jpeg', 'gif'])
MAX_CONTENT_LENGTH = 16 * 1024 * 1024

# Local file upload to var/uploads/
INSTA485_ROOT = pathlib.Path(__file__).resolve().parent.parent
UPLOAD_FOLDER = INSTA485_ROOT/"var/uploads"

# Database configuration
DATABASE_FILENAME = INSTA485_ROOT/"var/insta485.sqlite3"

Create another file config_dev.py, which will hold all of the configuration settings specific to the dev environment, and add the following code. One thing to note is that it is alright to hardcode your secret key in development because only you (and/or your team) will be able to see it and you are also the only person using the application during development. Use your own SECRET_KEY that you previously generated.

"""
Insta485 development configuration.
Andrew DeOrio <awdeorio@umich.edu>
"""

# Secret key for encrypting cookies.  Never hardcode this in production!
SECRET_KEY = (
    b'FIXME'
)

# Development Database Config
POSTGRESQL_DATABASE_HOST = "localhost"
POSTGRESQL_DATABASE_PORT = 5432
POSTGRESQL_DATABASE_USER = "awdeorio" # OS or WSL username
POSTGRESQL_DATABASE_PASSWORD = None
POSTGRESQL_DATABASE_DB = "insta485"

Create another file config_prod.py, which will hold all of the configuration settings specific to the prod environment, and add the following code. Note that it is extremely dangerous to leave your secret key as plaintext in your source code in production. If an attacker is able to gain access to the source code, then they might be able to create a backdoor channel and gain access to your application. Instead, the secret key should be set as an environment variable that is passed into the code, which is the solution we will be using in this lab. Or better yet, your secret should be stored in a service like AWS Secrets Manager and be fetched when needed.

Note: Even though we are adding in the AWS resource configs, you do not need to re-enable the resources just yet. We will re-enable everything in Lab 12.

Make sure to use your own AWS database host.

"""
Production configuration.
Andrew DeOrio <awdeorio@umich.edu>
"""
import os

# Secret key for encrypting cookies.  It's bad practice to save this in the
# source code, so in production, the secret key should be set as an environment
# variable.
SECRET_KEY = os.environ.get("FLASK_SECRET_KEY")
if not SECRET_KEY:
    raise ValueError("No FLASK_SECRET_KEY environment variable.")
POSTGRESQL_DATABASE_PASSWORD = os.environ.get("POSTGRESQL_DATABASE_PASSWORD")
if not POSTGRESQL_DATABASE_PASSWORD:
    raise ValueError("No POSTGRESQL_DATABASE_PASSWORD environment variable.")

# Production Database Config

# Use your own AWS database host
POSTGRESQL_DATABASE_HOST = "insta485.FIXME.us-east-2.rds.amazonaws.com" 
POSTGRESQL_DATABASE_PORT = 5432
POSTGRESQL_DATABASE_USER = "insta485"
POSTGRESQL_DATABASE_DB = "insta485"

# AWS S3 static files
# https://flask-s3.readthedocs.io/en/latest/
FLASKS3_DEBUG = True
FLASKS3_ACTIVE = True
FLASKS3_BUCKET_NAME = "uniqname.static.insta485.com" # Use your own AWS bucket name
FLASKS3_REGION = "us-east-2"
FLASKS3_FORCE_MIMETYPE = True
FLASKS3_USE_HTTPS = False
FLASKS3_CDN_DOMAIN = "FIXME.cloudfront.net" # Use your own CDN domain name

# AWS S3 file upload
AWS_S3_UPLOAD_BUCKET = "uniqname.uploads.insta485.com" # Use your own uploads S3 bucket name
AWS_S3_UPLOAD_REGION = "us-east-2"
AWS_S3_UPLOAD_FOLDER = "uploads"

Delete your old config.py as we will not need it anymore.

$ rm config.py

Lastly, we need to tell the Flask application to use these different configurations based on the environment. Replace the code in your __init__.py with the following.

"""
Insta485 package initializer.
Andrew DeOrio <awdeorio@umich.edu>
"""
import flask
import flask_s3

# app is a single object used by all the code modules in this package
app = flask.Flask(__name__)  # pylint: disable=invalid-name

# Read settings from config module (insta485/config.py)
app.config.from_object('insta485.config_common')

# Overlay production or development settings.  Set the environment variable
# FLASK_ENV=development for a development environment.  The default is
# production.
if app.config["ENV"] == "development":
    app.config.from_object('insta485.config_dev')
else:
    app.config.from_object('insta485.config_prod')

# Overlay settings read from a Python file whose path is set in the environment
# variable INSTA485_SETTINGS. Setting this environment variable is optional.
# Docs: http://flask.pocoo.org/docs/latest/config/
#
# EXAMPLE:
# $ export INSTA485_SETTINGS=secret_key_config.py
app.config.from_envvar('INSTA485_SETTINGS', silent=True)

s3 = flask_s3.FlaskS3(app)

# Tell our app about views and model.  This is dangerously close to a
# circular import, which is naughty, but Flask was designed that way.
# (Reference http://flask.pocoo.org/docs/0.12/patterns/packages/)  We're
# going to tell pylint and pycodestyle to ignore this coding style violation.
import insta485.api  # noqa: E402  pylint: disable=wrong-import-position
import insta485.views  # noqa: E402  pylint: disable=wrong-import-position
import insta485.model  # noqa: E402  pylint: disable=wrong-import-position

Sanity Check.

$ pwd 
/Users/awdeorio/src/eecs485/p3-insta485-clientside/insta485
$ tree --matchdirs -I '__pycache__|api|js|static|templates|views|model.py'
.
├── __init__.py
├── config_common.py
├── config_dev.py
└── config_prod.py

Separating Webpack Configuration

Similar to the Flask configurations, we will have 3 configurations for Webpack: 1 common, 1 development, and 1 production. If you want to learn more information about best practices and utilities for building a production application, take a look at the Webpack production documentation.

Some of the commonalities between our production and development Webpack configurations include the React entrypoint and the output directory. In your project 3 root directory, create a file named webpack.common.js and add the following code. We will later merge this common config with both the dev and prod configs.

// Webpack configuration shared by development and production builds
//
// Webpack Docs:
// https://webpack.js.org/guides/production/

const path = require('path');

module.exports = {
  entry: './insta485/js/main.jsx',
  output: {
    path: path.join(__dirname, '/insta485/static/js/'),
  },
  module: {
    rules: [
      {
        // Test for js or jsx files
        test: /\.jsx?$/,
        loader: 'babel-loader',
        options: {
          presets: ['@babel/preset-env', '@babel/preset-react'],
        },
      },
    ],
  },
  resolve: {
    extensions: ['.js', '.jsx'],
  },
};

Create a file named webpack.dev.js and add the following code. This code will merge in the common configuration we just defined and also set the output filename to bundle.dev.js, which should look very similar to the configuration you had in project 3.

// Webpack development configuration
//
// Webpack Docs:
// https://webpack.js.org/guides/production/

const merge = require('webpack-merge');
const common = require('./webpack.common.js');

module.exports = merge(common, {
  output: {
    filename: 'bundle.dev.js',
  },
  devtool: 'inline-source-map',
  mode: 'development',
});

The goal of having a separate Webpack configuration for production is to reduce the bundle size in prod using a minifier. To achieve this, create a file named webpack.prod.js and add the following code.

// Webpack production configuration
//
// Webpack Docs:
// https://webpack.js.org/guides/production/

const merge = require('webpack-merge');
const UglifyJSPlugin = require('uglifyjs-webpack-plugin');
const common = require('./webpack.common.js');

module.exports = merge(common, {
  output: {
    filename: 'bundle.min.js',
  },
  plugins: [
    new UglifyJSPlugin({
      sourceMap: true,
    }),
  ],
  mode: 'production',
});

Delete your old webpack.config.js as we do not need it anymore.

$ rm webpack.config.js

Replace the contents of your package.json.

{
  "name": "insta485",
  "version": "0.1.0",
  "description": "insta485 front end",
  "main": "insta485/js/main.jsx",
  "author": "awdeorio",
  "license": "MIT",
  "repository": {},
  "devDependencies": {
      "@babel/core": "7.10.3",
      "@babel/preset-env": "7.10.3",
      "@babel/preset-react": "7.10.1",
      "babel-eslint": "10.1.0",
      "babel-loader": "8.1.0",
      "eslint": "7.2.0",
      "eslint-config-airbnb": "18.2.0",
      "eslint-plugin-import": "2.21.2",
      "eslint-plugin-jsx-a11y": "6.3.1",
      "eslint-plugin-react": "7.20.0",
      "eslint-plugin-react-hooks": "4.0.4",
      "jsonlint": "1.6.3",
      "webpack": "4.44.2",
      "webpack-merge": "^4.2.2",
      "webpack-cli": "3.3.12",
      "uglifyjs-webpack-plugin": "^2.2.0"
  },
  "dependencies": {
      "moment": "2.27.0",
      "prop-types": "15.7.2",
      "react": "16.13.1",
      "react-dom": "16.13.1",
      "react-infinite-scroll-component": "5.0.5"
  }
}

Install the updated JS dependencies.

$ npm install .

Since we want to use a different bundle for the prod and dev environments, we also need to tell Flask to use the correct bundle based on the environment. If you look at your /insta485/templates/index.html, the source for the JavaScript code is always /js/bundle.js. However, now if we are in the dev environment we want to use /js/bundle.dev.js and if we are in the prod environment we want to use /js/bundle.min.js. In your /insta485/templates/index.html replace the code in the <script> tag specifying the bundle source with the following code.

<script
  type="text/javascript"
  {% if config['ENV'] == 'production' %}
  src="{{ url_for('static', filename='js/bundle.min.js') }}"
  {% else %}
  src="{{ url_for('static', filename='js/bundle.dev.js') }}"
  {% endif %}
></script>

Sanity Check

$ pwd
/Users/awdeorio/src/eecs485/p3-insta485-clientside
$ npx webpack --config webpack.dev.js
...
Hash: 8c778cb9bae0ff15e7f1
Version: webpack 4.43.0
Time: 8293ms
Built at: 08/19/2020 3:11:19 PM
        Asset      Size  Chunks             Chunk Names
bundle.dev.js  5.46 MiB    main  [emitted]  main
Entrypoint main = bundle.dev.js
[./insta485/js/comment-input.jsx] 5.68 KiB {main} [built]
...
$ ls insta485/static/js/
bundle.dev.js	... # Verify that bundle.dev.js exists
$ npx webpack --config webpack.prod.js
Hash: 4b88607644a9325ad858
Version: webpack 4.43.0
Time: 11916ms
Built at: 08/19/2020 3:46:41 PM
        Asset     Size  Chunks                    Chunk Names
bundle.min.js  500 KiB       0  [emitted]  [big]  main
Entrypoint main [big] = bundle.min.js
[145] (webpack)/buildin/module.js 552 bytes {0} [built]
...
$ ls insta485/static/js/
min.bundle.js ... # Verify that min.bundle.js exists

Writing Scripts to Run Each Environment

Now that we have all the Flask and Webpack configs separated, let’s write 2 Bash scripts to run each environment.

In your bin/ directory, replace the code in your insta485run script with the following.

#!/bin/bash
#
# insta485run
#
# Clean, build and start server
#
# Andrew DeOrio <awdeorio@umich.edu>

# Stop on errors, print commands
set -Eeuxo pipefail

# Create database if needed
if ! psql -lqt | grep -q insta485; then
    ./bin/insta485db create
fi

# Set up environment for Flask debug server
export FLASK_ENV=development
export FLASK_APP=insta485

# Compile in the background
npx webpack --config webpack.dev.js --watch &

# Run development server
flask run --port 8000

Run this script with ./bin/insta485run. If you navigate to the main feed page of your insta485 and view source, you should see that the source is bundle.dev.js.

In your bin/ directory, create a script named insta485run_prod and add the following code.

#!/bin/bash
#
# insta485run_prod
#
# Clean, build and start server
#
# Andrew DeOrio <awdeorio@umich.edu>

# Stop on errors, print commands
set -Eeuxo pipefail

# Create database if needed
if ! psql -lqt | grep -q insta485; then
    ./bin/insta485db create
fi

# Set up environment for Flask debug server
export FLASK_ENV=production
export FLASK_APP=insta485
export FLASK_SECRET_KEY=FIXME # Need to add this because of how we set up Flask prod config
export POSTGRESQL_DATABASE_PASSWORD=FIXME #Use your own AWS database password

# Compile in the background
npx webpack --config webpack.prod.js --watch &

# Run development server
flask run --port 8000

Add execute priviledges to the script by running chmod +x bin/insta485run_prod from your p3 root directory. You will not be able to run this script unless you re-enable all of your AWS resources from the previous labs. However, we will use this script later in lab 12.

You have now successfully separated out your environments!

Commit and Push Code Changes

If everything has worked for you up until this point, then go ahead and commit and push your changes.

$ git status
On branch aws-uniqname
Your branch is up to date with 'origin/aws-uniqname.
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)
    ...
$ git add .
$ git commit -m "Separate dev and prod Flask and Webpack environments"
$ git push

Docker

Docker is an open source containerization technology. Containers allow us to run our server-side dynamic pages code on AWS servers.

Installing Docker

MacOS

# Docker Server on macOS
brew cask install docker                # Install
open /Applications/Docker.app           # Start VM
docker version                          # Verify connection

Sanity check

$ docker run hello-world
...
Hello from Docker!
...

Pitfall If you see the following message

$ docker run hello-world
docker: Error response from daemon: dial unix docker.raw.sock: connect: connection refused.
See 'docker run --help'.

it means that Docker Desktop is not running and you can verify by checking the icon in your toolbar. If this is the case, then open up the Docker Desktop icon, wait for it to initialize, and then try again.

Linux/WSL

To use Docker on WSL, we first need to download Docker Desktop. Navigate to the link and select Download for Windows (stable).

Once the Open the installer and accept the default configurations by clicking Ok. Launch the Docker Desktop application. If you get an error about your WSL 2 kernel being out of date, navigate here and download the latest WSL2 Linux kernel.

If the dashboard does not appear, navigate to it, by right-clicking the Docker icon in the pop-up menu on your Windows desktop and clicking dashboard.

In the dashboard, click the settings icon. Then check Expose daemon on tcp://localhost:2375 without TLS and click Apply and Restart.

Now, open a terminal and run the following:

$ sudo apt update
$ sudo apt install apt-transport-https ca-certificates curl software-properties-common
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
$ sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable"
$ sudo apt update
$ sudo apt install docker-ce
$ sudo groupadd docker
$ sudo usermod -aG docker ${USER}
$ exit
# After opening a new terminal window
$ echo "export DOCKER_HOST=tcp://localhost:2375" >> ~/.bashrc && source ~/.bashrc

Verify you’re in the docker group.

$ groups
... docker

Sanity check.

$ docker run hello-world
Hello from Docker!
This message shows that your installation appears to be working correctly...

Writing a Manifest file

Your setup.py file describes the metadata about your project and how it can be easily installed by others including the dependencies. This file is used to package your python projects for distribution. However, when you distribute this project, you want to make sure that it is as lightweight as possible, so you should exclude all files that are not essential for actual distribution. In the context of project 3, your python package is your server-side code so you do not need to include the JavaScript files as they are static files (that can be served with AWS services like S3/CloudFront like you did in Lab 10) or dev/binary files as those will be generated when end users run the python code. To define what files will be added/removed to the source distribution, you will need to create a MANIFEST.in file in your p3 root directory and add the following code.

graft insta485

# Avoid JS files in insta485
global-exclude *.js
global-exclude *.jsx

# Avoid dev and and binary files
global-exclude *.pyc
global-exclude __pycache__

To learn more about the MANIFEST.in file and packaging in python, click here.

Writing a Dockerfile

A Dockerfile is a text document that contains all the commands a user would call on the command line to assemble an image. Using docker build users can create an automated build that executes several command-line instructions in succession. Think of these similar to the bash scripts we used in previous projects. They both run several commands to set up either an application or an image. We’ll be writing a Dockerfile for project 3 as well.

Create a new file in your project root folder named Dockerfile and add the following code:

# Configuration for insta485 web server Docker image
#
# Andrew DeOrio <awdeorio@umich.edu>

# Use a minimal Python image as a parent image
FROM python:3.8.1-slim-buster

# Copy list of production requirements into Docker container
COPY requirements-prod.txt /tmp

# Install Python package requirements
RUN pip install -r /tmp/requirements-prod.txt

# Copy application into image
COPY dist/insta485-0.1.0.tar.gz /tmp

# Install Insta485 web app
RUN pip install /tmp/insta485-0.1.0.tar.gz

# Run the web server in the container
CMD gunicorn \
  --workers 4 \
  --bind 0.0.0.0:8000 \
  insta485:app

Python requirements

You’re going to also need a requirements-prod.txt. Create this file in your p3-insta485-clientside (root) folder and add the following code

arrow==0.15.7
click==7.1.2
Flask==1.1.2
gunicorn==20.0.4
itsdangerous==1.1.0
Jinja2==2.11.2
MarkupSafe==1.1.1
python-dateutil==2.8.1
six==1.15.0
Werkzeug==1.0.1

Building a Docker image

With your virtual environment activated, run the following commands in your p3-insta485-clientside (root) folder:

$ pwd
/Users/awdeorio/src/eecs485/p3-insta485-clientside/
$ python3 setup.py sdist # Create a source distribution
$ ls dist/insta485-0.1.0.tar.gz 
dist/insta485-0.1.0.tar.gz
$ docker build --tag insta485:latest .
...
Successfully built d11400c99e16
Successfully tagged insta485:latest

Pitfall: If you can not successfully build the image, double check that your Manifest file, Dockerfile and requirements-prod.txt files are all in the p3-insta485-clientside (root) folder; not in the insta485 folder.

Sanity check: Check that your insta485 image exists in the list of images

$ docker images | grep insta485
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
insta485            latest              9f8303305987        9 seconds ago       206MB

Running a Docker image

Now that we’ve created a Docker image, let’s run it.

$ pwd
/Users/awdeorio/src/eecs485/p3-insta485-clientside/
$ docker run --publish 8000:8000 --env FLASK_SECRET_KEY=0123456789abcdef --env POSTGRESQL_DATABASE_PASSWORD=FIXME insta485:latest
[2020-08-03 22:09:21 +0000] [6] [INFO] Starting gunicorn 20.0.4
[2020-08-03 22:09:21 +0000] [6] [INFO] Listening at: http://0.0.0.0:8000 (6)
[2020-08-03 22:09:21 +0000] [6] [INFO] Using worker: sync
[2020-08-03 22:09:21 +0000] [8] [INFO] Booting worker with pid: 8
[2020-08-03 22:09:21 +0000] [9] [INFO] Booting worker with pid: 9
[2020-08-03 22:09:21 +0000] [10] [INFO] Booting worker with pid: 10
[2020-08-03 22:09:21 +0000] [11] [INFO] Booting worker with pid: 11

Sanity check: Open up another terminal window and verify the image is running by entering this command:

$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED              STATUS              PORTS                    NAMES
c113cbbaf59e        insta485:latest     "/bin/sh -c 'gunicor…"   About a minute ago   Up About a minute   0.0.0.0:8000->8000/tcp   magical_greider
$ curl -L localhost:8000
<!DOCTYPE html>
<html lang="en">
  <head>
    <title>Insta485</title>
...

Killing a Docker image

Now that we’ve built and run a Docker image, let’s kill our currently running image. In your second terminal, run the following command where container_id is the id associated with the insta485 image you got from docker ps:

$ docker kill <container_id> # ex: docker kill c113cbbaf59e
c113cbbaf59e
$ docker ps # should not have anything running!
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

Commit and Push Code Changes

If everything has worked for you up until this point, then go ahead and commit and push your changes.

$ git status
On branch aws-uniqname
Your branch is up to date with 'origin/aws-uniqname'.
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)
    ...
$ git add .
$ git commit -m "Add Docker support"
$ git push

Amazon ECR

Amazon Elastic Container Registry, or ECR, is a container registry that helps developers store and manage Docker container images. We will be using Amazon ECR to store our newly created Docker image.

Create an IAM user

If you have not done so already, create an IAM user and group using this IAM tutorial

AWS CLI

If you have not done so already, install and configure your AWS CLI using this AWS CLI tutorial

Deploying to ECR

During the project, we deployed our application directly to an EC2 instance. However, we had to individually clone our code repo, create a virtual environment, install all the dependencies and other tools, create our database, and run the application. If we want to scale this application, it would be quite tedious to replicate this process on multiple provisioned EC2 instances. Instead, by deploying a Docker image that specifies these instructions already to AWS, we can easily scale our application and the scaling process can even be automated by AWS in a fast and reliable way.

Authenticate Docker to AWS ECR. Run the following command, but use use your own AWS account ID (ours is 403881461553), which you find by going from console -> username -> My Account -> Account Id

$ aws ecr get-login-password --region us-east-2 | docker login --username AWS --password-stdin 403881461553.dkr.ecr.us-east-2.amazonaws.com
WARNING! Your password will be stored unencrypted in /Users/awdeorio/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

Create repository in AWS ECR

$ aws ecr create-repository \
    --repository-name insta485 \
    --image-scanning-configuration scanOnPush=true \
    --region us-east-2

THIS IS OUTPUT
{
    "repository": {
        "repositoryArn": "arn:aws:ecr:us-east-2:403881461553:repository/insta485",
        "registryId": "403881461553",
        "repositoryName": "insta485",
        "repositoryUri": "403881461553.dkr.ecr.us-east-2.amazonaws.com/insta485",
        "createdAt": "2020-06-24T19:40:49-04:00",
        "imageTagMutability": "MUTABLE",
        "imageScanningConfiguration": {
            "scanOnPush": true
        }
    }
}

Add a tag to the image with AWS info. Make sure to use your own account number. This tag helps AWS identify which images from your local repository should be pushed at a later point.

$ docker tag insta485:latest 403881461553.dkr.ecr.us-east-2.amazonaws.com/insta485:latest

Publish to ECR. Make sure to use your own account ID.

$ docker push 403881461553.dkr.ecr.us-east-2.amazonaws.com/insta485:latest

You should be able to now view your image on AWS

$ aws ecr describe-images --repository-name insta485
{
    "imageDetails": [
        {
            "registryId": "403881461553",
            "repositoryName": "insta485",
            ...
            "imageTags": [
                "latest"
            ],
            ...

You should also be able to see the repository in the AWS ECR Console.

Update Image on ECR

NOTE: You only need to worry about this section if you make a mistake with the initial Docker image you pushed to ECR.

First, make any necessary local changes.

Rebuild the docker image.

$ docker images
REPOSITORY                                              TAG                 IMAGE ID            CREATED             SIZE
943896243526.dkr.ecr.us-east-2.amazonaws.com/insta485   latest              154c64009ca9        31 minutes ago      364MB
insta485                                                latest              154c64009ca9        31 minutes ago      364MB
...
$ python3 setup.py sdist
$ ls dist/insta485-0.1.0.tar.gz 
dist/insta485-0.1.0.tar.gz
$ docker build --tag insta485:latest .
$ docker images # Notice that the image IDs are now different
REPOSITORY                                              TAG                 IMAGE ID            CREATED             SIZE
insta485                                                latest              213bd50edc70        9 seconds ago       364MB
943896243526.dkr.ecr.us-east-2.amazonaws.com/insta485   latest              154c64009ca9        33 minutes ago      364MB
...

Authenticate with AWS ECR. Don’t forget to change your account number.

$ aws ecr get-login-password --region us-east-2 | docker login --username AWS --password-stdin 943896243526.dkr.ecr.us-east-2.amazonaws.com
WARNING! Your password will be stored unencrypted in /Users/awdeorio/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

Add a tag to the image with AWS info. Don’t forget to change your account number.

$ docker tag insta485:latest 943896243526.dkr.ecr.us-east-2.amazonaws.com/insta485:latest
$ docker images # Notice that the latest AWS repository and insta485 repository have same Image ID
REPOSITORY                                              TAG                 IMAGE ID            CREATED             SIZE
943896243526.dkr.ecr.us-east-2.amazonaws.com/insta485   latest              213bd50edc70        5 minutes ago       364MB
insta485                                                latest              213bd50edc70        5 minutes ago       364MB
943896243526.dkr.ecr.us-east-2.amazonaws.com/insta485   <none>              154c64009ca9        38 minutes ago      364MB
...

Publish to ECR. Make sure to use your own account ID. Notice there are two images: one with an imageTag and one without

$ docker push 943896243526.dkr.ecr.us-east-2.amazonaws.com/insta485:latest

If you navigate to the ECR console and go into your insta485 repository, you should see 2 images: one with an image tag latest and one with image tag <untagged>. We will eventually remove the untagged image. You can also verify this with the CLI:

$ aws ecr list-images --repository-name insta485
{
    "imageIds": [
        {
            "imageDigest": "sha256:0bee367b4a6704dbdd78be0054cc3463e368ce6deb5ff02af09c1972516eb162"
        },
        {
            "imageDigest": "sha256:f2bd92fc77dc3f8dcfe1b9a196ac5e12cedb4e7a17e2c9a1c65c5052c8362740",
            "imageTag": "latest"
        }
    ]
}

Remove old image locally using the docker rmi command. Make sure to use the image ID of the image that does not have a tag.

$ docker images
REPOSITORY                                              TAG                 IMAGE ID            CREATED             SIZE
943896243526.dkr.ecr.us-east-2.amazonaws.com/insta485   latest              213bd50edc70        5 minutes ago       364MB
insta485                                                latest              213bd50edc70        5 minutes ago       364MB
943896243526.dkr.ecr.us-east-2.amazonaws.com/insta485   <none>              154c64009ca9        38 minutes ago      364MB
...
$ docker rmi -f 154c64009ca9
Untagged: 943896243526.dkr.ecr.us-east-2.amazonaws.com/insta485@sha256:0bee367b4a6704dbdd78be0054cc3463e368ce6deb5ff02af09c1972516eb162
Deleted: sha256:154c64009ca92cb9544ad8d17e279fb1883f17e7f5c06cd9d94756828e90f43a
Deleted: sha256:773dceb52980b9af6988008a8915585d9eae0164674e1fee9f0667221930b484
...
$ docker images
REPOSITORY                                              TAG                 IMAGE ID            CREATED             SIZE
943896243526.dkr.ecr.us-east-2.amazonaws.com/insta485   latest              213bd50edc70        5 minutes ago       364MB
insta485                                                latest              213bd50edc70        5 minutes ago       364MB
...

Remove old image from AWS ECR repository. You can either do this through the AWS ECR console by selecting the untagged images, clicking on delete, and typing in delete to confirm or through the CLI by using the batch-delete-image command. If you use the CLI to remove the image, make sure you use the imageDigest associated with the untagged image.

You should now only be able to see the one image in your insta485 repository in the AWS ECR console.

$ aws ecr list-images --repository-name insta485
{
    "imageIds": [
        {
            "imageDigest": "sha256:0bee367b4a6704dbdd78be0054cc3463e368ce6deb5ff02af09c1972516eb162"
        },
        {
            "imageDigest": "sha256:f2bd92fc77dc3f8dcfe1b9a196ac5e12cedb4e7a17e2c9a1c65c5052c8362740",
            "imageTag": "latest"
        }
    ]
}
$ aws ecr batch-delete-image --repository-name insta485 --image-ids imageDigest=sha256:0bee367b4a6704dbdd78be0054cc3463e368ce6deb5ff02af09c1972516eb162
{
    "imageIds": [
        {
            "imageDigest": "sha256:0bee367b4a6704dbdd78be0054cc3463e368ce6deb5ff02af09c1972516eb162"
        }
    ],
    "failures": []
}
$ aws ecr list-images --repository-name insta485
{
    "imageIds": [
        {
            "imageDigest": "sha256:f2bd92fc77dc3f8dcfe1b9a196ac5e12cedb4e7a17e2c9a1c65c5052c8362740",
            "imageTag": "latest"
        }
    ]
}

Completion Criteria

By the end of this lab, you should:

Lab Quiz

Complete the lab quiz by the due date.