p2-insta485-serverside

EECS485 P2: Server-side Dynamic Pages

Due 8pm Thursday February 6th. This is a group project to be completed in groups of two to three.

Please register your group on the Autograder. You must register your group to access the office hours queue. The synchronization between the autograder and office hours queue happens daily so you will need to wait a day before your group can access the OH queue.

Change Log

Initial Release for W20: Version 6.0

Version 6.1: An update has been made to the database tests that involves enabling foreign keys.

Version 6.2: Spec clarifications regarding /accounts/logout/, /accounts/edit/, following table and test_scripts.py. Update to account tests which had a bug.

Introduction

An Instagram clone implemented with server-side dynamic pages. This is the second of an EECS 485 three project sequence: a static site generator from templates, server-side dynamic pages, and client-side dynamic pages.

Build an interactive website using server-side dynamic pages. Reuse the templates from project 1, rendering them on-demand when a user loads a page. New features include creating, updating, and deleting users, posts, comments, and likes.

The learning goals of this project include server-side dynamic pages, CRUD (Create, Read, Update, Delete), sessions, and basic SQL database usage.

Here’s a preview of what your finished project will look like. A database-backed interactive website will work (mostly) like the real Instagram.

$ ./bin/insta485run
 * Serving Flask app "insta485"
 * Running on http://127.0.0.1:8000/ (Press CTRL+C to quit)

Then you will navigate to http://localhost:8000 and see working, multi-user, interactive website that you created. screenshot accounts login small

This project adds lots features. For example, users can add likes and comments.

screenshot accounts login small

This spec will walk you through several parts:

  1. Setup
  2. Building the database
  3. Creating the app and scripts
  4. Dynamic Server-side Insta485 specification
  5. Deploy to AWS
  6. Submitting and grading
  7. FAQ

Use the same local development tool chain for this project that you did in the previous project.

Setup

Group registration

Please register your group on the Autograder. You must register your group to access the office hours queue. The synchronization between the autograder and office hours queue happens daily so you will need to wait a day before your group can access the OH queue. The office hours queue will give first priority to groups asking a question for the first time in a day.

AWS account and instance

You will use Amazon Web Services (AWS) to deploy your project. AWS account setup may take up to 24 hours, so get started now. Create an account, launch and configure the instance. Don’t deploy yet. AWS Tutorial.

Project folder

Create a folder for this project (instructions). Your folder location might be different.

$ pwd
/Users/awdeorio/src/eecs485/p2-insta485-serverside

Version control

Set up version control using the Version control tutorial.

Be sure to check out the Version control for a team tutorial.

After you’re done, you should have a local repository with a “clean” status and your local repository should be connected to a remote GitLab repository.

$ pwd
/Users/awdeorio/src/eecs485/p2-insta485-serverside
$ git status
On branch master
Your branch is up-to-date with 'origin/master'.

nothing to commit, working tree clean
$ git remote -v
origin	https://gitlab.eecs.umich.edu/awdeorio/p2-insta485-serverside.git (fetch)
origin	https://gitlab.eecs.umich.edu/awdeorio/p2-insta485-serverside.git (push)

You should have a .gitignore file (instructions).

$ pwd
/Users/awdeorio/src/eecs485/p2-insta485-serverside
$ head .gitignore
This is a sample .gitignore file that's useful for EECS 485 projects.
...

Python virtual environment

Create a Python virtual environment using the Project 1 Python Virtual Environment Tutorial.

You should now have Python tools installed locally.

$ pwd
/Users/awdeorio/src/eecs485/p2-insta485-serverside
$ ls
env
$ source env/bin/activate
$ which python
/Users/awdeorio/src/eecs485/p2-insta485-serverside/env/bin/python
$ which pip
/Users/awdeorio/src/eecs485/p2-insta485-serverside/env/bin/pip

Install utilities

Linux and Windows 10 Subsystem for Linux

$ sudo apt-get install sqlite3 curl

MacOS

$ brew install sqlite3 curl

Starter files

Download and unpack the starter files.

$ pwd
/Users/awdeorio/src/eecs485/p2-insta485-serverside
$ wget https://eecs485staff.github.io/p2-insta485-serverside/starter_files.tar.gz
$ tar -xvzf starter_files.tar.gz

Move the starter files to your project directory and remove the original starter_files/ directory.

$ pwd
/Users/awdeorio/src/eecs485/p2-insta485-serverside
$ mv starter_files/* .
$ rm -rf starter_files starter_files.tar.gz

You should see these files.

$ tree --matchdirs -I env
.
├── VERSION
├── setup.py
├── sql
│   └── uploads
        ...
│       └── e1a7c5c32973862ee15173b0259e3efdb6a391af.jpg
└── tests
    ....
    └── util.py
VERSION Version of the starter files
setup.py Insta485 python package configuration
sql/uploads/ Sample image uploads
tests/ Public unit tests

Before making any changes to the clean starter files, it’s a good idea to make a commit to your Git repository.

Database

If you’re new to SQL, take a look at the w3Schools SQL Intro.

Complete the SQLite Tutorial. After the tutorial, you should have the sqlite3 command line utility installed. Your version might be different.

$ sqlite3 --version
3.29.0 2019-07-10 17:32:03 fc82b73eaac8b36950e527f12c4b5dc1e147e6f4ad2217ae43ad82882a88bfa6

You should have a script to manage the database. Your output might be slightly different.

$ pwd
/Users/awdeorio/src/eecs485/p2-insta485-serverside
$ ./bin/insta485db reset
+ rm -rf var/insta485.sqlite3 var/uploads
+ mkdir -p var/uploads
+ sqlite3 var/insta485.sqlite3 < sql/schema.sql
+ sqlite3 var/insta485.sqlite3 < sql/data.sql
+ cp sql/uploads/* var/uploads/

You have created these files.

$ pwd
/Users/awdeorio/src/eecs485/p2-insta485-serverside
$ tree sql var
sql
├── data.sql
├── schema.sql
└── uploads
    ...
    └── e1a7c5c32973862ee15173b0259e3efdb6a391af.jpg
var
├── insta485.sqlite3
└── uploads
    ...
    └── e1a7c5c32973862ee15173b0259e3efdb6a391af.jpg

Schema

Update schema.sql, which will create 5 tables: users, posts, following, comments and likes. The following list describes the tables and columns

To clarify for the following table, username1 follows username2.

Data

Update sql/data.sql to add all initial data. You can find a complete dump of the initial data in insta485db-dump.txt. Your timestamps will be different. The passwords are all set to password.

Testing

Run the public autograder testcases on your database schema and data. You will need to complete the Flask Tutorial before running this test.

$ pytest -v tests/test_database_public.py
...
========================== 2 passed in 1.76 seconds ===========================

Make sure these tests pass before moving on. The other unit tests rely on a fully functionally bin/insta485db script.

Flask app

Complete the Flask Tutorial if you have not already.

You should now have a directory containing an insta485 Python module.

$ tree insta485
insta485
├── __init__.py
├── config.py
├── model.py
├── static
│   └── css
│   │   └── style.css
├── templates
│   └── index.html
└── views
    ├── __init__.py
    └── index.py

The insta485run script starts a development server and you can browse to http://localhost:8000/ where you’ll see your “hello world” app.

$ ./bin/insta485run 
+ test -e var/insta485.sqlite3
+ export FLASK_DEBUG=True
+ FLASK_DEBUG=True
+ export FLASK_APP=insta485
+ FLASK_APP=insta485
+ export INSTA485_SETTINGS=config.py
+ INSTA485_SETTINGS=config.py
+ flask run --host 0.0.0.0 --port 8000
 * Serving Flask app "insta485" (lazy loading)
 * Environment: production
   WARNING: Do not use the development server in a production environment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on http://0.0.0.0:8000/ (Press CTRL+C to quit)

Testing

Compliant HTML

Automatically generated HTML shall be W3C HTML5 compliant. To test dynamically generated pages, the test_style.py::test_html test cases renders each page and saves it to a file. Then, it runs html5validator on the files.

$ pytest -vvs tests/test_style.py::test_html
...
GET /
GET /explore/
GET /u/awdeorio/
GET /p/3/
GET /u/jflinn/
GET /u/michjc/
GET /p/2/
GET /p/1/
GET /u/jag/
GET /accounts/edit/
GET /u/awdeorio/followers/
GET /u/awdeorio/following/
GET /u/jflinn/followers/
GET /u/jflinn/following/
GET /u/michjc/followers/
GET /u/michjc/following/
GET /u/jag/followers/
GET /u/jag/following/
GET /p/4/
GET /accounts/password/
GET /accounts/delete/
html5validator 0.3.3
html5validator --root tmp/localhost
PASSED

insta485test script

Similar to project 1, all Python code must be PEP8 compliant, comments shall be PEP257 compliant, and code shall pass a pylint static analysis.

Write another script called bin/insta485test that does this:

  1. Stop on errors and prints commands
  2. Run pycodestyle insta485
  3. Run pydocstyle insta485
  4. Run pylint --disable=cyclic-import insta485
  5. Run all unit tests using pytest -v tests

Don’t forget to check for shell script pitfalls.

$ file bin/*
bin/insta485db:        Bourne-Again shell script text executable, ASCII text
bin/insta485run:       Bourne-Again shell script text executable, ASCII text
bin/insta485test:      Bourne-Again shell script text executable, ASCII text

Unit tests

Now that you have the framework of the project in place and your utility scripts written, we can run some of the unit tests. We have provided the public Autograder tests.

Run the public autograder testcases on your utility scripts.

$ pytest -v tests/test_scripts.py
...
========================== 5 passed in 1.16 seconds ===========================

Note: if you get deprecation warnings from third party libraries, check out the pytest tutorial - deprecation warnings to suppress them.

Dynamic Server-side Insta485 specification

This project includes the same pages as project 1. The pages also include buttons to follow, unfollow, like, unlike and comment. We’ll also add pages for user account administration.

URLs

List of URLs from project 1, including screenshots with user awdeorio logged in.

List of new URLs:

Hint: When linking to pages or static files look into flask’s url_for() and send_from_directory() function.

All pages

Include a link to the main page /.

If not logged in, redirect to /accounts/login/ (unless your current page is the login page).

If logged in, include a link to /explore/.

If logged in, include a link to /u/<user_url_slug>/ where user_url_slug is the logged in user.

Access control

The server should accept POST requests from the logged in user only. To reject a request with a permissions error, use flask.abort(403).

The following examples assume you have a (mostly) working Insta485 project with a freshly reset database.

Normal example

This example is just like using a web browser to fill in the HTML login form rendered on the login page and then navigating to /. Then, it deletes a post.

Use curl to sign in to Insta485. This command issues a POST request to /accounts/login/ with a username and password.

$ curl -X POST http://localhost:8000/accounts/login/ \
  -F username=awdeorio \
  -F password=password \
  --cookie-jar cookies.txt

View the index / page by issuing a GET request.

$ curl -b cookies.txt http://localhost:8000/
... <index.html page content here>

Delete an Insta485 post. The user awdeorio can successfully delete his own post by issuing a POST request.

$ curl -X POST http://localhost:8000/p/1/ \
  -F postid=1 \
  -F 'delete=delete this post' \
  -b cookies.txt

Malicious example

Even though the “Delete post” button is hidden on posts that the logged in user doesn’t own, any user can use a tool like curl to send a POST request to try to delete an Insta485 post.

Try to delete a post created by jflinn using awdeorio’s cookies. We get a 403 Forbidden error. This is a good thing!

$ curl -X POST http://localhost:8000/p/2/ \
  -F postid=2 \
  -F 'delete=delete this post' \
  -b cookies.txt 
...
<title>403 Forbidden</title>
...

Index /

screenshot

The index page should include all posts from the logged in user and all other users that the logged in user is following. The most recent post should be at the top. For each post:

Form for “like” button

<!-- DO NOT CHANGE THIS (aside from where we say 'FIXME') -->
<form action="<FIXME CURRENT PAGE URL HERE>" method="post" enctype="multipart/form-data">
  <input type="hidden" name="postid" value="<FIXME postid HERE>"/>
  <input type="submit" name="like" value="like"/>
</form>

Form for “unlike” button

<!-- DO NOT CHANGE THIS (aside from where we say 'FIXME') -->
<form action="<FIXME CURRENT PAGE URL HERE>" method="post" enctype="multipart/form-data">
  <input type="hidden" name="postid" value="<FIXME postid HERE>"/>
  <input type="submit" name="unlike" value="unlike"/>
</form>

Form for “comment” button

<!-- DO NOT CHANGE THIS (aside from where we say 'FIXME') -->
<form action="<FIXME CURRENT PAGE URL HERE>" method="post" enctype="multipart/form-data">
  <input type="hidden" name="postid" value="<FIXME postid HERE>"/>
  <input type="text" name="text"/>
  <input type="submit" name="comment" value="comment"/>
</form>

/u/<user_url_slug>/

screenshot1 screenshot2

Be sure to include

For a user’s own page, also include

Form for follow button

<!-- DO NOT CHANGE THIS (aside from where we say 'FIXME') -->
<form action="<FIXME CURRENT PAGE URL HERE>" method="post" enctype="multipart/form-data">
  <input type="submit" name="follow" value="follow"/>
  <input type="hidden" name="username" value="<FIXME username>"/>
</form>

Form for unfollow button

<!-- DO NOT CHANGE THIS (aside from where we say 'FIXME') -->
<form action="<FIXME CURRENT PAGE URL HERE>" method="post" enctype="multipart/form-data">
  <input type="submit" name="unfollow" value="unfollow"/>
  <input type="hidden" name="username" value="<FIXME username>"/>
</form>

Form for file upload

<!-- DO NOT CHANGE THIS -->
<form action="<FIXME CURRENT PAGE URL HERE>" method="post" enctype="multipart/form-data">
  <input type="file" name="file">
  <input type="submit" name="create_post" value="upload new post"/>
</form>

We use the sha256 hash algorithm to compute a hash of file contents. You can do this at the command line like this:

$ wget http://andrewdeorio.com/assets/headshot.jpg -O awdeorio.jpg
$ sha256sum awdeorio.jpg
e1a7c5c32973862ee15173b0259e3efdb6a391af  awdeorio.jpg

Here’s how to compute filenames in your Flask app:

import os
import shutil
import tempfile
import flask

def sha256sum(filename):
    """Return sha256 hash of file content, similar to UNIX sha256sum."""
    content = open(filename, 'rb').read()
    sha256_obj = hashlib.sha256(content)
    return sha256_obj.hexdigest()


# Save POST request's file object to a temp file
dummy, temp_filename = tempfile.mkstemp()
file = flask.request.files["file"]
file.save(temp_filename)

# Compute filename
hash_txt = sha256sum(temp_filename)
dummy, suffix = os.path.splitext(file.filename)
hash_basename = hash_txt + suffix
hash_filename = insta485.app.config["UPLOAD_FOLDER"]/hash_basename

# Move temp file to permanent location
shutil.move(temp_filename, hash_filename)

/u/<user_url_slug>/followers/

screenshot

List the users that are following user_url_slug. For each, include:

/u/<user_url_slug>/following/

screenshot

List the users that user_url_slug is following. For each, include:

/p/<postid_url_slug>/

screenshot1 screenshot2

This page shows one post. Include the same information for this one post as is shown on the main page /.

Include a “delete” button next to each comment owned by the logged in user.

<!-- DO NOT CHANGE THIS (aside from where we say 'FIXME') -->
<form action="<FIXME CURRENT PAGE URL HERE>" method="post" enctype="multipart/form-data">
  <input type="hidden" name="commentid" value="<FIXME commentid"/>
  <input type="submit" name="uncomment" value="delete"/>
</form>

Include a “delete this post” button if the post is owned by the logged in user. Delete the file from the filesystem. Delete everything in the database related to this post. Redirect to /u/<user_url_slug>/ for the user whose post was deleted.

<!-- DO NOT CHANGE THIS (aside from where we say 'FIXME') -->
<form action="<FIXME CURRENT PAGE URL HERE>" method="post" enctype="multipart/form-data">
  <input type="hidden" name="postid" value="<FIXME postid>"/>
  <input type="submit" name="delete" value="delete this post"/>
</form>

/explore/

screenshot

This page lists all users not that the logged in user is not following and includes:

/accounts/login/

screenshot

If logged in, redirect to /.

Otherwise, include username and password inputs, and a login button. If someone tries to log in to a user that does not exist or provides incorrect credentials, abort(403).

Also include a link to /accounts/create/ in the page.

Use this HTML form code. Feel free to style it and include placeholders.

<!-- DO NOT CHANGE THIS (aside from styling) -->
<form action="<FIXME CURRENT PAGE URL HERE>" method="post" enctype="multipart/form-data">
  <input type="text" name="username"/>
  <input type="password" name="password"/>
  <input type="submit" value="login"/>
</form>

Warning: only store minimal information in a session cookie!

/accounts/logout/

Log out user on the server side. Immediately redirects to /accounts/login/, which is why there is no screenshot for /accounts/logout/.

/accounts/create/

screenshot

If a user is already logged in, redirect to /accounts/edit/.

If a user tries to create an account with an existing username in the database, abort(409) - this is an HTTP code indicating a Conflict Error.

If a user tries to create an account with an empty string as the password, abort(400) - this is an HTTP code indicating a Bad Request.

If the account is created successfully, log the user in, and redirect to the main page /.

Also include a link to /accounts/login/ in the page.

HTML form. Style as you like.

<!-- DO NOT CHANGE THIS (aside from styling) -->
<form action="<FIXME CURRENT PAGE URL HERE>" method="post" enctype="multipart/form-data">
  <input type="file" name="file">
  <input type="text" name="fullname"/>
  <input type="text" name="username"/>
  <input type="text" name="email"/>
  <input type="password" name="password"/>
  <input type="submit" name="signup" value="sign up"/>
</form>

See above for file upload and naming procedure.

A password entry in the database contains the algorithm, salt and password hash separated by $. Use the sha512 algorithm like this:

import uuid
import hashlib
algorithm = 'sha512'
salt = uuid.uuid4().hex
hash_obj = hashlib.new(algorithm)
password_salted = salt + password
hash_obj.update(password_salted.encode('utf-8'))
password_hash = hash_obj.hexdigest()
password_db_string = "$".join([algorithm, salt, password_hash])
print(password_db_string)

We’ve shipped one of the autograder tests so you can verify that login and logout mechanics work correctly. It’s in starter_files/tests/test_login_logout_public.py. Save it to tests/test_login_logout_public.py and run it like this:

$ pytest -v tests/test_login_logout_public.py
== test session starts ==
...
== 4 passed in 0.91 seconds ==

/accounts/delete/

screenshot

Confirmation page includes username and this form:

<!-- DO NOT CHANGE THIS -->
<form action="<FIXME CURRENT PAGE URL HERE>" method="post" enctype="multipart/form-data">
  <input type="submit" name="delete" value="confirm delete account"/>
</form>

Delete all post files created by this user. Delete user icon file. Delete all related entries in all tables. Hint: database tables set up properly with primary/foreign key relationships and CASCADE ON DELETE will do this automatically.

Upon successful submission, redirect to /accounts/create/.

/accounts/edit/

screenshot

Include user’s current photo and username. Include a form with photo upload, name and email. Name and email are automatically filled in with previous value. Username can not be edited.

If no photo file is included, then the server will update only the user’s name and email. If a photo file is included, then the server will update the user’s photo, name and email. Delete the old photo from the filesystem. See above for file upload and naming procedure.

Link to /accounts/password/.

Link to /accounts/delete/.

Use this form

<!-- DO NOT CHANGE THIS (aside from where we say 'FIXME') -->
<form action="<FIXME CURRENT PAGE URL HERE>" method="post" enctype="multipart/form-data">
  <input type="file" name="file">
  <input type="text" name="fullname" value="<FIXME full name here>"/>
  <input type="text" name="email" value="<FIXME email here>"/>
  <input type="submit" name="update" value="submit"/>
</form>

Upon successful submission, re-render the current page.

/accounts/password/

screenshot

Include this form:

<!-- DO NOT CHANGE THIS -->
<form action="<FIXME CURRENT PAGE URL HERE>" method="post" enctype="multipart/form-data">
  <input type="password" name="password"/>
  <input type="password" name="new_password1"/>
  <input type="password" name="new_password2"/>
  <input type="submit" name="update_password" value="submit"/>
</form>

Link to /accounts/edit/.

Check the user’s password and abort(403) if it fails.

Check if both new passwords match. abort(401) otherwise.

Update hashed password entry in database. (See above).

Upon successful submission, redirect to /accounts/edit/.

Static File Permissions

A user with the direct link to an uploaded file, /uploads/<filename>, should only be able to see that file if logged in. If an unauthenticated user attempts to access an uploaded file, abort(403).

Testing

Run the unit tests. Everything except for the deploy test should pass.

$ pytest -v

Deploy to AWS

You should have already created an AWS account and instance (instructions). Resume the AWS Tutorial - Deploy a web app.

After you have deployed your site, download the main page along with a log. Do this from your local machine.

$ pwd
/Users/awdeorio/src/eecs485/p2-insta485-serverside
$ curl -v <Public DNS (IPv4)>/accounts/login/ > deployed_insta485.html 2> deployed_insta485.log

Be sure to verify that the output in deployed_insta485.log doesn’t include errors like “Couldn’t connect to server”. If it does contain an error like this, it means curl couldn’t successfully connect with your flask app.

Testing

Test your deployed output.

$ pytest -v tests/test_deploy.py

Submitting and grading

One team member should register your group on the autograder using the create new invitation feature.

Submit a tarball to the autograder, which is linked from https://eecs485.org. Include the --disable-copyfile flag only on macOS.

$ tar \
  --disable-copyfile \
  --exclude '*__pycache__*' \
  -czvf submit.tar.gz \
  bin \
  insta485 \
  setup.py \
  sql \
  deployed_insta485.html \
  deployed_insta485.log

The autograder will run pip install -e YOUR_SOLUTION. The exact library versions in starter_files/setup.py are cached on the autograder, so be sure not to add extra library dependencies to setup.py.

Rubric

This is an approximate rubric.

Deliverable Value
Handcoded SQL 10%
Python and HTML style 10%
Scripts 10%
insta485 (public) 30%
insta485 (private) 40%

FAQ

Do trailing slashes in URLs matter?

Yes. Use them everywhere. See the “Unique URLs / Redirection Behavior” section in the Flask quickstart. Here’s a good example:

@insta485.app.route("/u/<username_url_slug>/", methods=["GET", "POST"])

Static files (like .jpg) should not have a trailing slash at the end. Use trailing slashes everywhere else.

Should I change HTML forms?

You can add HTML to style forms any way you choose. Don’t change the number, type or names of the inputs. This is because the autograder will make POST requests.

Use the flask.url_for function. Read documentation about this function online.

Can I disable any code style checks?

Do not disable any code style check from any code style tool (pycodestyle, pydocstyle, pylint). There are three exceptions listed here.

There are several pylint --disable options mentioned in the Testing Section, above. Use the exact commands provided.

In insta485/__init__.py, the Flask framework requires an import at the bottom of the file (reference). We’re going to tell pylint and pycodestyle to ignore this coding style violation.

import insta485.views  # noqa: E402  pylint: disable=wrong-import-position
import insta485.model  # noqa: E402  pylint: disable=wrong-import-position

In insta485/__init__.py, the Flask framework uses an object that is used throughout the module. Although its value never changes, it is not a constant in the classic sense. We’re going to tell pylint to ignore this coding style violation.

app = flask.Flask(__name__)  # pylint: disable=invalid-name

I’m getting “302 != 200” error from the autograder…

For this project, we will be using the same endpoints to do both ‘GET’ and ‘POST’ requests. Although it may be the same endpoint, these two types of requests will do different things (by definition), so please do not use redirects unnecessarily to handle each of the different types of requests. Instead, do the following:

DO:

if flask.request.method == 'POST':
  # POST specific code...
# Let execution fall to non-POST specific code

DON’T:

if flask.request.method == 'POST':
  # POST specific code...
return flask.redirect(flask.url_for( 'other_endpoint_for_nonpost' ))