p1-insta485-static

pytest Tutorial

pytest is a unit test utility that makes it easy to run unit tests. It is compatible with the Python unittest framework.

Quick start

TL;DR: for debugging, you probably want maximum verbose output (-vv), application stdout to pass through to pytest stdout (-s), and stop after the first error (-x).

$ pytest -vvsx

Step-by-step

This tutorial uses Project 1 as an example. It assumes you have some code and a directory called tests/.

$ pwd
/Users/awdeorio/src/eecs485/p1-insta485-static
$ ls
bin  hello      html      insta485generator           setup.py
env  hello_css  insta485  insta485generator.egg-info  tests

You have installed and activated your Python virtual environment (instructions) and installed pytest (pip install pytest if needed). Your version might be different.

$ pytest --version
This is pytest version 4.1.0, imported from /Users/awdeorio/src/eecs485/p1-insta485-static/env/lib/python3.7/site-packages/pytest.py

By default, pytest will search for tests and run them all.

$ pytest
============================= test session starts ==============================
platform darwin -- Python 3.7.7, pytest-5.4.1, py-1.8.1, pluggy-0.13.1
rootdir: /Users/awdeorio/src/eecs485/p1-insta485-static/solution
collected 35 items

tests/test_handcoded_html.py .                                           [  2%]
tests/test_handcoded_index.py ......                                     [ 20%]
tests/test_handcoded_user.py ......                                      [ 37%]
tests/test_insta485generator_hello.py .                                  [ 40%]
tests/test_insta485generator_hello_css.py .                              [ 42%]
tests/test_style.py ...                                                  [ 51%]
tests/test_template_followers.py .....                                   [ 65%]
tests/test_template_following.py .....                                   [ 80%]
tests/test_template_html.py .                                            [ 82%]
tests/test_template_index.py ......                                      [100%]

============================= 35 passed in 19.06s ==============================

Run all unit tests in one file.

$ pytest tests/test_template_followers.py

Run one unit test. You can find the exact name of a testcase by running them all with the verbose flag -v.

Common options

Print the name of each individual unit test with -v or --verbose. Pro-tip: -vv increases the output.

$ pytest -vv
============================= test session starts ==============================
platform darwin -- Python 3.7.7, pytest-5.4.1, py-1.8.1, pluggy-0.13.1 -- /Users/awdeorio/src/eecs485/p1-insta485-static/env/bin/python3
cachedir: .pytest_cache
rootdir: /Users/awdeorio/src/eecs485/p1-insta485-static/solution
collected 35 items

tests/test_handcoded_html.py::test_html PASSED                           [  2%]
tests/test_handcoded_index.py::test_files PASSED                         [  5%]
tests/test_handcoded_index.py::test_images PASSED                        [  8%]
tests/test_handcoded_index.py::test_links PASSED                         [ 11%]
tests/test_handcoded_index.py::test_likes PASSED                         [ 14%]
tests/test_handcoded_index.py::test_comments PASSED                      [ 17%]
tests/test_handcoded_index.py::test_timestamps PASSED                    [ 20%]
tests/test_handcoded_user.py::test_images PASSED                         [ 22%]
tests/test_handcoded_user.py::test_links PASSED                          [ 25%]
tests/test_handcoded_user.py::test_likes PASSED                          [ 28%]
tests/test_handcoded_user.py::test_comments PASSED                       [ 31%]
tests/test_handcoded_user.py::test_timestamps PASSED                     [ 34%]
tests/test_handcoded_user.py::test_user_info PASSED                      [ 37%]
tests/test_insta485generator_hello.py::test_hello PASSED                 [ 40%]
tests/test_insta485generator_hello_css.py::test_hello_css PASSED         [ 42%]
tests/test_style.py::test_pycodestyle PASSED                             [ 45%]
tests/test_style.py::test_pydocstyle PASSED                              [ 48%]
tests/test_style.py::test_pylint PASSED                                  [ 51%]
tests/test_template_followers.py::test_files PASSED                      [ 54%]
tests/test_template_followers.py::test_awdeorio_followers PASSED         [ 57%]
tests/test_template_followers.py::test_michjc_followers PASSED           [ 60%]
tests/test_template_followers.py::test_jag_followers PASSED              [ 62%]
tests/test_template_followers.py::test_jflinn_followers PASSED           [ 65%]
tests/test_template_following.py::test_files PASSED                      [ 68%]
tests/test_template_following.py::test_awdeorio_following PASSED         [ 71%]
tests/test_template_following.py::test_michjc_following PASSED           [ 74%]
tests/test_template_following.py::test_jag_following PASSED              [ 77%]
tests/test_template_following.py::test_jflinn_following PASSED           [ 80%]
tests/test_template_html.py::test_html PASSED                            [ 82%]
tests/test_template_index.py::test_files PASSED                          [ 85%]
tests/test_template_index.py::test_images PASSED                         [ 88%]
tests/test_template_index.py::test_links PASSED                          [ 91%]
tests/test_template_index.py::test_likes PASSED                          [ 94%]
tests/test_template_index.py::test_timestamps PASSED                     [ 97%]
tests/test_template_index.py::test_comments PASSED                       [100%]

============================= 35 passed in 18.78s ==============================

Running pytest with the additional -s flag passes output from the program (stdout) to the terminal, e.g., print() statements. Notice when we add -s, the output from Pylint (“Your code has been rated …”) appears.

$ pytest -vs tests/test_style.py::test_pylint
::test_pylint
--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)

PASSED

Stop after the first failure.

$ pytest -x

Debugging options

Rerun only the tests that failed last time

$ pytest --last-failed

Automatically start a debugger when a failure occurs.

$ pytest --pdb

Deprecation warnings

Sometimes, third party libraries trigger warnings that you have no control over. Here’s an example from Project 3. Certain library and Python versions cause this error.

$ pwd
/Users/awdeorio/src/eecs485/p3-insta485-clientside
$ python --version
Python 3.7.2
$ python -c 'import jinja2; print(jinja2.__version__)'
2.10
$ pytest tests/test_index.py::TestIndex::test_feed_load
============================= test session starts ==============================
tests/test_index.py::TestIndex::test_feed_load                     PASSED [100%]
=============================== warnings summary ===============================
...
/Users/awdeorio/src/eecs485/p3-insta485-clientside/env/lib/python3.7/site-packages/jinja2/runtime.py:318: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Mapping
...
===================== 1 passed, 5 warnings in 2.99 seconds =====================

Configure pytest with a pytest.ini file. We’ll configure it to ignore this warning. Here is a sample pytest.ini.

[pytest]
filterwarnings = ignore::DeprecationWarning

Run pytest again.

$ pwd
/Users/awdeorio/src/eecs485/p3-insta485-clientside
$ cat pytest.ini
[pytest]
filterwarnings = ignore:Using or importing the ABCs.*:DeprecationWarning
$ pytest tests/test_index.py::TestIndex::test_feed_load
...
=========================== 1 passed in 3.06 seconds ===========================

Further reading about pytest warnings can be found at https://docs.pytest.org/en/latest/warnings.html.