p1-insta485-static

`pytest` Tutorial

pytest is a unit test utility that makes it easy to run unit tests. It is compatible with the Python unittest framework.

Quick start

TL;DR: for debugging, you probably want maximum verbose output (-vv), application stdout to pass through to pytest stdout (-s), and stop after the first error (-x).

$ pytest -vvsx

Step-by-step

This tutorial uses Project 1 as an example. It assumes you have some code and a directory called tests/.

$ pwd
/Users/awdeorio/src/eecs485/p1-insta485-static
$ ls
bin  hello      html      insta485generator           pyproject.toml
env  hello_css  insta485  insta485generator.egg-info  tests

You have installed and activated your Python virtual environment (instructions) and installed pytest (pip install pytest if needed). Your version might be different.

$ pytest --version
This is pytest version 4.1.0, imported from /Users/awdeorio/src/eecs485/p1-insta485-static/env/lib/python3.7/site-packages/pytest.py

By default, pytest will search for tests and run them all.

$ pytest
============================= test session starts ==============================
platform darwin -- Python 3.7.7, pytest-5.4.1, py-1.8.1, pluggy-0.13.1
rootdir: /Users/awdeorio/src/eecs485/p1-insta485-static
collected 35 items

tests/test_handcoded_html.py .                                           [  2%]
tests/test_handcoded_index.py ......                                     [ 20%]
tests/test_handcoded_user.py ......                                      [ 37%]
tests/test_insta485generator_hello.py .                                  [ 40%]
tests/test_insta485generator_hello_css.py .                              [ 42%]
tests/test_style.py ...                                                  [ 51%]
tests/test_template_followers.py .....                                   [ 65%]
tests/test_template_following.py .....                                   [ 80%]
tests/test_template_html.py .                                            [ 82%]
tests/test_template_index.py ......                                      [100%]

============================= 35 passed in 19.06s ==============================

Run all unit tests in one file.

$ pytest tests/test_template_followers.py

Run one unit test. You can find the exact name of a testcase by running them all with the verbose flag -v.

tests/test_template_followers.py is a file

test_awdeorio_followers is one unit test, which is one function.

$ pytest tests/test_template_followers.py::test_awdeorio_followers

Common options

Print the name of each individual unit test with -v or --verbose. Adding -vv increases the output.

$ pytest -vv
============================= test session starts ==============================
platform darwin -- Python 3.7.7, pytest-5.4.1, py-1.8.1, pluggy-0.13.1 -- /Users/awdeorio/src/eecs485/p1-insta485-static/env/bin/python3
cachedir: .pytest_cache
rootdir: /Users/awdeorio/src/eecs485/p1-insta485-static
collected 35 items

tests/test_handcoded_html.py::test_html PASSED                           [  2%]
tests/test_handcoded_index.py::test_files PASSED                         [  5%]
tests/test_handcoded_index.py::test_images PASSED                        [  8%]
tests/test_handcoded_index.py::test_links PASSED                         [ 11%]
tests/test_handcoded_index.py::test_likes PASSED                         [ 14%]
tests/test_handcoded_index.py::test_comments PASSED                      [ 17%]
tests/test_handcoded_index.py::test_timestamps PASSED                    [ 20%]
tests/test_handcoded_user.py::test_images PASSED                         [ 22%]
tests/test_handcoded_user.py::test_links PASSED                          [ 25%]
tests/test_handcoded_user.py::test_likes PASSED                          [ 28%]
tests/test_handcoded_user.py::test_comments PASSED                       [ 31%]
tests/test_handcoded_user.py::test_timestamps PASSED                     [ 34%]
tests/test_handcoded_user.py::test_user_info PASSED                      [ 37%]
tests/test_insta485generator_hello.py::test_hello PASSED                 [ 40%]
tests/test_insta485generator_hello_css.py::test_hello_css PASSED         [ 42%]
tests/test_style.py::test_pycodestyle PASSED                             [ 45%]
tests/test_style.py::test_pydocstyle PASSED                              [ 48%]
tests/test_style.py::test_pylint PASSED                                  [ 51%]
tests/test_template_followers.py::test_files PASSED                      [ 54%]
tests/test_template_followers.py::test_awdeorio_followers PASSED         [ 57%]
tests/test_template_followers.py::test_michjc_followers PASSED           [ 60%]
tests/test_template_followers.py::test_jag_followers PASSED              [ 62%]
tests/test_template_followers.py::test_jflinn_followers PASSED           [ 65%]
tests/test_template_following.py::test_files PASSED                      [ 68%]
tests/test_template_following.py::test_awdeorio_following PASSED         [ 71%]
tests/test_template_following.py::test_michjc_following PASSED           [ 74%]
tests/test_template_following.py::test_jag_following PASSED              [ 77%]
tests/test_template_following.py::test_jflinn_following PASSED           [ 80%]
tests/test_template_html.py::test_html PASSED                            [ 82%]
tests/test_template_index.py::test_files PASSED                          [ 85%]
tests/test_template_index.py::test_images PASSED                         [ 88%]
tests/test_template_index.py::test_links PASSED                          [ 91%]
tests/test_template_index.py::test_likes PASSED                          [ 94%]
tests/test_template_index.py::test_timestamps PASSED                     [ 97%]
tests/test_template_index.py::test_comments PASSED                       [100%]

============================= 35 passed in 18.78s ==============================

Running pytest with the additional -s flag passes output from the program (stdout) to the terminal, e.g., print() statements. Notice when we add -s, the output from Pylint (“Your code has been rated …”) appears.

$ pytest -vs tests/test_style.py::test_pylint
::test_pylint
--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)

PASSED

Stop after the first failure.

$ pytest -x

Debugging options

Rerun only the tests that failed last time

$ pytest --last-failed

Automatically start a debugger when a failure occurs. The -s is required for PDB+. Don’t forget to add import pdbp.

$ pytest --pdb -s

Start a debugger on the first line of the test.

$ pytest --trace -s

Start PDB+ from a breakpoint().

$ pytest -s

Debugging with VSCode

If you’re using VSCode, you can leverage its visual debugger to debug your pytest tests. Follow these steps to set it up:

Open your project in Visual Studio Code.
Ensure that you have the Python extension installed and enabled.
Set breakpoints in your test files by clicking in the gutter area (next to the line numbers) where you want the debugger to pause.
Open the Command Palette (Ctrl+Shift+P or Cmd+Shift+P) and search for "Python: Configure Tests".
Select the option to configure your tests and choose "pytest" as the testing framework.
Select the "test" directory where your python tests live.
Once configured, a "Run Test" button will appear next to your test functions and classes in the editor.
Click the "Run Test" button next to the test you want to debug.
The debugger will pause execution at the breakpoints you’ve set, allowing you to inspect variables, evaluate expressions, and step through your code. 
Use the debug toolbar at the top of the editor to control the session (e.g., step into, step over, step out, continue).

By using these tools, you can streamline debugging and gain better insights into your how your code is performing on spefic tests.

Acknowledgments

Original document written by Andrew DeOrio awdeorio@umich.edu.

This document is licensed under a Creative Commons Attribution-NonCommercial 4.0 License. You’re free to copy and share this document, but not to sell it. You may not share source code provided with this document.