p3-insta485-clientside

End-to-End Testing

To test an entire website that uses client-side dynamic pages, we need mimic a web browser. We will use Google Chrome in headless mode, control it using Chrome Driver and the Selenium library.

If you’re running WSL, first take a look at the work arounds for WSL and Chromedriver.

Headless browser

Google Chrome will run our JavaScript in the background while a unit test runs. Chrome supports “headless” mode, where everything is hidden from the screen.

Make sure that you have Chrome version >59 by pasting chrome://settings/help into the search bar.

Google Chrome on Windows Subsystem for Linux

On WSL, even if you have the Chrome browser installed through Windows, you still need to install Chrome via apt-get.

$ wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | sudo apt-key add -
$ echo 'deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main' | sudo tee /etc/apt/sources.list.d/google-chrome.list
$ sudo apt-get update
$ sudo apt-get install google-chrome-stable

To open Linux GUI applications invoked through WSL, you’ll also need a Windows X server. Install and run Xming. Once it’s running, you will see in your system tray.

Xming system tray icon

Next, Edit your WSL .bashrc file and add the line export DISPLAY=:0. This environment variable contains the address of the X server display.

$ echo "export DISPLAY=:0" >> ~/.bashrc

Restart WSL. If you reboot your computer later, don’t forget that you need make sure Xming is running.

Selenium

Selenium allows us to automate finding elements in the DOM and navigating the web page under test (clicking on buttons, scrolling, etc.).

Make sure your virtual environment is activated. Activate it if necessary with source env/bin/activate.

$ echo $VIRTUAL_ENV
/Users/awdeorio/src/eecs485/p3-insta485-clientside/env
$ which python
/Users/awdeorio/src/eecs485/p3-insta485-clientside/env/bin/python

Make sure Selenium is installed. In EECS 485, this is just a sanity check, Selenium is included as a dependency in the provided setup.py. If you’re working on a side project, you might need to do pip install selenium.

$ pip show selenium
Name: selenium
Version: 3.141.0
...

Chromedriver

Chromedriver is a separate executable from the Google Chrome browser. Chromedriver is used by Selenium to control the Chrome browser.

macOS: Determine your Chrome version and set two environment variables.

$ /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --version
Google Chrome 79.0.3945.130
$ CHROMEDRIVER_ARCH=mac64
$ CHROME_MAJOR_VERSION=79  # FIXME change this to match your major version

Linux/WSL: Determine your Chrome version and set two environment variables.

$ google-chrome --version
Google Chrome 79.0.3945.130
$ CHROMEDRIVER_ARCH=linux64
$ CHROME_MAJOR_VERSION=79  # FIXME change this to match your major version

Download the version of Chromedriver that matches your Chrome install major version, 79 in this example.

$ pwd
/Users/awdeorio/src/eecs485/p3-insta485-clientside
$ echo $CHROMEDRIVER_ARCH  # Should match your OS
mac64
$ echo $CHROME_MAJOR_VERSION  # Should match your Chrome major version
79
$ CHROMEDRIVER_VERSION=`curl https://chromedriver.storage.googleapis.com/LATEST_RELEASE_${CHROME_MAJOR_VERSION}`
$ echo $CHROMEDRIVER_VERSION
79.0.3945.36
$ wget https://chromedriver.storage.googleapis.com/${CHROMEDRIVER_VERSION}/chromedriver_${CHROMEDRIVER_ARCH}.zip
...
chromedriver_mac64.zip saved

Unzip the file and move it into your Python virtual environment:

$ unzip chromedriver_${CHROMEDRIVER_ARCH}.zip
$ mv chromedriver env/bin/
$ rm chromedriver_*.zip

By moving the chromedriver into env/bin/ we have added it to our executable PATH whenever the virtual environment is active. We can see this if we enter our virtualenv:

$ pwd
/Users/awdeorio/src/eecs485/p3-insta485-clientside
$ echo $VIRTUAL_ENV
/Users/awdeorio/src/eecs485/p3-insta485-clientside/env
$ which chromedriver
/Users/awdeorio/src/eecs485/p3-insta485-clientside/env/bin/chromedriver

Double check that Chrome driver works. Your version may be different.

$ chromedriver --version
ChromeDriver 79.0.3945.36 (3582db32b33893869b8c1339e8f4d9ed1816f143-refs/branch-heads/3945@{#614})

Run a test

Run a simple test which performs a Google search for “hello world”.

$ wget https://eecs485staff.github.io/p3-insta485-clientside/test_selenium_hello.py
$ python3 test_selenium_hello.py
"Hello, World!" program - Wikipedia
HelloWorld - Digital promotions & loyalty programs for the ...
Hello, World! - Learn Python - Free Interactive Python Tutorial
The Hello World Collection
Hello World · GitHub Guides
The History of 'Hello, World' - HackerRank Blog
Hello World - Raspberry Pi
Computer Programming/Hello world - Wikibooks, open books ...

Next, run sanity test loads your HTML and JavaScript.

$ pytest -v --log-cli-level=INFO tests/test_index.py::test_anything
...
INFO     autograder:conftest.py:77 Setup test fixture 'app'
INFO     autograder:conftest.py:130 Setup test fixture 'base_driver'
INFO     autograder:conftest.py:160 IMPLICIT_WAIT_TIME=10
INFO     autograder:conftest.py:192 Setup test fixture 'driver'
INFO     werkzeug:_internal.py:122  * Running on http://localhost:50504/ (Press CTRL+C to quit)
INFO     werkzeug:_internal.py:122 127.0.0.1 - - [22/Jan/2020 08:25:52] "GET / HTTP/1.1" 302 -
INFO     werkzeug:_internal.py:122 127.0.0.1 - - [22/Jan/2020 08:25:52] "GET /accounts/login/ HTTP/1.1" 200 -
INFO     werkzeug:_internal.py:122 127.0.0.1 - - [22/Jan/2020 08:25:52] "GET / HTTP/1.1" 302 -
INFO     werkzeug:_internal.py:122 127.0.0.1 - - [22/Jan/2020 08:25:52] "GET /accounts/login/ HTTP/1.1" 200 -
INFO     werkzeug:_internal.py:122 127.0.0.1 - - [22/Jan/2020 08:25:52] "GET /static/css/style.css HTTP/1.1" 200 -
INFO     werkzeug:_internal.py:122 127.0.0.1 - - [22/Jan/2020 08:25:52] "GET /static/images/logo.png HTTP/1.1" 200 -
PASSED

WSL and Chromedriver

On WSL 1, frontend tests will sometimes fail with an error related to connecting to Chromedriver. This will be fixed in the forthcoming WSL 2.

selenium.common.exceptions.WebDriverException: Message: chrome not reachable
         (Session info: headless chrome=77.0.3865.90)

Workaround 1: CAEN Linux

The first work around is to run the tests that use Selenium on CAEN Linux. Installing your code on another machine is easy once you’ve created an install script.

$ ssh <uniquename>@login.engin.umich.edu
$ git clone <your git url>
$ cd p3-insta485-clientside
$ ./bin/insta485install
$ source env/bin/activate
$ pytest -v --log-cli-level=INFO tests/test_index.py::test_anything

If you run into a permission error when running ./bin/insta485install script, make sure you have made it executable using chmod +x ./bin/insta485install. Then delete the env/, node_modules/, insta485.egg-info(if it exists), insta485/statics/js/bundle.js and reinstall your solution ./bin/insta485install, reactivate the environment and run the test again.

Workaround 2: WSL 2

Selenium and headless Chrome work correctly on WSL 2. At the time of this writing, WSL 2 is part of the Windows Insider Program, which provides access to beta releases of Windows software. WSL 2 Install Instructions.

Understanding Selenium tests

Let’s take a deeper look at the previous example, test_selenium_hello.py.

First, add a breakpoint to the top of the test_selenium_hello() function in test_selenium_hello.py.

def test_selenium_hello():
    """Perform a Google search using Selenium and a headless Chrome browser."""
    import pdb; pdb.set_trace()
...

Pro-tip: Make sure you have pdbpp installed first! pip install pdbpp

Pro-tip: you can use breakpoint() instead of import pdb; pdb.set_trace() in Python 3.7+.

Next, comment out the line about headless chrome.

    options = selenium.webdriver.chrome.options.Options()
    # options.add_argument("headless")  # Comment out this line
    driver = selenium.webdriver.Chrome(options=options)

Run the test again and step through it one line at a time (n for next at the PDB/PDB++ prompt).

$ python3 test_selenium_hello.py
(Pdb++) n
(Pdb++) n
...

A Google Chrome window opens with a message about being controlled by automated test software.

The Selenium driver enters text into the search box.

Search results appear on the page.

Summary

Selenium automatically “drives” the browser, usually without showing a window (“headless” mode). For debugging, it’s useful to step through the test without the headless option, watching the results in the browser window.