Pytest with Python

Pytest provides a structured way for creating test cases with Python, using pytest we can pinpoint which functionalities are failing by creating a test for each functionality. We can also selectively run a test or fail test, run tests in parallel, and perform many more tasks.

The added advantage is, that we can install packages that can enhance pytest test cases and results, these packages are developed using hooks and fixtures of pytest. With pytest, you can arrange tests into different categories like sanity, regression, or other.

This is a long article, that will cover end to end of pytest

Make sure a test is not dependent on other test, complete isolation is required for better testing

Installation:

pip install pytest

Test Discovery with pytest:

Tests are like functions/methods but just vary in the name, and where the function is present, Below are a few points to remember while creating tests. (The underscores are optional, I added them for readability purposes)

• The function name must start with the word test, ex : test_sum
• The class name must start with the word Test, ex: TestMath
• It is not mandatory to put a test case inside a class
• The filing must start with the word test, ex: test+mathematics.py

What happens when the test is not prefixed to a function?

If a function is not prefixed with a test then that function will be considered as a python function and it will not be executed by pytest.

Running Pytest:

Create a file called test_sample.py and create the below piece of code in that.

def test_say_hello():
    print("Hello")

def welcome():
    print("welcome")

Execute the following command in the terminal:

pytest -vvv #-vvv for detailed log

# output

=================== test session starts ===========
collected 1 item                                                                                                                                    

test_sample.py::test_say_hello PASSED       [100%]

=================== 1 passed in 0.01s ==============

If you notice the above output, you can see only the function with the “test” prefix is executed not the other function. You do not see any output printed, we will learn out to print the output shortly.

Ordering/Sequencing test

You can order your test, ordering a test is nothing but setting the sequence of execution with pytest.

For this use a package called pytest-order, install pip install pytest-order

import pytest

@pytest.mark.math
class TestMath:

    @pytest.mark.order(2)
    def test_a(self):
        assert False

    @pytest.mark.order(1)
    def test_b(self):
        pass

Assertions in pytest:

In the above function, we have created just a print statement as part of the function. while creating test cases for functionalities we will be checking whether something matches our criteria or not if it does not match our criteria then we will fail the test case.

We use assertion to check whether our criteria are met. We can use the assert keyword from Python for checking.

Add the below code into the test file and run pytest on the terminal, both tests will fail but one with the default reason and the other one with user mentioned reason. So with assert, we can provide the reason why a particular test failed.

def test_sum():
    a = 2
    b = 4
    c = a + b
    assert c == 3

def test_multiplication():
    a = 2
    b = 4
    c = a * b
    assert c == 100, "The multiplication result is not 100"

Report:

So far we are seeing the result in our output screen, but in actuality this result might need to be shared with the team lead or with other peers, So in such cases, the console output is not enough.

We can generate reports in multiple ways 1. using (double – ) —junit-xml other with HTML result. Junit reports are widely accepted formats for generating further better reports, useful if you are using Jenkins.

Run the below command in the terminal to get junit xml type result.

pytest --junitxml junitresult.xml
# junitresult.xml is the filename, where the result should be stored

HTML report

For generating an HTML report we need to install a package, pip install pytest-html

Run the below command in the terminal, in the output you can notice the report path “Generated html report: file:///path/report.html”

pytest --html=report.html

# output

------ Generated html report: file:///path/pytest_examples/report.html ------
===== short test summary info =====
FAILED test_sample.py::test_sum - assert 6 == 3
FAILED test_sample.py::test_multiplication - AssertionError: The multiplication result is not 100
===== 2 failed in 0.02s =====

Collect only or dry run

If you want to check which tests will be executed without running actual tests, we can use the --collect-only parameter with pytest. This will be useful when you want to see why your test is not picked by the pytest.

pytest --collect-only
====== test session starts ========
collected 2 items                                                                                                                                   

<Module test_sample.py>
  <Function test_sum>
  <Function test_multiplication>

====== 2 tests collected in 0.00s =

Printing to screen with Pytest:

Normally when we execute the pytest, the print statements will not be part of our output. But if you using pytest for just learning or debugging then you might want the print statement to be part of the output.

We have to provide an argument -s along with pytest to print to the screen

pytest -s


test_sample.py result sum is  6
Fresult multi is  8
F

Execute a specific test, class, file

Let’s rewrite the code to have a class but keep the test in the same file

class TestMath:

    def test_sum(self):
        a, b = 2, 4
        c = a + b
        print(f"result sum is  {c}")
        assert c == 3

    def test_multiplication(self):
        a, b = 2, 4
        c = a * b
        print(f"result multi is  {c}")
        assert c == 100, "The multiplication result is not 100"

We can use the -k argument to run a specific test or a class or file.

pytest -k test_sum

# output
collected 2 items / 1 deselected / 1 selected  

# pytest -k "test_sum or test_multiplication"

#running class
pytest -k TestMath

# output
collected 2 items

#running file
pytest -k test_sample.py

# output
collected 2 items

You can use -k to avoid certain functions, classes, and files.

pytest -k "not test_sum". # test
pytest -k "not TestMath". # class
pytest -k "not test_sample.py"   # file

You can make combinations as well using and, not

pytest -k "test_multiplication and not test_sum"

Categorization of Tests, Tags or Markers

When you create a test framework, you need to segregate tests according to their priority and functionality.

• To categorize tests we will be using markers (also called tags) @pytest.mark.marker_name,
• Each test can be categorized
• Each Test class can be categorized
• Multiple markers can be associated with tests and class

import pytest

@pytest.mark.math
class TestMath:

    @pytest.mark.sum
    def test_sum(self):
        a, b = 2, 4
        c = a + b
        print(f"result sum is  {c}")
        assert c == 3

    @pytest.mark.multi
    @pytest.mark.xyz
    def test_multiplication(self):
        a, b = 2, 4
        c = a * b
        print(f"result multi is  {c}")
        assert c == 100, "The multiplication result is not 100"

Running Tags

We can use -m argument followed by a marker name to execute specific markers

pytest -m 'sum'

# output
collected 2 items / 1 deselected / 1 selected 

Parallels running with pytest

To save time we can run or test in parallel mode, this will save our time. One thing to remember, make sure your tests are not dependent on each other. The number of parallel threads depends on your system hardware configuration.

Install python package : pip install pytest-xdist

Then run the tests using -n argument followed by the number of parallels, if you are not sure whether your computer can handle a given number of parallels then use auto as value, auto will make sure, it opens as many as parallels your machine can handle

pytest -n 2              

# output
2 workers [2 items] # items here are tests


pytest -n auto

# output
8 workers [2 items] 

Repeat with counts

When I work on test creation, I try to make sure the test runs fine before I push it into the repository. Personally, I execute at least 5 times to see if it can pass all 5 times. But doing this manually is kind of boring.

We can use a package called pytest-repeat, install package : pip install pytest-repeat

We have to pass an argument –count followed by a number, the number denoting how many times this test has to run.

pytest --count 10

# there is 2 test, so total tests are 2 *10 = 20

Timeout

Some tests may take time but if you want to fail/limit test if it takes more time, then we can use a package pytest-timeout, install pip install pytest-timeout.

pytest --timeout=5

If you want to assign for each test case then you add a marker

    @pytest.mark.timeout(7)
    def test_multiplication(self):
        sleep(10)
        a, b = 2, 4
        c = a * b
        print(f"result multi is  {c}")
        assert c == 100, "The multiplication result is not 100"

Output

Dependency

You can write your test case to depend on other test cases, use the package pytest-depends, install pip install pytest-depends

I personally avoid writing test cases with dependency.

Note: pytest-dependency is another package same purpose, but it did not work out

import pytest

@pytest.mark.math
class TestMath:

    # @pytest.mark.xfail(reason="deliberate fail")
    def test_a(self):
        assert False

    def test_b(self):
        pass

    @pytest.mark.depends(on=["test_a"])
    def test_c(self):
        pass

    @pytest.mark.depends(on=["test_b", "test_c"])
    def test_e(self):
        pass

collected 4 items                                                                                                                                   

test_sample.py::TestMath::test_a FAILED                                                                                                       [ 25%]
test_sample.py::TestMath::test_b PASSED                                                                                                       [ 50%]
test_sample.py::TestMath::test_c SKIPPED (test_sample.py::TestMath::test_c depends on test_sample.py::TestMath::test_a)                       [ 75%]
test_sample.py::TestMath::test_e SKIPPED (test_sample.py::TestMath::test_e depends on test_sample.py::TestMath::test_c)                       [100%]

Expected Failures:

There will be times when you know some test will fail (maybe because of an existing bug) but you do not want this to be part of the reports as failed, in such cases, you can use xfail(), which makes the test become xfailed than fail if it fails

import pytest
class TestMath:

    @pytest.mark.xfail()
    def test_multiplication(self):
        a, b = 2, 4
        c = a * b
        print(f"result multi is  {c}")
        assert c == 100, "The multiplication result is not 100"

Skipping tests:

If you want to skip a test, you can use skip marker.

class TestMath:

    def test_sum(self):
        a, b = 2, 4
        c = a + b
        print(f"result sum is  {c}")
        assert c == 3

    @pytest.mark.skip("skip this test, I am the reason")
    def test_multiplication(self):
        a, b = 2, 4
        c = a * b
        print(f"result multi is  {c}")
        assert c == 100, "The multiplication result is not 100"

pytest -v

# output
collected 2 items                                                                                                                                   

test_sample.py::TestMath::test_sum FAILED                                                                                                     [ 50%]
test_sample.py::TestMath::test_multiplication SKIPPED (skip this test, I am the reason)                                                       [100%]

You can skip a test based on a criteria, for this using skipif marker

import pytest
class TestMath:

    def test_sum(self):
        a, b = 2, 4
        c = a + b
        print(f"result sum is  {c}")
        assert c == 3

    @pytest.mark.skipif(3 > 6, reason="skipping if criteria matches")
    def test_multiplication(self):
        a, b = 2, 4
        c = a * b
        print(f"result multi is  {c}")
        assert c == 100, "The multiplication result is not 100"

pytest -v

# output 
collected 2 items                                                                                                                                   

test_sample.py::TestMath::test_sum FAILED                                                                                                     [ 50%]
test_sample.py::TestMath::test_multiplication FAILED                                                                                          [100%]

Retry failed tests

You can run the failed tests again withing same execution using pytest-retry package, install pip install pytest-retry

import pytest

@pytest.mark.math
class TestMath:

    @pytest.mark.order(2)
    def test_a(self):
        assert False

    @pytest.mark.order(1)
    def test_b(self):
        pass

Run command:

pytest -v --retries 10

In the below screenshot, you can see only one retry count, but in reality, it tried 10 times (I did not include it due to the length of the trace).