Pyjo.Test - Writing test scripts¶
from Pyjo.Test import *
import os
if not os.getenv('TEST_ONLINE'):
plan(skip_all='set TEST_ONLINE to enable this test')
ok(got == expected, 'test name')
is_ok(got, expected, 'test name')
isnt_ok(got, unexpected, 'test name')
like_ok(got, r'expected', 'test name')
unlike_ok(got, r'unexpected', 'test name')
cmp_ok(got, '==', expected, 'test name')
is_deeply_ok(got_complex_structure, expected_complex_structure, 'test name')
isa_ok(obj, cls)
throws_ok(lambda: int('x'), ValueError, 'test name')
that(got).is_(expected, 'test name').unlike(r'unexpected', 'test name')
with subtest('Parse'):
is_ok(url.host, 'example.com', 'right host')
@subtest('Format')
def _():
is_ok(str(url), 'http://example.com', 'right format')
diag("here's what went wrong")
note('what happens next')
done_testing()
Pyjo.Test is a module for writing test scripts which print their results
in the Test Anything Protocol (TAP). It is
inspired by Perl’s Test::More, which the tests of Mojolicious are written
with, but it is not a port of it.
Each test script is a separate Python program, usually placed in the t
directory.
Running tests¶
The test scripts run with Pyjo.Prove:
$ python -m Pyjo.Prove -I . -r t
or with Perl’s prove, as they print standard TAP:
$ PYTHONPATH=. prove --ext=py --exec=python -r t
pytest runs them too, each script as a test of pytest, through
Pyjo.Test.pytest_plugin.
A single script runs on its own as well:
$ PYTHONPATH=. python t/Pyjo/URL.py
Plans¶
A test script declares how many tests it runs, either up front with
plan() or at the end with done_testing(). A script that runs tests
without a plan fails.
plan() is a context manager too, which issues the plan at the end of its
block, as done_testing() does, unless the block raises an exception:
with plan():
ok(True, 'first')
ok(True, 'second')
Subtests¶
A subtest groups tests under a name. Its tests are numbered on their own and
printed indented, and the subtest as a whole counts as one test of its parent,
which passes when all of its tests pass. subtest() works as a context
manager, as a decorator which runs the function at once, and as a function
which runs a callback:
with subtest('Simple'):
ok(True, 'first')
@subtest('Decorated')
def _():
ok(True, 'second')
subtest('Callback', lambda: ok(True, 'third'))
A subtest has an implicit done_testing(). plan(skip_all=...) skips
the rest of the subtest, and a subtest that runs no tests fails. A subtest of
a remote host runs again after a failure with reruns of subtest().
Chains¶
The functions which test a value return a Result, which is true
when the test passed, so that if not is_ok(...): works, and whose
methods test the same value again, with the names of Perl’s Test::More,
so that the tests of a value chain as the assertions of
Pyjo.Test.Pyjo do:
is_ok(url.host, 'example.com', 'right host').like(r'\.com$', 'top-level domain')
that() makes one of a value without a test, so that a chain starts
with the first of its methods:
(
that(answer)
.isa(int)
.cmp('>', 0, 'positive')
.is_(42, 'meaning of life')
)
Each method is a test of its own, which runs also after a test of the chain failed, and a failure shows the line of the method.
Extensions¶
A failed test shows the line of the first frame outside of Pyjo.Test
and of the modules that leave their frames out of the traceback of pytest
with __tracebackhide__, so that the assertions of a module of their own,
as a subclass of Pyjo.Test.Pyjo, show the line of the test that calls
them:
# MyTest.py
import Pyjo.Test
__tracebackhide__ = Pyjo.Test.__tracebackhide__
def host_is(url, host, desc=None):
return Pyjo.Test.is_ok(url.host, host, desc or f'host is {host}')
Exit status¶
The exit status of a test script is 0 when every test passed, the number of
failed tests (up to 254), 254 when tests ran without a plan, and 255
when the number of tests does not match the plan, when the script died with an
exception, or after bail_out().
Exceptions¶
After raise_failures(), the functions do not print TAP, so that they
work in the tests of another test runner, as pytest or unittest, where
Pyjo.Test.pytest_plugin calls it. A failed test raises
AssertionError with the name of the test and the diagnostic message,
plan(skip_all=...) raises unittest.SkipTest, which both runners
report as a skipped test, or in a subtest leaves the subtest, and
bail_out() raises BailOut. A subtest is a block, the plans are
not checked, skip() does nothing, and diag() and note()
print their messages, which the runner captures.
A module of tests of unittest.TestCase calls it in setUpModule,
with unittest.enterModuleContext(), which restores the mode that it
replaced at the end of the module, as the one of pytest, which runs such a
module too:
import unittest
import Pyjo.Test
import Pyjo.URL
def setUpModule():
unittest.enterModuleContext(Pyjo.Test.raise_failures())
class TestURL(unittest.TestCase):
def test_host(self):
url = Pyjo.URL.new('HtTp://Example.Com')
Pyjo.Test.is_ok(url.host, 'Example.Com', 'right host')
Without it, a failed test prints TAP, which unittest does not read, so it
reports the test as passed, and the process ends with the exit status of a
test script without a plan. unittest has no bail out: BailOut is an
error of its test, and the other tests run.
Functions¶
- class Pyjo.Test.Result(got: object, passed: bool)[source]¶
The result of a test of a value, returned by
ok(),is_ok()and the other functions which test one, or of no test, returned bythat(). Its methods test the same value again, and it is true when all the tests of the chain passed:isa_ok(answer, str).is_('42', 'meaning of life') or diag('the answer is lost')
- cmp(operator: str, expected: Any, test_name: str | None = None) Self[source]¶
ok(count, 'some').cmp('<', 10, 'small enough')
The same as
cmp_ok().
- got: Any¶
got = result.got
The value that the tests check: the first argument of the function, or the exception that the callback of
throws_ok()raised, without its traceback.
- in_(elem: object, test_name: str | None = None) Self[source]¶
isa_ok(names, list).in_('foo', "there's a foo")
The same as
in_ok().
- is_(expected: object, test_name: str | None = None) Self[source]¶
isa_ok(answer, int).is_(42, 'meaning of life')
The same as
is_ok().
- is_deeply(expected: object, test_name: str | None = None) Self[source]¶
isa_ok(got, dict).is_deeply({'a': [1, 2]}, 'deep structure')
The same as
is_deeply_ok().
- isa(cls: type | tuple[type, ...], test_name: str | None = None) Self[source]¶
throws_ok(lambda: {}['x'], KeyError).isa(LookupError)
The same as
isa_ok().
- isnt(expected: object, test_name: str | None = None) Self[source]¶
isa_ok(foo, str).isnt('', 'got some foo')
The same as
isnt_ok().
- like(expected: _Pattern, test_name: str | None = None) Self[source]¶
isa_ok(got, str).like(r'expected', 'test name')
The same as
like_ok().
- none(test_name: str | None = None) Self[source]¶
isnt_ok(req.headers.transfer_encoding, 'chunked').none('no "Transfer-Encoding" value')
The same as
none_ok().
- not_in(elem: object, test_name: str | None = None) Self[source]¶
in_ok(names, 'foo').not_in('bar', 'no bar')
The same as
not_in_ok().
- ok(test_name: str | None = None) Self[source]¶
isa_ok(items, list).ok('not empty')
The same as
ok()of the value.
- passed¶
boolean = result.passed
True if all the tests of the chain passed.
- unlike(expected: _Pattern, test_name: str | None = None) Self[source]¶
like_ok(got, r'expected').unlike(r'unexpected', 'test name')
The same as
unlike_ok().
- class Pyjo.Test.RaiseFailures(previous: bool)[source]¶
The mode of
raise_failures(), which it sets at once. It is a context manager: at the end of its block, the functions raise exceptions or print TAP as they did beforeraise_failures(), also whereunittest.enterModuleContext()orunittest.TestCase.enterClassContext()ends it.
- class Pyjo.Test.Subtest(name: str, reruns: int = 0, reruns_delay: float = 0)[source]¶
A subtest, returned by
subtest()without a callback. It is a context manager and a decorator; after it ends,passedtells whether all of its tests passed.
- class Pyjo.Test.BailOut[source]¶
Exception raised by
bail_out()afterraise_failures(), with the reason.
- Pyjo.Test.bail_out(reason: str = '') NoReturn[source]¶
if not connected: bail_out('No database connection')
Stop all testing at once, here and in the test harness, which shows the reason. The exit status is
255.
- Pyjo.Test.cmp_ok(got: Any, operator: str, expected: Any, test_name: str | None = None) Result[source]¶
# ok(got == expected) cmp_ok(got, '==', expected, 'this == that')
Compare two values with a binary operator:
==,!=,<,<=,>,>=,is,is not,inornot in. The test passes if the comparison is true.
- Pyjo.Test.diag(*messages: object) bool[source]¶
if not in_ok(users, 'foo', "There's a foo user"): diag("Since there's no foo, check that /etc/bar is set up right")
Print a diagnostic message to the standard error, which does not interfere with the test output. Returns
False, so a failure is preserved.
- Pyjo.Test.done_testing(how_many: int | None = None) bool[source]¶
ok(True) done_testing() ok(True) done_testing(1)
Issue the plan when the tests are done, instead of declaring it up front with
plan(). The number of tests is optional; when it is given, it has to match the number of tests run.
- Pyjo.Test.fail_ok(test_name: str | None = None) bool[source]¶
fail_ok('This should not happen')
A test that always fails, the same as
ok(False).
- Pyjo.Test.in_ok(got: Any, elem: object, test_name: str | None = None) Result[source]¶
in_ok(['foo', 'bar'], 'foo', 'foo is in the list')
Check if the element is in the container. The same as
ok(elem in got), with a better diagnostic message.
- Pyjo.Test.is_deeply_ok(got: object, expected: object, test_name: str | None = None) Result[source]¶
is_deeply_ok([[1, 2], [3, [4, 5]]], [[1, 2], [3]], 'deep structure')
Compare two data structures, walking through lists, tuples, sets and dicts. If they differ, the diagnostic message shows where they start differing.
- Pyjo.Test.is_ok(got: object, expected: object, test_name: str | None = None) Result[source]¶
is_ok(ultimate_answer(), 42, 'meaning of life')
Compare two values with
==.
- Pyjo.Test.isa_ok(got: object, cls: type | tuple[type, ...], test_name: str | None = None) Result[source]¶
obj = SomeClass() isa_ok(obj, SomeClass) isa_ok(123, (int, float))
Check if the object is an instance of the class, or of one of the classes.
- Pyjo.Test.isnt_ok(got: object, expected: object, test_name: str | None = None) Result[source]¶
isnt_ok(foo, '', 'got some foo')
Compare two values with
!=.
- Pyjo.Test.like_ok(got: Any, expected: _Pattern, test_name: str | None = None) Result[source]¶
like_ok(got, r'expected', 'test name') like_ok(got, (r'expected', 'i'), 'test name') like_ok(got, re.compile(r'expected'), 'test name')
Check if the value matches the regular expression. The expression can be compiled already, or given with flags as a tuple:
a,i,m,sandx.
- Pyjo.Test.none_ok(got: object, test_name: str | None = None) Result[source]¶
none_ok(req.headers.transfer_encoding, 'no "Transfer-Encoding" value')
Check if the value is
None.
- Pyjo.Test.not_in_ok(got: Any, elem: object, test_name: str | None = None) Result[source]¶
not_in_ok(os.environ, 'DISPLAY', 'DISPLAY is not set')
The opposite of
in_ok(): check if the element is not in the container.
- Pyjo.Test.note(*messages: object) None[source]¶
note(f'Tempfile is {tempfile}')
Print a message to the standard output as a TAP comment, which the test harness hides unless asked to show it.
- Pyjo.Test.ok(check: object, test_name: str | None = None) Result[source]¶
ok(3**4 == 81, 'simple exponential') ok(isinstance(film, Film), 'set_db()') ok(None not in items, 'all items defined')
The basic test: it passes if the value is true and fails otherwise. Returns whether the test passed.
- Pyjo.Test.pass_ok(test_name: str | None = None) bool[source]¶
pass_ok('step 1')
A test that always passes, the same as
ok(True).
- Pyjo.Test.plan(tests: int | None = None, skip_all: str | None = None) Plan[source]¶
plan(tests=3) if not os.getenv('TEST_ONLINE'): plan(skip_all='set TEST_ONLINE to enable this test') with plan(): ok(True) with plan(tests=1): ok(True)
Declare how many tests the script is going to run, which protects against a premature exit, or skip all of them with a reason. In a subtest,
skip_allskips the rest of the subtest.It returns a
Plan, a context manager which callsdone_testing()at the end of its block, so that the number of tests can be left out.
- Pyjo.Test.plan_skip_all(why: str) Plan[source]¶
plan_skip_all('set TEST_ONLINE to enable this test')
The same as
plan(skip_all=why).
- Pyjo.Test.raise_failures(enable: bool = True) RaiseFailures[source]¶
raise_failures() raise_failures(False) with raise_failures(): is_ok(1, 1, 'same')
Make the functions raise exceptions instead of printing TAP, as the section Exceptions describes, or print TAP again with
False. It returns aRaiseFailures, whose block ends in the mode that it replaced.
- Pyjo.Test.skip(why: str | None = None, how_many: int = 1) bool[source]¶
if os.getenv('TEST_ONLINE'): ok(Pyjo.UserAgent.new().get('https://mojolicious.org').result.is_success, 'online request') else: skip('set TEST_ONLINE to enable this test', 1)
Skip the given number of tests, which still count for the plan.
- Pyjo.Test.subtest(name: str, fn: None = None, *, reruns: int = 0, reruns_delay: float = 0) Subtest[source]¶
- Pyjo.Test.subtest(name: str, fn: Callable[[], object], *, reruns: int = 0, reruns_delay: float = 0) bool
with subtest('Parse'): ok(True) @subtest('Parse') def _(): ok(True) passed = subtest('Parse', lambda: ok(True))
Run a group of tests as a single test of the parent. Without a callback it returns a
Subtest, which works as a context manager and as a decorator; with a callback it runs it at once and returns whether the subtest passed.A subtest that fails or raises an exception runs again, up to
rerunstimes, afterreruns_delayseconds, as a test of a remote host does after a failure of the network:@subtest('Remote host', reruns=2, reruns_delay=10) def _(): ok(Pyjo.UserAgent.new().get('https://mojolicious.org').result.is_success, 'online request')
Only its last run counts, for the parent and its plan; the runs before it are diagnostic messages. Such a subtest is a decorator or has a callback, as the block of a
withstatement runs only once.
- Pyjo.Test.that(got: object) Result[source]¶
that(answer).isa(int).is_(42, 'meaning of life')
Make a
Resultof the value without a test, which is true, so that a chain of tests of the value starts with the first of its methods.
- Pyjo.Test.throws_ok(cb: Callable[[], object], expected: type[BaseException] | str | Pattern[str], test_name: str | None = None) Result[source]¶
throws_ok(lambda: [1, 2, 3][3], IndexError, 'no more elements') throws_ok(lambda: int('x'), "invalid literal for int() with base 10: 'x'", 'right message') throws_ok(lambda: int('x'), re.compile(r'invalid literal'), 'right message')
Check if the callback raises an exception: of the given class, with the given message, or with a message matching the regular expression.