Test guidelines

Testing is based on sqa-baseline criteria, tox automation is used to simplify the testing process.

To install it with pip use:

$ pip install tox

To run unit and functional tests together with reports use:

$ tox
...
clean: commands succeeded
stylecheck: commands succeeded
bandit: commands succeeded
docs: commands succeeded
py36: commands succeeded
report: commands succeeded
...

The last coverage report output is produced at htmlcov which can be displayed in html format accessing to index.html.

The last Pep8 report produced by flake8 at the output file flake8.log.

However, you can also run different test configurations using different tox environments:

Code Style [QC.Sty]

Test pep8 maintenance style conventions based on pylint format. To run stylecheck use:

$ tox -e stylecheck
...
stylecheck: commands succeeded
...

Unit Testing [QC.Uni]

All unit tests are placed inside the package (./o3skim/test). This helps to test easily functions at low level and ensure the functions have the expected behavior. To run unit tests use:

$ tox -e unittesting
...
unittesting: commands succeeded
...

This environment also provide a coverage term report for the tests. The design of Unit Tests is based on the python unittest framework, a simple and extended test framework which ships by default together with python.

The usage is very simple and straight forward for simple tests, but the difficulty to parametrize and combine multiple test fixtures makes it not suitable for Black-Box testing without a very complex customization.

Functional Testing [QC.Fun]

Located inside tests package folder (./tests). Functional testing is used to test the system from a general overview of the application. To run functional tests use:

$ tox -e functional
...
functional: commands succeeded
...

This environment also provide a coverage term report for the tests. The framework used is pytest to provide a simple syntax to test all possible combinations from the user point of view.

Pytest detects directly all tests following the test_discovery naming conventions. Therefore all functional tests should be located on the tests folder at the package root and start with test. For example test_sources.py.

More than 500 test combinations are generated using which otherwise might not be feasible using other python test frameworks.

Security [QC.Sec]

Security checks are performed by bandit, a tool designed to find common security issues in Python code. To run security checks use:

$ tox -e functional
...
functional: commands succeeded
...

Documentation [QC.Doc]

Documentation is build using sphinx, a tool designed to create documentation based on code. To run documentation build checks use:

$ tox -e docs
...
docs: commands succeeded
...

The HTML pages are build inside in docs/_build.