Mercurial > libervia-backend
changeset 3430:4ba31dd1f0f5
doc: documentation for testing in new `contributing` section
| author | Goffi <goffi@goffi.org> |
|---|---|
| date | Fri, 27 Nov 2020 16:40:26 +0100 |
| parents | d4558f3cbf13 |
| children | 5d7569378914 |
| files | doc/conf.py doc/contribuing/index.rst doc/contribuing/testing.rst doc/index.rst |
| diffstat | 4 files changed, 230 insertions(+), 1 deletions(-) [+] |
line wrap: on
line diff
--- a/doc/conf.py Fri Nov 27 16:39:40 2020 +0100 +++ b/doc/conf.py Fri Nov 27 16:40:26 2020 +0100 @@ -22,7 +22,7 @@ # -- Project information ----------------------------------------------------- project = u'Salut à Toi' -copyright = u'2019, Salut à Toi' +copyright = u'2019,2020 Salut à Toi' author = u'Salut à Toi team' doc_dir = os.path.dirname(os.path.abspath(__file__))
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/doc/contribuing/index.rst Fri Nov 27 16:40:26 2020 +0100 @@ -0,0 +1,15 @@ +.. _contribuing: + +=========== +contribuing +=========== + +This part of the documentation is for people willing to contribute to the development of +Salut à Toi and its ecosystem, it is not needed to read this if you only plan to use the +software. + +.. toctree:: + :glob: + :maxdepth: 2 + + testing
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/doc/contribuing/testing.rst Fri Nov 27 16:40:26 2020 +0100 @@ -0,0 +1,213 @@ +======= +testing +======= + +You'll find here the documentation to run tests on Salut à Toi. If you plan to contribute +to the ecosystem, you should use them to check that your modification is not breaking +anything, and if possible you should extend them to cover any new feature. + +.. _contributing-overview: + +overview +======== + +Tests are run using `pytest`_ and are located in the ``tests`` directory. +You'll also find legacy tests in ``sat/test`` but those one are old, not maintained and +only kept there temporarily until they are ported to the new system. + +For now, emphasis is put on end-2-end tests, as they are covering the whole ecosystem, and +are quite easy to write. The downside is that they are quite long to run. + +Several `fixtures`_ are available in the various ``conftest.py`` files, it's a good idea +to have an overview of them if you're willing to write your own tests. + +.. _pytest: https://www.pytest.org +.. _fixtures: https://docs.pytest.org/en/latest/fixture.html + +end-to-end tests +================ + +End-to-end tests are located in ``tests/e2e``. They are launched in a well defined +environment managed through Docker. The ``docker/docker-compose_e2e.yml`` is used to +create the suitable containers. + +A script is available at ``tests/e2e/run_e2e.py`` to launch the tests. It will create the +containers, bind the current code to them, and set environment variables according to +arguments. + +The arguments set to this script are the ``pytest`` options, thus you can have a verbose +mode with ``-v`` and select specific test with ``-k EXPRESSION`` (see ``pytest --help`` for +details). + +In addition to pytest option, some flags can be set with the following arguments: + + +``--visual`` + Launch a VNC viewer to see in real time browser based tests. You must have ``vncviewer`` + executable available in your path (this is part of `TigerVNC`_) + +``--keep-containers`` + Do no remove Docker container after the end of tests. + +``--keep-profiles`` + Do not delete test profiles after the end of tests + +``--keep-vnc`` + Do not stop VNC viewer after the end of tests. This argument implies ``--visual``. + +``--keep-browser`` + Do not kill the browser inside the container after tests are done. This argument implies + ``--keep-container`` and ``--keep-vnc``. + +``--dev-mode`` + Shortcut for ``--keep-containers``, ``--keep-profiles`` and ``--keep-vnc``. This is + useful, as you guess with its names, for development of tests. User can then log-in into + the ``sat`` container, launch a Python console, and work with the automated browser in + real-time. Basic commands to launch a browser and log-in with test account are printed + at the end of the tests. Note that if you want to have profiles created, or extra tools + like the fake SMTP server, you'll have to launch at least one test which require them. + To log-in into the ``sat`` container, you can use the following command, from + ``/docker`` directory:: + + $ docker-compose -f docker-compose_e2e.yml exec sat /bin/bash + + Then run a python console with given instructions + +.. _TigerVNC: https://tigervnc.org + +common fixtures +--------------- + +Here are the fixture common to all e2e tests which are good to know: + +``test_profiles`` + Creates a bunch of test accounts which are available during the whole test session. + Those account are destroyed once all the tests are finished (successful or not), except + if you set the ``SAT_TEST_E2E_KEEP_PROFILES`` environment variable (or use the + ``--keep-profiles`` flag in ``run_e2e.py``. + + The profiles created are in the form ``accountX`` for account on the ``server1.test``, + or ``accountX_sY`` for account on other servers (see the docstring for details). + + This fixture should be used on top of each e2e test module. + +``pubsub_nodes`` + Create 2 pubsub nodes with ``open`` access model and named ``test`` (one on ``account1`` + PEP service, and the other one on ``pubsub.server1.test``, created with the same + account). + + Those node are created for the scope of the class. + +``fake_file`` + Create files filled with random bytes, and check them. + + A file is created by calling ``fake_file.size(size)``, and by default files of the same + size are re-used (set ``use_cache=False`` to create new files). This method returns a + ``pathlib.Path``. SHA-256 hash of the created file can be retrieved using + ``fake_file.get_source_hash(source_file_path)`` with the file path as argument. + + ``fake_file.new_dest_file()`` will return a Path to a randomly named destination file, + and ``fake_file.get_dest_hash(dest_file_path)`` will generate its hash once written. + +``sent_emails`` + When used, a fake SMTP server (already configured in container's ``sat.conf``) will be + launched if it's not already, and all message sent to it since the beginning of the test + will be available in the given list. Message are subclasses of + ``email.message.EmailMessage`` with the extra properties ``from_``, ``to``, ``subject`` + and ``body`` available for easy access to their content. + + The SMTP server is terminated at the end of the test session. + +jp e2e tests +------------ + +End-to-end tests for ``jp`` are a good way to tests backend features without having to +deal with frontends UI. Those tests use extensively the ``sh`` module, which helps +writing ``jp`` commands like if they where methods. + +Among the helping fixture (check the various ``conftest.py`` files for details), the +following are specially good to know: + +``jp_json`` + Set the ``json_raw`` output are parse it. When you use this instead of the normal ``jp``, + you'll get a Python object that you can manipulate easily. + +``jp_elt`` + Set the ``xml_raw`` output and parse it as a Twisted ``domish.Element``. When you use a + command which can return XML, it is useful to get this object which is easy to + manipulate in Python. + +``editor`` + Create a fake editor (replacing the one normally set in ``EDITOR`` environment + variable), to automatically modify and/or check the text sent by a command. You can + specify Python code to execute to modify the received text with the ``set_filter`` + method (this code is in a string which will be executed by Python interpreter, where the + ``content`` variable is the received text). By default, the text is kept unmodified. + + After ``editor`` has been used by the ``jp`` command, you can check its + ``original_content`` property to see the text that it received, and ``new_content`` + property to see the text that has been written after updating the original content with + the code set in ``set_filter``. + +Libervia e2e tests +------------------ + +E2e tests for Libervia are executed, as it is common in web world, with `Selenium`_: user +actions are simulated in automated browser, and results are checked. + +To make the tests as easy to write as possible, and as natural to read as possible, the +higher level `Helium`_ Python module is used. Thanks to it, the tests can read pretty much +like instructions we would give to a human user. Helium makes also easy to do some tasks +more complicated with Selenium alone, like dropping a file to an element. + +If a test is failing, a screenshot of the browser is taken. If you run the tests though +the ``run_e2e.py`` command (which you should), you'll find the screenshots in the +``report_*`` directory which is created in working dir in case of failure. + +Here are the helping fixtures which are notably good to know, you should always use either +``log_in_account1`` or ``nobody_logged_in``: + +``log_in_account1`` + Start the test with the main test account logged. + +``nobody_logged_in`` + Start the test without anybody logged (this is done by clearing all cookies). + +.. _Selenium: https://www.selenium.dev +.. _Helium: https://github.com/mherrmann/selenium-python-helium + +examples +-------- + +Following examples have to be run from ``tests/e2e`` directory. + +Run all tests for ``jp``:: + + $ ./run_e2e.py -k jp + +Run all tests for ``Libervia`` with real-time visual feedback (note that you need to have +``vncviewer`` installed and available in path, see above):: + + $ ./run_e2e.py -k libervia --visual + + +Run all tests with verbose mode (useful to know which test is currently running):: + + $ ./run_e2e.py -v + +Run pubsub tests in verbose mode:: + + $ ./run_e2e.py -k pubsub -v + +Run in dev mode, to work on new tests, note that we run the ``user_can_create_account`` +test to be sure to have test profiles created and fake SMTP server run…:: + + $ ./run_e2e.py -k user_can_create_account --dev-mode + +…then to go into the ``sat`` container and work with the browser (to be run in ``docker`` +directory)…:: + + $ /sbin/docker-compose -f docker-compose_e2e.yml exec sat /bin/bash + +…and, inside the container, you can now run ``python3`` and enter instruction prints at +the end of the test session.