libervia-backend: doc/contributing/testing.rst annotate

annotate doc/contributing/testing.rst @ 4067:3f62c2445df1

plugin XEP-0339: "Source-Specific Media Attributes in Jingle" implementation: rel 439

author	Goffi <goffi@goffi.org>
date	Thu, 01 Jun 2023 20:53:33 +0200
parents	e65395e5f649
children	d6837db456fd

rev	line source
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	1 =======
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	2 testing
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	3 =======
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	4
3505 4705f80b6e23 doc: more renaming Goffi <goffi@goffi.org> parents: 3498 diff changeset	5 You'll find here the documentation to run tests on Libervia. If you plan to contribute
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	6 to the ecosystem, you should use them to check that your modification is not breaking
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	7 anything, and if possible you should extend them to cover any new feature.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	8
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	9 .. _contributing-overview:
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	10
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	11 overview
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	12 ========
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	13
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	14 Tests are run using `pytest`_ and are located in the ``tests`` directory.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	15 You'll also find legacy tests in ``sat/test`` but those one are old, not maintained and
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	16 only kept there temporarily until they are ported to the new system.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	17
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	18 For now, emphasis is put on end-2-end tests, as they are covering the whole ecosystem, and
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	19 are quite easy to write. The downside is that they are quite long to run.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	20
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	21 Several `fixtures`_ are available in the various ``conftest.py`` files, it's a good idea
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	22 to have an overview of them if you're willing to write your own tests.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	23
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	24 .. _pytest: https://www.pytest.org
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	25 .. _fixtures: https://docs.pytest.org/en/latest/fixture.html
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	26
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	27 end-to-end tests
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	28 ================
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	29
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	30 End-to-end tests are located in ``tests/e2e``. They are launched in a well defined
3644 062a09705d43 doc (contributing): fix `docker-compose-e2e.yml` spelling Goffi <goffi@goffi.org> parents: 3505 diff changeset	31 environment managed through Docker. The ``docker/docker-compose-e2e.yml`` is used to
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	32 create the suitable containers.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	33
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	34 A script is available at ``tests/e2e/run_e2e.py`` to launch the tests. It will create the
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	35 containers, bind the current code to them, and set environment variables according to
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	36 arguments.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	37
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	38 The arguments set to this script are the ``pytest`` options, thus you can have a verbose
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	39 mode with ``-v`` and select specific test with ``-k EXPRESSION`` (see ``pytest --help`` for
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	40 details).
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	41
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	42 In addition to pytest option, some flags can be set with the following arguments:
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	43
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	44 ``--visual``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	45 Launch a VNC viewer to see in real time browser based tests. You must have ``vncviewer``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	46 executable available in your path (this is part of `TigerVNC`_)
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	47
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	48 ``--keep-containers``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	49 Do no remove Docker container after the end of tests.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	50
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	51 ``--keep-profiles``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	52 Do not delete test profiles after the end of tests
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	53
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	54 ``--keep-vnc``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	55 Do not stop VNC viewer after the end of tests. This argument implies ``--visual``.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	56
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	57 ``--keep-browser``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	58 Do not kill the browser inside the container after tests are done. This argument implies
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	59 ``--keep-container`` and ``--keep-vnc``.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	60
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	61 ``--dev-mode``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	62 Shortcut for ``--keep-containers``, ``--keep-profiles`` and ``--keep-vnc``. This is
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	63 useful, as you guess with its names, for development of tests. User can then log-in into
3498 d78b5eae912a tests: update following names change Goffi <goffi@goffi.org> parents: 3439 diff changeset	64 the ``backend`` container, launch a Python console, and work with the automated browser in
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	65 real-time. Basic commands to launch a browser and log-in with test account are printed
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	66 at the end of the tests. Note that if you want to have profiles created, or extra tools
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	67 like the fake SMTP server, you'll have to launch at least one test which require them.
3498 d78b5eae912a tests: update following names change Goffi <goffi@goffi.org> parents: 3439 diff changeset	68 To log-in into the ``backend`` container, you can use the following command, from
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	69 ``/docker`` directory::
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	70
3644 062a09705d43 doc (contributing): fix `docker-compose-e2e.yml` spelling Goffi <goffi@goffi.org> parents: 3505 diff changeset	71 $ docker-compose -f docker-compose-e2e.yml exec backend /bin/bash
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	72
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	73 Then run a python console with given instructions
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	74
3439 b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc Goffi <goffi@goffi.org> parents: 3431 diff changeset	75 It's also good to know that in the e2e test environment, the following pytest plugins are
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc Goffi <goffi@goffi.org> parents: 3431 diff changeset	76 installed and used:
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc Goffi <goffi@goffi.org> parents: 3431 diff changeset	77
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc Goffi <goffi@goffi.org> parents: 3431 diff changeset	78 `pytest-timeout`_
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc Goffi <goffi@goffi.org> parents: 3431 diff changeset	79 To avoid having test stuck, it's good to terminate them after a while. A timeout of 60s
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc Goffi <goffi@goffi.org> parents: 3431 diff changeset	80 is set by default for each test (lower value can give false negatives, as some e2e tests
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc Goffi <goffi@goffi.org> parents: 3431 diff changeset	81 can be long, notably with Selenium).
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc Goffi <goffi@goffi.org> parents: 3431 diff changeset	82
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc Goffi <goffi@goffi.org> parents: 3431 diff changeset	83 `pytest-dependency`_
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc Goffi <goffi@goffi.org> parents: 3431 diff changeset	84 Even if good testing practice normally means that tests can be run independently, in the
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc Goffi <goffi@goffi.org> parents: 3431 diff changeset	85 case of e2e tests we are using a real environment, and some tests do create files,
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc Goffi <goffi@goffi.org> parents: 3431 diff changeset	86 PubSub nodes, accounts, etc. It would be resource consuming to delete then recreate them
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc Goffi <goffi@goffi.org> parents: 3431 diff changeset	87 only to have standalone tests, thus to keep tests short and simple, some of them must be
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc Goffi <goffi@goffi.org> parents: 3431 diff changeset	88 run in order. The dependecy plugin is used to manage that, and will skip tests if one
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc Goffi <goffi@goffi.org> parents: 3431 diff changeset	89 of their dependencies is failing. The markup help also to document the tests order.
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc Goffi <goffi@goffi.org> parents: 3431 diff changeset	90
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	91 .. _TigerVNC: https://tigervnc.org
3439 b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc Goffi <goffi@goffi.org> parents: 3431 diff changeset	92 .. _pytest-timeout: https://github.com/pytest-dev/pytest-timeout
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc Goffi <goffi@goffi.org> parents: 3431 diff changeset	93 .. _pytest-dependency: https://github.com/RKrahl/pytest-dependency
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	94
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	95 common fixtures
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	96 ---------------
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	97
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	98 Here are the fixture common to all e2e tests which are good to know:
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	99
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	100 ``test_profiles``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	101 Creates a bunch of test accounts which are available during the whole test session.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	102 Those account are destroyed once all the tests are finished (successful or not), except
3498 d78b5eae912a tests: update following names change Goffi <goffi@goffi.org> parents: 3439 diff changeset	103 if you set the ``LIBERVIA_TEST_E2E_KEEP_PROFILES`` environment variable (or use the
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	104 ``--keep-profiles`` flag in ``run_e2e.py``.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	105
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	106 The profiles created are in the form ``accountX`` for account on the ``server1.test``,
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	107 or ``accountX_sY`` for account on other servers (see the docstring for details).
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	108
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	109 This fixture should be used on top of each e2e test module.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	110
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	111 ``pubsub_nodes``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	112 Create 2 pubsub nodes with ``open`` access model and named ``test`` (one on ``account1``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	113 PEP service, and the other one on ``pubsub.server1.test``, created with the same
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	114 account).
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	115
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	116 Those node are created for the scope of the class.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	117
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	118 ``fake_file``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	119 Create files filled with random bytes, and check them.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	120
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	121 A file is created by calling ``fake_file.size(size)``, and by default files of the same
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	122 size are re-used (set ``use_cache=False`` to create new files). This method returns a
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	123 ``pathlib.Path``. SHA-256 hash of the created file can be retrieved using
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	124 ``fake_file.get_source_hash(source_file_path)`` with the file path as argument.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	125
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	126 ``fake_file.new_dest_file()`` will return a Path to a randomly named destination file,
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	127 and ``fake_file.get_dest_hash(dest_file_path)`` will generate its hash once written.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	128
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	129 ``sent_emails``
3498 d78b5eae912a tests: update following names change Goffi <goffi@goffi.org> parents: 3439 diff changeset	130 When used, a fake SMTP server (already configured in container's ``libervia.conf``) will be
3431 5d7569378914 doc (testing): typos Goffi <goffi@goffi.org> parents: 3430 diff changeset	131 launched if it's not already, and all messages sent to it since the beginning of the test
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	132 will be available in the given list. Message are subclasses of
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	133 ``email.message.EmailMessage`` with the extra properties ``from_``, ``to``, ``subject``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	134 and ``body`` available for easy access to their content.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	135
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	136 The SMTP server is terminated at the end of the test session.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	137
3498 d78b5eae912a tests: update following names change Goffi <goffi@goffi.org> parents: 3439 diff changeset	138 libervia-cli e2e tests
d78b5eae912a tests: update following names change Goffi <goffi@goffi.org> parents: 3439 diff changeset	139 ----------------------
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	140
3498 d78b5eae912a tests: update following names change Goffi <goffi@goffi.org> parents: 3439 diff changeset	141 End-to-end tests for ``libervia-cli`` are a good way to tests backend features without having to
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	142 deal with frontends UI. Those tests use extensively the ``sh`` module, which helps
3498 d78b5eae912a tests: update following names change Goffi <goffi@goffi.org> parents: 3439 diff changeset	143 writing ``libervia-cli`` commands like if they where methods.
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	144
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	145 Among the helping fixture (check the various ``conftest.py`` files for details), the
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	146 following are specially good to know:
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	147
3498 d78b5eae912a tests: update following names change Goffi <goffi@goffi.org> parents: 3439 diff changeset	148 ``li_json``
d78b5eae912a tests: update following names change Goffi <goffi@goffi.org> parents: 3439 diff changeset	149 Set the ``json_raw`` output are parse it. When you use this instead of the normal ``libervia-cli``,
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	150 you'll get a Python object that you can manipulate easily.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	151
3498 d78b5eae912a tests: update following names change Goffi <goffi@goffi.org> parents: 3439 diff changeset	152 ``li_elt``
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	153 Set the ``xml_raw`` output and parse it as a Twisted ``domish.Element``. When you use a
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	154 command which can return XML, it is useful to get this object which is easy to
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	155 manipulate in Python.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	156
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	157 ``editor``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	158 Create a fake editor (replacing the one normally set in ``EDITOR`` environment
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	159 variable), to automatically modify and/or check the text sent by a command. You can
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	160 specify Python code to execute to modify the received text with the ``set_filter``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	161 method (this code is in a string which will be executed by Python interpreter, where the
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	162 ``content`` variable is the received text). By default, the text is kept unmodified.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	163
3498 d78b5eae912a tests: update following names change Goffi <goffi@goffi.org> parents: 3439 diff changeset	164 After ``editor`` has been used by the ``libervia-cli`` command, you can check its
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	165 ``original_content`` property to see the text that it received, and ``new_content``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	166 property to see the text that has been written after updating the original content with
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	167 the code set in ``set_filter``.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	168
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	169 Libervia e2e tests
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	170 ------------------
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	171
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	172 E2e tests for Libervia are executed, as it is common in web world, with `Selenium`_: user
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	173 actions are simulated in automated browser, and results are checked.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	174
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	175 To make the tests as easy to write as possible, and as natural to read as possible, the
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	176 higher level `Helium`_ Python module is used. Thanks to it, the tests can read pretty much
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	177 like instructions we would give to a human user. Helium makes also easy to do some tasks
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	178 more complicated with Selenium alone, like dropping a file to an element.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	179
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	180 If a test is failing, a screenshot of the browser is taken. If you run the tests though
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	181 the ``run_e2e.py`` command (which you should), you'll find the screenshots in the
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	182 ``report_*`` directory which is created in working dir in case of failure.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	183
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	184 Here are the helping fixtures which are notably good to know, you should always use either
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	185 ``log_in_account1`` or ``nobody_logged_in``:
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	186
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	187 ``log_in_account1``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	188 Start the test with the main test account logged.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	189
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	190 ``nobody_logged_in``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	191 Start the test without anybody logged (this is done by clearing all cookies).
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	192
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	193 .. _Selenium: https://www.selenium.dev
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	194 .. _Helium: https://github.com/mherrmann/selenium-python-helium
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	195
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	196 examples
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	197 --------
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	198
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	199 Following examples have to be run from ``tests/e2e`` directory.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	200
3498 d78b5eae912a tests: update following names change Goffi <goffi@goffi.org> parents: 3439 diff changeset	201 Run all tests for ``Libervia CLI``::
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	202
3498 d78b5eae912a tests: update following names change Goffi <goffi@goffi.org> parents: 3439 diff changeset	203 $ ./run_e2e.py -k libervia-cli
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	204
3498 d78b5eae912a tests: update following names change Goffi <goffi@goffi.org> parents: 3439 diff changeset	205 Run all tests for ``Libervia Web`` with real-time visual feedback (note that you need to have
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	206 ``vncviewer`` installed and available in path, see above)::
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	207
3498 d78b5eae912a tests: update following names change Goffi <goffi@goffi.org> parents: 3439 diff changeset	208 $ ./run_e2e.py -k libervia-web --visual
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	209
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	210
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	211 Run all tests with verbose mode (useful to know which test is currently running)::
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	212
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	213 $ ./run_e2e.py -v
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	214
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	215 Run pubsub tests in verbose mode::
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	216
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	217 $ ./run_e2e.py -k pubsub -v
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	218
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	219 Run in dev mode, to work on new tests, note that we run the ``user_can_create_account``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	220 test to be sure to have test profiles created and fake SMTP server run…::
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	221
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	222 $ ./run_e2e.py -k user_can_create_account --dev-mode
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	223
3498 d78b5eae912a tests: update following names change Goffi <goffi@goffi.org> parents: 3439 diff changeset	224 …then to go into the ``backend`` container and work with the browser (to be run in ``docker``
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	225 directory)…::
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	226
3644 062a09705d43 doc (contributing): fix `docker-compose-e2e.yml` spelling Goffi <goffi@goffi.org> parents: 3505 diff changeset	227 $ docker-compose -f docker-compose-e2e.yml exec backend /bin/bash
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	228
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	229 …and, inside the container, you can now run ``python3`` and enter instruction prints at
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	230 the end of the test session.

Mercurial > libervia-backend

annotate doc/contributing/testing.rst @ 4067:3f62c2445df1