libervia-backend: doc/contributing/testing.rst annotate

annotate doc/contributing/testing.rst @ 3732:0fac164ff2d8

tools (xml_tools): fix `widget_args` modification in `_dataFormField2XMLUIData`: in `textbox` widget, the first arg was modified with all values, lettings other ones unchanger, resulting in invalid number of arguments. This has been fixed by replacing all args with the new value.

author	Goffi <goffi@goffi.org>
date	Mon, 31 Jan 2022 18:35:52 +0100
parents	e65395e5f649
children	d6837db456fd

rev	line source
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	1 =======
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	2 testing
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	3 =======
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	4
3505 4705f80b6e23 doc: more renaming Goffi <goffi@goffi.org> parents: 3498 diff changeset	5 You'll find here the documentation to run tests on Libervia. If you plan to contribute
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	6 to the ecosystem, you should use them to check that your modification is not breaking
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	7 anything, and if possible you should extend them to cover any new feature.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	8
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	9 .. _contributing-overview:
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	10
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	11 overview
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	12 ========
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	13
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	14 Tests are run using `pytest`_ and are located in the ``tests`` directory.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	15 You'll also find legacy tests in ``sat/test`` but those one are old, not maintained and
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	16 only kept there temporarily until they are ported to the new system.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	17
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	18 For now, emphasis is put on end-2-end tests, as they are covering the whole ecosystem, and
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	19 are quite easy to write. The downside is that they are quite long to run.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	20
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	21 Several `fixtures`_ are available in the various ``conftest.py`` files, it's a good idea
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	22 to have an overview of them if you're willing to write your own tests.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	23
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	24 .. _pytest: https://www.pytest.org
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	25 .. _fixtures: https://docs.pytest.org/en/latest/fixture.html
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	26
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	27 end-to-end tests
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	28 ================
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	29
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	30 End-to-end tests are located in ``tests/e2e``. They are launched in a well defined
3644 062a09705d43 doc (contributing): fix `docker-compose-e2e.yml` spelling Goffi <goffi@goffi.org> parents: 3505 diff changeset	31 environment managed through Docker. The ``docker/docker-compose-e2e.yml`` is used to
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	32 create the suitable containers.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	33
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	34 A script is available at ``tests/e2e/run_e2e.py`` to launch the tests. It will create the
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	35 containers, bind the current code to them, and set environment variables according to
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	36 arguments.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	37
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	38 The arguments set to this script are the ``pytest`` options, thus you can have a verbose
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	39 mode with ``-v`` and select specific test with ``-k EXPRESSION`` (see ``pytest --help`` for
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	40 details).
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	41
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	42 In addition to pytest option, some flags can be set with the following arguments:
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	43
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	44 ``--visual``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	45 Launch a VNC viewer to see in real time browser based tests. You must have ``vncviewer``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	46 executable available in your path (this is part of `TigerVNC`_)
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	47
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	48 ``--keep-containers``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	49 Do no remove Docker container after the end of tests.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	50
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	51 ``--keep-profiles``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	52 Do not delete test profiles after the end of tests
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	53
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	54 ``--keep-vnc``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	55 Do not stop VNC viewer after the end of tests. This argument implies ``--visual``.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	56
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	57 ``--keep-browser``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	58 Do not kill the browser inside the container after tests are done. This argument implies
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	59 ``--keep-container`` and ``--keep-vnc``.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	60
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	61 ``--dev-mode``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	62 Shortcut for ``--keep-containers``, ``--keep-profiles`` and ``--keep-vnc``. This is
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	63 useful, as you guess with its names, for development of tests. User can then log-in into
3498 d78b5eae912a tests: update following names change Goffi <goffi@goffi.org> parents: 3439 diff changeset	64 the ``backend`` container, launch a Python console, and work with the automated browser in
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	65 real-time. Basic commands to launch a browser and log-in with test account are printed
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	66 at the end of the tests. Note that if you want to have profiles created, or extra tools
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	67 like the fake SMTP server, you'll have to launch at least one test which require them.
3498 d78b5eae912a tests: update following names change Goffi <goffi@goffi.org> parents: 3439 diff changeset	68 To log-in into the ``backend`` container, you can use the following command, from
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	69 ``/docker`` directory::
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	70
3644 062a09705d43 doc (contributing): fix `docker-compose-e2e.yml` spelling Goffi <goffi@goffi.org> parents: 3505 diff changeset	71 $ docker-compose -f docker-compose-e2e.yml exec backend /bin/bash
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	72
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	73 Then run a python console with given instructions
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	74
3439 b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc Goffi <goffi@goffi.org> parents: 3431 diff changeset	75 It's also good to know that in the e2e test environment, the following pytest plugins are
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc Goffi <goffi@goffi.org> parents: 3431 diff changeset	76 installed and used:
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc Goffi <goffi@goffi.org> parents: 3431 diff changeset	77
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc Goffi <goffi@goffi.org> parents: 3431 diff changeset	78 `pytest-timeout`_
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc Goffi <goffi@goffi.org> parents: 3431 diff changeset	79 To avoid having test stuck, it's good to terminate them after a while. A timeout of 60s
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc Goffi <goffi@goffi.org> parents: 3431 diff changeset	80 is set by default for each test (lower value can give false negatives, as some e2e tests
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc Goffi <goffi@goffi.org> parents: 3431 diff changeset	81 can be long, notably with Selenium).
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc Goffi <goffi@goffi.org> parents: 3431 diff changeset	82
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc Goffi <goffi@goffi.org> parents: 3431 diff changeset	83 `pytest-dependency`_
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc Goffi <goffi@goffi.org> parents: 3431 diff changeset	84 Even if good testing practice normally means that tests can be run independently, in the
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc Goffi <goffi@goffi.org> parents: 3431 diff changeset	85 case of e2e tests we are using a real environment, and some tests do create files,
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc Goffi <goffi@goffi.org> parents: 3431 diff changeset	86 PubSub nodes, accounts, etc. It would be resource consuming to delete then recreate them
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc Goffi <goffi@goffi.org> parents: 3431 diff changeset	87 only to have standalone tests, thus to keep tests short and simple, some of them must be
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc Goffi <goffi@goffi.org> parents: 3431 diff changeset	88 run in order. The dependecy plugin is used to manage that, and will skip tests if one
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc Goffi <goffi@goffi.org> parents: 3431 diff changeset	89 of their dependencies is failing. The markup help also to document the tests order.
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc Goffi <goffi@goffi.org> parents: 3431 diff changeset	90
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	91 .. _TigerVNC: https://tigervnc.org
3439 b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc Goffi <goffi@goffi.org> parents: 3431 diff changeset	92 .. _pytest-timeout: https://github.com/pytest-dev/pytest-timeout
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc Goffi <goffi@goffi.org> parents: 3431 diff changeset	93 .. _pytest-dependency: https://github.com/RKrahl/pytest-dependency
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	94
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	95 common fixtures
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	96 ---------------
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	97
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	98 Here are the fixture common to all e2e tests which are good to know:
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	99
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	100 ``test_profiles``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	101 Creates a bunch of test accounts which are available during the whole test session.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	102 Those account are destroyed once all the tests are finished (successful or not), except
3498 d78b5eae912a tests: update following names change Goffi <goffi@goffi.org> parents: 3439 diff changeset	103 if you set the ``LIBERVIA_TEST_E2E_KEEP_PROFILES`` environment variable (or use the
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	104 ``--keep-profiles`` flag in ``run_e2e.py``.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	105
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	106 The profiles created are in the form ``accountX`` for account on the ``server1.test``,
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	107 or ``accountX_sY`` for account on other servers (see the docstring for details).
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	108
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	109 This fixture should be used on top of each e2e test module.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	110
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	111 ``pubsub_nodes``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	112 Create 2 pubsub nodes with ``open`` access model and named ``test`` (one on ``account1``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	113 PEP service, and the other one on ``pubsub.server1.test``, created with the same
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	114 account).
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	115
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	116 Those node are created for the scope of the class.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	117
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	118 ``fake_file``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	119 Create files filled with random bytes, and check them.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	120
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	121 A file is created by calling ``fake_file.size(size)``, and by default files of the same
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	122 size are re-used (set ``use_cache=False`` to create new files). This method returns a
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	123 ``pathlib.Path``. SHA-256 hash of the created file can be retrieved using
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	124 ``fake_file.get_source_hash(source_file_path)`` with the file path as argument.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	125
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	126 ``fake_file.new_dest_file()`` will return a Path to a randomly named destination file,
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	127 and ``fake_file.get_dest_hash(dest_file_path)`` will generate its hash once written.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	128
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	129 ``sent_emails``
3498 d78b5eae912a tests: update following names change Goffi <goffi@goffi.org> parents: 3439 diff changeset	130 When used, a fake SMTP server (already configured in container's ``libervia.conf``) will be
3431 5d7569378914 doc (testing): typos Goffi <goffi@goffi.org> parents: 3430 diff changeset	131 launched if it's not already, and all messages sent to it since the beginning of the test
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	132 will be available in the given list. Message are subclasses of
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	133 ``email.message.EmailMessage`` with the extra properties ``from_``, ``to``, ``subject``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	134 and ``body`` available for easy access to their content.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	135
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	136 The SMTP server is terminated at the end of the test session.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	137
3498 d78b5eae912a tests: update following names change Goffi <goffi@goffi.org> parents: 3439 diff changeset	138 libervia-cli e2e tests
d78b5eae912a tests: update following names change Goffi <goffi@goffi.org> parents: 3439 diff changeset	139 ----------------------
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	140
3498 d78b5eae912a tests: update following names change Goffi <goffi@goffi.org> parents: 3439 diff changeset	141 End-to-end tests for ``libervia-cli`` are a good way to tests backend features without having to
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	142 deal with frontends UI. Those tests use extensively the ``sh`` module, which helps
3498 d78b5eae912a tests: update following names change Goffi <goffi@goffi.org> parents: 3439 diff changeset	143 writing ``libervia-cli`` commands like if they where methods.
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	144
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	145 Among the helping fixture (check the various ``conftest.py`` files for details), the
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	146 following are specially good to know:
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	147
3498 d78b5eae912a tests: update following names change Goffi <goffi@goffi.org> parents: 3439 diff changeset	148 ``li_json``
d78b5eae912a tests: update following names change Goffi <goffi@goffi.org> parents: 3439 diff changeset	149 Set the ``json_raw`` output are parse it. When you use this instead of the normal ``libervia-cli``,
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	150 you'll get a Python object that you can manipulate easily.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	151
3498 d78b5eae912a tests: update following names change Goffi <goffi@goffi.org> parents: 3439 diff changeset	152 ``li_elt``
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	153 Set the ``xml_raw`` output and parse it as a Twisted ``domish.Element``. When you use a
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	154 command which can return XML, it is useful to get this object which is easy to
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	155 manipulate in Python.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	156
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	157 ``editor``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	158 Create a fake editor (replacing the one normally set in ``EDITOR`` environment
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	159 variable), to automatically modify and/or check the text sent by a command. You can
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	160 specify Python code to execute to modify the received text with the ``set_filter``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	161 method (this code is in a string which will be executed by Python interpreter, where the
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	162 ``content`` variable is the received text). By default, the text is kept unmodified.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	163
3498 d78b5eae912a tests: update following names change Goffi <goffi@goffi.org> parents: 3439 diff changeset	164 After ``editor`` has been used by the ``libervia-cli`` command, you can check its
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	165 ``original_content`` property to see the text that it received, and ``new_content``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	166 property to see the text that has been written after updating the original content with
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	167 the code set in ``set_filter``.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	168
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	169 Libervia e2e tests
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	170 ------------------
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	171
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	172 E2e tests for Libervia are executed, as it is common in web world, with `Selenium`_: user
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	173 actions are simulated in automated browser, and results are checked.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	174
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	175 To make the tests as easy to write as possible, and as natural to read as possible, the
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	176 higher level `Helium`_ Python module is used. Thanks to it, the tests can read pretty much
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	177 like instructions we would give to a human user. Helium makes also easy to do some tasks
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	178 more complicated with Selenium alone, like dropping a file to an element.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	179
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	180 If a test is failing, a screenshot of the browser is taken. If you run the tests though
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	181 the ``run_e2e.py`` command (which you should), you'll find the screenshots in the
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	182 ``report_*`` directory which is created in working dir in case of failure.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	183
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	184 Here are the helping fixtures which are notably good to know, you should always use either
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	185 ``log_in_account1`` or ``nobody_logged_in``:
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	186
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	187 ``log_in_account1``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	188 Start the test with the main test account logged.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	189
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	190 ``nobody_logged_in``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	191 Start the test without anybody logged (this is done by clearing all cookies).
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	192
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	193 .. _Selenium: https://www.selenium.dev
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	194 .. _Helium: https://github.com/mherrmann/selenium-python-helium
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	195
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	196 examples
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	197 --------
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	198
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	199 Following examples have to be run from ``tests/e2e`` directory.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	200
3498 d78b5eae912a tests: update following names change Goffi <goffi@goffi.org> parents: 3439 diff changeset	201 Run all tests for ``Libervia CLI``::
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	202
3498 d78b5eae912a tests: update following names change Goffi <goffi@goffi.org> parents: 3439 diff changeset	203 $ ./run_e2e.py -k libervia-cli
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	204
3498 d78b5eae912a tests: update following names change Goffi <goffi@goffi.org> parents: 3439 diff changeset	205 Run all tests for ``Libervia Web`` with real-time visual feedback (note that you need to have
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	206 ``vncviewer`` installed and available in path, see above)::
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	207
3498 d78b5eae912a tests: update following names change Goffi <goffi@goffi.org> parents: 3439 diff changeset	208 $ ./run_e2e.py -k libervia-web --visual
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	209
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	210
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	211 Run all tests with verbose mode (useful to know which test is currently running)::
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	212
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	213 $ ./run_e2e.py -v
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	214
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	215 Run pubsub tests in verbose mode::
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	216
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	217 $ ./run_e2e.py -k pubsub -v
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	218
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	219 Run in dev mode, to work on new tests, note that we run the ``user_can_create_account``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	220 test to be sure to have test profiles created and fake SMTP server run…::
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	221
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	222 $ ./run_e2e.py -k user_can_create_account --dev-mode
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	223
3498 d78b5eae912a tests: update following names change Goffi <goffi@goffi.org> parents: 3439 diff changeset	224 …then to go into the ``backend`` container and work with the browser (to be run in ``docker``
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	225 directory)…::
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	226
3644 062a09705d43 doc (contributing): fix `docker-compose-e2e.yml` spelling Goffi <goffi@goffi.org> parents: 3505 diff changeset	227 $ docker-compose -f docker-compose-e2e.yml exec backend /bin/bash
3430 4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	228
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	229 …and, inside the container, you can now run ``python3`` and enter instruction prints at
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section Goffi <goffi@goffi.org> parents: diff changeset	230 the end of the test session.

Mercurial > libervia-backend

annotate doc/contributing/testing.rst @ 3732:0fac164ff2d8