Mercurial > libervia-backend

comparison doc/contributing/testing.rst @ 3645:e65395e5f649

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
doc (contributing): typo
author Goffi <goffi@goffi.org>
date Wed, 08 Sep 2021 11:15:41 +0200
parents doc/contribuing/testing.rst@062a09705d43
children d6837db456fd
comparison
equal deleted inserted replaced
3644:062a09705d43 3645:e65395e5f649
1 =======
2 testing
3 =======
4
5 You'll find here the documentation to run tests on Libervia. If you plan to contribute
6 to the ecosystem, you should use them to check that your modification is not breaking
7 anything, and if possible you should extend them to cover any new feature.
8
9 .. _contributing-overview:
10
11 overview
12 ========
13
14 Tests are run using `pytest`_ and are located in the ``tests`` directory.
15 You'll also find legacy tests in ``sat/test`` but those one are old, not maintained and
16 only kept there temporarily until they are ported to the new system.
17
18 For now, emphasis is put on end-2-end tests, as they are covering the whole ecosystem, and
19 are quite easy to write. The downside is that they are quite long to run.
20
21 Several `fixtures`_ are available in the various ``conftest.py`` files, it's a good idea
22 to have an overview of them if you're willing to write your own tests.
23
24 .. _pytest: https://www.pytest.org
25 .. _fixtures: https://docs.pytest.org/en/latest/fixture.html
26
27 end-to-end tests
28 ================
29
30 End-to-end tests are located in ``tests/e2e``. They are launched in a well defined
31 environment managed through Docker. The ``docker/docker-compose-e2e.yml`` is used to
32 create the suitable containers.
33
34 A script is available at ``tests/e2e/run_e2e.py`` to launch the tests. It will create the
35 containers, bind the current code to them, and set environment variables according to
36 arguments.
37
38 The arguments set to this script are the ``pytest`` options, thus you can have a verbose
39 mode with ``-v`` and select specific test with ``-k EXPRESSION`` (see ``pytest --help`` for
40 details).
41
42 In addition to pytest option, some flags can be set with the following arguments:
43
44 ``--visual``
45 Launch a VNC viewer to see in real time browser based tests. You must have ``vncviewer``
46 executable available in your path (this is part of `TigerVNC`_)
47
48 ``--keep-containers``
49 Do no remove Docker container after the end of tests.
50
51 ``--keep-profiles``
52 Do not delete test profiles after the end of tests
53
54 ``--keep-vnc``
55 Do not stop VNC viewer after the end of tests. This argument implies ``--visual``.
56
57 ``--keep-browser``
58 Do not kill the browser inside the container after tests are done. This argument implies
59 ``--keep-container`` and ``--keep-vnc``.
60
61 ``--dev-mode``
62 Shortcut for ``--keep-containers``, ``--keep-profiles`` and ``--keep-vnc``. This is
63 useful, as you guess with its names, for development of tests. User can then log-in into
64 the ``backend`` container, launch a Python console, and work with the automated browser in
65 real-time. Basic commands to launch a browser and log-in with test account are printed
66 at the end of the tests. Note that if you want to have profiles created, or extra tools
67 like the fake SMTP server, you'll have to launch at least one test which require them.
68 To log-in into the ``backend`` container, you can use the following command, from
69 ``/docker`` directory::
70
71 $ docker-compose -f docker-compose-e2e.yml exec backend /bin/bash
72
73 Then run a python console with given instructions
74
75 It's also good to know that in the e2e test environment, the following pytest plugins are
76 installed and used:
77
78 `pytest-timeout`_
79 To avoid having test stuck, it's good to terminate them after a while. A timeout of 60s
80 is set by default for each test (lower value can give false negatives, as some e2e tests
81 can be long, notably with Selenium).
82
83 `pytest-dependency`_
84 Even if good testing practice normally means that tests can be run independently, in the
85 case of e2e tests we are using a real environment, and some tests do create files,
86 PubSub nodes, accounts, etc. It would be resource consuming to delete then recreate them
87 only to have standalone tests, thus to keep tests short and simple, some of them must be
88 run in order. The *dependecy* plugin is used to manage that, and will skip tests if one
89 of their dependencies is failing. The markup help also to document the tests order.
90
91 .. _TigerVNC: https://tigervnc.org
92 .. _pytest-timeout: https://github.com/pytest-dev/pytest-timeout
93 .. _pytest-dependency: https://github.com/RKrahl/pytest-dependency
94
95 common fixtures
96 ---------------
97
98 Here are the fixture common to all e2e tests which are good to know:
99
100 ``test_profiles``
101 Creates a bunch of test accounts which are available during the whole test session.
102 Those account are destroyed once all the tests are finished (successful or not), except
103 if you set the ``LIBERVIA_TEST_E2E_KEEP_PROFILES`` environment variable (or use the
104 ``--keep-profiles`` flag in ``run_e2e.py``.
105
106 The profiles created are in the form ``accountX`` for account on the ``server1.test``,
107 or ``accountX_sY`` for account on other servers (see the docstring for details).
108
109 This fixture should be used on top of each e2e test module.
110
111 ``pubsub_nodes``
112 Create 2 pubsub nodes with ``open`` access model and named ``test`` (one on ``account1``
113 PEP service, and the other one on ``pubsub.server1.test``, created with the same
114 account).
115
116 Those node are created for the scope of the class.
117
118 ``fake_file``
119 Create files filled with random bytes, and check them.
120
121 A file is created by calling ``fake_file.size(size)``, and by default files of the same
122 size are re-used (set ``use_cache=False`` to create new files). This method returns a
123 ``pathlib.Path``. SHA-256 hash of the created file can be retrieved using
124 ``fake_file.get_source_hash(source_file_path)`` with the file path as argument.
125
126 ``fake_file.new_dest_file()`` will return a Path to a randomly named destination file,
127 and ``fake_file.get_dest_hash(dest_file_path)`` will generate its hash once written.
128
129 ``sent_emails``
130 When used, a fake SMTP server (already configured in container's ``libervia.conf``) will be
131 launched if it's not already, and all messages sent to it since the beginning of the test
132 will be available in the given list. Message are subclasses of
133 ``email.message.EmailMessage`` with the extra properties ``from_``, ``to``, ``subject``
134 and ``body`` available for easy access to their content.
135
136 The SMTP server is terminated at the end of the test session.
137
138 libervia-cli e2e tests
139 ----------------------
140
141 End-to-end tests for ``libervia-cli`` are a good way to tests backend features without having to
142 deal with frontends UI. Those tests use extensively the ``sh`` module, which helps
143 writing ``libervia-cli`` commands like if they where methods.
144
145 Among the helping fixture (check the various ``conftest.py`` files for details), the
146 following are specially good to know:
147
148 ``li_json``
149 Set the ``json_raw`` output are parse it. When you use this instead of the normal ``libervia-cli``,
150 you'll get a Python object that you can manipulate easily.
151
152 ``li_elt``
153 Set the ``xml_raw`` output and parse it as a Twisted ``domish.Element``. When you use a
154 command which can return XML, it is useful to get this object which is easy to
155 manipulate in Python.
156
157 ``editor``
158 Create a fake editor (replacing the one normally set in ``EDITOR`` environment
159 variable), to automatically modify and/or check the text sent by a command. You can
160 specify Python code to execute to modify the received text with the ``set_filter``
161 method (this code is in a string which will be executed by Python interpreter, where the
162 ``content`` variable is the received text). By default, the text is kept unmodified.
163
164 After ``editor`` has been used by the ``libervia-cli`` command, you can check its
165 ``original_content`` property to see the text that it received, and ``new_content``
166 property to see the text that has been written after updating the original content with
167 the code set in ``set_filter``.
168
169 Libervia e2e tests
170 ------------------
171
172 E2e tests for Libervia are executed, as it is common in web world, with `Selenium`_: user
173 actions are simulated in automated browser, and results are checked.
174
175 To make the tests as easy to write as possible, and as natural to read as possible, the
176 higher level `Helium`_ Python module is used. Thanks to it, the tests can read pretty much
177 like instructions we would give to a human user. Helium makes also easy to do some tasks
178 more complicated with Selenium alone, like dropping a file to an element.
179
180 If a test is failing, a screenshot of the browser is taken. If you run the tests though
181 the ``run_e2e.py`` command (which you should), you'll find the screenshots in the
182 ``report_*`` directory which is created in working dir in case of failure.
183
184 Here are the helping fixtures which are notably good to know, you should always use either
185 ``log_in_account1`` or ``nobody_logged_in``:
186
187 ``log_in_account1``
188 Start the test with the main test account logged.
189
190 ``nobody_logged_in``
191 Start the test without anybody logged (this is done by clearing all cookies).
192
193 .. _Selenium: https://www.selenium.dev
194 .. _Helium: https://github.com/mherrmann/selenium-python-helium
195
196 examples
197 --------
198
199 Following examples have to be run from ``tests/e2e`` directory.
200
201 Run all tests for ``Libervia CLI``::
202
203 $ ./run_e2e.py -k libervia-cli
204
205 Run all tests for ``Libervia Web`` with real-time visual feedback (note that you need to have
206 ``vncviewer`` installed and available in path, see above)::
207
208 $ ./run_e2e.py -k libervia-web --visual
209
210
211 Run all tests with verbose mode (useful to know which test is currently running)::
212
213 $ ./run_e2e.py -v
214
215 Run pubsub tests in verbose mode::
216
217 $ ./run_e2e.py -k pubsub -v
218
219 Run in dev mode, to work on new tests, note that we run the ``user_can_create_account``
220 test to be sure to have test profiles created and fake SMTP server run…::
221
222 $ ./run_e2e.py -k user_can_create_account --dev-mode
223
224 …then to go into the ``backend`` container and work with the browser (to be run in ``docker``
225 directory)…::
226
227 $ docker-compose -f docker-compose-e2e.yml exec backend /bin/bash
228
229 …and, inside the container, you can now run ``python3`` and enter instruction prints at
230 the end of the test session.