Mercurial > libervia-backend

comparison doc/contribuing/testing.rst @ 3430:4ba31dd1f0f5

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
doc: documentation for testing in new `contributing` section
author Goffi <goffi@goffi.org>
date Fri, 27 Nov 2020 16:40:26 +0100
parents
children 5d7569378914
comparison
equal deleted inserted replaced
3429:d4558f3cbf13 3430:4ba31dd1f0f5
1 =======
2 testing
3 =======
4
5 You'll find here the documentation to run tests on Salut à Toi. If you plan to contribute
6 to the ecosystem, you should use them to check that your modification is not breaking
7 anything, and if possible you should extend them to cover any new feature.
8
9 .. _contributing-overview:
10
11 overview
12 ========
13
14 Tests are run using `pytest`_ and are located in the ``tests`` directory.
15 You'll also find legacy tests in ``sat/test`` but those one are old, not maintained and
16 only kept there temporarily until they are ported to the new system.
17
18 For now, emphasis is put on end-2-end tests, as they are covering the whole ecosystem, and
19 are quite easy to write. The downside is that they are quite long to run.
20
21 Several `fixtures`_ are available in the various ``conftest.py`` files, it's a good idea
22 to have an overview of them if you're willing to write your own tests.
23
24 .. _pytest: https://www.pytest.org
25 .. _fixtures: https://docs.pytest.org/en/latest/fixture.html
26
27 end-to-end tests
28 ================
29
30 End-to-end tests are located in ``tests/e2e``. They are launched in a well defined
31 environment managed through Docker. The ``docker/docker-compose_e2e.yml`` is used to
32 create the suitable containers.
33
34 A script is available at ``tests/e2e/run_e2e.py`` to launch the tests. It will create the
35 containers, bind the current code to them, and set environment variables according to
36 arguments.
37
38 The arguments set to this script are the ``pytest`` options, thus you can have a verbose
39 mode with ``-v`` and select specific test with ``-k EXPRESSION`` (see ``pytest --help`` for
40 details).
41
42 In addition to pytest option, some flags can be set with the following arguments:
43
44
45 ``--visual``
46 Launch a VNC viewer to see in real time browser based tests. You must have ``vncviewer``
47 executable available in your path (this is part of `TigerVNC`_)
48
49 ``--keep-containers``
50 Do no remove Docker container after the end of tests.
51
52 ``--keep-profiles``
53 Do not delete test profiles after the end of tests
54
55 ``--keep-vnc``
56 Do not stop VNC viewer after the end of tests. This argument implies ``--visual``.
57
58 ``--keep-browser``
59 Do not kill the browser inside the container after tests are done. This argument implies
60 ``--keep-container`` and ``--keep-vnc``.
61
62 ``--dev-mode``
63 Shortcut for ``--keep-containers``, ``--keep-profiles`` and ``--keep-vnc``. This is
64 useful, as you guess with its names, for development of tests. User can then log-in into
65 the ``sat`` container, launch a Python console, and work with the automated browser in
66 real-time. Basic commands to launch a browser and log-in with test account are printed
67 at the end of the tests. Note that if you want to have profiles created, or extra tools
68 like the fake SMTP server, you'll have to launch at least one test which require them.
69 To log-in into the ``sat`` container, you can use the following command, from
70 ``/docker`` directory::
71
72 $ docker-compose -f docker-compose_e2e.yml exec sat /bin/bash
73
74 Then run a python console with given instructions
75
76 .. _TigerVNC: https://tigervnc.org
77
78 common fixtures
79 ---------------
80
81 Here are the fixture common to all e2e tests which are good to know:
82
83 ``test_profiles``
84 Creates a bunch of test accounts which are available during the whole test session.
85 Those account are destroyed once all the tests are finished (successful or not), except
86 if you set the ``SAT_TEST_E2E_KEEP_PROFILES`` environment variable (or use the
87 ``--keep-profiles`` flag in ``run_e2e.py``.
88
89 The profiles created are in the form ``accountX`` for account on the ``server1.test``,
90 or ``accountX_sY`` for account on other servers (see the docstring for details).
91
92 This fixture should be used on top of each e2e test module.
93
94 ``pubsub_nodes``
95 Create 2 pubsub nodes with ``open`` access model and named ``test`` (one on ``account1``
96 PEP service, and the other one on ``pubsub.server1.test``, created with the same
97 account).
98
99 Those node are created for the scope of the class.
100
101 ``fake_file``
102 Create files filled with random bytes, and check them.
103
104 A file is created by calling ``fake_file.size(size)``, and by default files of the same
105 size are re-used (set ``use_cache=False`` to create new files). This method returns a
106 ``pathlib.Path``. SHA-256 hash of the created file can be retrieved using
107 ``fake_file.get_source_hash(source_file_path)`` with the file path as argument.
108
109 ``fake_file.new_dest_file()`` will return a Path to a randomly named destination file,
110 and ``fake_file.get_dest_hash(dest_file_path)`` will generate its hash once written.
111
112 ``sent_emails``
113 When used, a fake SMTP server (already configured in container's ``sat.conf``) will be
114 launched if it's not already, and all message sent to it since the beginning of the test
115 will be available in the given list. Message are subclasses of
116 ``email.message.EmailMessage`` with the extra properties ``from_``, ``to``, ``subject``
117 and ``body`` available for easy access to their content.
118
119 The SMTP server is terminated at the end of the test session.
120
121 jp e2e tests
122 ------------
123
124 End-to-end tests for ``jp`` are a good way to tests backend features without having to
125 deal with frontends UI. Those tests use extensively the ``sh`` module, which helps
126 writing ``jp`` commands like if they where methods.
127
128 Among the helping fixture (check the various ``conftest.py`` files for details), the
129 following are specially good to know:
130
131 ``jp_json``
132 Set the ``json_raw`` output are parse it. When you use this instead of the normal ``jp``,
133 you'll get a Python object that you can manipulate easily.
134
135 ``jp_elt``
136 Set the ``xml_raw`` output and parse it as a Twisted ``domish.Element``. When you use a
137 command which can return XML, it is useful to get this object which is easy to
138 manipulate in Python.
139
140 ``editor``
141 Create a fake editor (replacing the one normally set in ``EDITOR`` environment
142 variable), to automatically modify and/or check the text sent by a command. You can
143 specify Python code to execute to modify the received text with the ``set_filter``
144 method (this code is in a string which will be executed by Python interpreter, where the
145 ``content`` variable is the received text). By default, the text is kept unmodified.
146
147 After ``editor`` has been used by the ``jp`` command, you can check its
148 ``original_content`` property to see the text that it received, and ``new_content``
149 property to see the text that has been written after updating the original content with
150 the code set in ``set_filter``.
151
152 Libervia e2e tests
153 ------------------
154
155 E2e tests for Libervia are executed, as it is common in web world, with `Selenium`_: user
156 actions are simulated in automated browser, and results are checked.
157
158 To make the tests as easy to write as possible, and as natural to read as possible, the
159 higher level `Helium`_ Python module is used. Thanks to it, the tests can read pretty much
160 like instructions we would give to a human user. Helium makes also easy to do some tasks
161 more complicated with Selenium alone, like dropping a file to an element.
162
163 If a test is failing, a screenshot of the browser is taken. If you run the tests though
164 the ``run_e2e.py`` command (which you should), you'll find the screenshots in the
165 ``report_*`` directory which is created in working dir in case of failure.
166
167 Here are the helping fixtures which are notably good to know, you should always use either
168 ``log_in_account1`` or ``nobody_logged_in``:
169
170 ``log_in_account1``
171 Start the test with the main test account logged.
172
173 ``nobody_logged_in``
174 Start the test without anybody logged (this is done by clearing all cookies).
175
176 .. _Selenium: https://www.selenium.dev
177 .. _Helium: https://github.com/mherrmann/selenium-python-helium
178
179 examples
180 --------
181
182 Following examples have to be run from ``tests/e2e`` directory.
183
184 Run all tests for ``jp``::
185
186 $ ./run_e2e.py -k jp
187
188 Run all tests for ``Libervia`` with real-time visual feedback (note that you need to have
189 ``vncviewer`` installed and available in path, see above)::
190
191 $ ./run_e2e.py -k libervia --visual
192
193
194 Run all tests with verbose mode (useful to know which test is currently running)::
195
196 $ ./run_e2e.py -v
197
198 Run pubsub tests in verbose mode::
199
200 $ ./run_e2e.py -k pubsub -v
201
202 Run in dev mode, to work on new tests, note that we run the ``user_can_create_account``
203 test to be sure to have test profiles created and fake SMTP server run…::
204
205 $ ./run_e2e.py -k user_can_create_account --dev-mode
206
207 …then to go into the ``sat`` container and work with the browser (to be run in ``docker``
208 directory)…::
209
210 $ /sbin/docker-compose -f docker-compose_e2e.yml exec sat /bin/bash
211
212 …and, inside the container, you can now run ``python3`` and enter instruction prints at
213 the end of the test session.