Mercurial > libervia-backend

annotate doc/contributing/testing.rst @ 3942:a92eef737703

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
plugin XEP-0373: download public keys if they are not found in local storage: public keys were only obtained from PEP notifications, however this wasn't working if the entity was not in our roster. Now if no public key is retrieved from local storage, the public key node is requested, and an error is raised if nothing is found. This allows the use of OX with entities which are not in roster. rel 380
author Goffi <goffi@goffi.org>
date Sat, 15 Oct 2022 20:38:33 +0200
parents e65395e5f649
children d6837db456fd
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
3430
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
1 =======
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
2 testing
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
3 =======
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
4
3505
4705f80b6e23 doc: more renaming
Goffi <goffi@goffi.org>
parents: 3498
diff changeset
5 You'll find here the documentation to run tests on Libervia. If you plan to contribute
3430
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
6 to the ecosystem, you should use them to check that your modification is not breaking
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
7 anything, and if possible you should extend them to cover any new feature.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
8
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
9 .. _contributing-overview:
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
10
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
11 overview
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
12 ========
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
13
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
14 Tests are run using `pytest`_ and are located in the ``tests`` directory.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
15 You'll also find legacy tests in ``sat/test`` but those one are old, not maintained and
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
16 only kept there temporarily until they are ported to the new system.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
17
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
18 For now, emphasis is put on end-2-end tests, as they are covering the whole ecosystem, and
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
19 are quite easy to write. The downside is that they are quite long to run.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
20
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
21 Several `fixtures`_ are available in the various ``conftest.py`` files, it's a good idea
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
22 to have an overview of them if you're willing to write your own tests.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
23
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
24 .. _pytest: https://www.pytest.org
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
25 .. _fixtures: https://docs.pytest.org/en/latest/fixture.html
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
26
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
27 end-to-end tests
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
28 ================
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
29
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
30 End-to-end tests are located in ``tests/e2e``. They are launched in a well defined
3644
062a09705d43 doc (contributing): fix `docker-compose-e2e.yml` spelling
Goffi <goffi@goffi.org>
parents: 3505
diff changeset
31 environment managed through Docker. The ``docker/docker-compose-e2e.yml`` is used to
3430
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
32 create the suitable containers.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
33
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
34 A script is available at ``tests/e2e/run_e2e.py`` to launch the tests. It will create the
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
35 containers, bind the current code to them, and set environment variables according to
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
36 arguments.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
37
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
38 The arguments set to this script are the ``pytest`` options, thus you can have a verbose
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
39 mode with ``-v`` and select specific test with ``-k EXPRESSION`` (see ``pytest --help`` for
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
40 details).
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
41
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
42 In addition to pytest option, some flags can be set with the following arguments:
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
43
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
44 ``--visual``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
45 Launch a VNC viewer to see in real time browser based tests. You must have ``vncviewer``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
46 executable available in your path (this is part of `TigerVNC`_)
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
47
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
48 ``--keep-containers``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
49 Do no remove Docker container after the end of tests.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
50
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
51 ``--keep-profiles``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
52 Do not delete test profiles after the end of tests
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
53
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
54 ``--keep-vnc``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
55 Do not stop VNC viewer after the end of tests. This argument implies ``--visual``.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
56
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
57 ``--keep-browser``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
58 Do not kill the browser inside the container after tests are done. This argument implies
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
59 ``--keep-container`` and ``--keep-vnc``.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
60
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
61 ``--dev-mode``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
62 Shortcut for ``--keep-containers``, ``--keep-profiles`` and ``--keep-vnc``. This is
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
63 useful, as you guess with its names, for development of tests. User can then log-in into
3498
d78b5eae912a tests: update following names change
Goffi <goffi@goffi.org>
parents: 3439
diff changeset
64 the ``backend`` container, launch a Python console, and work with the automated browser in
3430
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
65 real-time. Basic commands to launch a browser and log-in with test account are printed
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
66 at the end of the tests. Note that if you want to have profiles created, or extra tools
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
67 like the fake SMTP server, you'll have to launch at least one test which require them.
3498
d78b5eae912a tests: update following names change
Goffi <goffi@goffi.org>
parents: 3439
diff changeset
68 To log-in into the ``backend`` container, you can use the following command, from
3430
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
69 ``/docker`` directory::
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
70
3644
062a09705d43 doc (contributing): fix `docker-compose-e2e.yml` spelling
Goffi <goffi@goffi.org>
parents: 3505
diff changeset
71 $ docker-compose -f docker-compose-e2e.yml exec backend /bin/bash
3430
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
72
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
73 Then run a python console with given instructions
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
74
3439
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc
Goffi <goffi@goffi.org>
parents: 3431
diff changeset
75 It's also good to know that in the e2e test environment, the following pytest plugins are
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc
Goffi <goffi@goffi.org>
parents: 3431
diff changeset
76 installed and used:
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc
Goffi <goffi@goffi.org>
parents: 3431
diff changeset
77
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc
Goffi <goffi@goffi.org>
parents: 3431
diff changeset
78 `pytest-timeout`_
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc
Goffi <goffi@goffi.org>
parents: 3431
diff changeset
79 To avoid having test stuck, it's good to terminate them after a while. A timeout of 60s
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc
Goffi <goffi@goffi.org>
parents: 3431
diff changeset
80 is set by default for each test (lower value can give false negatives, as some e2e tests
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc
Goffi <goffi@goffi.org>
parents: 3431
diff changeset
81 can be long, notably with Selenium).
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc
Goffi <goffi@goffi.org>
parents: 3431
diff changeset
82
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc
Goffi <goffi@goffi.org>
parents: 3431
diff changeset
83 `pytest-dependency`_
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc
Goffi <goffi@goffi.org>
parents: 3431
diff changeset
84 Even if good testing practice normally means that tests can be run independently, in the
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc
Goffi <goffi@goffi.org>
parents: 3431
diff changeset
85 case of e2e tests we are using a real environment, and some tests do create files,
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc
Goffi <goffi@goffi.org>
parents: 3431
diff changeset
86 PubSub nodes, accounts, etc. It would be resource consuming to delete then recreate them
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc
Goffi <goffi@goffi.org>
parents: 3431
diff changeset
87 only to have standalone tests, thus to keep tests short and simple, some of them must be
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc
Goffi <goffi@goffi.org>
parents: 3431
diff changeset
88 run in order. The *dependecy* plugin is used to manage that, and will skip tests if one
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc
Goffi <goffi@goffi.org>
parents: 3431
diff changeset
89 of their dependencies is failing. The markup help also to document the tests order.
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc
Goffi <goffi@goffi.org>
parents: 3431
diff changeset
90
3430
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
91 .. _TigerVNC: https://tigervnc.org
3439
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc
Goffi <goffi@goffi.org>
parents: 3431
diff changeset
92 .. _pytest-timeout: https://github.com/pytest-dev/pytest-timeout
b166a59b1724 tests, doc(testing): added `pytest-dependency` plugin to `backend_e2e` image + doc
Goffi <goffi@goffi.org>
parents: 3431
diff changeset
93 .. _pytest-dependency: https://github.com/RKrahl/pytest-dependency
3430
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
94
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
95 common fixtures
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
96 ---------------
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
97
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
98 Here are the fixture common to all e2e tests which are good to know:
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
99
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
100 ``test_profiles``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
101 Creates a bunch of test accounts which are available during the whole test session.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
102 Those account are destroyed once all the tests are finished (successful or not), except
3498
d78b5eae912a tests: update following names change
Goffi <goffi@goffi.org>
parents: 3439
diff changeset
103 if you set the ``LIBERVIA_TEST_E2E_KEEP_PROFILES`` environment variable (or use the
3430
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
104 ``--keep-profiles`` flag in ``run_e2e.py``.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
105
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
106 The profiles created are in the form ``accountX`` for account on the ``server1.test``,
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
107 or ``accountX_sY`` for account on other servers (see the docstring for details).
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
108
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
109 This fixture should be used on top of each e2e test module.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
110
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
111 ``pubsub_nodes``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
112 Create 2 pubsub nodes with ``open`` access model and named ``test`` (one on ``account1``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
113 PEP service, and the other one on ``pubsub.server1.test``, created with the same
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
114 account).
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
115
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
116 Those node are created for the scope of the class.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
117
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
118 ``fake_file``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
119 Create files filled with random bytes, and check them.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
120
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
121 A file is created by calling ``fake_file.size(size)``, and by default files of the same
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
122 size are re-used (set ``use_cache=False`` to create new files). This method returns a
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
123 ``pathlib.Path``. SHA-256 hash of the created file can be retrieved using
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
124 ``fake_file.get_source_hash(source_file_path)`` with the file path as argument.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
125
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
126 ``fake_file.new_dest_file()`` will return a Path to a randomly named destination file,
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
127 and ``fake_file.get_dest_hash(dest_file_path)`` will generate its hash once written.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
128
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
129 ``sent_emails``
3498
d78b5eae912a tests: update following names change
Goffi <goffi@goffi.org>
parents: 3439
diff changeset
130 When used, a fake SMTP server (already configured in container's ``libervia.conf``) will be
3431
5d7569378914 doc (testing): typos
Goffi <goffi@goffi.org>
parents: 3430
diff changeset
131 launched if it's not already, and all messages sent to it since the beginning of the test
3430
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
132 will be available in the given list. Message are subclasses of
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
133 ``email.message.EmailMessage`` with the extra properties ``from_``, ``to``, ``subject``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
134 and ``body`` available for easy access to their content.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
135
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
136 The SMTP server is terminated at the end of the test session.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
137
3498
d78b5eae912a tests: update following names change
Goffi <goffi@goffi.org>
parents: 3439
diff changeset
138 libervia-cli e2e tests
d78b5eae912a tests: update following names change
Goffi <goffi@goffi.org>
parents: 3439
diff changeset
139 ----------------------
3430
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
140
3498
d78b5eae912a tests: update following names change
Goffi <goffi@goffi.org>
parents: 3439
diff changeset
141 End-to-end tests for ``libervia-cli`` are a good way to tests backend features without having to
3430
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
142 deal with frontends UI. Those tests use extensively the ``sh`` module, which helps
3498
d78b5eae912a tests: update following names change
Goffi <goffi@goffi.org>
parents: 3439
diff changeset
143 writing ``libervia-cli`` commands like if they where methods.
3430
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
144
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
145 Among the helping fixture (check the various ``conftest.py`` files for details), the
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
146 following are specially good to know:
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
147
3498
d78b5eae912a tests: update following names change
Goffi <goffi@goffi.org>
parents: 3439
diff changeset
148 ``li_json``
d78b5eae912a tests: update following names change
Goffi <goffi@goffi.org>
parents: 3439
diff changeset
149 Set the ``json_raw`` output are parse it. When you use this instead of the normal ``libervia-cli``,
3430
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
150 you'll get a Python object that you can manipulate easily.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
151
3498
d78b5eae912a tests: update following names change
Goffi <goffi@goffi.org>
parents: 3439
diff changeset
152 ``li_elt``
3430
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
153 Set the ``xml_raw`` output and parse it as a Twisted ``domish.Element``. When you use a
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
154 command which can return XML, it is useful to get this object which is easy to
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
155 manipulate in Python.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
156
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
157 ``editor``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
158 Create a fake editor (replacing the one normally set in ``EDITOR`` environment
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
159 variable), to automatically modify and/or check the text sent by a command. You can
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
160 specify Python code to execute to modify the received text with the ``set_filter``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
161 method (this code is in a string which will be executed by Python interpreter, where the
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
162 ``content`` variable is the received text). By default, the text is kept unmodified.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
163
3498
d78b5eae912a tests: update following names change
Goffi <goffi@goffi.org>
parents: 3439
diff changeset
164 After ``editor`` has been used by the ``libervia-cli`` command, you can check its
3430
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
165 ``original_content`` property to see the text that it received, and ``new_content``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
166 property to see the text that has been written after updating the original content with
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
167 the code set in ``set_filter``.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
168
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
169 Libervia e2e tests
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
170 ------------------
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
171
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
172 E2e tests for Libervia are executed, as it is common in web world, with `Selenium`_: user
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
173 actions are simulated in automated browser, and results are checked.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
174
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
175 To make the tests as easy to write as possible, and as natural to read as possible, the
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
176 higher level `Helium`_ Python module is used. Thanks to it, the tests can read pretty much
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
177 like instructions we would give to a human user. Helium makes also easy to do some tasks
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
178 more complicated with Selenium alone, like dropping a file to an element.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
179
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
180 If a test is failing, a screenshot of the browser is taken. If you run the tests though
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
181 the ``run_e2e.py`` command (which you should), you'll find the screenshots in the
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
182 ``report_*`` directory which is created in working dir in case of failure.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
183
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
184 Here are the helping fixtures which are notably good to know, you should always use either
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
185 ``log_in_account1`` or ``nobody_logged_in``:
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
186
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
187 ``log_in_account1``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
188 Start the test with the main test account logged.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
189
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
190 ``nobody_logged_in``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
191 Start the test without anybody logged (this is done by clearing all cookies).
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
192
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
193 .. _Selenium: https://www.selenium.dev
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
194 .. _Helium: https://github.com/mherrmann/selenium-python-helium
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
195
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
196 examples
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
197 --------
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
198
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
199 Following examples have to be run from ``tests/e2e`` directory.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
200
3498
d78b5eae912a tests: update following names change
Goffi <goffi@goffi.org>
parents: 3439
diff changeset
201 Run all tests for ``Libervia CLI``::
3430
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
202
3498
d78b5eae912a tests: update following names change
Goffi <goffi@goffi.org>
parents: 3439
diff changeset
203 $ ./run_e2e.py -k libervia-cli
3430
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
204
3498
d78b5eae912a tests: update following names change
Goffi <goffi@goffi.org>
parents: 3439
diff changeset
205 Run all tests for ``Libervia Web`` with real-time visual feedback (note that you need to have
3430
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
206 ``vncviewer`` installed and available in path, see above)::
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
207
3498
d78b5eae912a tests: update following names change
Goffi <goffi@goffi.org>
parents: 3439
diff changeset
208 $ ./run_e2e.py -k libervia-web --visual
3430
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
209
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
210
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
211 Run all tests with verbose mode (useful to know which test is currently running)::
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
212
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
213 $ ./run_e2e.py -v
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
214
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
215 Run pubsub tests in verbose mode::
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
216
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
217 $ ./run_e2e.py -k pubsub -v
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
218
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
219 Run in dev mode, to work on new tests, note that we run the ``user_can_create_account``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
220 test to be sure to have test profiles created and fake SMTP server run…::
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
221
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
222 $ ./run_e2e.py -k user_can_create_account --dev-mode
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
223
3498
d78b5eae912a tests: update following names change
Goffi <goffi@goffi.org>
parents: 3439
diff changeset
224 …then to go into the ``backend`` container and work with the browser (to be run in ``docker``
3430
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
225 directory)…::
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
226
3644
062a09705d43 doc (contributing): fix `docker-compose-e2e.yml` spelling
Goffi <goffi@goffi.org>
parents: 3505
diff changeset
227 $ docker-compose -f docker-compose-e2e.yml exec backend /bin/bash
3430
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
228
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
229 …and, inside the container, you can now run ``python3`` and enter instruction prints at
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
230 the end of the test session.