Mercurial > libervia-backend

annotate doc/contribuing/testing.rst @ 3430:4ba31dd1f0f5

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
doc: documentation for testing in new `contributing` section
author Goffi <goffi@goffi.org>
date Fri, 27 Nov 2020 16:40:26 +0100
parents
children 5d7569378914
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
3430
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
1 =======
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
2 testing
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
3 =======
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
4
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
5 You'll find here the documentation to run tests on Salut à Toi. If you plan to contribute
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
6 to the ecosystem, you should use them to check that your modification is not breaking
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
7 anything, and if possible you should extend them to cover any new feature.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
8
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
9 .. _contributing-overview:
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
10
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
11 overview
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
12 ========
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
13
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
14 Tests are run using `pytest`_ and are located in the ``tests`` directory.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
15 You'll also find legacy tests in ``sat/test`` but those one are old, not maintained and
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
16 only kept there temporarily until they are ported to the new system.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
17
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
18 For now, emphasis is put on end-2-end tests, as they are covering the whole ecosystem, and
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
19 are quite easy to write. The downside is that they are quite long to run.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
20
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
21 Several `fixtures`_ are available in the various ``conftest.py`` files, it's a good idea
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
22 to have an overview of them if you're willing to write your own tests.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
23
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
24 .. _pytest: https://www.pytest.org
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
25 .. _fixtures: https://docs.pytest.org/en/latest/fixture.html
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
26
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
27 end-to-end tests
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
28 ================
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
29
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
30 End-to-end tests are located in ``tests/e2e``. They are launched in a well defined
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
31 environment managed through Docker. The ``docker/docker-compose_e2e.yml`` is used to
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
32 create the suitable containers.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
33
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
34 A script is available at ``tests/e2e/run_e2e.py`` to launch the tests. It will create the
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
35 containers, bind the current code to them, and set environment variables according to
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
36 arguments.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
37
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
38 The arguments set to this script are the ``pytest`` options, thus you can have a verbose
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
39 mode with ``-v`` and select specific test with ``-k EXPRESSION`` (see ``pytest --help`` for
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
40 details).
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
41
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
42 In addition to pytest option, some flags can be set with the following arguments:
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
43
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
44
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
45 ``--visual``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
46 Launch a VNC viewer to see in real time browser based tests. You must have ``vncviewer``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
47 executable available in your path (this is part of `TigerVNC`_)
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
48
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
49 ``--keep-containers``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
50 Do no remove Docker container after the end of tests.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
51
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
52 ``--keep-profiles``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
53 Do not delete test profiles after the end of tests
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
54
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
55 ``--keep-vnc``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
56 Do not stop VNC viewer after the end of tests. This argument implies ``--visual``.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
57
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
58 ``--keep-browser``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
59 Do not kill the browser inside the container after tests are done. This argument implies
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
60 ``--keep-container`` and ``--keep-vnc``.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
61
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
62 ``--dev-mode``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
63 Shortcut for ``--keep-containers``, ``--keep-profiles`` and ``--keep-vnc``. This is
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
64 useful, as you guess with its names, for development of tests. User can then log-in into
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
65 the ``sat`` container, launch a Python console, and work with the automated browser in
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
66 real-time. Basic commands to launch a browser and log-in with test account are printed
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
67 at the end of the tests. Note that if you want to have profiles created, or extra tools
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
68 like the fake SMTP server, you'll have to launch at least one test which require them.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
69 To log-in into the ``sat`` container, you can use the following command, from
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
70 ``/docker`` directory::
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
71
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
72 $ docker-compose -f docker-compose_e2e.yml exec sat /bin/bash
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
73
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
74 Then run a python console with given instructions
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
75
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
76 .. _TigerVNC: https://tigervnc.org
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
77
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
78 common fixtures
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
79 ---------------
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
80
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
81 Here are the fixture common to all e2e tests which are good to know:
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
82
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
83 ``test_profiles``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
84 Creates a bunch of test accounts which are available during the whole test session.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
85 Those account are destroyed once all the tests are finished (successful or not), except
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
86 if you set the ``SAT_TEST_E2E_KEEP_PROFILES`` environment variable (or use the
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
87 ``--keep-profiles`` flag in ``run_e2e.py``.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
88
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
89 The profiles created are in the form ``accountX`` for account on the ``server1.test``,
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
90 or ``accountX_sY`` for account on other servers (see the docstring for details).
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
91
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
92 This fixture should be used on top of each e2e test module.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
93
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
94 ``pubsub_nodes``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
95 Create 2 pubsub nodes with ``open`` access model and named ``test`` (one on ``account1``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
96 PEP service, and the other one on ``pubsub.server1.test``, created with the same
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
97 account).
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
98
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
99 Those node are created for the scope of the class.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
100
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
101 ``fake_file``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
102 Create files filled with random bytes, and check them.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
103
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
104 A file is created by calling ``fake_file.size(size)``, and by default files of the same
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
105 size are re-used (set ``use_cache=False`` to create new files). This method returns a
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
106 ``pathlib.Path``. SHA-256 hash of the created file can be retrieved using
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
107 ``fake_file.get_source_hash(source_file_path)`` with the file path as argument.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
108
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
109 ``fake_file.new_dest_file()`` will return a Path to a randomly named destination file,
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
110 and ``fake_file.get_dest_hash(dest_file_path)`` will generate its hash once written.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
111
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
112 ``sent_emails``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
113 When used, a fake SMTP server (already configured in container's ``sat.conf``) will be
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
114 launched if it's not already, and all message sent to it since the beginning of the test
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
115 will be available in the given list. Message are subclasses of
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
116 ``email.message.EmailMessage`` with the extra properties ``from_``, ``to``, ``subject``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
117 and ``body`` available for easy access to their content.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
118
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
119 The SMTP server is terminated at the end of the test session.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
120
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
121 jp e2e tests
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
122 ------------
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
123
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
124 End-to-end tests for ``jp`` are a good way to tests backend features without having to
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
125 deal with frontends UI. Those tests use extensively the ``sh`` module, which helps
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
126 writing ``jp`` commands like if they where methods.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
127
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
128 Among the helping fixture (check the various ``conftest.py`` files for details), the
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
129 following are specially good to know:
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
130
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
131 ``jp_json``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
132 Set the ``json_raw`` output are parse it. When you use this instead of the normal ``jp``,
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
133 you'll get a Python object that you can manipulate easily.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
134
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
135 ``jp_elt``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
136 Set the ``xml_raw`` output and parse it as a Twisted ``domish.Element``. When you use a
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
137 command which can return XML, it is useful to get this object which is easy to
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
138 manipulate in Python.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
139
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
140 ``editor``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
141 Create a fake editor (replacing the one normally set in ``EDITOR`` environment
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
142 variable), to automatically modify and/or check the text sent by a command. You can
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
143 specify Python code to execute to modify the received text with the ``set_filter``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
144 method (this code is in a string which will be executed by Python interpreter, where the
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
145 ``content`` variable is the received text). By default, the text is kept unmodified.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
146
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
147 After ``editor`` has been used by the ``jp`` command, you can check its
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
148 ``original_content`` property to see the text that it received, and ``new_content``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
149 property to see the text that has been written after updating the original content with
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
150 the code set in ``set_filter``.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
151
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
152 Libervia e2e tests
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
153 ------------------
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
154
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
155 E2e tests for Libervia are executed, as it is common in web world, with `Selenium`_: user
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
156 actions are simulated in automated browser, and results are checked.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
157
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
158 To make the tests as easy to write as possible, and as natural to read as possible, the
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
159 higher level `Helium`_ Python module is used. Thanks to it, the tests can read pretty much
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
160 like instructions we would give to a human user. Helium makes also easy to do some tasks
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
161 more complicated with Selenium alone, like dropping a file to an element.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
162
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
163 If a test is failing, a screenshot of the browser is taken. If you run the tests though
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
164 the ``run_e2e.py`` command (which you should), you'll find the screenshots in the
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
165 ``report_*`` directory which is created in working dir in case of failure.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
166
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
167 Here are the helping fixtures which are notably good to know, you should always use either
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
168 ``log_in_account1`` or ``nobody_logged_in``:
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
169
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
170 ``log_in_account1``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
171 Start the test with the main test account logged.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
172
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
173 ``nobody_logged_in``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
174 Start the test without anybody logged (this is done by clearing all cookies).
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
175
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
176 .. _Selenium: https://www.selenium.dev
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
177 .. _Helium: https://github.com/mherrmann/selenium-python-helium
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
178
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
179 examples
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
180 --------
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
181
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
182 Following examples have to be run from ``tests/e2e`` directory.
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
183
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
184 Run all tests for ``jp``::
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
185
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
186 $ ./run_e2e.py -k jp
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
187
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
188 Run all tests for ``Libervia`` with real-time visual feedback (note that you need to have
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
189 ``vncviewer`` installed and available in path, see above)::
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
190
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
191 $ ./run_e2e.py -k libervia --visual
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
192
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
193
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
194 Run all tests with verbose mode (useful to know which test is currently running)::
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
195
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
196 $ ./run_e2e.py -v
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
197
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
198 Run pubsub tests in verbose mode::
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
199
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
200 $ ./run_e2e.py -k pubsub -v
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
201
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
202 Run in dev mode, to work on new tests, note that we run the ``user_can_create_account``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
203 test to be sure to have test profiles created and fake SMTP server run…::
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
204
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
205 $ ./run_e2e.py -k user_can_create_account --dev-mode
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
206
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
207 …then to go into the ``sat`` container and work with the browser (to be run in ``docker``
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
208 directory)…::
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
209
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
210 $ /sbin/docker-compose -f docker-compose_e2e.yml exec sat /bin/bash
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
211
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
212 …and, inside the container, you can now run ``python3`` and enter instruction prints at
4ba31dd1f0f5 doc: documentation for testing in new `contributing` section
Goffi <goffi@goffi.org>
parents:
diff changeset
213 the end of the test session.