annotate tests/e2e/run_e2e.py @ 3415:814e118d9ef3

tests: end-2-end tests first draft: - e2e tests are launched inside the new docker e2e test environment - `run_e2e.py` launch the docker container, mount the current code base in it, launch the e2e tests and print report in real time - `conftest.py` are pytest fixtures managing many things such as account creation, fake files management, JSON or Domish.Element parsing, fake editor, etc. - `test_jp.py` are end-to-end test done with `jp`. `sh` library is used to make tests writting as user-friendly as possible. The `SAT_TEST_ENV_E2E` environment variable is checked, and tests will be skipped if it's not set.
author Goffi <goffi@goffi.org>
date Thu, 12 Nov 2020 14:53:16 +0100
parents
children d4558f3cbf13
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
3415
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
1 #!/usr/bin/env python3
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
2
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
3 # SàT: an XMPP client
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
4 # Copyright (C) 2009-2020 Jérôme Poisson (goffi@goffi.org)
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
5
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
6 # This program is free software: you can redistribute it and/or modify
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
7 # it under the terms of the GNU Affero General Public License as published by
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
8 # the Free Software Foundation, either version 3 of the License, or
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
9 # (at your option) any later version.
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
10
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
11 # This program is distributed in the hope that it will be useful,
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
12 # but WITHOUT ANY WARRANTY; without even the implied warranty of
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
13 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
14 # GNU Affero General Public License for more details.
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
15
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
16 # You should have received a copy of the GNU Affero General Public License
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
17 # along with this program. If not, see <http://www.gnu.org/licenses/>.
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
18
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
19 """Run end-to-end tests in appropriate Docker environment"""
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
20
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
21 import sys
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
22 from pathlib import Path
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
23 import tempfile
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
24 from textwrap import dedent
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
25 import sh
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
26 from sat.core import exceptions
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
27
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
28 KEEP_OPT = "--keep"
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
29
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
30
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
31 def live_out(data):
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
32 sys.stdout.write(data)
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
33 sys.stdout.flush()
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
34
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
35
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
36 def live_err(data):
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
37 sys.stderr.write(data)
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
38 sys.stderr.flush()
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
39
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
40
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
41 def use_e2e_env():
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
42 if KEEP_OPT in sys.argv:
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
43 keep_containers = True
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
44 sys.argv.remove(KEEP_OPT)
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
45 else:
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
46 keep_containers = False
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
47 for p in Path.cwd().parents:
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
48 package_path = p / "sat"
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
49 docker_path = p / "docker"
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
50 if package_path.is_dir() and docker_path.is_dir():
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
51 sat_root_path = p
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
52 break
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
53 else:
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
54 raise exceptions.NotFound(
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
55 "Can't find root of SàT code, are you sure that you are running the test "
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
56 "from the backend repository?"
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
57 )
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
58
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
59 compose_e2e_path = docker_path / "docker-compose_e2e.yml"
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
60 if not compose_e2e_path.is_file():
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
61 raise exceptions.NotFound('"docker-compose_e2e.yml" file can\'t be found')
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
62
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
63 with tempfile.TemporaryDirectory() as temp_dir:
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
64 override_path = Path(temp_dir) / "test_override.yml"
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
65 with override_path.open("w") as f:
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
66 f.write(dedent(f"""\
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
67 version: "3.6"
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
68 services:
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
69 sat:
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
70 volumes:
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
71 - type: bind
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
72 source: {sat_root_path}
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
73 target: /src/sat
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
74 read_only: true
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
75 """))
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
76
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
77 docker_compose = sh.docker_compose.bake(
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
78 "-f", compose_e2e_path, "-f", override_path)
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
79 docker_compose.up("-d")
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
80
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
81 try:
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
82 docker_compose.exec(
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
83 "-T", "--workdir", "/src/sat/tests", "sat",
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
84 "pytest", "-o", "cache_dir=/tmp", *sys.argv[1:], color="yes",
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
85 _in=sys.stdin, _out=live_out, _out_bufsize=0, _err=live_err, _err_bufsize=0
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
86 )
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
87 except sh.ErrorReturnCode as e:
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
88 sys.exit(e.exit_code)
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
89 finally:
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
90 if not keep_containers:
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
91 docker_compose.down(volumes=True)
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
92
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
93 if __name__ == "__main__":
814e118d9ef3 tests: end-2-end tests first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
94 use_e2e_env()