Mercurial > libervia-backend

annotate doc/developer.rst @ 3946:f2a5936f2496

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
tests (e2e/cli): add test for pubsub encryption: test encryption and keys sharing. rel 380
author Goffi <goffi@goffi.org>
date Sat, 15 Oct 2022 20:38:33 +0200
parents 432aaba0d7fe
children 524856bd7b19
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
3606
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
1 .. _developer:
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
2
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
3 =======================
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
4 Developer Documentation
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
5 =======================
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
6
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
7 This documentation is intended for people who wants to contribute or work with the
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
8 internals of the project, it is not for end-users.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
9
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
10 Storage
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
11 =======
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
12
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
13 Since version 0.9, Libervia uses SQLAlchemy_ with its Object–Relational Mapping as a
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
14 backend to store persistent data, and Alembic_ is used to handle schema and data
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
15 migrations.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
16
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
17 SQLite_ is currently the only supported database, but it is planned to add support for
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
18 other ones (notably PostgreSQL), probably during the development of 0.9 version.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
19
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
20 The mapping is done in ``sat.memory.sqla_mapping`` and working with database is done
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
21 through high level methods found in ``sat.memory.sqla``.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
22
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
23 Before the move to SQLAlchemy, there was a strict separation between database
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
24 implementation and the rest of the code. With 0.9, objects mapping to database can be used
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
25 and manipulated directly outside of ``sat.memory.sqla`` to take profit of SQLAlchemy
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
26 possibilities.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
27
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
28 Database state is detected when the backend starts, and the database will be created or
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
29 migrated automatically if necessary.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
30
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
31 To create a new migration script, ``Alembic`` may be used directly. To do so, be sure to
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
32 have an up-to-date database (and a backup in case of troubles), then activate the virtual
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
33 environment where Libervia is installed (Alembic needs to access ORM mapping), go to
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
34 ``sat/memory/migration`` directory, and enter the following command::
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
35
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
36 alembic revision --autogenerate -m "some revision message"
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
37
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
38 This will create a base migration file in ``versions`` directory. Adapt it to your needs,
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
39 try to create both ``upgrade`` and ``downgrade`` method whenever possible, and be sure to
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
40 test it in both directions (``alembic upgrade head`` and ``alembic downgrade
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
41 <previous_revision>``). Please check Alembic documentation for more details.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
42
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
43 .. _SQLALchemy: https://www.sqlalchemy.org/
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
44 .. _Alembic: https://alembic.sqlalchemy.org/
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
45 .. _SQLite: https://sqlite.org
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
46
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
47 Pubsub Cache
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
48 ============
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
49
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
50 There is an internal cache for pubsub nodes and items, which is done in
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
51 ``plugin_pubsub_cache``. The ``PubsubNode`` and ``PubsubItem`` class are the one mapping
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
52 the database.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
53
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
54 The cache is operated transparently to end-user, when a pubsub request is done, it uses a
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
55 trigger to check if the requested node is or must be cached, and if possible returns
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
56 result directly from database, otherwise it lets the normal workflow continue and query the
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
57 pubsub service.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
58
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
59 To save resources, not all nodes are fully cached. When a node is checked, a series of
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
60 analysers are checked, and the first one matching is used to determine if the node must be
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
61 synchronised or not.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
62
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
63 Analysers can be registered by any plugins using ``registerAnalyser`` method:
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
64
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
65 .. automethod:: sat.plugins.plugin_pubsub_cache.PubsubCache.registerAnalyser
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
66
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
67 If no analyser is found, ``to_sync`` is false, or an error happens during the caching,
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
68 the node won't be synchronised and the pubsub service will always be requested.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
69
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
70 Specifying an optional **parser** will store parsed data in addition to the raw XML of the
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
71 items. This is more space consuming, but may be desired for the following reasons:
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
72
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
73 * the parsing is resource consuming (network call or some CPU intensive operations are
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
74 done)
3828
432aaba0d7fe doc (developer): typo
Goffi <goffi@goffi.org>
parents: 3606
diff changeset
75 * it is desirable to do queries on parsed data. Indeed the parsed data are stored in a
3606
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
76 JSON_ field and its keys may be queried individually.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
77
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
78 The Raw XML is kept as the cache operates transparently, and a plugin may need raw data, or
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
79 an user may be doing a low-level pubsub request.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
80
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
81 .. _JSON: https://docs.sqlalchemy.org/en/14/core/type_basics.html#sqlalchemy.types.JSON
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
82