Mercurial > libervia-backend

annotate doc/developer.rst @ 4219:1b5cf2ee1d86

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
plugin XEP-0384, XEP-0391: download missing devices list: when a peer jid was not in our roster, devices list was not retrieved, resulting in failed en/decryption. This patch does check it and download missing devices list in necessary. There is no subscription managed yet, so the list won't be updated in case of new devices, this should be addressed at some point.
author Goffi <goffi@goffi.org>
date Tue, 05 Mar 2024 17:31:36 +0100
parents d6837db456fd
children 00852dd54695
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
3606
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
1 .. _developer:
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
2
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
3 =======================
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
4 Developer Documentation
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
5 =======================
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
6
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
7 This documentation is intended for people who wants to contribute or work with the
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
8 internals of the project, it is not for end-users.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
9
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
10 Storage
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
11 =======
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
12
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
13 Since version 0.9, Libervia uses SQLAlchemy_ with its Object–Relational Mapping as a
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
14 backend to store persistent data, and Alembic_ is used to handle schema and data
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
15 migrations.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
16
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
17 SQLite_ is currently the only supported database, but it is planned to add support for
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
18 other ones (notably PostgreSQL), probably during the development of 0.9 version.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
19
4077
d6837db456fd refactoring: fix names in doc following modules hierarchy refactoring
Goffi <goffi@goffi.org>
parents: 4037
diff changeset
20 The mapping is done in ``libervia.backend.memory.sqla_mapping`` and working with database is done
d6837db456fd refactoring: fix names in doc following modules hierarchy refactoring
Goffi <goffi@goffi.org>
parents: 4037
diff changeset
21 through high level methods found in ``libervia.backend.memory.sqla``.
3606
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
22
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
23 Before the move to SQLAlchemy, there was a strict separation between database
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
24 implementation and the rest of the code. With 0.9, objects mapping to database can be used
4077
d6837db456fd refactoring: fix names in doc following modules hierarchy refactoring
Goffi <goffi@goffi.org>
parents: 4037
diff changeset
25 and manipulated directly outside of ``libervia.backend.memory.sqla`` to take profit of SQLAlchemy
3606
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
26 possibilities.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
27
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
28 Database state is detected when the backend starts, and the database will be created or
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
29 migrated automatically if necessary.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
30
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
31 To create a new migration script, ``Alembic`` may be used directly. To do so, be sure to
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
32 have an up-to-date database (and a backup in case of troubles), then activate the virtual
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
33 environment where Libervia is installed (Alembic needs to access ORM mapping), go to
4077
d6837db456fd refactoring: fix names in doc following modules hierarchy refactoring
Goffi <goffi@goffi.org>
parents: 4037
diff changeset
34 ``libervia/backend/memory/migration`` directory, and enter the following command::
3606
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
35
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
36 alembic revision --autogenerate -m "some revision message"
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
37
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
38 This will create a base migration file in ``versions`` directory. Adapt it to your needs,
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
39 try to create both ``upgrade`` and ``downgrade`` method whenever possible, and be sure to
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
40 test it in both directions (``alembic upgrade head`` and ``alembic downgrade
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
41 <previous_revision>``). Please check Alembic documentation for more details.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
42
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
43 .. _SQLALchemy: https://www.sqlalchemy.org/
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
44 .. _Alembic: https://alembic.sqlalchemy.org/
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
45 .. _SQLite: https://sqlite.org
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
46
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
47 Pubsub Cache
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
48 ============
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
49
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
50 There is an internal cache for pubsub nodes and items, which is done in
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
51 ``plugin_pubsub_cache``. The ``PubsubNode`` and ``PubsubItem`` class are the one mapping
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
52 the database.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
53
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
54 The cache is operated transparently to end-user, when a pubsub request is done, it uses a
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
55 trigger to check if the requested node is or must be cached, and if possible returns
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
56 result directly from database, otherwise it lets the normal workflow continue and query the
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
57 pubsub service.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
58
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
59 To save resources, not all nodes are fully cached. When a node is checked, a series of
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
60 analysers are checked, and the first one matching is used to determine if the node must be
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
61 synchronised or not.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
62
4037
524856bd7b19 massive refactoring to switch from camelCase to snake_case:
Goffi <goffi@goffi.org>
parents: 3828
diff changeset
63 Analysers can be registered by any plugins using ``register_analyser`` method:
3606
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
64
4077
d6837db456fd refactoring: fix names in doc following modules hierarchy refactoring
Goffi <goffi@goffi.org>
parents: 4037
diff changeset
65 .. automethod:: libervia.backend.plugins.plugin_pubsub_cache.PubsubCache.register_analyser
3606
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
66
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
67 If no analyser is found, ``to_sync`` is false, or an error happens during the caching,
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
68 the node won't be synchronised and the pubsub service will always be requested.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
69
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
70 Specifying an optional **parser** will store parsed data in addition to the raw XML of the
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
71 items. This is more space consuming, but may be desired for the following reasons:
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
72
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
73 * the parsing is resource consuming (network call or some CPU intensive operations are
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
74 done)
3828
432aaba0d7fe doc (developer): typo
Goffi <goffi@goffi.org>
parents: 3606
diff changeset
75 * it is desirable to do queries on parsed data. Indeed the parsed data are stored in a
3606
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
76 JSON_ field and its keys may be queried individually.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
77
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
78 The Raw XML is kept as the cache operates transparently, and a plugin may need raw data, or
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
79 an user may be doing a low-level pubsub request.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
80
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
81 .. _JSON: https://docs.sqlalchemy.org/en/14/core/type_basics.html#sqlalchemy.types.JSON
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
82