Mercurial > libervia-backend
annotate doc/developer.rst @ 4306:94e0968987cd
plugin XEP-0033: code modernisation, improve delivery, data validation:
- Code has been rewritten using Pydantic models and `async` coroutines for data validation
and cleaner element parsing/generation.
- Delivery has been completely rewritten. It now works even if server doesn't support
multicast, and send to local multicast service first. Delivering to local multicast
service first is due to bad support of XEP-0033 in server (notably Prosody which has an
incomplete implementation), and the current impossibility to detect if a sub-domain
service handles fully multicast or only for local domains. This is a workaround to have
a good balance between backward compatilibity and use of bandwith, and to make it work
with the incoming email gateway implementation (the gateway will only deliver to
entities of its own domain).
- disco feature checking now uses `async` corountines. `host` implementation still use
Deferred return values for compatibility with legacy code.
rel 450
author | Goffi <goffi@goffi.org> |
---|---|
date | Thu, 26 Sep 2024 16:12:01 +0200 |
parents | 00852dd54695 |
children |
rev | line source |
---|---|
3606
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
1 .. _developer: |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
2 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
3 ======================= |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
4 Developer Documentation |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
5 ======================= |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
6 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
7 This documentation is intended for people who wants to contribute or work with the |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
8 internals of the project, it is not for end-users. |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
9 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
10 Storage |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
11 ======= |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
12 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
13 Since version 0.9, Libervia uses SQLAlchemy_ with its Object–Relational Mapping as a |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
14 backend to store persistent data, and Alembic_ is used to handle schema and data |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
15 migrations. |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
16 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
17 SQLite_ is currently the only supported database, but it is planned to add support for |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
18 other ones (notably PostgreSQL), probably during the development of 0.9 version. |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
19 |
4077
d6837db456fd
refactoring: fix names in doc following modules hierarchy refactoring
Goffi <goffi@goffi.org>
parents:
4037
diff
changeset
|
20 The mapping is done in ``libervia.backend.memory.sqla_mapping`` and working with database is done |
d6837db456fd
refactoring: fix names in doc following modules hierarchy refactoring
Goffi <goffi@goffi.org>
parents:
4037
diff
changeset
|
21 through high level methods found in ``libervia.backend.memory.sqla``. |
3606
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
22 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
23 Before the move to SQLAlchemy, there was a strict separation between database |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
24 implementation and the rest of the code. With 0.9, objects mapping to database can be used |
4077
d6837db456fd
refactoring: fix names in doc following modules hierarchy refactoring
Goffi <goffi@goffi.org>
parents:
4037
diff
changeset
|
25 and manipulated directly outside of ``libervia.backend.memory.sqla`` to take profit of SQLAlchemy |
3606
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
26 possibilities. |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
27 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
28 Database state is detected when the backend starts, and the database will be created or |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
29 migrated automatically if necessary. |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
30 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
31 To create a new migration script, ``Alembic`` may be used directly. To do so, be sure to |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
32 have an up-to-date database (and a backup in case of troubles), then activate the virtual |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
33 environment where Libervia is installed (Alembic needs to access ORM mapping), go to |
4077
d6837db456fd
refactoring: fix names in doc following modules hierarchy refactoring
Goffi <goffi@goffi.org>
parents:
4037
diff
changeset
|
34 ``libervia/backend/memory/migration`` directory, and enter the following command:: |
3606
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
35 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
36 alembic revision --autogenerate -m "some revision message" |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
37 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
38 This will create a base migration file in ``versions`` directory. Adapt it to your needs, |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
39 try to create both ``upgrade`` and ``downgrade`` method whenever possible, and be sure to |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
40 test it in both directions (``alembic upgrade head`` and ``alembic downgrade |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
41 <previous_revision>``). Please check Alembic documentation for more details. |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
42 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
43 .. _SQLALchemy: https://www.sqlalchemy.org/ |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
44 .. _Alembic: https://alembic.sqlalchemy.org/ |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
45 .. _SQLite: https://sqlite.org |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
46 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
47 Pubsub Cache |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
48 ============ |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
49 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
50 There is an internal cache for pubsub nodes and items, which is done in |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
51 ``plugin_pubsub_cache``. The ``PubsubNode`` and ``PubsubItem`` class are the one mapping |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
52 the database. |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
53 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
54 The cache is operated transparently to end-user, when a pubsub request is done, it uses a |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
55 trigger to check if the requested node is or must be cached, and if possible returns |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
56 result directly from database, otherwise it lets the normal workflow continue and query the |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
57 pubsub service. |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
58 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
59 To save resources, not all nodes are fully cached. When a node is checked, a series of |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
60 analysers are checked, and the first one matching is used to determine if the node must be |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
61 synchronised or not. |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
62 |
4037
524856bd7b19
massive refactoring to switch from camelCase to snake_case:
Goffi <goffi@goffi.org>
parents:
3828
diff
changeset
|
63 Analysers can be registered by any plugins using ``register_analyser`` method: |
3606
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
64 |
4077
d6837db456fd
refactoring: fix names in doc following modules hierarchy refactoring
Goffi <goffi@goffi.org>
parents:
4037
diff
changeset
|
65 .. automethod:: libervia.backend.plugins.plugin_pubsub_cache.PubsubCache.register_analyser |
3606
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
66 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
67 If no analyser is found, ``to_sync`` is false, or an error happens during the caching, |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
68 the node won't be synchronised and the pubsub service will always be requested. |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
69 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
70 Specifying an optional **parser** will store parsed data in addition to the raw XML of the |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
71 items. This is more space consuming, but may be desired for the following reasons: |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
72 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
73 * the parsing is resource consuming (network call or some CPU intensive operations are |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
74 done) |
3828 | 75 * it is desirable to do queries on parsed data. Indeed the parsed data are stored in a |
3606
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
76 JSON_ field and its keys may be queried individually. |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
77 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
78 The Raw XML is kept as the cache operates transparently, and a plugin may need raw data, or |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
79 an user may be doing a low-level pubsub request. |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
80 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
81 .. _JSON: https://docs.sqlalchemy.org/en/14/core/type_basics.html#sqlalchemy.types.JSON |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
82 |
4248
00852dd54695
doc: documentation to create a Libervia app, first draft
Goffi <goffi@goffi.org>
parents:
4077
diff
changeset
|
83 |
00852dd54695
doc: documentation to create a Libervia app, first draft
Goffi <goffi@goffi.org>
parents:
4077
diff
changeset
|
84 Libervia Apps |
00852dd54695
doc: documentation to create a Libervia app, first draft
Goffi <goffi@goffi.org>
parents:
4077
diff
changeset
|
85 ============= |
00852dd54695
doc: documentation to create a Libervia app, first draft
Goffi <goffi@goffi.org>
parents:
4077
diff
changeset
|
86 |
00852dd54695
doc: documentation to create a Libervia app, first draft
Goffi <goffi@goffi.org>
parents:
4077
diff
changeset
|
87 Libervia has a mechanism to embed applications and configure them to integrate with the |
00852dd54695
doc: documentation to create a Libervia app, first draft
Goffi <goffi@goffi.org>
parents:
4077
diff
changeset
|
88 ecosystem. This is notably used to embed docker applications. Description of the |
00852dd54695
doc: documentation to create a Libervia app, first draft
Goffi <goffi@goffi.org>
parents:
4077
diff
changeset
|
89 application description file is available at :ref:`libervia-app-config`. |