Mercurial > libervia-backend

annotate doc/developer.rst @ 3942:a92eef737703

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
plugin XEP-0373: download public keys if they are not found in local storage: public keys were only obtained from PEP notifications, however this wasn't working if the entity was not in our roster. Now if no public key is retrieved from local storage, the public key node is requested, and an error is raised if nothing is found. This allows the use of OX with entities which are not in roster. rel 380
author Goffi <goffi@goffi.org>
date Sat, 15 Oct 2022 20:38:33 +0200
parents 432aaba0d7fe
children 524856bd7b19
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
3606
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
1 .. _developer:
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
2
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
3 =======================
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
4 Developer Documentation
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
5 =======================
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
6
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
7 This documentation is intended for people who wants to contribute or work with the
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
8 internals of the project, it is not for end-users.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
9
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
10 Storage
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
11 =======
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
12
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
13 Since version 0.9, Libervia uses SQLAlchemy_ with its Object–Relational Mapping as a
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
14 backend to store persistent data, and Alembic_ is used to handle schema and data
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
15 migrations.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
16
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
17 SQLite_ is currently the only supported database, but it is planned to add support for
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
18 other ones (notably PostgreSQL), probably during the development of 0.9 version.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
19
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
20 The mapping is done in ``sat.memory.sqla_mapping`` and working with database is done
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
21 through high level methods found in ``sat.memory.sqla``.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
22
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
23 Before the move to SQLAlchemy, there was a strict separation between database
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
24 implementation and the rest of the code. With 0.9, objects mapping to database can be used
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
25 and manipulated directly outside of ``sat.memory.sqla`` to take profit of SQLAlchemy
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
26 possibilities.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
27
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
28 Database state is detected when the backend starts, and the database will be created or
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
29 migrated automatically if necessary.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
30
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
31 To create a new migration script, ``Alembic`` may be used directly. To do so, be sure to
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
32 have an up-to-date database (and a backup in case of troubles), then activate the virtual
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
33 environment where Libervia is installed (Alembic needs to access ORM mapping), go to
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
34 ``sat/memory/migration`` directory, and enter the following command::
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
35
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
36 alembic revision --autogenerate -m "some revision message"
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
37
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
38 This will create a base migration file in ``versions`` directory. Adapt it to your needs,
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
39 try to create both ``upgrade`` and ``downgrade`` method whenever possible, and be sure to
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
40 test it in both directions (``alembic upgrade head`` and ``alembic downgrade
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
41 <previous_revision>``). Please check Alembic documentation for more details.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
42
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
43 .. _SQLALchemy: https://www.sqlalchemy.org/
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
44 .. _Alembic: https://alembic.sqlalchemy.org/
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
45 .. _SQLite: https://sqlite.org
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
46
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
47 Pubsub Cache
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
48 ============
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
49
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
50 There is an internal cache for pubsub nodes and items, which is done in
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
51 ``plugin_pubsub_cache``. The ``PubsubNode`` and ``PubsubItem`` class are the one mapping
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
52 the database.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
53
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
54 The cache is operated transparently to end-user, when a pubsub request is done, it uses a
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
55 trigger to check if the requested node is or must be cached, and if possible returns
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
56 result directly from database, otherwise it lets the normal workflow continue and query the
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
57 pubsub service.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
58
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
59 To save resources, not all nodes are fully cached. When a node is checked, a series of
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
60 analysers are checked, and the first one matching is used to determine if the node must be
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
61 synchronised or not.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
62
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
63 Analysers can be registered by any plugins using ``registerAnalyser`` method:
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
64
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
65 .. automethod:: sat.plugins.plugin_pubsub_cache.PubsubCache.registerAnalyser
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
66
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
67 If no analyser is found, ``to_sync`` is false, or an error happens during the caching,
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
68 the node won't be synchronised and the pubsub service will always be requested.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
69
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
70 Specifying an optional **parser** will store parsed data in addition to the raw XML of the
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
71 items. This is more space consuming, but may be desired for the following reasons:
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
72
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
73 * the parsing is resource consuming (network call or some CPU intensive operations are
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
74 done)
3828
432aaba0d7fe doc (developer): typo
Goffi <goffi@goffi.org>
parents: 3606
diff changeset
75 * it is desirable to do queries on parsed data. Indeed the parsed data are stored in a
3606
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
76 JSON_ field and its keys may be queried individually.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
77
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
78 The Raw XML is kept as the cache operates transparently, and a plugin may need raw data, or
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
79 an user may be doing a low-level pubsub request.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
80
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
81 .. _JSON: https://docs.sqlalchemy.org/en/14/core/type_basics.html#sqlalchemy.types.JSON
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
82