Mercurial > libervia-backend

comparison doc/developer.rst @ 3606:25d3677701e7

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
doc: developer documentation explaining storage and pubsub cache
author Goffi <goffi@goffi.org>
date Thu, 29 Jul 2021 22:51:01 +0200
parents
children 432aaba0d7fe
comparison
equal deleted inserted replaced
3605:62e81b1058f8 3606:25d3677701e7
1 .. _developer:
2
3 =======================
4 Developer Documentation
5 =======================
6
7 This documentation is intended for people who wants to contribute or work with the
8 internals of the project, it is not for end-users.
9
10 Storage
11 =======
12
13 Since version 0.9, Libervia uses SQLAlchemy_ with its Object–Relational Mapping as a
14 backend to store persistent data, and Alembic_ is used to handle schema and data
15 migrations.
16
17 SQLite_ is currently the only supported database, but it is planned to add support for
18 other ones (notably PostgreSQL), probably during the development of 0.9 version.
19
20 The mapping is done in ``sat.memory.sqla_mapping`` and working with database is done
21 through high level methods found in ``sat.memory.sqla``.
22
23 Before the move to SQLAlchemy, there was a strict separation between database
24 implementation and the rest of the code. With 0.9, objects mapping to database can be used
25 and manipulated directly outside of ``sat.memory.sqla`` to take profit of SQLAlchemy
26 possibilities.
27
28 Database state is detected when the backend starts, and the database will be created or
29 migrated automatically if necessary.
30
31 To create a new migration script, ``Alembic`` may be used directly. To do so, be sure to
32 have an up-to-date database (and a backup in case of troubles), then activate the virtual
33 environment where Libervia is installed (Alembic needs to access ORM mapping), go to
34 ``sat/memory/migration`` directory, and enter the following command::
35
36 alembic revision --autogenerate -m "some revision message"
37
38 This will create a base migration file in ``versions`` directory. Adapt it to your needs,
39 try to create both ``upgrade`` and ``downgrade`` method whenever possible, and be sure to
40 test it in both directions (``alembic upgrade head`` and ``alembic downgrade
41 <previous_revision>``). Please check Alembic documentation for more details.
42
43 .. _SQLALchemy: https://www.sqlalchemy.org/
44 .. _Alembic: https://alembic.sqlalchemy.org/
45 .. _SQLite: https://sqlite.org
46
47 Pubsub Cache
48 ============
49
50 There is an internal cache for pubsub nodes and items, which is done in
51 ``plugin_pubsub_cache``. The ``PubsubNode`` and ``PubsubItem`` class are the one mapping
52 the database.
53
54 The cache is operated transparently to end-user, when a pubsub request is done, it uses a
55 trigger to check if the requested node is or must be cached, and if possible returns
56 result directly from database, otherwise it lets the normal workflow continue and query the
57 pubsub service.
58
59 To save resources, not all nodes are fully cached. When a node is checked, a series of
60 analysers are checked, and the first one matching is used to determine if the node must be
61 synchronised or not.
62
63 Analysers can be registered by any plugins using ``registerAnalyser`` method:
64
65 .. automethod:: sat.plugins.plugin_pubsub_cache.PubsubCache.registerAnalyser
66
67 If no analyser is found, ``to_sync`` is false, or an error happens during the caching,
68 the node won't be synchronised and the pubsub service will always be requested.
69
70 Specifying an optional **parser** will store parsed data in addition to the raw XML of the
71 items. This is more space consuming, but may be desired for the following reasons:
72
73 * the parsing is resource consuming (network call or some CPU intensive operations are
74 done)
75 * it is desirable to to queries on parsed data. Indeed the parsed data is stored in a
76 JSON_ field and its keys may be queried individually.
77
78 The Raw XML is kept as the cache operates transparently, and a plugin may need raw data, or
79 an user may be doing a low-level pubsub request.
80
81 .. _JSON: https://docs.sqlalchemy.org/en/14/core/type_basics.html#sqlalchemy.types.JSON
82