Mercurial > libervia-backend

annotate doc/developer.rst @ 4158:83d8d8500bc2

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
tools (common/data_objects): expose message's `extra` field.
author Goffi <goffi@goffi.org>
date Wed, 22 Nov 2023 15:11:25 +0100
parents d6837db456fd
children 00852dd54695
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
3606
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
1 .. _developer:
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
2
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
3 =======================
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
4 Developer Documentation
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
5 =======================
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
6
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
7 This documentation is intended for people who wants to contribute or work with the
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
8 internals of the project, it is not for end-users.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
9
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
10 Storage
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
11 =======
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
12
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
13 Since version 0.9, Libervia uses SQLAlchemy_ with its Object–Relational Mapping as a
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
14 backend to store persistent data, and Alembic_ is used to handle schema and data
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
15 migrations.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
16
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
17 SQLite_ is currently the only supported database, but it is planned to add support for
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
18 other ones (notably PostgreSQL), probably during the development of 0.9 version.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
19
4077
d6837db456fd refactoring: fix names in doc following modules hierarchy refactoring
Goffi <goffi@goffi.org>
parents: 4037
diff changeset
20 The mapping is done in ``libervia.backend.memory.sqla_mapping`` and working with database is done
d6837db456fd refactoring: fix names in doc following modules hierarchy refactoring
Goffi <goffi@goffi.org>
parents: 4037
diff changeset
21 through high level methods found in ``libervia.backend.memory.sqla``.
3606
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
22
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
23 Before the move to SQLAlchemy, there was a strict separation between database
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
24 implementation and the rest of the code. With 0.9, objects mapping to database can be used
4077
d6837db456fd refactoring: fix names in doc following modules hierarchy refactoring
Goffi <goffi@goffi.org>
parents: 4037
diff changeset
25 and manipulated directly outside of ``libervia.backend.memory.sqla`` to take profit of SQLAlchemy
3606
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
26 possibilities.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
27
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
28 Database state is detected when the backend starts, and the database will be created or
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
29 migrated automatically if necessary.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
30
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
31 To create a new migration script, ``Alembic`` may be used directly. To do so, be sure to
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
32 have an up-to-date database (and a backup in case of troubles), then activate the virtual
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
33 environment where Libervia is installed (Alembic needs to access ORM mapping), go to
4077
d6837db456fd refactoring: fix names in doc following modules hierarchy refactoring
Goffi <goffi@goffi.org>
parents: 4037
diff changeset
34 ``libervia/backend/memory/migration`` directory, and enter the following command::
3606
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
35
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
36 alembic revision --autogenerate -m "some revision message"
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
37
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
38 This will create a base migration file in ``versions`` directory. Adapt it to your needs,
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
39 try to create both ``upgrade`` and ``downgrade`` method whenever possible, and be sure to
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
40 test it in both directions (``alembic upgrade head`` and ``alembic downgrade
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
41 <previous_revision>``). Please check Alembic documentation for more details.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
42
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
43 .. _SQLALchemy: https://www.sqlalchemy.org/
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
44 .. _Alembic: https://alembic.sqlalchemy.org/
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
45 .. _SQLite: https://sqlite.org
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
46
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
47 Pubsub Cache
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
48 ============
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
49
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
50 There is an internal cache for pubsub nodes and items, which is done in
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
51 ``plugin_pubsub_cache``. The ``PubsubNode`` and ``PubsubItem`` class are the one mapping
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
52 the database.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
53
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
54 The cache is operated transparently to end-user, when a pubsub request is done, it uses a
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
55 trigger to check if the requested node is or must be cached, and if possible returns
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
56 result directly from database, otherwise it lets the normal workflow continue and query the
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
57 pubsub service.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
58
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
59 To save resources, not all nodes are fully cached. When a node is checked, a series of
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
60 analysers are checked, and the first one matching is used to determine if the node must be
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
61 synchronised or not.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
62
4037
524856bd7b19 massive refactoring to switch from camelCase to snake_case:
Goffi <goffi@goffi.org>
parents: 3828
diff changeset
63 Analysers can be registered by any plugins using ``register_analyser`` method:
3606
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
64
4077
d6837db456fd refactoring: fix names in doc following modules hierarchy refactoring
Goffi <goffi@goffi.org>
parents: 4037
diff changeset
65 .. automethod:: libervia.backend.plugins.plugin_pubsub_cache.PubsubCache.register_analyser
3606
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
66
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
67 If no analyser is found, ``to_sync`` is false, or an error happens during the caching,
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
68 the node won't be synchronised and the pubsub service will always be requested.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
69
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
70 Specifying an optional **parser** will store parsed data in addition to the raw XML of the
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
71 items. This is more space consuming, but may be desired for the following reasons:
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
72
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
73 * the parsing is resource consuming (network call or some CPU intensive operations are
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
74 done)
3828
432aaba0d7fe doc (developer): typo
Goffi <goffi@goffi.org>
parents: 3606
diff changeset
75 * it is desirable to do queries on parsed data. Indeed the parsed data are stored in a
3606
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
76 JSON_ field and its keys may be queried individually.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
77
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
78 The Raw XML is kept as the cache operates transparently, and a plugin may need raw data, or
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
79 an user may be doing a low-level pubsub request.
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
80
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
81 .. _JSON: https://docs.sqlalchemy.org/en/14/core/type_basics.html#sqlalchemy.types.JSON
25d3677701e7 doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff changeset
82