Mercurial > libervia-backend
annotate doc/developer.rst @ 4094:c3b68fdc2de7
component AP gateway: fix handling of XMPP comments authors:
the gateway was supposing that comments where emitted from PEP of author. While this is
the case for most blog posts, it's not for comments. Instead the component is now using
`author_jid` which is retrieved by XEP-0277 plugin, and reject the item if the auhor is
not verified (i.e. if `publisher` attribute is not set by XMPP service).
author | Goffi <goffi@goffi.org> |
---|---|
date | Mon, 12 Jun 2023 14:50:43 +0200 |
parents | d6837db456fd |
children | 00852dd54695 |
rev | line source |
---|---|
3606
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
1 .. _developer: |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
2 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
3 ======================= |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
4 Developer Documentation |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
5 ======================= |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
6 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
7 This documentation is intended for people who wants to contribute or work with the |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
8 internals of the project, it is not for end-users. |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
9 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
10 Storage |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
11 ======= |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
12 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
13 Since version 0.9, Libervia uses SQLAlchemy_ with its Object–Relational Mapping as a |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
14 backend to store persistent data, and Alembic_ is used to handle schema and data |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
15 migrations. |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
16 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
17 SQLite_ is currently the only supported database, but it is planned to add support for |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
18 other ones (notably PostgreSQL), probably during the development of 0.9 version. |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
19 |
4077
d6837db456fd
refactoring: fix names in doc following modules hierarchy refactoring
Goffi <goffi@goffi.org>
parents:
4037
diff
changeset
|
20 The mapping is done in ``libervia.backend.memory.sqla_mapping`` and working with database is done |
d6837db456fd
refactoring: fix names in doc following modules hierarchy refactoring
Goffi <goffi@goffi.org>
parents:
4037
diff
changeset
|
21 through high level methods found in ``libervia.backend.memory.sqla``. |
3606
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
22 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
23 Before the move to SQLAlchemy, there was a strict separation between database |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
24 implementation and the rest of the code. With 0.9, objects mapping to database can be used |
4077
d6837db456fd
refactoring: fix names in doc following modules hierarchy refactoring
Goffi <goffi@goffi.org>
parents:
4037
diff
changeset
|
25 and manipulated directly outside of ``libervia.backend.memory.sqla`` to take profit of SQLAlchemy |
3606
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
26 possibilities. |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
27 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
28 Database state is detected when the backend starts, and the database will be created or |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
29 migrated automatically if necessary. |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
30 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
31 To create a new migration script, ``Alembic`` may be used directly. To do so, be sure to |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
32 have an up-to-date database (and a backup in case of troubles), then activate the virtual |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
33 environment where Libervia is installed (Alembic needs to access ORM mapping), go to |
4077
d6837db456fd
refactoring: fix names in doc following modules hierarchy refactoring
Goffi <goffi@goffi.org>
parents:
4037
diff
changeset
|
34 ``libervia/backend/memory/migration`` directory, and enter the following command:: |
3606
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
35 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
36 alembic revision --autogenerate -m "some revision message" |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
37 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
38 This will create a base migration file in ``versions`` directory. Adapt it to your needs, |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
39 try to create both ``upgrade`` and ``downgrade`` method whenever possible, and be sure to |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
40 test it in both directions (``alembic upgrade head`` and ``alembic downgrade |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
41 <previous_revision>``). Please check Alembic documentation for more details. |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
42 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
43 .. _SQLALchemy: https://www.sqlalchemy.org/ |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
44 .. _Alembic: https://alembic.sqlalchemy.org/ |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
45 .. _SQLite: https://sqlite.org |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
46 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
47 Pubsub Cache |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
48 ============ |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
49 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
50 There is an internal cache for pubsub nodes and items, which is done in |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
51 ``plugin_pubsub_cache``. The ``PubsubNode`` and ``PubsubItem`` class are the one mapping |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
52 the database. |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
53 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
54 The cache is operated transparently to end-user, when a pubsub request is done, it uses a |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
55 trigger to check if the requested node is or must be cached, and if possible returns |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
56 result directly from database, otherwise it lets the normal workflow continue and query the |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
57 pubsub service. |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
58 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
59 To save resources, not all nodes are fully cached. When a node is checked, a series of |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
60 analysers are checked, and the first one matching is used to determine if the node must be |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
61 synchronised or not. |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
62 |
4037
524856bd7b19
massive refactoring to switch from camelCase to snake_case:
Goffi <goffi@goffi.org>
parents:
3828
diff
changeset
|
63 Analysers can be registered by any plugins using ``register_analyser`` method: |
3606
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
64 |
4077
d6837db456fd
refactoring: fix names in doc following modules hierarchy refactoring
Goffi <goffi@goffi.org>
parents:
4037
diff
changeset
|
65 .. automethod:: libervia.backend.plugins.plugin_pubsub_cache.PubsubCache.register_analyser |
3606
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
66 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
67 If no analyser is found, ``to_sync`` is false, or an error happens during the caching, |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
68 the node won't be synchronised and the pubsub service will always be requested. |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
69 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
70 Specifying an optional **parser** will store parsed data in addition to the raw XML of the |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
71 items. This is more space consuming, but may be desired for the following reasons: |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
72 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
73 * the parsing is resource consuming (network call or some CPU intensive operations are |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
74 done) |
3828 | 75 * it is desirable to do queries on parsed data. Indeed the parsed data are stored in a |
3606
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
76 JSON_ field and its keys may be queried individually. |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
77 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
78 The Raw XML is kept as the cache operates transparently, and a plugin may need raw data, or |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
79 an user may be doing a low-level pubsub request. |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
80 |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
81 .. _JSON: https://docs.sqlalchemy.org/en/14/core/type_basics.html#sqlalchemy.types.JSON |
25d3677701e7
doc: developer documentation explaining storage and pubsub cache
Goffi <goffi@goffi.org>
parents:
diff
changeset
|
82 |