Mercurial > libervia-backend
changeset 3606:25d3677701e7
doc: developer documentation explaining storage and pubsub cache
author | Goffi <goffi@goffi.org> |
---|---|
date | Thu, 29 Jul 2021 22:51:01 +0200 (2021-07-29) |
parents | 62e81b1058f8 |
children | bc043afedbac |
files | doc/developer.rst doc/index.rst |
diffstat | 2 files changed, 83 insertions(+), 0 deletions(-) [+] |
line wrap: on
line diff
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/doc/developer.rst Thu Jul 29 22:51:01 2021 +0200 @@ -0,0 +1,82 @@ +.. _developer: + +======================= +Developer Documentation +======================= + +This documentation is intended for people who wants to contribute or work with the +internals of the project, it is not for end-users. + +Storage +======= + +Since version 0.9, Libervia uses SQLAlchemy_ with its Object–Relational Mapping as a +backend to store persistent data, and Alembic_ is used to handle schema and data +migrations. + +SQLite_ is currently the only supported database, but it is planned to add support for +other ones (notably PostgreSQL), probably during the development of 0.9 version. + +The mapping is done in ``sat.memory.sqla_mapping`` and working with database is done +through high level methods found in ``sat.memory.sqla``. + +Before the move to SQLAlchemy, there was a strict separation between database +implementation and the rest of the code. With 0.9, objects mapping to database can be used +and manipulated directly outside of ``sat.memory.sqla`` to take profit of SQLAlchemy +possibilities. + +Database state is detected when the backend starts, and the database will be created or +migrated automatically if necessary. + +To create a new migration script, ``Alembic`` may be used directly. To do so, be sure to +have an up-to-date database (and a backup in case of troubles), then activate the virtual +environment where Libervia is installed (Alembic needs to access ORM mapping), go to +``sat/memory/migration`` directory, and enter the following command:: + + alembic revision --autogenerate -m "some revision message" + +This will create a base migration file in ``versions`` directory. Adapt it to your needs, +try to create both ``upgrade`` and ``downgrade`` method whenever possible, and be sure to +test it in both directions (``alembic upgrade head`` and ``alembic downgrade +<previous_revision>``). Please check Alembic documentation for more details. + +.. _SQLALchemy: https://www.sqlalchemy.org/ +.. _Alembic: https://alembic.sqlalchemy.org/ +.. _SQLite: https://sqlite.org + +Pubsub Cache +============ + +There is an internal cache for pubsub nodes and items, which is done in +``plugin_pubsub_cache``. The ``PubsubNode`` and ``PubsubItem`` class are the one mapping +the database. + +The cache is operated transparently to end-user, when a pubsub request is done, it uses a +trigger to check if the requested node is or must be cached, and if possible returns +result directly from database, otherwise it lets the normal workflow continue and query the +pubsub service. + +To save resources, not all nodes are fully cached. When a node is checked, a series of +analysers are checked, and the first one matching is used to determine if the node must be +synchronised or not. + +Analysers can be registered by any plugins using ``registerAnalyser`` method: + +.. automethod:: sat.plugins.plugin_pubsub_cache.PubsubCache.registerAnalyser + +If no analyser is found, ``to_sync`` is false, or an error happens during the caching, +the node won't be synchronised and the pubsub service will always be requested. + +Specifying an optional **parser** will store parsed data in addition to the raw XML of the +items. This is more space consuming, but may be desired for the following reasons: + +* the parsing is resource consuming (network call or some CPU intensive operations are + done) +* it is desirable to to queries on parsed data. Indeed the parsed data is stored in a + JSON_ field and its keys may be queried individually. + +The Raw XML is kept as the cache operates transparently, and a plugin may need raw data, or +an user may be doing a low-level pubsub request. + +.. _JSON: https://docs.sqlalchemy.org/en/14/core/type_basics.html#sqlalchemy.types.JSON +