.. _libervia-cli_pubsub_cache:

=====================================
pubsub/cache: PubSub Cache Management
=====================================

Libervia runs transparently a cache for pubsub. That means that according to internal
criteria, some pubsub items are stored locally.

The ``cache`` subcommands let user inspect and manipulate the internal cache.

get
===

Retrieve items from internal cache only. Most end-users won't need to use this command, as
the usual ``pubsub get`` command will use cache transparently. However, it may be useful
to inspect local cache, notably for debugging.

The parameters are basically the same as for :ref:`li_pubsub_get`.

example
-------

Retrieve the last 2 cached items for personal blog::

    $ li pubsub cache get -n urn:xmpp:microblog:0 -M 2

.. _li_pubsub_cache_sync:

sync
====

Synchronise or resynchronise a pubsub node. If the node is already in cache, it will be
deleted then re-cached. Node will be put in cache even if internal policy doesn't request
a synchronisation for this kind of nodes. Node will be (re-)subscribed to keep cache
synchronised.

All items of the node (up to the internal limit which is high), will be retrieved and put
in cache, even if a previous version of those items have been deleted by the
:ref:`li_pubsub_cache_purge` command.


example
-------

Resynchronise personal blog::

    $ li pubusb cache sync -n urn:xmpp:microblog:0

.. _li_pubsub_cache_purge:

purge
=====

Remove items from cache. This may be desirable to save resource, notably disk space.

Note that once a pubsub node is cached, the cache is the source of trust. That means that
if cache is not explicitly bypassed when retrieving items of a pubsub node (notably with
the ``-C, --no-cache`` option of :ref:`li_pubsub_get`), only items found in cache will be
returned, thus purged items won't be used or returned anymore even if they still exists on
the original pubsub service.

If you have purged items by mistake, it is possible to retrieve them either node by node
using :ref:`li_pubsub_cache_sync`, or by resetting the whole pubsub cache with
:ref:`li_pubsub_cache_reset`.

If you have a node or a profile (e.g. a component) caching a lot of items frequently, you
may use this command using a scheduler like cron_.

.. _cron: https://en.wikipedia.org/wiki/Cron

examples
--------

Remove all blog and event items from cache if they haven't been updated since 6 months::

    $ li pubsub cache purge -t blog -t event -b "6 months ago"

Remove items from profile ``ap_gateway`` if they have been created more that 2 months
ago::

    $ li pubsub cache purge -p ap_gateway --created-before "2 months ago"

.. _li_pubsub_cache_reset:

reset
=====

Reset the whole pubsub cache. This means that all nodes and all them items will be removed
from cache. After this command, cache will be re-filled progressively as if it where a new
one.

.. note::

    Use this command with caution: even if cache will be re-constructed with time, that
    means that items will have to be retrieved again, that may be resource intensive both
    for your machine and for the pubsub services which will be used. That also means that
    searching items will return less results until all desired items are cached again.

    Also note that all items of cached nodes are retrieved, even if you have previously
    purged items, they will be retrieved again.

example
-------

Reset the whole pubsub cache::

    $ li pubsub cache reset

search
======

Search items into pubsub cache. The search is done on the whole cache, it's not restricted
to a single node/profile (even if it may be if suitable filters are specified). Full-Text
Search can be done with ``-f FTS, --fts FTS`` argument, as well as filtering on parsed
data (with ``-F PATH OPERATOR VALUE, --field PATH OPERATOR VALUE``, see below).

By default, parsed data are returned, with the 3 additional keys ``pubsub_service``,
``pubsub_items`` (the search being done on the whole cache, those data are here to get the
full location of each item) and ``node_profile``.

"Parsed data" are the result of the parsing of the items XML payload by feature aware
plugins. Those data are usually more readable and easier to work with. Parsed data are
only stored when a parser is registered for a specific feature, that means that a Pubsub
item in cache may not have parsed data at all, in which case an empty dict will be used
instead (and ``-P, --payload`` argument should be used to get content of the item).

The dates are normally stored as `Unix time`_ in database, but the default output convert
the ``updated``, ``created`` and ``published`` fields to human readable local time. Use
``--output simple`` if you want to keep the float (or int) value.

XML item payload is not returned by default, but it can be added to the ``item_payload``
field if ``-P, --payload`` argument is set. You can also use the ``--output xml`` (or
``xml_raw`` if you don't want prettifying) to output directly the highlighted XML
— without the parsed data —, to have an output similar to the one of ``li pubsub get``.

If you are interested only in a specific data (e.g. item id and title), the ``-k KEY,
--key KEY`` can be used.

You'll probably want to limit result size by using ``-l LIMIT, --limit LIMIT``, and do
pagination using ``-i INDEX, --index INDEX``.

.. _Unix time: https://en.wikipedia.org/wiki/Unix_time

Filters
-------

By default search returns all items in cache, you have to use filter to specify what you
are looking after. We can split filters in 3 categories: nodes/items metadata,
Full-Text Search query and parsed metadata.

Nodes/items metadata are the generic information you have on a node: which profile it
belong too, which pubsub service it's coming from, what's the name or type of the node,
etc.

Arguments there should be self-explanatory. Type (set with ``-t TYPE, --type TYPE``) and
subtype (set with ``-S SUBTYPE, --subtype SUBTYPE``) are values dependent of the
plugin/feature associated with the node, so we can't list them in an exhaustive way here.
The most common type is probably ``blog``, from which a subtype can be ``comment``. An
empty string can be used to find items with (sub)type not set.

It's usually a good idea to specify a profile with ``-p PROFILE, --profile PROFILE``,
otherwise you may get duplicated results.

Full-Text Search
----------------

You can specify a Full-Text Search query with the ``-f FTS_QUERY, --fts FTS_QUERY``
argument. The engine is currently SQLite FTS5, and you can check its `query syntax`_.
FTS is done on the whole raw XML payload, that means that all data there can be matched
(including XML tags and attributes).

FTS queries are indexed, that means that they are fast and efficient.

.. note::

  Futures version of Libervia will probably include other FTS engines (support for
  PostgreSQL and MySQL/MariaDB is planned). Thus the syntax may vary depending on the
  engine, or a common syntax may be implemented for all engines in the future. Keep that
  in mind if you plan to use FTS capabilities in long-term queries, e.g. in scripts.

.. _query syntax: https://sqlite.org/fts5.html#full_text_query_syntax

Parsed Metadata Filters
-----------------------

It is possible to filter on any field of parsed data. This is done with the ``-F PATH
OPERATOR VALUE, --field PATH OPERATOR VALUE`` (be careful that the short option is an
uppercase ``F``, the lower case one being used for Full-Text Search).

.. note::

  Parsed Metadata Filters are not indexed, that means that using them is less efficient
  than using e.g. Full-Text Search. If you want to filter on a text field, it's often a
  good idea to pre-filter using Full-Text Search to have more efficient queries.

``PATH`` and ``VALUE`` can be either specified as string, or using JSON syntax (if the
value can't be decoded as JSON, it is used as plain text).

``PATH`` is the name of the field to use. If you must go beyond root level fields, you can
use a JSON array to specify each element of the path. If a string is used, it's an object
key, if a number is used it's an array index. Thus you can use ``title`` to access the
root title key, or ``'"title"'`` (JSON string escaped for shell) or ``'["title"]'`` (JSON
array with the "title" string, escaped for shell).

.. note::

  The extra fields ``pubsub_service``, ``pubsub_node`` and  ``node_profile`` are added to
  the result after the query, thus they can't be used as fields for filtering (use the
  direct arguments for that).

``OPERATOR`` indicate how to use the value to make a filter. The currently supported
operators are:

``==`` or ``eq``
  Equality operator, true if field value is the same as given value.

``!=`` or ``ne``
  Inequality operator, true if the field value is different from given value.

``>`` or ``gt``
  Greater than, true if the field value is higher than given value. For string, this is
  according to alphabetical order.

  Time Pattern can be used here, see below.

``<`` or ``lt``
  Lesser than, true if the field value is lower than given value. For string, this is
  according to alphabetical order.

  Time Pattern can be used here, see below.

``between``
  Given value must be an array with 2 elements. The condition is true if field value is
  between the 2 elements (for string, this is according to alphabetical order).

  Time Pattern can be used here, see below.

``in``
  Given value must be an array of elements. Field value must be one of them to make the
  condition true.

``not_in``
  Given value must be an array of elements. Field value must not be any of them the make
  the condition true.

``overlap``
  This can be used only on array fields.

  If given value is not already an array, it is put in an array. Condition is true if any
  element of field value match any element of given value. Notably useful to filter on
  tags.

``ioverlap``
  Same as ``overlap`` but done in a case insensitive way.

``disjoint``
  This can be used only on array fields.

  If given value is not already an array, it is put in an array. Condition is true if no
  element of field value match any element of given value. Notably useful to filter out
  tags.

``idisjoint``
  Same as ``disjoint`` but done in a case insensitive way.

``like``
  Does pattern matching on a string. ``%`` can be used to match zero or more characters
  and ``_`` can be used to match any single character.

  If you're not looking after a specific field, it's better to use Full-Text Search when
  possible.

``ilike``
  Like ``like`` but done in a case insensitive way.


``not_like``
  Same as ``like`` except that condition is true when pattern is **not** matching.

``not_ilike``
  Same as ``not_like`` but done in a case insensitive way.


For ``gt``/``>``, ``lt``/``<`` and ``between``, you can use :ref:`time_pattern` by using
the syntax ``TP(<time pattern>)`` (see examples below).

Ordering
--------

Result ordering can be done by a well know order, or using a parsed data field. Ordering
default to ``created`` (see below), but this may be changed with ``-o ORDER [FIELD]
[DIRECTION], --order-by ORDER [FIELD] [DIRECTION]``.

``ORDER`` can be one of the following:

``creation``
  Order by item creation date. Note that is this the date of creation of the item in cache
  (which most of time should correspond to order of creation of the item in the source
  pubsub service), and this may differ from the date of publication as specified with some
  feature (like blog). This is important when old items are imported, e.g. when they're
  coming from an other blog engine.

``modification``
  Order by the date when item has last been modified. Modification date is the same as
  creation date if the item has never been modified since it is in cache. The same warning
  as for ``creation`` applies: this is the date of last modification in cache, not the one
  advertised in parsed data.

``item_id``
  Order by XMPP id of the item. Notably useful when user-friendly ID are used (like it is
  often the case with blogs).

``rank``
  Order item by Full-Text Search rank. This one can only be used when Full-Text Search is
  used (via ``-f FTS_QUERY, --fts FTS_QUERY``). Rank is a value indicating how well an
  item match the query. This usually needs to be used with ``desc`` direction, so you get
  most relevant items first.

``field``
  This special order indicates that the ordering must be done on an parsed data field. The
  following argument is then the path of the field to used (which can be a plain text name
  of a root field, or a JSON encoded array). An optional direction can be specified as a
  third argument. See examples below.

examples
--------

Search for blog items cached for the profile ``louise`` which contain the word
``Slovakia``::

  $ li pubsub cache search -t blog -p louise -f Slovakia

Show title, publication date and id of blog articles (excluding comments) which have been
published on Louise's blog during the last 6 months, order them by item id. Here we use an
empty string as a subtype to exclude comments (for which subtype is ``comment``)::

  $ li pubsub cache search -t blog -S "" -p louise -s louise@example.net -n urn:xmpp:microblog:0 -F published gt 'TP(6 months ago)' -k id -k published -k title -o item_id

Show all blog items from anywhere which are tagged as XMPP or ActivityPub (case
insensitive) and which have been published in the last month (according to advertised
publishing date, not cache creation date).

We want to order them by descending publication date (again the advertised publication
date, not cache creation), and we don't want more than 50 results.

We do a FTS query there even if it's not mandatory, because it will do an efficient
pre-filtering::

  $ li pubsub cache search -f "xmpp OR activitypub" -F tags ioverlap '["xmpp", "activitypub"]' -F published gt 'TP(1 month ago)' -o field published desc -l 50