.. _components:

===================
Libervia Components
===================

Libervia can act as an XMPP server component, which can be seen as a generic plugin for
XMPP servers.

This page explains which components are available and how to use them.

Running a component
===================

Components are linked to a Libervia profile in the same way as normal clients.

To run a component, you'll need to know its *entry point*, which is the name of the import
name of plugin managing it. The entry point to use will be specified in the component
installation documentation.

You'll also have to declare the component on your XMPP server, this is a server dependent
step and you'll have to check your server documentation for details. You'll have to
specify a **shared secret** (can also be named simply *password*) that must be set both on
the XMPP server and as the XMPP password of the Libervia profile.

Here is a list of relevant documentation for most common servers:

ejabberd
  https://docs.ejabberd.im/admin/configuration/listen-options/

MongooseIm
  https://esl.github.io/MongooseDocs/latest/configuration/listen/#xmpp-components-listenservice

OpenFire
  use the web-based admin panel

Prosody
  https://prosody.im/doc/components

Tigase
  https://docs.tigase.net/tigase-server/stable-snapshot/Administration_Guide/webhelp/externalComponentConfiguration.html


On Libervia, setup is done with Libervia CLI's :ref:`profile create <li_profile_create>`
command.

You'll usually want to have the component to start automatically when the backend
is started, for this you must unset the profile password (not to be confused with the XMPP
password which is the one also set on the server configuration) with ``-p ""`` and set
auto-connection with ``-A``.

You'll specify the XMPP password (also named *shared secret* in `XEP-0144`_ terminology)
with ``-x <your_shared_secret>`` and the JID to use with ``-j
<component_subdomain>.<server.tld>``.

The component entry point is specified with ``-C <entry_point>``.

.. _XEP-0144: https://xmpp.org/extensions/xep-0114.html

example
-------

Louise wants to run an ActivityPub gateway on her server ``example.org`` with the JID
``ap.example.org``. The shared secret is ``xmpp_rocks`` and she wants the component to
start automatically with the backend, thus she doesn't set a profile password. The
entry-point for ActivityPub component is ``ap-gateway``, and she wants to use the same
name for the profile. To do this, she enters the following command::

  $ li profile create ap-gateway -j ap.example.org -p "" -x xmpp_rocks -C ap-gateway -A

The component will then be started next time Libervia Backend is launched. If Louise
wants to connect it immediately, she can use::

  $ li profile connect -cp ap-gateway

Available Components
====================

Below is a list of currently available components in Libervia, and instructions on what
they do and how to use them.


File Sharing
------------

**entry_point:** ``file-sharing``

File Sharing component manage the hosting of user files. Users can upload file there using
either `Jingle File Transfer`_ or `HTTP File Upload`_.

There is no limit to the size of files which can be uploaded, but administrators can set a
quota to limit the space that can be used.

Files can be retrieved using `File Information Sharing`_, and deleted using `Ad-Hoc Commands`_.

Files can be shared with a public HTTP link, or made available only to a specified list of
entities (JIDs). Permissions can be set through Ad-Hoc Commands.

.. _Jingle File Transfer: https://xmpp.org/extensions/
.. _HTTP File Upload: https://xmpp.org/extensions/xep-0363.html
.. _File Information Sharing: https://xmpp.org/extensions/xep-0329.html
.. _Ad-Hoc Commands: https://xmpp.org/extensions/xep-0050.html

Configuration
~~~~~~~~~~~~~

All options are to be set in ``[component file-sharing]`` section.

``http_upload_port``
  port to use for HTTP File Upload

  **default**: 8888

``http_upload_connection_type``
  either ``http`` or ``https``.

  **default**: ``https``

  Note that HTTP Upload should always be ``https`` to end-user, the ``http`` option is to
  be used only if you use a HTTP server as a proxy, and this server is already set for
  TLS.

``http_upload_public_facing_url``
  must be set to the URL that end-user will see. Notably useful if the component is behind
  a proxy.

  **default**: ``https://<component host>:<http_upload_port``

``quotas_json``
  a JSON object indicating quotas to use for users. The object can have 3 keys:

  ``admins``
    quotas to use for administrators (i.e. profiles set in ``admins_list``)

  ``users``
    quotas to use for normal users (i.e. non admin profiles)

  ``jids``
    per-jid specific quotas. The value is a JSON object where key is a user bare jid and
    value is a quota.

  Quotas can be either ``null`` for unlimited space, or a size value (`SI prefixes and
  binary prefixes`_ can be used).

  example::

    quotas_json = {
      "admins": null,
      "users": "50 Mio",
      "jids": {"pierre@example.org": "1 Gio"}
    }

  .. _SI prefixes and binary prefixes: https://en.wikipedia.org/wiki/Octet_(computing)#Unit_multiples


ActivityPub Gateway
-------------------

**entry_point:** ``ap-gateway``

.. note::

  this component is currently in active development, and not yet fully functional. This
  documentation will be updated during evolution of component.

  You can follow the development by reading `Libervia Progress Notes`_.

  .. _Libervia Progress Notes: https://www.goffi.org/tag/Libervia%20progress

This gateway will provide a bidirectional gateway between XMPP and `ActivityPub`_ (or AP
below). That means that user from XMPP will be able to follow actors or comments messages
from any software compatible with ActivityPub protocol, and vice versa.

.. _ActivityPub: https://activitypub.rocks/

.. note::

  this component is mostly tested with Prosody as XMPP server reference, and Mastodon as
  AP server reference, but it should work with any XMPP or AP server.

The component launches a HTTP server (necessary to communicate with AP software). This
server needs to handle HTTP requests made at paths ``/.well-known/webfinger`` and ``/_ap``
(or the ``ap_path`` set in configuration, see below). If the component is not directly
facing internet (e.g. integrated in an existing website though a proxy), you'll have to
redirect the requests made to those path to the HTTP server (i.e. to component host at the
port set at ``http_port``, see configuration below). Please check your HTTP server
documentation to find how this must be done.


Configuration
~~~~~~~~~~~~~

All options are to be set in ``[component ap-gateway]`` section.

``public_url``
  Main user-facing domain of the HTTP server, this will be used to construct all AP URLs

  **default**: if not set, ``xmpp_domain`` is used. If ``xmpp_domain`` is not set either,
  an error is raised.

``http_port``
  port where the HTTP server should listen. Port ``80`` is not used directly as it would
  require root privileges, and it is strongly recommended against launching Libervia under
  a privileged account. An HTTP Proxy or a port redirection should be set to redirect the
  ``80`` port to the port specified here.

  **default**: ``8123``

``http_connection_type``
  either ``http`` or ``https``. If you have a HTTP proxy such as Apache or NGINX which
  already handles HTTPS, you may want to use ``http``.

  **default**: ``https``

  Note that the HTTP server should always use ``https`` with end-user, the ``http`` option
  is only to be used with an HTTPS proxy.

``local_only``
  A boolean value indicating if the gateway is allowed to convert pubsub node from
  external XMPP service or not. A JID is considered external if its domain part doesn't
  end with the gateway's server. For instance, if a gateway ``ap.example.org`` is set on
  the server ``example.org``, the JIDs ``pierre@example.org`` or
  ``some-node@pubsub.example.org`` will be considered local, but
  ``buenaventura@example.net`` won't (note the different domain).

  Most of time, ``local_only`` should be used.

  **default**: ``true``

``ap_path``
  Path prefix to use for ActivityPub request. It's usually not necessary to change the
  default value.

  **default**: ``_ap``

``comments_max_depth``
  An integer value indicating the maximum number of comment nodes that can be created. See
  :ref:`ap-xmpp-threads-conversion`


.. _ap-actor-from-xmpp:

How to Address an AP Actor from XMPP
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

When addressing an ActivityPub actor from XMPP, you must use a JID corresponding to the
actor. The domain part of the JID correspond to the gateway JID (the one set in gateway
profile), while the local part (before the ``@``) is used to specify the AP actor.

`XEP-0106`_ (JID Escaping) is used to indicate the AP actor identifier, thus the ``@``
must be escaped with ``\40``.

.. _XEP-0106: JID Escaping

**example**

If Louise wants to talk to Pierre which is on the ``example.net`` AP server, she can use
her XMPP AP gateway which is at ``ap.example.org``. Pierre AP's actor identifier is
``pierre@example.net``, Louise can access it via the JID
``pierre\40example.net@ap.example.org``.

Of course, this is a bit cumbersome to do by hand, it is expected that XMPP clients will
do the (un)escaping automatically for end-user, in a way that Louise could enter
``pierre@example.net`` directly, with an indicator to show that this is an ActivityPub
actor identifier rather than an XMPP JID.


.. _xmpp-node-from-ap:

How to Address an XMPP Entity from AP
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To access an XMPP entity, it is a little bit more complicated for 2 reasons:

- XMPP use a wider range of allowed characters than most AP implementations [#AP_chars]_.
- to get XMPP items, we need 2 data: the entity JID, and a pubsub node


However, Libervia AP gateway tries to make it as user friendly as possible, thus it works
like this:

- in the most common case, one just wants to access the personal blog of a user, and basic
  ASCII characters (with possibly ``-``, ``_`` or ``.``) are used. in this case, the XMPP
  JID can be directly used as AP actor handle
- when a pubsub node needs to be specified it is most of time with a pubsub JID which has
  not user part (something like ``pubsub.example.org``). In this case, the pubsub node can
  be used as AP actor handle's user part, Libervia will use XMPP discovery to know that
  it's a pubsub service. So if you want to access the blog named ``xmpp_news`` at
  ``pubsub.example.org``, you can use the handle ``xmpp_news@pubsub.example.org`` (be sure
  that the domain ``pubsub.example.org`` links to the Libervia AP gateway HTTP server)
- if you want to use a specific node with an entity which already has a user part, then a
  special encoding must be used, where ``---`` (three dashes) are used to separate node
  from entity: ``some_node--some_user@example.org``
- if you need to use special characters, then you'll have to use ``___`` followed by the
  special encoding (see below).

The encoding is explained in the documentation of the following method:

.. automethod:: sat.plugins.plugin_comp_ap_gateway.APGateway.getJIDAndNode


.. [#AP_chars] Most if not all AP implementations use webfinger `acct` URI as a de-facto
   standard to manage user-friendly handles (something like ``user@example.org``). Those
   handles, according to `RFC7565`_, should
   manage a wide variety of characters thanks to the support of percent-encoding.

   Unfortunately, at least one of the most-used AP implementation (Mastodon, which is used
   a reference implementation for this gateway), only uses a limited subset of allowed
   characters. In addition, Mastodon needs an associated handle [#m_wf]_ thus an alternate
   way to encode characters had to be found.

   An issue has been opened to improve this situation on Mastodon bug tracker:
   https://github.com/mastodon/mastodon/issues/17222

   .. _RFC7565: https://datatracker.ietf.org/doc/html/rfc7565
   .. [#m_wf] https://docs.joinmastodon.org/spec/webfinger/

**example**

If Pierre wants to talk to Louise, he can directly use the JID which is the same as the AP
actor identifier, i.e. ``louise@example.org`` (note that on AP software, a ``@`` prefix is
often required, thus Pierre will look for ``@louise@example.org``).

.. note::

   The actor endpoint can also be used directly in AP software (in the example above, it
   would be by default ``https://example.org/_ap/actor/louise%40example.org``).


Getting AP Message from XMPP
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To retrieve ActivityPub messages of an actor from an XMPP client with blogging
capabilities (like Libervia or Movim), just use the associated JID as explained in
:ref:`ap-actor-from-xmpp`. The actor messages (from ``outbox`` collection) should appear
as regular XMPP blog items.

.. note::

   Due to limitation of `ActivityStream Collection Paging`_, the conversion from XMPP
   `RSM`_ requests is inneficient beyond first or last page. This problem is avoided if
   anybody subscribe to the gateway node (i.e. follow the AP actor), as the collection
   will then be cached, and efficiently delivered.

   .. _ActivityStream Collection Paging: https://www.w3.org/TR/activitystreams-core/#h-paging
   .. _RSM: https://xmpp.org/extensions/xep-0059.html

Getting XMPP Items from ActivityPub
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To get XMPP items from an ActivityPub implementation, just use the handle as explained at
:ref:`xmpp-node-from-ap` (often handle searches are triggered with a ``@`` before the
handle in AP implementations, e.g.: ``@louise@example.org``).

.. note::

   Some AP implementations such as Mastodon don't retrieve existing items, but only keep
   new ones once an actor is followed. That means that you won't see any message published
   before your entity is followed. Other implementations may work differently.


.. _ap-xmpp-threads-conversion:

ActivyPub to XMPP Discussion Threads Conversion
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

By default, each (micro)blog publication converted from ActivityPub to XMPP Pubsub is
associated with a comment node (see `XEP-0277 comments`_) to allow user to post a reply on
any on them. This result in a "tree-like" comments threading model, which is similar to
what you can see on popular website such as Reddit, Hacker News, or LinuxFr.

However, some XMPP clients may not play nicely with this kind of threading model. To work
around this, the ``comments_max_depth`` option allows to limit the maximum number of
thread. It is an integer value which indicate how many comments nodes may exist for a root
post. If set to ``1``, only one comment node will be made, and ActivityPub items below
this level will be moved to this single XMPP pubsub comment node.

The default value of ``0`` means unlimited max depth.

An example can probably make it more clear. Imagine that you have a message ``root
message``, with a comment to it named ``1``, a comment to ``1`` named ``2`` and so on
until ``5``. With ``comments_max_depth`` set to ``0``, you'll have one comment node per
item, resulting in following threads model::

  root message ┑
               ┕ 1 ┑
                   ┕ 2 ┑
                       ┕ 3 ┑
                           ┕ 4 ┑
                               ┕ 5

With ``comments_max_depth`` set to ``2``, only 2 nodes will be created, and items below
depth ``2`` will be put on the same level::

  root message ┑
               ┕ 1 ┑
                   ┝ 2
                   ┝ 3
                   ┝ 4
                   ┕ 5

This way, admins can configure the model which suits best the clients which is expected to
be mainly used on the instance.

.. _XEP-0277 comments: https://xmpp.org/extensions/xep-0277.html#comments

Publishing an Item
~~~~~~~~~~~~~~~~~~

To publish a new item (e.g. a blog post), you just need to publish normally on your own
PEP/pubsub node, AP actors following you will be notified. To reply to an AP item, just
publish to the corresponding pubsub node managed by the gateway. This is transparent for
AP and XMPP end users.

For instance, if Pierre has posted an interesting message on his AP server, and Louise
wants to reply to it, she just use a client to reply on the comments node of this message,
this will be delivered as an AP object to Pierre's AP server.

On the other hand, if Louise is publishing a new blog post on her XMPP server, Pierre will
receive corresponding AP object because he's following her. If Pierre answer using his AP
client, the corresponding message will be published on the comments node of the message
that Louise has initially published.

Following, Subscribing and Cache
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

When you try to access an uncached AP collection from XMPP (e.g. blog posts), a best
effort is done to translate XMPP pagination (which uses `XEP-0059 (Result Set
Management)`_) to the less powerful `AP Collection Paging`_. This is inefficient due to
technical limitations (page size can't be specified in AP, there is not standard way to
request item after or before a specific ID, implementations may not implement reverse
pagination).

That's one of the reason why whenever possible, collections are cached locally. Once
cached, it's easier to return items according to complex requests.

However, to cache correctly an AP collection, you need to keep it in sync, and thus to
receive update when something change (e.g. a new blog item is published).

In AP, this is done by following an actor, in XMPP this correspond to a node subscription.

When you subscribe to a blog node managed by this gateway, this will be translated to a
*follow* activity on AP side, and vice versa.

When an AP actor is followed, its *outbox* collection (i.e. message published), are
automatically cached, and will be updated when events will be received. That means that
you can use pubsub cache search on followed actors, e.g. to retrieve all items about a
specific topic or published at specific time range.

Reciprocally, unsubscribing from a node will *unfollow* the corresponding AP actor.

If an AP actor is following or unfollowing an actor mapping an XMPP entity, they nodes
will be subscribed to or unsubscribed from.

All subscriptions are made public as specified by `XEP-0465 (Pubsub Public Subscriptions)`_.

.. _XEP-0059 (Result Set Management): https://xmpp.org/extensions/xep-0059.html
.. _AP Collection Paging: https://www.w3.org/TR/activitystreams-core/#h-paging
.. _XEP-0465 (Pubsub Public Subscriptions): https://xmpp.org/extensions/inbox/pubsub-public-subscriptions.html

Following/Followers Collections and Public Pubsub Subscription
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The AP *following* collection is mapped to `XEP-0465 (Pubsub Public Subscriptions)`_.

In the same spirit, the AP *followers* collection correspond to public subscribers to the
microblog node.

Because AP doesn't send any event when *following* or *followers* collections are
modified, those collections can't be cached, and thus the translation to public pubsub
subscriptions is done as best as possible given the constraints.

Using the Component (for developers)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Publication of AP items can be tested using the following method (with can be accessed
through the ``APSend`` bridge method, client is then replaced by the ``profile`` name, as
last argument):

.. automethod:: sat.plugins.plugin_comp_ap_gateway.APGateway.publishMessage

The method can be used either with CLI's :ref:`debug bridge method
<li_debug_bridge_method>` or with any D-Bus tool like ``qdbus`` or ``d-feet`` (only if you
use the D-Bus bridge).

example
~~~~~~~

On its ``example.net`` Mastodon instance, Pierre has published a message with the id
``https://example.net/@pierre/106986412193109832``. To send a reply to this message,
Louise can use the following command::

  $ li debug bridge method -c APSend '"{\"node\": \"https://example.net/@pierre/106986412193109832\", \"content\": \"A lille hello from XMPP\"}","pierre\\40example.net@ap.example.org", "louise"'

Note the double escaping, one for the shell argument, and the other to specify JSON
object.