libervia-backend: sat/plugins/plugin_comp_ap_gateway/__init_

annotate sat/plugins/plugin_comp_ap_gateway/init.py @ 3743:54c249ec35ce

core (memory/migration): ignore FTS table when autogenerating script for migration: SQLite Full-Text Search stable are not associated to Python object and can't be detected by Alembic. To avoid the generation of unwanted drop commands, they are now ignored when autogenerating migration scripts. rel 364

author	Goffi <goffi@goffi.org>
date	Tue, 22 Mar 2022 17:00:42 +0100
parents	bf0505d41c09
children	a8c7e5cef0cb

rev	line source
3682 7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	1 #!/usr/bin/env python3
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	2
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	3 # Libervia ActivityPub Gateway
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	4 # Copyright (C) 2009-2021 Jérôme Poisson (goffi@goffi.org)
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	5
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	6 # This program is free software: you can redistribute it and/or modify
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	7 # it under the terms of the GNU Affero General Public License as published by
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	8 # the Free Software Foundation, either version 3 of the License, or
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	9 # (at your option) any later version.
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	10
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	11 # This program is distributed in the hope that it will be useful,
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	12 # but WITHOUT ANY WARRANTY; without even the implied warranty of
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	13 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	14 # GNU Affero General Public License for more details.
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	15
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	16 # You should have received a copy of the GNU Affero General Public License
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	17 # along with this program. If not, see <http://www.gnu.org/licenses/>.
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	18
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	19 import base64
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	20 import hashlib
3728 b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	21 import json
3682 7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	22 from pathlib import Path
3728 b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	23 from typing import Optional, Dict, Tuple, List, Union
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	24 from urllib import parse
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	25 import calendar
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	26 import re
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	27
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	28 import dateutil
3682 7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	29 from cryptography.hazmat.primitives import serialization
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	30 from cryptography.hazmat.primitives import hashes
3728 b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	31 from cryptography.hazmat.primitives.asymmetric import rsa
3682 7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	32 from cryptography.hazmat.primitives.asymmetric import padding
3728 b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	33 import shortuuid
3682 7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	34 import treq
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	35 from treq.response import _Response as TReqResponse
3728 b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	36 from twisted.internet import defer, reactor, threads
3729 86eea17cafa7 component AP gateway: split plugin in several files: Goffi <goffi@goffi.org> parents: 3728 diff changeset	37 from twisted.web import http
3728 b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	38 from twisted.words.protocols.jabber import jid, error
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	39 from twisted.words.xish import domish
3729 86eea17cafa7 component AP gateway: split plugin in several files: Goffi <goffi@goffi.org> parents: 3728 diff changeset	40 from wokkel import rsm
3728 b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	41
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	42 from sat.core import exceptions
3682 7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	43 from sat.core.constants import Const as C
3728 b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	44 from sat.core.core_types import SatXMPPEntity
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	45 from sat.core.i18n import _
3682 7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	46 from sat.core.log import getLogger
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	47 from sat.tools import utils
3728 b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	48 from sat.tools.common import data_format, tls
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	49 from sat.tools.common.async_utils import async_lru
3729 86eea17cafa7 component AP gateway: split plugin in several files: Goffi <goffi@goffi.org> parents: 3728 diff changeset	50
86eea17cafa7 component AP gateway: split plugin in several files: Goffi <goffi@goffi.org> parents: 3728 diff changeset	51 from .constants import (IMPORT_NAME, CONF_SECTION, TYPE_ACTOR, TYPE_ITEM, MEDIA_TYPE_AP,
86eea17cafa7 component AP gateway: split plugin in several files: Goffi <goffi@goffi.org> parents: 3728 diff changeset	52 AP_MB_MAP, LRU_MAX_SIZE)
86eea17cafa7 component AP gateway: split plugin in several files: Goffi <goffi@goffi.org> parents: 3728 diff changeset	53 from .http_server import HTTPServer
86eea17cafa7 component AP gateway: split plugin in several files: Goffi <goffi@goffi.org> parents: 3728 diff changeset	54 from .pubsub_service import APPubsubService
3682 7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	55
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	56
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	57 log = getLogger(__name__)
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	58
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	59 IMPORT_NAME = "ap-gateway"
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	60
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	61 PLUGIN_INFO = {
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	62 C.PI_NAME: "ActivityPub Gateway component",
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	63 C.PI_IMPORT_NAME: IMPORT_NAME,
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	64 C.PI_MODES: [C.PLUG_MODE_COMPONENT],
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	65 C.PI_TYPE: C.PLUG_TYPE_ENTRY_POINT,
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	66 C.PI_PROTOCOLS: [],
3728 b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	67 C.PI_DEPENDENCIES: ["XEP-0106", "XEP-0277", "XEP-0060"],
3682 7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	68 C.PI_RECOMMENDATIONS: [],
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	69 C.PI_MAIN: "APGateway",
3728 b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	70 C.PI_HANDLER: C.BOOL_TRUE,
3682 7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	71 C.PI_DESCRIPTION: _(
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	72 "Gateway for bidirectional communication between XMPP and ActivityPub."
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	73 ),
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	74 }
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	75
3728 b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	76 HEXA_ENC = r"(?P<hex>[0-9a-fA-f]{2})"
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	77 RE_PERIOD_ENC = re.compile(f"\\.{HEXA_ENC}")
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	78 RE_PERCENT_ENC = re.compile(f"%{HEXA_ENC}")
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	79 RE_ALLOWED_UNQUOTED = re.compile(r"^[a-zA-Z0-9_-]+$")
3682 7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	80
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	81
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	82 class APGateway:
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	83
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	84 def __init__(self, host):
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	85 self.host = host
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	86 self.initialised = False
3728 b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	87 self._m = host.plugins["XEP-0277"]
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	88 self._p = host.plugins["XEP-0060"]
3682 7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	89
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	90 host.bridge.addMethod(
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	91 "APSend",
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	92 ".plugin",
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	93 in_sign="sss",
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	94 out_sign="",
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	95 method=self._publishMessage,
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	96 async_=True,
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	97 )
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	98
3728 b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	99 def getHandler(self, __):
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	100 return APPubsubService(self)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	101
3682 7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	102 async def init(self, client):
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	103 if self.initialised:
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	104 return
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	105
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	106 self.initialised = True
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	107 log.info(_("ActivityPub Gateway initialization"))
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	108
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	109 # RSA keys
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	110 stored_data = await self.host.memory.storage.getPrivates(
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	111 IMPORT_NAME, ["rsa_key"], profile=client.profile
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	112 )
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	113 private_key_pem = stored_data.get("rsa_key")
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	114 if private_key_pem is None:
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	115 self.private_key = await threads.deferToThread(
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	116 rsa.generate_private_key,
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	117 public_exponent=65537,
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	118 key_size=4096,
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	119 )
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	120 private_key_pem = self.private_key.private_bytes(
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	121 encoding=serialization.Encoding.PEM,
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	122 format=serialization.PrivateFormat.PKCS8,
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	123 encryption_algorithm=serialization.NoEncryption()
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	124 ).decode()
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	125 await self.host.memory.storage.setPrivateValue(
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	126 IMPORT_NAME, "rsa_key", private_key_pem, profile=client.profile
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	127 )
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	128 else:
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	129 self.private_key = serialization.load_pem_private_key(
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	130 private_key_pem.encode(),
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	131 password=None,
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	132 )
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	133 self.public_key = self.private_key.public_key()
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	134 self.public_key_pem = self.public_key.public_bytes(
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	135 encoding=serialization.Encoding.PEM,
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	136 format=serialization.PublicFormat.SubjectPublicKeyInfo
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	137 ).decode()
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	138
3728 b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	139 # params
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	140 # URL and port
3682 7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	141 self.public_url = self.host.memory.getConfig(
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	142 CONF_SECTION, "public_url"
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	143 ) or self.host.memory.getConfig(
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	144 CONF_SECTION, "xmpp_domain"
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	145 )
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	146 if self.public_url is None:
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	147 log.error(
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	148 '"public_url" not set in configuration, this is mandatory to have'
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	149 "ActivityPub Gateway running. Please set this option it to public facing "
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	150 f"url in {CONF_SECTION!r} configuration section."
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	151 )
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	152 return
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	153 if parse.urlparse(self.public_url).scheme:
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	154 log.error(
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	155 "Scheme must not be specified in \"public_url\", please remove it from "
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	156 "\"public_url\" configuration option. ActivityPub Gateway won't be run."
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	157 )
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	158 return
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	159 self.http_port = int(self.host.memory.getConfig(
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	160 CONF_SECTION, 'http_port', 8123))
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	161 connection_type = self.host.memory.getConfig(
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	162 CONF_SECTION, 'http_connection_type', 'https')
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	163 if connection_type not in ('http', 'https'):
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	164 raise exceptions.ConfigError(
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	165 'bad ap-gateay http_connection_type, you must use one of "http" or '
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	166 '"https"'
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	167 )
3728 b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	168 self.max_items = self.host.memory.getConfig(
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	169 CONF_SECTION, 'new_node_max_items', 50
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	170
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	171 )
3682 7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	172 self.ap_path = self.host.memory.getConfig(CONF_SECTION, 'ap_path', '_ap')
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	173 self.base_ap_url = parse.urljoin(f"https://{self.public_url}", f"{self.ap_path}/")
3728 b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	174 # True (default) if we provide gateway only to entities/services from our server
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	175 self.local_only = C.bool(
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	176 self.host.memory.getConfig(CONF_SECTION, 'local_only', C.BOOL_TRUE)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	177 )
3682 7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	178
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	179 # HTTP server launch
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	180 self.server = HTTPServer(self)
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	181 if connection_type == 'http':
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	182 reactor.listenTCP(self.http_port, self.server)
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	183 else:
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	184 options = tls.getOptionsFromConfig(
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	185 self.host.memory.config, CONF_SECTION)
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	186 tls.TLSOptionsCheck(options)
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	187 context_factory = tls.getTLSContextFactory(options)
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	188 reactor.listenSSL(self.http_port, self.server, context_factory)
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	189
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	190 async def profileConnecting(self, client):
3728 b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	191 self.client = client
3682 7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	192 await self.init(client)
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	193
3728 b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	194 async def apGet(self, url: str) -> dict:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	195 """Retrieve AP JSON from given URL
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	196
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	197 @raise error.StanzaError: "service-unavailable" is sent when something went wrong
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	198 with AP server
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	199 """
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	200 try:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	201 return await treq.json_content(await treq.get(
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	202 url,
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	203 headers = {
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	204 "Accept": [MEDIA_TYPE_AP],
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	205 "Content-Type": [MEDIA_TYPE_AP],
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	206 }
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	207 ))
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	208 except Exception as e:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	209 raise error.StanzaError(
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	210 "service-unavailable",
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	211 text=f"Can't get AP data at {url}: {e}"
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	212 )
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	213
3742 bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	214 @overload
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	215 async def apGetObject(self, data: dict, key: str) -> Optional[dict]:
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	216 ...
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	217
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	218 @overload
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	219 async def apGetObject(
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	220 self, data: Union[str, dict], key: None = None
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	221 ) -> dict:
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	222 ...
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	223
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	224 async def apGetObject(self, data, key = None):
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	225 """Retrieve an AP object, dereferencing when necessary
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	226
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	227 This method is to be used with attributes marked as "Functional" in
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	228 https://www.w3.org/TR/activitystreams-vocabulary
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	229 @param data: AP object where an other object is looked for, or the object itself
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	230 @param key: name of the object to look for, or None if data is the object directly
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	231 @return: found object if any
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	232 """
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	233 if key is not None:
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	234 value = data.get(key)
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	235 else:
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	236 value = data
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	237 if value is None:
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	238 if key is None:
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	239 raise ValueError("None can't be used with apGetObject is key is None")
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	240 return None
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	241 elif isinstance(value, dict):
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	242 return value
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	243 elif isinstance(value, str):
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	244 return await self.apGet(value)
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	245 else:
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	246 raise NotImplementedError(
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	247 "was expecting a string or a dict, got {type(value)}: {value!r}}"
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	248 )
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	249
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	250 async def apGetList(self, data: dict, key: str) -> Optional[List[dict]]:
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	251 """Retrieve a list of objects from AP data, dereferencing when necessary
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	252
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	253 This method is to be used with non functional vocabularies. Use ``apGetObject``
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	254 otherwise.
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	255 If the value is a dictionary, it will be wrapped in a list
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	256 @param data: AP object where a list of objects is looked for
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	257 @param key: key of the list to look for
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	258 @return: list of objects, or None if the key is not present
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	259 """
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	260 value = data.get(key)
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	261 if value is None:
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	262 return None
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	263 elif isinstance(value, str):
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	264 value = await self.apGet(value)
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	265 if isinstance(value, dict):
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	266 return [value]
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	267 if not isinstance(value, list):
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	268 raise ValueError(f"A list was expected, got {type(value)}: {value!r}")
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	269 return [await self.apGetObject(i) for i in value]
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	270
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	271 async def apGetActors(
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	272 self,
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	273 data: dict,
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	274 key: str,
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	275 as_account: bool = True
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	276 ) -> List[str]:
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	277 """Retrieve AP actors from data
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	278
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	279 @param data: AP object containing a field with actors
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	280 @param key: field to use to retrieve actors
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	281 @param as_account: if True returns account handles, otherwise will return actor
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	282 IDs
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	283 @raise exceptions.DataError: there is not actor data or it is invalid
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	284 """
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	285 value = data.get(key)
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	286 if value is None:
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	287 raise exceptions.DataError(
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	288 f"no actor associated to object {data.get('id')!r}"
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	289 )
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	290 elif isinstance(value, dict):
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	291 actor_id = value.get("id")
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	292 if actor_id is None:
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	293 raise exceptions.DataError(
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	294 f"invalid actor associated to object {data.get('id')!r}: {value!r}"
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	295 )
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	296 value = [actor_id]
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	297 elif isinstance(value, str):
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	298 value = [value]
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	299 elif isinstance(value, list):
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	300 try:
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	301 value = [a if isinstance(a, str) else a["id"] for a in value]
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	302 except (TypeError, KeyError):
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	303 raise exceptions.DataError(
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	304 f"invalid actors list to object {data.get('id')!r}: {value!r}"
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	305 )
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	306 if not value:
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	307 raise exceptions.DataError(
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	308 f"list of actors is empty"
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	309 )
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	310 if as_account:
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	311 return [await self.getAPAccountFromId(actor_id) for actor_id in value]
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	312 else:
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	313 return value
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	314
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	315 async def apGetSenderActor(
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	316 self,
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	317 data: dict,
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	318 ) -> str:
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	319 """Retrieve actor who sent data
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	320
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	321 This is done by checking "attributedTo" field first, then "actor" field.
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	322 Only the first found actor is taken into accoun
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	323 @param data: AP object
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	324 @return: actor id of the sender
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	325 @raise exceptions.NotFound: no actor has been found in data
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	326 """
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	327 try:
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	328 actors = await self.apGetActors(data, "attributedTo", as_account=False)
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	329 except exceptions.DataError:
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	330 actors = None
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	331 if not actors:
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	332 try:
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	333 actors = await self.apGetActors(data, "actor", as_account=False)
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	334 except exceptions.DataError:
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	335 raise exceptions.NotFound(
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	336 'actor not specified in "attributedTo" or "actor"'
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	337 )
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	338 try:
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	339 return actors[0]
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	340 except IndexError:
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	341 raise exceptions.NotFound("list of actors is empty")
bf0505d41c09 comp AP: helper methods to get AP objects: Goffi <goffi@goffi.org> parents: 3729 diff changeset	342
3728 b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	343 def mustEncode(self, text: str) -> bool:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	344 """Indicate if a text must be period encoded"""
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	345 return (
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	346 not RE_ALLOWED_UNQUOTED.match(text)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	347 or text.startswith("___")
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	348 or "---" in text
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	349 )
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	350
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	351 def periodEncode(self, text: str) -> str:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	352 """Period encode a text
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	353
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	354 see [getJIDAndNode] for reasons of period encoding
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	355 """
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	356 return (
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	357 parse.quote(text, safe="")
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	358 .replace("---", "%2d%2d%2d")
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	359 .replace("___", "%5f%5f%5f")
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	360 .replace(".", "%2e")
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	361 .replace("~", "%7e")
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	362 .replace("%", ".")
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	363 )
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	364
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	365 async def getAPAccountFromJidAndNode(
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	366 self,
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	367 jid_: jid.JID,
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	368 node: Optional[str]
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	369 ) -> str:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	370 """Construct AP account from JID and node
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	371
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	372 The account construction will use escaping when necessary
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	373 """
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	374 if not node or node == self._m.namespace:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	375 node = None
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	376
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	377 if node and not jid_.user and not self.mustEncode(node):
3729 86eea17cafa7 component AP gateway: split plugin in several files: Goffi <goffi@goffi.org> parents: 3728 diff changeset	378 is_pubsub = await self.isPubsub(jid_)
3728 b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	379 # when we have a pubsub service, the user part can be used to set the node
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	380 # this produces more user-friendly AP accounts
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	381 if is_pubsub:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	382 jid_.user = node
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	383 node = None
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	384
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	385 is_local = self.isLocal(jid_)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	386 user = jid_.user if is_local else jid_.userhost()
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	387 if user is None:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	388 user = ""
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	389 account_elts = []
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	390 if node and self.mustEncode(node) or self.mustEncode(user):
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	391 account_elts = ["___"]
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	392 if node:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	393 node = self.periodEncode(node)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	394 user = self.periodEncode(user)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	395
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	396 if not user:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	397 raise exceptions.InternalError("there should be a user part")
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	398
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	399 if node:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	400 account_elts.extend((node, "---"))
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	401
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	402 account_elts.extend((
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	403 user, "@", jid_.host if is_local else self.client.jid.userhost()
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	404 ))
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	405 return "".join(account_elts)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	406
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	407 def isLocal(self, jid_: jid.JID) -> bool:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	408 """Returns True if jid_ use a domain or subdomain of gateway's host"""
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	409 local_host = self.client.host.split(".")
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	410 assert local_host
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	411 return jid_.host.split(".")[-len(local_host):] == local_host
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	412
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	413 async def isPubsub(self, jid_: jid.JID) -> bool:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	414 """Indicate if a JID is a Pubsub service"""
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	415 host_disco = await self.host.getDiscoInfos(self.client, jid_)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	416 return (
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	417 ("pubsub", "service") in host_disco.identities
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	418 and not ("pubsub", "pep") in host_disco.identities
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	419 )
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	420
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	421 async def getJIDAndNode(self, ap_account: str) -> Tuple[jid.JID, Optional[str]]:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	422 """Decode raw AP account handle to get XMPP JID and Pubsub Node
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	423
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	424 Username are case insensitive.
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	425
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	426 By default, the username correspond to local username (i.e. username from
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	427 component's server).
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	428
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	429 If local name's domain is a pubsub service (and not PEP), the username is taken as
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	430 a pubsub node.
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	431
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	432 If ``---`` is present in username, the part before is used as pubsub node, and the
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	433 rest as a JID user part.
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	434
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	435 If username starts with ``___``, characters are encoded using period encoding
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	436 (i.e. percent encoding where a ``.`` is used instead of ``%``).
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	437
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	438 This horror is necessary due to limitation in some AP implementation (notably
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	439 Mastodon), cf. https://github.com/mastodon/mastodon/issues/17222
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	440
3729 86eea17cafa7 component AP gateway: split plugin in several files: Goffi <goffi@goffi.org> parents: 3728 diff changeset	441 examples:
3728 b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	442
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	443 ``toto@example.org`` => JID = toto@example.org, node = None
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	444
3729 86eea17cafa7 component AP gateway: split plugin in several files: Goffi <goffi@goffi.org> parents: 3728 diff changeset	445 ``___toto.40example.net@example.org`` => JID = toto@example.net (this one is a
86eea17cafa7 component AP gateway: split plugin in several files: Goffi <goffi@goffi.org> parents: 3728 diff changeset	446 non-local JID, and will work only if setings ``local_only`` is False), node = None
3728 b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	447
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	448 ``toto@pubsub.example.org`` (with pubsub.example.org being a pubsub service) =>
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	449 JID = pubsub.example.org, node = toto
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	450
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	451 ``tata---toto@example.org`` => JID = toto@example.org, node = tata
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	452
3729 86eea17cafa7 component AP gateway: split plugin in several files: Goffi <goffi@goffi.org> parents: 3728 diff changeset	453 ``___urn.3axmpp.3amicroblog.3a0@pubsub.example.org`` (with pubsub.example.org
3728 b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	454 being a pubsub service) ==> JID = pubsub.example.org, node = urn:xmpp:microblog:0
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	455
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	456 @param ap_account: ActivityPub account handle (``username@domain.tld``)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	457 @return: service JID and pubsub node
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	458 if pubsub is None, default microblog pubsub node (and possibly other nodes
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	459 that plugins may hanlde) will be used
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	460 @raise ValueError: invalid account
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	461 @raise PermissionError: non local jid is used when gateway doesn't allow them
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	462 """
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	463 if ap_account.count("@") != 1:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	464 raise ValueError("Invalid AP account")
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	465 if ap_account.startswith("___"):
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	466 encoded = True
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	467 ap_account = ap_account[3:]
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	468 else:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	469 encoded = False
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	470
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	471 username, domain = ap_account.split("@")
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	472
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	473 if "---" in username:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	474 node, username = username.rsplit("---", 1)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	475 else:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	476 node = None
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	477
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	478 if encoded:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	479 username = parse.unquote(
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	480 RE_PERIOD_ENC.sub(r"%\g<hex>", username),
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	481 errors="strict"
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	482 )
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	483 if node:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	484 node = parse.unquote(
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	485 RE_PERIOD_ENC.sub(r"%\g<hex>", node),
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	486 errors="strict"
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	487 )
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	488
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	489 if "@" in username:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	490 username, domain = username.rsplit("@", 1)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	491
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	492 if not node:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	493 # we need to check host disco, because disco request to user may be
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	494 # blocked for privacy reason (see
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	495 # https://xmpp.org/extensions/xep-0030.html#security)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	496 is_pubsub = await self.isPubsub(jid.JID(domain))
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	497
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	498 if is_pubsub:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	499 # if the host is a pubsub service and not a PEP, we consider that username
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	500 # is in fact the node name
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	501 node = username
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	502 username = None
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	503
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	504 jid_s = f"{username}@{domain}" if username else domain
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	505 try:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	506 jid_ = jid.JID(jid_s)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	507 except RuntimeError:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	508 raise ValueError(f"Invalid jid: {jid_s!r}")
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	509
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	510 if self.local_only and not self.isLocal(jid_):
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	511 raise exceptions.PermissionError(
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	512 "This gateway is configured to map only local entities and services"
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	513 )
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	514
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	515 return jid_, node
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	516
3682 7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	517 def parseAPURL(self, url: str) -> Tuple[str, str]:
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	518 """Parse an URL leading to an AP endpoint
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	519
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	520 @param url: URL to parse (schema is not mandatory)
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	521 @return: endpoint type and AP account
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	522 """
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	523 path = parse.urlparse(url).path.lstrip("/")
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	524 type_, account = path[len(self.ap_path):].lstrip("/").split("/", 1)
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	525 return type_, parse.unquote(account)
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	526
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	527 def buildAPURL(self, type_:str , *args: str) -> str:
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	528 """Build an AP endpoint URL
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	529
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	530 @param type_: type of AP endpoing
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	531 @param arg: endpoint dependant arguments
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	532 """
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	533 return parse.urljoin(
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	534 self.base_ap_url,
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	535 str(Path(type_).joinpath(*(parse.quote_plus(a) for a in args)))
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	536 )
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	537
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	538 async def signAndPost(self, url: str, url_actor: str, doc: dict) -> TReqResponse:
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	539 """Sign a documentent and post it to AP server
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	540
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	541 @param url: AP server endpoint
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	542 @param url_actor: URL generated by this gateway for local actor
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	543 @param doc: document to send
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	544 """
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	545 p_url = parse.urlparse(url)
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	546 date = http.datetimeToString().decode()
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	547 body = json.dumps(doc).encode()
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	548 digest_hash = base64.b64encode(hashlib.sha256(body).digest()).decode()
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	549 digest = f"sha-256={digest_hash}"
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	550 to_sign = (
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	551 f"(request-target): post {p_url.path}\nhost: {p_url.hostname}\n"
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	552 f"date: {date}\ndigest: {digest}"
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	553 )
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	554 signature = base64.b64encode(self.private_key.sign(
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	555 to_sign.encode(),
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	556 # we have to use PKCS1v15 padding to be compatible with Mastodon
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	557 padding.PKCS1v15(),
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	558 hashes.SHA256()
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	559 )).decode()
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	560 h_signature = (
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	561 f'keyId="{url_actor}",headers="(request-target) host date digest",'
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	562 f'signature="{signature}"'
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	563 )
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	564 return await treq.post(
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	565 url,
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	566 body,
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	567 headers={
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	568 "Host": [p_url.hostname],
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	569 "Date": [date],
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	570 "Digest": [digest],
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	571 "Signature": [h_signature],
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	572 }
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	573 )
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	574
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	575 def _publishMessage(self, mess_data_s: str, service_s: str, profile: str):
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	576 mess_data: dict = data_format.deserialise(mess_data_s) # type: ignore
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	577 service = jid.JID(service_s)
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	578 client = self.host.getClient(profile)
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	579 return defer.ensureDeferred(self.publishMessage(client, mess_data, service))
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	580
3728 b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	581 async def getAPActorIdFromAccount(self, account: str) -> str:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	582 """Retrieve account ID from it's handle using WebFinger
3682 7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	583
3728 b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	584 @param account: AP handle (user@domain.tld)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	585 @return: Actor ID (which is an URL)
3682 7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	586 """
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	587 if account.count("@") != 1 or "/" in account:
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	588 raise ValueError("Invalid account: {account!r}")
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	589 host = account.split("@")[1]
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	590 try:
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	591 finger_data = await treq.json_content(await treq.get(
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	592 f"https://{host}/.well-known/webfinger?"
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	593 f"resource=acct:{parse.quote_plus(account)}",
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	594 ))
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	595 except Exception as e:
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	596 raise exceptions.DataError(f"Can't get webfinger data: {e}")
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	597 for link in finger_data.get("links", []):
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	598 if (
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	599 link.get("type") == "application/activity+json"
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	600 and link.get("rel") == "self"
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	601 ):
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	602 href = link.get("href", "").strip()
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	603 if not href:
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	604 raise ValueError(
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	605 f"Invalid webfinger data for {account:r}: missing href"
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	606 )
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	607 break
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	608 else:
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	609 raise ValueError(
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	610 f"No ActivityPub link found for {account!r}"
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	611 )
3728 b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	612 return href
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	613
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	614 async def getAPActorDataFromId(self, account: str) -> dict:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	615 """Retrieve ActivityPub Actor data
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	616
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	617 @param account: ActivityPub Actor identifier
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	618 """
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	619 href = await self.getAPActorIdFromAccount(account)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	620 return await self.apGet(href)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	621
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	622 @async_lru(maxsize=LRU_MAX_SIZE)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	623 async def getAPAccountFromId(self, actor_id: str):
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	624 """Retrieve AP account from the ID URL
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	625
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	626 @param actor_id: AP ID of the actor (URL to the actor data)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	627 """
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	628 url_parsed = parse.urlparse(actor_id)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	629 actor_data = await self.apGet(actor_id)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	630 username = actor_data.get("preferredUsername")
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	631 if not username:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	632 raise exceptions.DataError(
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	633 'No "preferredUsername" field found, can\'t retrieve actor account'
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	634 )
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	635 account = f"{username}@{url_parsed.hostname}"
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	636 # we try to retrieve the actor ID from the account to check it
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	637 found_id = await self.getAPActorIdFromAccount(account)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	638 if found_id != actor_id:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	639 # cf. https://socialhub.activitypub.rocks/t/how-to-retrieve-user-server-tld-handle-from-actors-url/2196
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	640 msg = (
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	641 f"Account ID found on WebFinger {found_id!r} doesn't match our actor ID "
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	642 f"({actor_id!r}). This AP instance doesn't seems to use "
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	643 '"preferredUsername" as we expect.'
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	644 )
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	645 log.warning(msg)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	646 raise exceptions.DataError(msg)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	647 return account
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	648
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	649 async def getAPItems(
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	650 self,
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	651 account: str,
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	652 max_items: Optional[int] = None,
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	653 chronological_pagination: bool = True,
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	654 after_id: Optional[str] = None,
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	655 start_index: Optional[int] = None,
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	656 ) -> Tuple[List[domish.Element], rsm.RSMResponse]:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	657 """Retrieve AP items and convert them to XMPP items
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	658
3729 86eea17cafa7 component AP gateway: split plugin in several files: Goffi <goffi@goffi.org> parents: 3728 diff changeset	659 @param account: AP account handle to get items from
3728 b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	660 @param max_items: maximum number of items to retrieve
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	661 retrieve all items by default
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	662 @param chronological_pagination: get pages in chronological order
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	663 AP use reversed chronological order for pagination, "first" page returns more
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	664 recent items. If "chronological_pagination" is True, "last" AP page will be
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	665 retrieved first.
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	666 @param after_id: if set, retrieve items starting from given ID
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	667 Due to ActivityStream Collection Paging limitations, this is inefficient and
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	668 if ``after_id`` is not already in cache, we have to retrieve every page until
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	669 we find it.
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	670 In most common cases, ``after_id`` should be in cache though (client usually
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	671 use known ID when in-order pagination is used).
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	672 @param start_index: start retrieving items from the one with given index
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	673 Due to ActivityStream Collection Paging limitations, this is inefficient and
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	674 all pages before the requested index will be retrieved to count items.
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	675 @return: XMPP Pubsub items and corresponding RSM Response
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	676 Items are always returned in chronological order in the result
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	677 """
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	678 actor_data = await self.getAPActorDataFromId(account)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	679 outbox = actor_data.get("outbox")
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	680 rsm_resp: Dict[str, Union[bool, int]] = {}
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	681 if not outbox:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	682 raise exceptions.DataError(f"No outbox found for actor {account}")
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	683 outbox_data = await self.apGet(outbox)
3682 7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	684 try:
3728 b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	685 count = outbox_data["totalItems"]
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	686 except KeyError:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	687 log.warning(
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	688 f'"totalItems" not found in outbox of {account}, defaulting to 20'
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	689 )
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	690 count = 20
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	691 else:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	692 log.info(f"{account}'s outbox has {count} item(s)")
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	693 rsm_resp["count"] = count
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	694
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	695 if start_index is not None:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	696 assert chronological_pagination and after_id is None
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	697 if start_index >= count:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	698 return [], rsm_resp
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	699 elif start_index == 0:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	700 # this is the default behaviour
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	701 pass
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	702 elif start_index > 5000:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	703 raise error.StanzaError(
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	704 "feature-not-implemented",
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	705 text="Maximum limit for previous_index has been reached, this limit"
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	706 "is set to avoid DoS"
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	707 )
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	708 else:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	709 # we'll convert "start_index" to "after_id", thus we need the item just
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	710 # before "start_index"
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	711 previous_index = start_index - 1
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	712 retrieved_items = 0
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	713 current_page = outbox_data["last"]
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	714 while retrieved_items < count:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	715 page_data, items = await self.parseAPPage(current_page)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	716 if not items:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	717 log.warning(f"found an empty AP page at {current_page}")
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	718 return [], rsm_resp
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	719 page_start_idx = retrieved_items
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	720 retrieved_items += len(items)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	721 if previous_index <= retrieved_items:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	722 after_id = items[previous_index - page_start_idx]["id"]
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	723 break
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	724 try:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	725 current_page = page_data["prev"]
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	726 except KeyError:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	727 log.warning(
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	728 f"missing previous page link at {current_page}: {page_data!r}"
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	729 )
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	730 raise error.StanzaError(
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	731 "service-unavailable",
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	732 "Error while retrieving previous page from AP service at "
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	733 f"{current_page}"
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	734 )
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	735
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	736 init_page = "last" if chronological_pagination else "first"
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	737 page = outbox_data.get(init_page)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	738 if not page:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	739 raise exceptions.DataError(
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	740 f"Initial page {init_page!r} not found for outbox {outbox}"
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	741 )
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	742 items = []
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	743 page_items = []
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	744 retrieved_items = 0
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	745 found_after_id = False
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	746
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	747 while retrieved_items < count:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	748 __, page_items = await self.parseAPPage(page)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	749 retrieved_items += len(page_items)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	750 if after_id is not None and not found_after_id:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	751 # if we have an after_id, we ignore all items until the requested one is
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	752 # found
3729 86eea17cafa7 component AP gateway: split plugin in several files: Goffi <goffi@goffi.org> parents: 3728 diff changeset	753 try:
86eea17cafa7 component AP gateway: split plugin in several files: Goffi <goffi@goffi.org> parents: 3728 diff changeset	754 limit_idx = [i["id"] for i in page_items].index(after_id)
86eea17cafa7 component AP gateway: split plugin in several files: Goffi <goffi@goffi.org> parents: 3728 diff changeset	755 except ValueError:
3728 b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	756 # if "after_id" is not found, we don't add any item from this page
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	757 log.debug(f"{after_id!r} not found at {page}, skipping")
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	758 else:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	759 found_after_id = True
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	760 if chronological_pagination:
3729 86eea17cafa7 component AP gateway: split plugin in several files: Goffi <goffi@goffi.org> parents: 3728 diff changeset	761 start_index = retrieved_items - len(page_items) + limit_idx + 1
3728 b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	762 page_items = page_items[limit_idx+1:]
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	763 else:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	764 start_index = count - (retrieved_items - len(page_items) +
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	765 limit_idx + 1)
3729 86eea17cafa7 component AP gateway: split plugin in several files: Goffi <goffi@goffi.org> parents: 3728 diff changeset	766 page_items = page_items[:limit_idx]
3728 b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	767 items.extend(page_items)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	768 else:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	769 items.extend(page_items)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	770 if max_items is not None and len(items) >= max_items:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	771 if chronological_pagination:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	772 items = items[:max_items]
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	773 else:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	774 items = items[-max_items:]
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	775 break
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	776 page = outbox_data.get("prev" if chronological_pagination else "next")
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	777 if not page:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	778 break
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	779
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	780 if after_id is not None and not found_after_id:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	781 raise error.StanzaError("item-not-found")
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	782
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	783 if after_id is None:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	784 rsm_resp["index"] = 0 if chronological_pagination else count - len(items)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	785
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	786 if start_index is not None:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	787 rsm_resp["index"] = start_index
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	788 elif after_id is not None:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	789 log.warning("Can't determine index of first element")
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	790 elif chronological_pagination:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	791 rsm_resp["index"] = 0
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	792 else:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	793 rsm_resp["index"] = count - len(items)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	794 if items:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	795 rsm_resp.update({
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	796 "first": items[0]["id"],
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	797 "last": items[-1]["id"]
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	798 })
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	799
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	800 return items, rsm.RSMResponse(**rsm_resp)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	801
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	802 async def parseAPPage(self, url: str) -> Tuple[dict, List[domish.Element]]:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	803 """Convert AP objects from an AP page to XMPP items
3682 7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	804
3728 b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	805 @param url: url linking and AP page
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	806 @return: page data, pubsub items
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	807 """
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	808 page_data = await self.apGet(url)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	809 ap_items = page_data.get("orderedItems")
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	810 if not ap_items:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	811 log.warning('No "orderedItems" collection found')
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	812 return page_data, []
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	813 items = []
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	814 # AP Collections are in antichronological order, but we expect chronological in
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	815 # Pubsub, thus we reverse it
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	816 for ap_item in reversed(ap_items):
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	817 try:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	818 ap_object, mb_data = await self.apItem2MBdata(ap_item)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	819 except (exceptions.DataError, NotImplementedError, error.StanzaError):
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	820 continue
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	821
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	822 item_elt = await self._m.data2entry(
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	823 self.client, mb_data, ap_object["id"], None, self._m.namespace
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	824 )
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	825 item_elt["publisher"] = mb_data["author_jid"].full()
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	826 items.append(item_elt)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	827
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	828 return page_data, items
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	829
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	830 async def apItem2MBdata(self, ap_item: dict) -> Tuple[dict, dict]:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	831 """Convert AP item to microblog data
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	832
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	833 @return: AP Item's Object and microblog data
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	834 @raise exceptions.DataError: something is invalid in the AP item
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	835 @raise NotImplemented: some AP data is not handled yet
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	836 @raise error.StanzaError: error while contacting the AP server
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	837 """
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	838 ap_object = ap_item.get("object")
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	839 if not ap_object:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	840 log.warning(f'No "object" found in AP item {ap_item!r}')
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	841 raise exceptions.DataError
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	842 if isinstance(ap_object, str):
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	843 ap_object = await self.apGet(ap_object)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	844 obj_id = ap_object.get("id")
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	845 if not obj_id:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	846 log.warning(f'No "id" found in AP object: {ap_object!r}')
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	847 raise exceptions.DataError
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	848 if ap_object.get("inReplyTo") is not None:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	849 raise NotImplementedError
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	850 mb_data = {}
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	851 for ap_key, mb_key in AP_MB_MAP.items():
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	852 data = ap_object.get(ap_key)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	853 if data is None:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	854 continue
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	855 mb_data[mb_key] = data
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	856
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	857 # content
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	858 try:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	859 language, content_xhtml = ap_object["contentMap"].popitem()
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	860 except (KeyError, AttributeError):
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	861 try:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	862 mb_data["content_xhtml"] = mb_data["content"]
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	863 except KeyError:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	864 log.warning(f"no content found:\n{ap_object!r}")
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	865 raise exceptions.DataError
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	866 else:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	867 mb_data["language"] = language
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	868 mb_data["content_xhtml"] = content_xhtml
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	869
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	870 # author
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	871 actor = ap_item.get("actor")
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	872 if not actor:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	873 log.warning(f"no actor associated to object id {obj_id!r}")
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	874 raise exceptions.DataError
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	875 elif isinstance(actor, list):
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	876 # we only keep first item of list as author
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	877 # TODO: handle multiple actors
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	878 if len(actor) > 1:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	879 log.warning("multiple actors are not managed")
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	880 actor = actor[0]
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	881
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	882 if isinstance(actor, dict):
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	883 actor = actor.get("id")
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	884 if not actor:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	885 log.warning(f"no actor id found: {actor!r}")
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	886 raise exceptions.DataError
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	887
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	888 if isinstance(actor, str):
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	889 account = await self.getAPAccountFromId(actor)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	890 mb_data["author"] = account.split("@", 1)[0]
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	891 author_jid = mb_data["author_jid"] = jid.JID(
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	892 None,
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	893 (
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	894 self.host.plugins["XEP-0106"].escape(account),
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	895 self.client.jid.host,
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	896 None
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	897 )
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	898 )
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	899 else:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	900 log.warning(f"unknown actor type found: {actor!r}")
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	901 raise exceptions.DataError
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	902
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	903 # published/updated
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	904 for field in ("published", "updated"):
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	905 value = ap_object.get(field)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	906 if not value and field == "updated":
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	907 value = ap_object.get("published")
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	908 if value:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	909 try:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	910 mb_data[field] = calendar.timegm(
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	911 dateutil.parser.parse(str(value)).utctimetuple()
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	912 )
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	913 except dateutil.parser.ParserError as e:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	914 log.warning(f"Can't parse {field!r} field: {e}")
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	915 return ap_object, mb_data
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	916
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	917 async def mbdata2APitem(self, client: SatXMPPEntity, mb_data: dict) -> dict:
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	918 """Convert Libervia Microblog Data to ActivityPub item"""
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	919 if not mb_data.get("id"):
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	920 mb_data["id"] = shortuuid.uuid()
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	921 if not mb_data.get("author_jid"):
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	922 mb_data["author_jid"] = client.jid
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	923 ap_account = await self.getAPAccountFromJidAndNode(
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	924 jid.JID(mb_data["author_jid"]),
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	925 None
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	926 )
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	927 url_actor = self.buildAPURL(TYPE_ACTOR, ap_account)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	928 url_item = self.buildAPURL(TYPE_ITEM, ap_account, mb_data["id"])
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	929 return {
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	930 "@context": "https://www.w3.org/ns/activitystreams",
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	931 "id": url_item,
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	932 "type": "Create",
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	933 "actor": url_actor,
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	934
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	935 "object": {
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	936 "id": url_item,
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	937 "type": "Note",
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	938 "published": utils.xmpp_date(mb_data["published"]),
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	939 "attributedTo": url_actor,
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	940 "content": mb_data.get("content_xhtml") or mb_data["content"],
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	941 "to": "https://www.w3.org/ns/activitystreams#Public"
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	942 }
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	943 }
3682 7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	944
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	945 async def publishMessage(
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	946 self,
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	947 client: SatXMPPEntity,
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	948 mess_data: dict,
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	949 service: jid.JID
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	950 ) -> None:
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	951 """Send an AP message
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	952
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	953 .. note::
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	954
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	955 This is a temporary method used for development only
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	956
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	957 @param mess_data: message data. Following keys must be set:
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	958
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	959 ``node``
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	960 identifier of message which is being replied (this will
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	961 correspond to pubsub node in the future)
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	962
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	963 ``content_xhtml`` or ``content``
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	964 message body (respectively in XHTML or plain text)
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	965
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	966 @param service: JID corresponding to the AP actor.
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	967 """
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	968 if not service.user:
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	969 raise ValueError("service must have a local part")
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	970 account = self.host.plugins["XEP-0106"].unescape(service.user)
3728 b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	971 ap_actor_data = await self.getAPActorDataFromId(account)
3682 7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	972
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	973 try:
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	974 inbox_url = ap_actor_data["endpoints"]["sharedInbox"]
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	975 except KeyError:
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	976 raise exceptions.DataError("Can't get ActivityPub actor inbox")
7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	977
3728 b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	978 item_data = await self.mbdata2APitem(client, mess_data)
b15644cae50d component AP gateway: JID/node ⟺ AP outbox conversion: Goffi <goffi@goffi.org> parents: 3684 diff changeset	979 url_actor = item_data["object"]["attributedTo"]
3682 7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	980 resp = await self.signAndPost(inbox_url, url_actor, item_data)
3684 8353cc3b8db9 component AP gateway: fix wrong operator when testing HTTP code Goffi <goffi@goffi.org> parents: 3682 diff changeset	981 if resp.code != 202:
3682 7c990aaa49d3 comp AP Gateway: ActivityPub Component first draft: Goffi <goffi@goffi.org> parents: diff changeset	982 raise exceptions.NetworkError(f"unexpected return code: {resp.code}")

Mercurial > libervia-backend

annotate sat/plugins/plugin_comp_ap_gateway/__init__.py @ 3743:54c249ec35ce

annotate sat/plugins/plugin_comp_ap_gateway/init.py @ 3743:54c249ec35ce