libervia-backend: libervia/backend/plugins/plugin_misc

annotate libervia/backend/plugins/plugin_misc_download.py @ 4306:94e0968987cd

plugin XEP-0033: code modernisation, improve delivery, data validation: - Code has been rewritten using Pydantic models and `async` coroutines for data validation and cleaner element parsing/generation. - Delivery has been completely rewritten. It now works even if server doesn't support multicast, and send to local multicast service first. Delivering to local multicast service first is due to bad support of XEP-0033 in server (notably Prosody which has an incomplete implementation), and the current impossibility to detect if a sub-domain service handles fully multicast or only for local domains. This is a workaround to have a good balance between backward compatilibity and use of bandwith, and to make it work with the incoming email gateway implementation (the gateway will only deliver to entities of its own domain). - disco feature checking now uses `async` corountines. `host` implementation still use Deferred return values for compatibility with legacy code. rel 450

author	Goffi <goffi@goffi.org>
date	Thu, 26 Sep 2024 16:12:01 +0200
parents	0d7bb4df2343
children

rev	line source
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	1 #!/usr/bin/env python3
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	2
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	3 # SAT plugin for downloading files
3479 be6d91572633 date update Goffi <goffi@goffi.org> parents: 3211 diff changeset	4 # Copyright (C) 2009-2021 Jérôme Poisson (goffi@goffi.org)
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	5
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	6 # This program is free software: you can redistribute it and/or modify
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	7 # it under the terms of the GNU Affero General Public License as published by
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	8 # the Free Software Foundation, either version 3 of the License, or
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	9 # (at your option) any later version.
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	10
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	11 # This program is distributed in the hope that it will be useful,
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	12 # but WITHOUT ANY WARRANTY; without even the implied warranty of
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	13 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	14 # GNU Affero General Public License for more details.
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	15
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	16 # You should have received a copy of the GNU Affero General Public License
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	17 # along with this program. If not, see <http://www.gnu.org/licenses/>.
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	18
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	19 import hashlib
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	20 from pathlib import Path
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	21 from typing import Any, Dict, Optional, Union, Tuple, Callable
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	22 from urllib.parse import unquote, urlparse
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	23
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	24 import treq
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	25 from twisted.internet import defer
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	26 from twisted.words.protocols.jabber import error as jabber_error
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	27
4071 4b842c1fb686 refactoring: renamed `sat` package to `libervia.backend` Goffi <goffi@goffi.org> parents: 4037 diff changeset	28 from libervia.backend.core import exceptions
4b842c1fb686 refactoring: renamed `sat` package to `libervia.backend` Goffi <goffi@goffi.org> parents: 4037 diff changeset	29 from libervia.backend.core.constants import Const as C
4b842c1fb686 refactoring: renamed `sat` package to `libervia.backend` Goffi <goffi@goffi.org> parents: 4037 diff changeset	30 from libervia.backend.core.core_types import SatXMPPEntity
4b842c1fb686 refactoring: renamed `sat` package to `libervia.backend` Goffi <goffi@goffi.org> parents: 4037 diff changeset	31 from libervia.backend.core.i18n import D_, _
4b842c1fb686 refactoring: renamed `sat` package to `libervia.backend` Goffi <goffi@goffi.org> parents: 4037 diff changeset	32 from libervia.backend.core.log import getLogger
4b842c1fb686 refactoring: renamed `sat` package to `libervia.backend` Goffi <goffi@goffi.org> parents: 4037 diff changeset	33 from libervia.backend.tools import xml_tools
4b842c1fb686 refactoring: renamed `sat` package to `libervia.backend` Goffi <goffi@goffi.org> parents: 4037 diff changeset	34 from libervia.backend.tools import stream
4b842c1fb686 refactoring: renamed `sat` package to `libervia.backend` Goffi <goffi@goffi.org> parents: 4037 diff changeset	35 from libervia.backend.tools.common import data_format
4b842c1fb686 refactoring: renamed `sat` package to `libervia.backend` Goffi <goffi@goffi.org> parents: 4037 diff changeset	36 from libervia.backend.tools.web import treq_client_no_ssl
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	37
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	38 log = getLogger(__name__)
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	39
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	40
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	41 PLUGIN_INFO = {
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	42 C.PI_NAME: "File Download",
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	43 C.PI_IMPORT_NAME: "DOWNLOAD",
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	44 C.PI_TYPE: C.PLUG_TYPE_MISC,
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	45 C.PI_MODES: C.PLUG_MODE_BOTH,
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	46 C.PI_MAIN: "DownloadPlugin",
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	47 C.PI_HANDLER: "no",
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	48 C.PI_DESCRIPTION: _("""File download management"""),
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	49 }
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	50
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	51
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	52 class DownloadPlugin(object):
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	53
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	54 def __init__(self, host):
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	55 log.info(_("plugin Download initialization"))
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	56 self.host = host
4037 524856bd7b19 massive refactoring to switch from camelCase to snake_case: Goffi <goffi@goffi.org> parents: 3922 diff changeset	57 host.bridge.add_method(
524856bd7b19 massive refactoring to switch from camelCase to snake_case: Goffi <goffi@goffi.org> parents: 3922 diff changeset	58 "file_download",
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	59 ".plugin",
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	60 in_sign="ssss",
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	61 out_sign="s",
4037 524856bd7b19 massive refactoring to switch from camelCase to snake_case: Goffi <goffi@goffi.org> parents: 3922 diff changeset	62 method=self._file_download,
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	63 async_=True,
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	64 )
4037 524856bd7b19 massive refactoring to switch from camelCase to snake_case: Goffi <goffi@goffi.org> parents: 3922 diff changeset	65 host.bridge.add_method(
524856bd7b19 massive refactoring to switch from camelCase to snake_case: Goffi <goffi@goffi.org> parents: 3922 diff changeset	66 "file_download_complete",
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	67 ".plugin",
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	68 in_sign="ssss",
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	69 out_sign="s",
4037 524856bd7b19 massive refactoring to switch from camelCase to snake_case: Goffi <goffi@goffi.org> parents: 3922 diff changeset	70 method=self._file_download_complete,
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	71 async_=True,
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	72 )
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	73 self._download_callbacks = {}
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	74 self._scheme_callbacks = {}
4270 0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	75 self.register_scheme("http", self.download_http)
0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	76 self.register_scheme("https", self.download_http)
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	77
4037 524856bd7b19 massive refactoring to switch from camelCase to snake_case: Goffi <goffi@goffi.org> parents: 3922 diff changeset	78 def _file_download(
4270 0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	79 self, attachment_s: str, dest_path: str, extra_s: str, profile: str
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	80 ) -> defer.Deferred:
4270 0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	81 d = defer.ensureDeferred(
0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	82 self.file_download(
0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	83 self.host.get_client(profile),
0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	84 data_format.deserialise(attachment_s),
0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	85 Path(dest_path),
0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	86 data_format.deserialise(extra_s),
0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	87 )
0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	88 )
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	89 d.addCallback(lambda ret: data_format.serialise(ret))
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	90 return d
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	91
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	92 async def file_download(
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	93 self,
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	94 client: SatXMPPEntity,
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	95 attachment: Dict[str, Any],
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	96 dest_path: Path,
4270 0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	97 extra: Optional[Dict[str, Any]] = None,
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	98 ) -> Dict[str, Any]:
3187 d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	99 """Download a file using best available method
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	100
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	101 parameters are the same as for [download]
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	102 @return (dict): action dictionary, with progress id in case of success, else xmlui
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	103 message
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	104 """
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	105 try:
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	106 progress_id, __ = await self.download(client, attachment, dest_path, extra)
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	107 except Exception as e:
4270 0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	108 if (
0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	109 isinstance(e, jabber_error.StanzaError)
0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	110 and e.condition == "not-acceptable"
0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	111 ):
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	112 reason = e.text
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	113 else:
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	114 reason = str(e)
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	115 msg = D_("Can't download file: {reason}").format(reason=reason)
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	116 log.warning(msg)
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	117 return {
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	118 "xmlui": xml_tools.note(
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	119 msg, D_("Can't download file"), C.XMLUI_DATA_LVL_WARNING
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	120 ).toXml()
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	121 }
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	122 else:
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	123 return {"progress": progress_id}
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	124
4037 524856bd7b19 massive refactoring to switch from camelCase to snake_case: Goffi <goffi@goffi.org> parents: 3922 diff changeset	125 def _file_download_complete(
4270 0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	126 self, attachment_s: str, dest_path: str, extra_s: str, profile: str
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	127 ) -> defer.Deferred:
4270 0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	128 d = defer.ensureDeferred(
0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	129 self.file_download_complete(
0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	130 self.host.get_client(profile),
0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	131 data_format.deserialise(attachment_s),
0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	132 Path(dest_path),
0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	133 data_format.deserialise(extra_s),
0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	134 )
0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	135 )
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	136 d.addCallback(lambda path: str(path))
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	137 return d
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	138
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	139 async def file_download_complete(
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	140 self,
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	141 client: SatXMPPEntity,
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	142 attachment: Dict[str, Any],
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	143 dest_path: Path,
4270 0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	144 extra: Optional[Dict[str, Any]] = None,
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	145 ) -> str:
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	146 """Helper method to fully download a file and return its path
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	147
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	148 parameters are the same as for [download]
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	149 @return (str): path to the downloaded file
3187 d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	150 use empty string to store the file in cache
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	151 """
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	152 __, download_d = await self.download(client, attachment, dest_path, extra)
3187 d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	153 dest_path = await download_d
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	154 return dest_path
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	155
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	156 async def download_uri(
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	157 self,
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	158 client: SatXMPPEntity,
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	159 uri: str,
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	160 dest_path: Union[Path, str],
4270 0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	161 extra: Optional[Dict[str, Any]] = None,
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	162 ) -> Tuple[str, defer.Deferred]:
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	163 if extra is None:
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	164 extra = {}
4270 0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	165 uri_parsed = urlparse(uri, "http")
3187 d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	166 if dest_path:
d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	167 dest_path = Path(dest_path)
3211 4252176ad993 plugin download: clean unfinished files and re-raise exception in case of download error Goffi <goffi@goffi.org> parents: 3205 diff changeset	168 cache_uid = None
3187 d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	169 else:
d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	170 filename = Path(unquote(uri_parsed.path)).name.strip() or C.FILE_DEFAULT_NAME
d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	171 # we don't use Path.suffixes because we don't want to have more than 2
d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	172 # suffixes, but we still want to handle suffixes like "tar.gz".
4270 0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	173 stem, *suffixes = filename.rsplit(".", 2)
3187 d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	174 # we hash the URL to have an unique identifier, and avoid double download
d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	175 url_hash = hashlib.sha256(uri_parsed.geturl().encode()).hexdigest()
3211 4252176ad993 plugin download: clean unfinished files and re-raise exception in case of download error Goffi <goffi@goffi.org> parents: 3205 diff changeset	176 cache_uid = f"{stem}_{url_hash}"
4037 524856bd7b19 massive refactoring to switch from camelCase to snake_case: Goffi <goffi@goffi.org> parents: 3922 diff changeset	177 cache_data = client.cache.get_metadata(cache_uid)
3187 d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	178 if cache_data is not None:
d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	179 # file is already in cache, we return it
4270 0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	180 download_d = defer.succeed(cache_data["path"])
0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	181 return "", download_d
3187 d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	182 else:
d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	183 # the file is not in cache
4270 0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	184 unique_name = ".".join([cache_uid] + suffixes)
4037 524856bd7b19 massive refactoring to switch from camelCase to snake_case: Goffi <goffi@goffi.org> parents: 3922 diff changeset	185 with client.cache.cache_data(
4270 0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	186 "DOWNLOAD", cache_uid, filename=unique_name
0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	187 ) as f:
3187 d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	188 # we close the file and only use its name, the file will be opened
d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	189 # by the registered callback
3211 4252176ad993 plugin download: clean unfinished files and re-raise exception in case of download error Goffi <goffi@goffi.org> parents: 3205 diff changeset	190 dest_path = Path(f.name)
3205 2c0628f3927e plugin download, aesgcm: disable TLS check if `check_certificate` setting is disabled Goffi <goffi@goffi.org> parents: 3187 diff changeset	191
2c0628f3927e plugin download, aesgcm: disable TLS check if `check_certificate` setting is disabled Goffi <goffi@goffi.org> parents: 3187 diff changeset	192 # should we check certificates?
4037 524856bd7b19 massive refactoring to switch from camelCase to snake_case: Goffi <goffi@goffi.org> parents: 3922 diff changeset	193 check_certificate = self.host.memory.param_get_a(
4270 0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	194 "check_certificate", "Connection", profile_key=client.profile
0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	195 )
3205 2c0628f3927e plugin download, aesgcm: disable TLS check if `check_certificate` setting is disabled Goffi <goffi@goffi.org> parents: 3187 diff changeset	196 if not check_certificate:
4270 0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	197 extra["ignore_tls_errors"] = True
0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	198 log.warning(_("certificate check disabled for download, this is dangerous!"))
3205 2c0628f3927e plugin download, aesgcm: disable TLS check if `check_certificate` setting is disabled Goffi <goffi@goffi.org> parents: 3187 diff changeset	199
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	200 try:
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	201 callback = self._scheme_callbacks[uri_parsed.scheme]
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	202 except KeyError:
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	203 raise exceptions.NotFound(f"Can't find any handler for uri {uri}")
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	204 else:
3211 4252176ad993 plugin download: clean unfinished files and re-raise exception in case of download error Goffi <goffi@goffi.org> parents: 3205 diff changeset	205 try:
4252176ad993 plugin download: clean unfinished files and re-raise exception in case of download error Goffi <goffi@goffi.org> parents: 3205 diff changeset	206 progress_id, download_d = await callback(
4270 0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	207 client, uri_parsed, dest_path, extra
0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	208 )
3211 4252176ad993 plugin download: clean unfinished files and re-raise exception in case of download error Goffi <goffi@goffi.org> parents: 3205 diff changeset	209 except Exception as e:
4270 0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	210 log.warning(
0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	211 _("Can't download URI {uri}: {reason}").format(uri=uri, reason=e)
0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	212 )
3211 4252176ad993 plugin download: clean unfinished files and re-raise exception in case of download error Goffi <goffi@goffi.org> parents: 3205 diff changeset	213 if cache_uid is not None:
4037 524856bd7b19 massive refactoring to switch from camelCase to snake_case: Goffi <goffi@goffi.org> parents: 3922 diff changeset	214 client.cache.remove_from_cache(cache_uid)
3211 4252176ad993 plugin download: clean unfinished files and re-raise exception in case of download error Goffi <goffi@goffi.org> parents: 3205 diff changeset	215 elif dest_path.exists():
4252176ad993 plugin download: clean unfinished files and re-raise exception in case of download error Goffi <goffi@goffi.org> parents: 3205 diff changeset	216 dest_path.unlink()
4252176ad993 plugin download: clean unfinished files and re-raise exception in case of download error Goffi <goffi@goffi.org> parents: 3205 diff changeset	217 raise e
3187 d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	218 download_d.addCallback(lambda __: dest_path)
d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	219 return progress_id, download_d
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	220
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	221 async def download(
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	222 self,
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	223 client: SatXMPPEntity,
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	224 attachment: Dict[str, Any],
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	225 dest_path: Union[Path, str],
4270 0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	226 extra: Optional[Dict[str, Any]] = None,
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	227 ) -> Tuple[str, defer.Deferred]:
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	228 """Download a file from URI using suitable method
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	229
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	230 @param uri: URI to the file to download
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	231 @param dest_path: where the file must be downloaded
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	232 if empty string, the file will be stored in local path
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	233 @param extra: options depending on scheme handler
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	234 Some common options:
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	235 - ignore_tls_errors(bool): True to ignore SSL/TLS certificate verification
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	236 used only if HTTPS transport is needed
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	237 @return: ``progress_id`` and a Deferred which fire download URL when download is
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	238 finished.
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	239 ``progress_id`` can be empty string if the file already exist and is not
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	240 downloaded again (can happen if cache is used with empty ``dest_path``).
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	241 """
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	242 uri = attachment.get("uri")
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	243 if uri:
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	244 return await self.download_uri(client, uri, dest_path, extra)
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	245 else:
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	246 for source in attachment.get("sources", []):
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	247 source_type = source.get("type")
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	248 if not source_type:
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	249 log.warning(
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	250 "source type is missing for source: {source}\nattachment: "
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	251 f"{attachment}"
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	252 )
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	253 continue
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	254 try:
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	255 cb = self._download_callbacks[source_type]
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	256 except KeyError:
4270 0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	257 log.warning(f"no source handler registered for {source_type!r}")
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	258 else:
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	259 try:
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	260 return await cb(client, attachment, source, dest_path, extra)
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	261 except exceptions.CancelError as e:
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	262 # the handler can't or doesn't want to handle this source
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	263 log.debug(
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	264 f"Following source handling by {cb} has been cancelled ({e}):"
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	265 f"{source}"
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	266 )
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	267
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	268 log.warning(
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	269 "no source could be handled, we can't download the attachment:\n"
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	270 f"{attachment}"
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	271 )
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	272 raise exceptions.FeatureNotFound("no handler could manage the attachment")
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	273
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	274 def register_download_handler(
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	275 self,
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	276 source_type: str,
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	277 callback: Callable[
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	278 [
4270 0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	279 SatXMPPEntity,
0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	280 Dict[str, Any],
0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	281 Dict[str, Any],
0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	282 Union[str, Path],
0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	283 Dict[str, Any],
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	284 ],
4270 0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	285 Tuple[str, defer.Deferred],
0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	286 ],
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	287 ) -> None:
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	288 """Register a handler to manage a type of attachment source
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	289
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	290 @param source_type: ``type`` of source handled
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	291 This is usually the namespace of the protocol used
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	292 @param callback: method to call to manage the source.
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	293 Call arguments are the same as for [download], with an extra ``source`` dict
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	294 which is used just after ``attachment`` to give a quick reference to the
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	295 source used.
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	296 The callabke must return a tuple with:
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	297 - progress ID
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	298 - a Deferred which fire whant the file is fully downloaded
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	299 """
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	300 if source_type is self._download_callbacks:
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	301 raise exceptions.ConflictError(
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	302 f"The is already a callback registered for source type {source_type!r}"
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	303 )
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	304 self._download_callbacks[source_type] = callback
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	305
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	306 def register_scheme(self, scheme: str, download_cb: Callable) -> None:
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	307 """Register an URI scheme handler
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	308
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	309 @param scheme: URI scheme this callback is handling
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	310 @param download_cb: callback to download a file
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	311 arguments are:
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	312 - (SatXMPPClient) client
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	313 - (urllib.parse.SplitResult) parsed URI
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	314 - (Path) destination path where the file must be downloaded
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	315 - (dict) options
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	316 must return a tuple with progress_id and a Deferred which fire when download
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	317 is finished
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	318 """
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	319 if scheme in self._scheme_callbacks:
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	320 raise exceptions.ConflictError(
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	321 f"A method with scheme {scheme!r} is already registered"
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	322 )
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	323 self._scheme_callbacks[scheme] = download_cb
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	324
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	325 def unregister(self, scheme):
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	326 try:
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	327 del self._scheme_callbacks[scheme]
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	328 except KeyError:
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	329 raise exceptions.NotFound(f"No callback registered for scheme {scheme!r}")
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	330
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	331 def errback_download(self, file_obj, download_d, resp):
3186 84b0c8b4dee0 plugin download, aesgcm: fixed handling of HTTP errors Goffi <goffi@goffi.org> parents: 3136 diff changeset	332 """Set file_obj and download deferred appropriatly after a network error
84b0c8b4dee0 plugin download, aesgcm: fixed handling of HTTP errors Goffi <goffi@goffi.org> parents: 3136 diff changeset	333
84b0c8b4dee0 plugin download, aesgcm: fixed handling of HTTP errors Goffi <goffi@goffi.org> parents: 3136 diff changeset	334 @param file_obj(SatFile): file where the download must be done
84b0c8b4dee0 plugin download, aesgcm: fixed handling of HTTP errors Goffi <goffi@goffi.org> parents: 3136 diff changeset	335 @param download_d(Deferred): deffered which must be fired on complete download
84b0c8b4dee0 plugin download, aesgcm: fixed handling of HTTP errors Goffi <goffi@goffi.org> parents: 3136 diff changeset	336 @param resp(treq.response.IResponse): treq response
84b0c8b4dee0 plugin download, aesgcm: fixed handling of HTTP errors Goffi <goffi@goffi.org> parents: 3136 diff changeset	337 """
84b0c8b4dee0 plugin download, aesgcm: fixed handling of HTTP errors Goffi <goffi@goffi.org> parents: 3136 diff changeset	338 msg = f"HTTP error ({resp.code}): {resp.phrase.decode()}"
84b0c8b4dee0 plugin download, aesgcm: fixed handling of HTTP errors Goffi <goffi@goffi.org> parents: 3136 diff changeset	339 file_obj.close(error=msg)
84b0c8b4dee0 plugin download, aesgcm: fixed handling of HTTP errors Goffi <goffi@goffi.org> parents: 3136 diff changeset	340 download_d.errback(exceptions.NetworkError(msg))
84b0c8b4dee0 plugin download, aesgcm: fixed handling of HTTP errors Goffi <goffi@goffi.org> parents: 3136 diff changeset	341
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	342 async def download_http(self, client, uri_parsed, dest_path, options):
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	343 url = uri_parsed.geturl()
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	344
4270 0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	345 if options.get("ignore_tls_errors", False):
0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	346 log.warning("TLS certificate check disabled, this is highly insecure")
3205 2c0628f3927e plugin download, aesgcm: disable TLS check if `check_certificate` setting is disabled Goffi <goffi@goffi.org> parents: 3187 diff changeset	347 treq_client = treq_client_no_ssl
2c0628f3927e plugin download, aesgcm: disable TLS check if `check_certificate` setting is disabled Goffi <goffi@goffi.org> parents: 3187 diff changeset	348 else:
2c0628f3927e plugin download, aesgcm: disable TLS check if `check_certificate` setting is disabled Goffi <goffi@goffi.org> parents: 3187 diff changeset	349 treq_client = treq
2c0628f3927e plugin download, aesgcm: disable TLS check if `check_certificate` setting is disabled Goffi <goffi@goffi.org> parents: 3187 diff changeset	350
3211 4252176ad993 plugin download: clean unfinished files and re-raise exception in case of download error Goffi <goffi@goffi.org> parents: 3205 diff changeset	351 head_data = await treq_client.head(url)
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	352 try:
4270 0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	353 content_length = int(head_data.headers.getRawHeaders("content-length")[0])
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	354 except (KeyError, TypeError, IndexError):
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	355 content_length = None
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	356 log.debug(f"No content lenght found at {url}")
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	357 file_obj = stream.SatFile(
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	358 self.host,
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	359 client,
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	360 dest_path,
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	361 mode="wb",
4270 0d7bb4df2343 Reformatted code base using black. Goffi <goffi@goffi.org> parents: 4071 diff changeset	362 size=content_length,
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	363 )
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	364
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	365 progress_id = file_obj.uid
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	366
3205 2c0628f3927e plugin download, aesgcm: disable TLS check if `check_certificate` setting is disabled Goffi <goffi@goffi.org> parents: 3187 diff changeset	367 resp = await treq_client.get(url, unbuffered=True)
3186 84b0c8b4dee0 plugin download, aesgcm: fixed handling of HTTP errors Goffi <goffi@goffi.org> parents: 3136 diff changeset	368 if resp.code == 200:
84b0c8b4dee0 plugin download, aesgcm: fixed handling of HTTP errors Goffi <goffi@goffi.org> parents: 3136 diff changeset	369 d = treq.collect(resp, file_obj.write)
84b0c8b4dee0 plugin download, aesgcm: fixed handling of HTTP errors Goffi <goffi@goffi.org> parents: 3136 diff changeset	370 d.addBoth(lambda _: file_obj.close())
84b0c8b4dee0 plugin download, aesgcm: fixed handling of HTTP errors Goffi <goffi@goffi.org> parents: 3136 diff changeset	371 else:
84b0c8b4dee0 plugin download, aesgcm: fixed handling of HTTP errors Goffi <goffi@goffi.org> parents: 3136 diff changeset	372 d = defer.Deferred()
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	373 self.errback_download(file_obj, d, resp)
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	374 return progress_id, d

Mercurial > libervia-backend

annotate libervia/backend/plugins/plugin_misc_download.py @ 4306:94e0968987cd