libervia-backend: libervia/backend/plugins/plugin_misc

annotate libervia/backend/plugins/plugin_misc_download.py @ 4094:c3b68fdc2de7

component AP gateway: fix handling of XMPP comments authors: the gateway was supposing that comments where emitted from PEP of author. While this is the case for most blog posts, it's not for comments. Instead the component is now using `author_jid` which is retrieved by XEP-0277 plugin, and reject the item if the auhor is not verified (i.e. if `publisher` attribute is not set by XMPP service).

author	Goffi <goffi@goffi.org>
date	Mon, 12 Jun 2023 14:50:43 +0200
parents	4b842c1fb686
children	0d7bb4df2343

rev	line source
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	1 #!/usr/bin/env python3
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	2
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	3 # SAT plugin for downloading files
3479 be6d91572633 date update Goffi <goffi@goffi.org> parents: 3211 diff changeset	4 # Copyright (C) 2009-2021 Jérôme Poisson (goffi@goffi.org)
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	5
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	6 # This program is free software: you can redistribute it and/or modify
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	7 # it under the terms of the GNU Affero General Public License as published by
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	8 # the Free Software Foundation, either version 3 of the License, or
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	9 # (at your option) any later version.
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	10
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	11 # This program is distributed in the hope that it will be useful,
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	12 # but WITHOUT ANY WARRANTY; without even the implied warranty of
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	13 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	14 # GNU Affero General Public License for more details.
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	15
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	16 # You should have received a copy of the GNU Affero General Public License
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	17 # along with this program. If not, see <http://www.gnu.org/licenses/>.
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	18
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	19 import hashlib
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	20 from pathlib import Path
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	21 from typing import Any, Dict, Optional, Union, Tuple, Callable
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	22 from urllib.parse import unquote, urlparse
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	23
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	24 import treq
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	25 from twisted.internet import defer
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	26 from twisted.words.protocols.jabber import error as jabber_error
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	27
4071 4b842c1fb686 refactoring: renamed `sat` package to `libervia.backend` Goffi <goffi@goffi.org> parents: 4037 diff changeset	28 from libervia.backend.core import exceptions
4b842c1fb686 refactoring: renamed `sat` package to `libervia.backend` Goffi <goffi@goffi.org> parents: 4037 diff changeset	29 from libervia.backend.core.constants import Const as C
4b842c1fb686 refactoring: renamed `sat` package to `libervia.backend` Goffi <goffi@goffi.org> parents: 4037 diff changeset	30 from libervia.backend.core.core_types import SatXMPPEntity
4b842c1fb686 refactoring: renamed `sat` package to `libervia.backend` Goffi <goffi@goffi.org> parents: 4037 diff changeset	31 from libervia.backend.core.i18n import D_, _
4b842c1fb686 refactoring: renamed `sat` package to `libervia.backend` Goffi <goffi@goffi.org> parents: 4037 diff changeset	32 from libervia.backend.core.log import getLogger
4b842c1fb686 refactoring: renamed `sat` package to `libervia.backend` Goffi <goffi@goffi.org> parents: 4037 diff changeset	33 from libervia.backend.tools import xml_tools
4b842c1fb686 refactoring: renamed `sat` package to `libervia.backend` Goffi <goffi@goffi.org> parents: 4037 diff changeset	34 from libervia.backend.tools import stream
4b842c1fb686 refactoring: renamed `sat` package to `libervia.backend` Goffi <goffi@goffi.org> parents: 4037 diff changeset	35 from libervia.backend.tools.common import data_format
4b842c1fb686 refactoring: renamed `sat` package to `libervia.backend` Goffi <goffi@goffi.org> parents: 4037 diff changeset	36 from libervia.backend.tools.web import treq_client_no_ssl
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	37
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	38 log = getLogger(__name__)
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	39
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	40
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	41 PLUGIN_INFO = {
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	42 C.PI_NAME: "File Download",
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	43 C.PI_IMPORT_NAME: "DOWNLOAD",
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	44 C.PI_TYPE: C.PLUG_TYPE_MISC,
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	45 C.PI_MODES: C.PLUG_MODE_BOTH,
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	46 C.PI_MAIN: "DownloadPlugin",
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	47 C.PI_HANDLER: "no",
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	48 C.PI_DESCRIPTION: _("""File download management"""),
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	49 }
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	50
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	51
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	52 class DownloadPlugin(object):
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	53
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	54 def __init__(self, host):
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	55 log.info(_("plugin Download initialization"))
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	56 self.host = host
4037 524856bd7b19 massive refactoring to switch from camelCase to snake_case: Goffi <goffi@goffi.org> parents: 3922 diff changeset	57 host.bridge.add_method(
524856bd7b19 massive refactoring to switch from camelCase to snake_case: Goffi <goffi@goffi.org> parents: 3922 diff changeset	58 "file_download",
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	59 ".plugin",
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	60 in_sign="ssss",
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	61 out_sign="s",
4037 524856bd7b19 massive refactoring to switch from camelCase to snake_case: Goffi <goffi@goffi.org> parents: 3922 diff changeset	62 method=self._file_download,
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	63 async_=True,
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	64 )
4037 524856bd7b19 massive refactoring to switch from camelCase to snake_case: Goffi <goffi@goffi.org> parents: 3922 diff changeset	65 host.bridge.add_method(
524856bd7b19 massive refactoring to switch from camelCase to snake_case: Goffi <goffi@goffi.org> parents: 3922 diff changeset	66 "file_download_complete",
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	67 ".plugin",
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	68 in_sign="ssss",
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	69 out_sign="s",
4037 524856bd7b19 massive refactoring to switch from camelCase to snake_case: Goffi <goffi@goffi.org> parents: 3922 diff changeset	70 method=self._file_download_complete,
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	71 async_=True,
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	72 )
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	73 self._download_callbacks = {}
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	74 self._scheme_callbacks = {}
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	75 self.register_scheme('http', self.download_http)
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	76 self.register_scheme('https', self.download_http)
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	77
4037 524856bd7b19 massive refactoring to switch from camelCase to snake_case: Goffi <goffi@goffi.org> parents: 3922 diff changeset	78 def _file_download(
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	79 self, attachment_s: str, dest_path: str, extra_s: str, profile: str
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	80 ) -> defer.Deferred:
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	81 d = defer.ensureDeferred(self.file_download(
4037 524856bd7b19 massive refactoring to switch from camelCase to snake_case: Goffi <goffi@goffi.org> parents: 3922 diff changeset	82 self.host.get_client(profile),
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	83 data_format.deserialise(attachment_s),
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	84 Path(dest_path),
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	85 data_format.deserialise(extra_s)
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	86 ))
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	87 d.addCallback(lambda ret: data_format.serialise(ret))
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	88 return d
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	89
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	90 async def file_download(
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	91 self,
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	92 client: SatXMPPEntity,
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	93 attachment: Dict[str, Any],
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	94 dest_path: Path,
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	95 extra: Optional[Dict[str, Any]] = None
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	96 ) -> Dict[str, Any]:
3187 d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	97 """Download a file using best available method
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	98
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	99 parameters are the same as for [download]
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	100 @return (dict): action dictionary, with progress id in case of success, else xmlui
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	101 message
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	102 """
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	103 try:
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	104 progress_id, __ = await self.download(client, attachment, dest_path, extra)
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	105 except Exception as e:
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	106 if (isinstance(e, jabber_error.StanzaError)
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	107 and e.condition == 'not-acceptable'):
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	108 reason = e.text
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	109 else:
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	110 reason = str(e)
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	111 msg = D_("Can't download file: {reason}").format(reason=reason)
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	112 log.warning(msg)
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	113 return {
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	114 "xmlui": xml_tools.note(
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	115 msg, D_("Can't download file"), C.XMLUI_DATA_LVL_WARNING
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	116 ).toXml()
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	117 }
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	118 else:
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	119 return {"progress": progress_id}
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	120
4037 524856bd7b19 massive refactoring to switch from camelCase to snake_case: Goffi <goffi@goffi.org> parents: 3922 diff changeset	121 def _file_download_complete(
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	122 self, attachment_s: str, dest_path: str, extra_s: str, profile: str
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	123 ) -> defer.Deferred:
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	124 d = defer.ensureDeferred(self.file_download_complete(
4037 524856bd7b19 massive refactoring to switch from camelCase to snake_case: Goffi <goffi@goffi.org> parents: 3922 diff changeset	125 self.host.get_client(profile),
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	126 data_format.deserialise(attachment_s),
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	127 Path(dest_path),
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	128 data_format.deserialise(extra_s)
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	129 ))
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	130 d.addCallback(lambda path: str(path))
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	131 return d
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	132
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	133 async def file_download_complete(
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	134 self,
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	135 client: SatXMPPEntity,
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	136 attachment: Dict[str, Any],
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	137 dest_path: Path,
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	138 extra: Optional[Dict[str, Any]] = None
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	139 ) -> str:
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	140 """Helper method to fully download a file and return its path
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	141
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	142 parameters are the same as for [download]
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	143 @return (str): path to the downloaded file
3187 d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	144 use empty string to store the file in cache
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	145 """
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	146 __, download_d = await self.download(client, attachment, dest_path, extra)
3187 d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	147 dest_path = await download_d
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	148 return dest_path
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	149
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	150 async def download_uri(
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	151 self,
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	152 client: SatXMPPEntity,
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	153 uri: str,
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	154 dest_path: Union[Path, str],
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	155 extra: Optional[Dict[str, Any]] = None
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	156 ) -> Tuple[str, defer.Deferred]:
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	157 if extra is None:
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	158 extra = {}
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	159 uri_parsed = urlparse(uri, 'http')
3187 d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	160 if dest_path:
d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	161 dest_path = Path(dest_path)
3211 4252176ad993 plugin download: clean unfinished files and re-raise exception in case of download error Goffi <goffi@goffi.org> parents: 3205 diff changeset	162 cache_uid = None
3187 d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	163 else:
d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	164 filename = Path(unquote(uri_parsed.path)).name.strip() or C.FILE_DEFAULT_NAME
d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	165 # we don't use Path.suffixes because we don't want to have more than 2
d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	166 # suffixes, but we still want to handle suffixes like "tar.gz".
d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	167 stem, *suffixes = filename.rsplit('.', 2)
d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	168 # we hash the URL to have an unique identifier, and avoid double download
d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	169 url_hash = hashlib.sha256(uri_parsed.geturl().encode()).hexdigest()
3211 4252176ad993 plugin download: clean unfinished files and re-raise exception in case of download error Goffi <goffi@goffi.org> parents: 3205 diff changeset	170 cache_uid = f"{stem}_{url_hash}"
4037 524856bd7b19 massive refactoring to switch from camelCase to snake_case: Goffi <goffi@goffi.org> parents: 3922 diff changeset	171 cache_data = client.cache.get_metadata(cache_uid)
3187 d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	172 if cache_data is not None:
d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	173 # file is already in cache, we return it
d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	174 download_d = defer.succeed(cache_data['path'])
d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	175 return '', download_d
d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	176 else:
d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	177 # the file is not in cache
3211 4252176ad993 plugin download: clean unfinished files and re-raise exception in case of download error Goffi <goffi@goffi.org> parents: 3205 diff changeset	178 unique_name = '.'.join([cache_uid] + suffixes)
4037 524856bd7b19 massive refactoring to switch from camelCase to snake_case: Goffi <goffi@goffi.org> parents: 3922 diff changeset	179 with client.cache.cache_data(
3211 4252176ad993 plugin download: clean unfinished files and re-raise exception in case of download error Goffi <goffi@goffi.org> parents: 3205 diff changeset	180 "DOWNLOAD", cache_uid, filename=unique_name) as f:
3187 d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	181 # we close the file and only use its name, the file will be opened
d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	182 # by the registered callback
3211 4252176ad993 plugin download: clean unfinished files and re-raise exception in case of download error Goffi <goffi@goffi.org> parents: 3205 diff changeset	183 dest_path = Path(f.name)
3205 2c0628f3927e plugin download, aesgcm: disable TLS check if `check_certificate` setting is disabled Goffi <goffi@goffi.org> parents: 3187 diff changeset	184
2c0628f3927e plugin download, aesgcm: disable TLS check if `check_certificate` setting is disabled Goffi <goffi@goffi.org> parents: 3187 diff changeset	185 # should we check certificates?
4037 524856bd7b19 massive refactoring to switch from camelCase to snake_case: Goffi <goffi@goffi.org> parents: 3922 diff changeset	186 check_certificate = self.host.memory.param_get_a(
3205 2c0628f3927e plugin download, aesgcm: disable TLS check if `check_certificate` setting is disabled Goffi <goffi@goffi.org> parents: 3187 diff changeset	187 "check_certificate", "Connection", profile_key=client.profile)
2c0628f3927e plugin download, aesgcm: disable TLS check if `check_certificate` setting is disabled Goffi <goffi@goffi.org> parents: 3187 diff changeset	188 if not check_certificate:
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	189 extra['ignore_tls_errors'] = True
3205 2c0628f3927e plugin download, aesgcm: disable TLS check if `check_certificate` setting is disabled Goffi <goffi@goffi.org> parents: 3187 diff changeset	190 log.warning(
2c0628f3927e plugin download, aesgcm: disable TLS check if `check_certificate` setting is disabled Goffi <goffi@goffi.org> parents: 3187 diff changeset	191 _("certificate check disabled for download, this is dangerous!"))
2c0628f3927e plugin download, aesgcm: disable TLS check if `check_certificate` setting is disabled Goffi <goffi@goffi.org> parents: 3187 diff changeset	192
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	193 try:
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	194 callback = self._scheme_callbacks[uri_parsed.scheme]
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	195 except KeyError:
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	196 raise exceptions.NotFound(f"Can't find any handler for uri {uri}")
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	197 else:
3211 4252176ad993 plugin download: clean unfinished files and re-raise exception in case of download error Goffi <goffi@goffi.org> parents: 3205 diff changeset	198 try:
4252176ad993 plugin download: clean unfinished files and re-raise exception in case of download error Goffi <goffi@goffi.org> parents: 3205 diff changeset	199 progress_id, download_d = await callback(
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	200 client, uri_parsed, dest_path, extra)
3211 4252176ad993 plugin download: clean unfinished files and re-raise exception in case of download error Goffi <goffi@goffi.org> parents: 3205 diff changeset	201 except Exception as e:
4252176ad993 plugin download: clean unfinished files and re-raise exception in case of download error Goffi <goffi@goffi.org> parents: 3205 diff changeset	202 log.warning(_(
4252176ad993 plugin download: clean unfinished files and re-raise exception in case of download error Goffi <goffi@goffi.org> parents: 3205 diff changeset	203 "Can't download URI {uri}: {reason}").format(
4252176ad993 plugin download: clean unfinished files and re-raise exception in case of download error Goffi <goffi@goffi.org> parents: 3205 diff changeset	204 uri=uri, reason=e))
4252176ad993 plugin download: clean unfinished files and re-raise exception in case of download error Goffi <goffi@goffi.org> parents: 3205 diff changeset	205 if cache_uid is not None:
4037 524856bd7b19 massive refactoring to switch from camelCase to snake_case: Goffi <goffi@goffi.org> parents: 3922 diff changeset	206 client.cache.remove_from_cache(cache_uid)
3211 4252176ad993 plugin download: clean unfinished files and re-raise exception in case of download error Goffi <goffi@goffi.org> parents: 3205 diff changeset	207 elif dest_path.exists():
4252176ad993 plugin download: clean unfinished files and re-raise exception in case of download error Goffi <goffi@goffi.org> parents: 3205 diff changeset	208 dest_path.unlink()
4252176ad993 plugin download: clean unfinished files and re-raise exception in case of download error Goffi <goffi@goffi.org> parents: 3205 diff changeset	209 raise e
3187 d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	210 download_d.addCallback(lambda __: dest_path)
d92a144f3589 plugin download: use cache if dest_path is empty: Goffi <goffi@goffi.org> parents: 3186 diff changeset	211 return progress_id, download_d
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	212
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	213
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	214 async def download(
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	215 self,
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	216 client: SatXMPPEntity,
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	217 attachment: Dict[str, Any],
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	218 dest_path: Union[Path, str],
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	219 extra: Optional[Dict[str, Any]] = None
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	220 ) -> Tuple[str, defer.Deferred]:
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	221 """Download a file from URI using suitable method
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	222
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	223 @param uri: URI to the file to download
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	224 @param dest_path: where the file must be downloaded
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	225 if empty string, the file will be stored in local path
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	226 @param extra: options depending on scheme handler
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	227 Some common options:
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	228 - ignore_tls_errors(bool): True to ignore SSL/TLS certificate verification
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	229 used only if HTTPS transport is needed
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	230 @return: ``progress_id`` and a Deferred which fire download URL when download is
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	231 finished.
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	232 ``progress_id`` can be empty string if the file already exist and is not
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	233 downloaded again (can happen if cache is used with empty ``dest_path``).
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	234 """
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	235 uri = attachment.get("uri")
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	236 if uri:
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	237 return await self.download_uri(client, uri, dest_path, extra)
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	238 else:
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	239 for source in attachment.get("sources", []):
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	240 source_type = source.get("type")
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	241 if not source_type:
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	242 log.warning(
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	243 "source type is missing for source: {source}\nattachment: "
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	244 f"{attachment}"
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	245 )
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	246 continue
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	247 try:
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	248 cb = self._download_callbacks[source_type]
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	249 except KeyError:
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	250 log.warning(
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	251 f"no source handler registered for {source_type!r}"
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	252 )
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	253 else:
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	254 try:
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	255 return await cb(client, attachment, source, dest_path, extra)
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	256 except exceptions.CancelError as e:
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	257 # the handler can't or doesn't want to handle this source
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	258 log.debug(
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	259 f"Following source handling by {cb} has been cancelled ({e}):"
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	260 f"{source}"
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	261 )
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	262
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	263 log.warning(
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	264 "no source could be handled, we can't download the attachment:\n"
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	265 f"{attachment}"
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	266 )
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	267 raise exceptions.FeatureNotFound("no handler could manage the attachment")
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	268
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	269 def register_download_handler(
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	270 self,
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	271 source_type: str,
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	272 callback: Callable[
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	273 [
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	274 SatXMPPEntity, Dict[str, Any], Dict[str, Any], Union[str, Path],
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	275 Dict[str, Any]
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	276 ],
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	277 Tuple[str, defer.Deferred]
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	278 ]
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	279 ) -> None:
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	280 """Register a handler to manage a type of attachment source
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	281
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	282 @param source_type: ``type`` of source handled
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	283 This is usually the namespace of the protocol used
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	284 @param callback: method to call to manage the source.
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	285 Call arguments are the same as for [download], with an extra ``source`` dict
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	286 which is used just after ``attachment`` to give a quick reference to the
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	287 source used.
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	288 The callabke must return a tuple with:
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	289 - progress ID
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	290 - a Deferred which fire whant the file is fully downloaded
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	291 """
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	292 if source_type is self._download_callbacks:
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	293 raise exceptions.ConflictError(
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	294 f"The is already a callback registered for source type {source_type!r}"
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	295 )
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	296 self._download_callbacks[source_type] = callback
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	297
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	298 def register_scheme(self, scheme: str, download_cb: Callable) -> None:
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	299 """Register an URI scheme handler
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	300
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	301 @param scheme: URI scheme this callback is handling
0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	302 @param download_cb: callback to download a file
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	303 arguments are:
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	304 - (SatXMPPClient) client
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	305 - (urllib.parse.SplitResult) parsed URI
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	306 - (Path) destination path where the file must be downloaded
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	307 - (dict) options
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	308 must return a tuple with progress_id and a Deferred which fire when download
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	309 is finished
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	310 """
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	311 if scheme in self._scheme_callbacks:
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	312 raise exceptions.ConflictError(
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	313 f"A method with scheme {scheme!r} is already registered"
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	314 )
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	315 self._scheme_callbacks[scheme] = download_cb
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	316
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	317 def unregister(self, scheme):
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	318 try:
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	319 del self._scheme_callbacks[scheme]
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	320 except KeyError:
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	321 raise exceptions.NotFound(f"No callback registered for scheme {scheme!r}")
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	322
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	323 def errback_download(self, file_obj, download_d, resp):
3186 84b0c8b4dee0 plugin download, aesgcm: fixed handling of HTTP errors Goffi <goffi@goffi.org> parents: 3136 diff changeset	324 """Set file_obj and download deferred appropriatly after a network error
84b0c8b4dee0 plugin download, aesgcm: fixed handling of HTTP errors Goffi <goffi@goffi.org> parents: 3136 diff changeset	325
84b0c8b4dee0 plugin download, aesgcm: fixed handling of HTTP errors Goffi <goffi@goffi.org> parents: 3136 diff changeset	326 @param file_obj(SatFile): file where the download must be done
84b0c8b4dee0 plugin download, aesgcm: fixed handling of HTTP errors Goffi <goffi@goffi.org> parents: 3136 diff changeset	327 @param download_d(Deferred): deffered which must be fired on complete download
84b0c8b4dee0 plugin download, aesgcm: fixed handling of HTTP errors Goffi <goffi@goffi.org> parents: 3136 diff changeset	328 @param resp(treq.response.IResponse): treq response
84b0c8b4dee0 plugin download, aesgcm: fixed handling of HTTP errors Goffi <goffi@goffi.org> parents: 3136 diff changeset	329 """
84b0c8b4dee0 plugin download, aesgcm: fixed handling of HTTP errors Goffi <goffi@goffi.org> parents: 3136 diff changeset	330 msg = f"HTTP error ({resp.code}): {resp.phrase.decode()}"
84b0c8b4dee0 plugin download, aesgcm: fixed handling of HTTP errors Goffi <goffi@goffi.org> parents: 3136 diff changeset	331 file_obj.close(error=msg)
84b0c8b4dee0 plugin download, aesgcm: fixed handling of HTTP errors Goffi <goffi@goffi.org> parents: 3136 diff changeset	332 download_d.errback(exceptions.NetworkError(msg))
84b0c8b4dee0 plugin download, aesgcm: fixed handling of HTTP errors Goffi <goffi@goffi.org> parents: 3136 diff changeset	333
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	334 async def download_http(self, client, uri_parsed, dest_path, options):
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	335 url = uri_parsed.geturl()
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	336
3205 2c0628f3927e plugin download, aesgcm: disable TLS check if `check_certificate` setting is disabled Goffi <goffi@goffi.org> parents: 3187 diff changeset	337 if options.get('ignore_tls_errors', False):
2c0628f3927e plugin download, aesgcm: disable TLS check if `check_certificate` setting is disabled Goffi <goffi@goffi.org> parents: 3187 diff changeset	338 log.warning(
2c0628f3927e plugin download, aesgcm: disable TLS check if `check_certificate` setting is disabled Goffi <goffi@goffi.org> parents: 3187 diff changeset	339 "TLS certificate check disabled, this is highly insecure"
2c0628f3927e plugin download, aesgcm: disable TLS check if `check_certificate` setting is disabled Goffi <goffi@goffi.org> parents: 3187 diff changeset	340 )
2c0628f3927e plugin download, aesgcm: disable TLS check if `check_certificate` setting is disabled Goffi <goffi@goffi.org> parents: 3187 diff changeset	341 treq_client = treq_client_no_ssl
2c0628f3927e plugin download, aesgcm: disable TLS check if `check_certificate` setting is disabled Goffi <goffi@goffi.org> parents: 3187 diff changeset	342 else:
2c0628f3927e plugin download, aesgcm: disable TLS check if `check_certificate` setting is disabled Goffi <goffi@goffi.org> parents: 3187 diff changeset	343 treq_client = treq
2c0628f3927e plugin download, aesgcm: disable TLS check if `check_certificate` setting is disabled Goffi <goffi@goffi.org> parents: 3187 diff changeset	344
3211 4252176ad993 plugin download: clean unfinished files and re-raise exception in case of download error Goffi <goffi@goffi.org> parents: 3205 diff changeset	345 head_data = await treq_client.head(url)
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	346 try:
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	347 content_length = int(head_data.headers.getRawHeaders('content-length')[0])
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	348 except (KeyError, TypeError, IndexError):
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	349 content_length = None
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	350 log.debug(f"No content lenght found at {url}")
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	351 file_obj = stream.SatFile(
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	352 self.host,
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	353 client,
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	354 dest_path,
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	355 mode="wb",
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	356 size = content_length,
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	357 )
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	358
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	359 progress_id = file_obj.uid
d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	360
3205 2c0628f3927e plugin download, aesgcm: disable TLS check if `check_certificate` setting is disabled Goffi <goffi@goffi.org> parents: 3187 diff changeset	361 resp = await treq_client.get(url, unbuffered=True)
3186 84b0c8b4dee0 plugin download, aesgcm: fixed handling of HTTP errors Goffi <goffi@goffi.org> parents: 3136 diff changeset	362 if resp.code == 200:
84b0c8b4dee0 plugin download, aesgcm: fixed handling of HTTP errors Goffi <goffi@goffi.org> parents: 3136 diff changeset	363 d = treq.collect(resp, file_obj.write)
84b0c8b4dee0 plugin download, aesgcm: fixed handling of HTTP errors Goffi <goffi@goffi.org> parents: 3136 diff changeset	364 d.addBoth(lambda _: file_obj.close())
84b0c8b4dee0 plugin download, aesgcm: fixed handling of HTTP errors Goffi <goffi@goffi.org> parents: 3136 diff changeset	365 else:
84b0c8b4dee0 plugin download, aesgcm: fixed handling of HTTP errors Goffi <goffi@goffi.org> parents: 3136 diff changeset	366 d = defer.Deferred()
3922 0ff265725489 plugin XEP-0447: handle attachment and download: Goffi <goffi@goffi.org> parents: 3479 diff changeset	367 self.errback_download(file_obj, d, resp)
3088 d1464548055a plugin file download: meta plugin to handle file download: Goffi <goffi@goffi.org> parents: diff changeset	368 return progress_id, d

Mercurial > libervia-backend

annotate libervia/backend/plugins/plugin_misc_download.py @ 4094:c3b68fdc2de7