libervia-backend: sat/plugins/plugin_misc_text

annotate sat/plugins/plugin_misc_text_syntaxes.py @ 4002:5245b675f7ad

plugin XEP-0313: don't wait for MAM to be retrieved in connection workflow: MAM retrieval can be long, and can be done after connection, message just need to be sorted when being inserted (i.e. frontends must do insort). To avoid blocking connection for too long and result in bad UX and timeout risk, one2one MAM message are not retrieved in background.

author	Goffi <goffi@goffi.org>
date	Fri, 10 Mar 2023 17:22:45 +0100
parents	33d75cd3c371
children	524856bd7b19

rev	line source
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2878 diff changeset	1 #!/usr/bin/env python3
3137 559a625a236b fixed shebangs Goffi <goffi@goffi.org> parents: 3136 diff changeset	2
665 6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	3
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	4 # SAT plugin for managing various text syntaxes
3479 be6d91572633 date update Goffi <goffi@goffi.org> parents: 3137 diff changeset	5 # Copyright (C) 2009-2021 Jérôme Poisson (goffi@goffi.org)
665 6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	6
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	7 # This program is free software: you can redistribute it and/or modify
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	8 # it under the terms of the GNU Affero General Public License as published by
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	9 # the Free Software Foundation, either version 3 of the License, or
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	10 # (at your option) any later version.
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	11
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	12 # This program is distributed in the hope that it will be useful,
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	13 # but WITHOUT ANY WARRANTY; without even the implied warranty of
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	14 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	15 # GNU Affero General Public License for more details.
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	16
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	17 # You should have received a copy of the GNU Affero General Public License
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	18 # along with this program. If not, see <http://www.gnu.org/licenses/>.
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	19
3693 0bbdc50aa405 plugin text syntaxes: remove `id` attributes and whitelist allowed classes: Goffi <goffi@goffi.org> parents: 3479 diff changeset	20 from functools import partial
3075 501a1a3c8594 plugin text syntaxes: don't use anymore deprecated cgi.escape Goffi <goffi@goffi.org> parents: 3040 diff changeset	21 from html import escape
3693 0bbdc50aa405 plugin text syntaxes: remove `id` attributes and whitelist allowed classes: Goffi <goffi@goffi.org> parents: 3479 diff changeset	22 import re
0bbdc50aa405 plugin text syntaxes: remove `id` attributes and whitelist allowed classes: Goffi <goffi@goffi.org> parents: 3479 diff changeset	23 from typing import Set
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	24
665 6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	25 from twisted.internet import defer
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	26 from twisted.internet.threads import deferToThread
3693 0bbdc50aa405 plugin text syntaxes: remove `id` attributes and whitelist allowed classes: Goffi <goffi@goffi.org> parents: 3479 diff changeset	27
705 6c8a119dcc94 plugin text syntaxes: clean_xhtml now accept lxml's HtmlElement to avoid parsing two times the same xml Goffi <goffi@goffi.org> parents: 702 diff changeset	28 from sat.core import exceptions
3693 0bbdc50aa405 plugin text syntaxes: remove `id` attributes and whitelist allowed classes: Goffi <goffi@goffi.org> parents: 3479 diff changeset	29 from sat.core.constants import Const as C
0bbdc50aa405 plugin text syntaxes: remove `id` attributes and whitelist allowed classes: Goffi <goffi@goffi.org> parents: 3479 diff changeset	30 from sat.core.i18n import D_, _
0bbdc50aa405 plugin text syntaxes: remove `id` attributes and whitelist allowed classes: Goffi <goffi@goffi.org> parents: 3479 diff changeset	31 from sat.core.log import getLogger
2782 b17e6fa1e607 core (XMLUI): new XHTMLBox widget: Goffi <goffi@goffi.org> parents: 2781 diff changeset	32 from sat.tools import xml_tools
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	33
1542 94901070478e plugins: added new MissingModule exceptions to plugins using third party modules Goffi <goffi@goffi.org> parents: 1458 diff changeset	34 try:
94901070478e plugins: added new MissingModule exceptions to plugins using third party modules Goffi <goffi@goffi.org> parents: 1458 diff changeset	35 from lxml import html
94901070478e plugins: added new MissingModule exceptions to plugins using third party modules Goffi <goffi@goffi.org> parents: 1458 diff changeset	36 from lxml.html import clean
2786 be8405795e09 plugin text syntaxes: handle empty content in cleanXHTML + don't use self-closing tags for non-void elements. Goffi <goffi@goffi.org> parents: 2782 diff changeset	37 from lxml import etree
1542 94901070478e plugins: added new MissingModule exceptions to plugins using third party modules Goffi <goffi@goffi.org> parents: 1458 diff changeset	38 except ImportError:
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	39 raise exceptions.MissingModule(
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2878 diff changeset	40 "Missing module lxml, please download/install it from http://lxml.de/"
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	41 )
674 fb0b1100c908 plugin text_syntaxes: fixed clean_xhml (it now return XHTML instead of HTML) Goffi <goffi@goffi.org> parents: 665 diff changeset	42
2873 e1207b8ad97c plugin text syntaxes: disable raw HTML parsing in mardown by default Goffi <goffi@goffi.org> parents: 2869 diff changeset	43 log = getLogger(__name__)
665 6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	44
771 bfabeedbf32e core: i18n refactoring: Goffi <goffi@goffi.org> parents: 744 diff changeset	45 CATEGORY = D_("Composition")
665 6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	46 NAME = "Syntax"
2869 148d30147890 plugin text syntaxes: fixed default syntax Goffi <goffi@goffi.org> parents: 2786 diff changeset	47 _SYNTAX_XHTML = "xhtml" # must be lower case
744 312a2842b2b8 plugins text-syntaxes: added a default value to use the current user syntax in convert souliane <souliane@mailoo.org> parents: 705 diff changeset	48 _SYNTAX_CURRENT = "@CURRENT@"
312a2842b2b8 plugins text-syntaxes: added a default value to use the current user syntax in convert souliane <souliane@mailoo.org> parents: 705 diff changeset	49
692 e98db42cd78c plugin text syntaxes: styles sanitisation Goffi <goffi@goffi.org> parents: 674 diff changeset	50 # TODO: check/adapt following list
1805 3c40fa0dcd7a pluging text syntaxes: various minor improvments: Goffi <goffi@goffi.org> parents: 1803 diff changeset	51 # list initialy based on feedparser list (http://pythonhosted.org/feedparser/html-sanitization.html)
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	52 STYLES_WHITELIST = (
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	53 "azimuth",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	54 "background-color",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	55 "border-bottom-color",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	56 "border-collapse",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	57 "border-color",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	58 "border-left-color",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	59 "border-right-color",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	60 "border-top-color",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	61 "clear",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	62 "color",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	63 "cursor",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	64 "direction",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	65 "display",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	66 "elevation",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	67 "float",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	68 "font",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	69 "font-family",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	70 "font-size",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	71 "font-style",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	72 "font-variant",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	73 "font-weight",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	74 "height",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	75 "letter-spacing",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	76 "line-height",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	77 "overflow",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	78 "pause",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	79 "pause-after",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	80 "pause-before",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	81 "pitch",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	82 "pitch-range",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	83 "richness",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	84 "speak",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	85 "speak-header",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	86 "speak-numeral",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	87 "speak-punctuation",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	88 "speech-rate",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	89 "stress",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	90 "text-align",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	91 "text-decoration",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	92 "text-indent",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	93 "unicode-bidi",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	94 "vertical-align",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	95 "voice-family",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	96 "volume",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	97 "white-space",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	98 "width",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	99 )
665 6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	100
2786 be8405795e09 plugin text syntaxes: handle empty content in cleanXHTML + don't use self-closing tags for non-void elements. Goffi <goffi@goffi.org> parents: 2782 diff changeset	101 # cf. https://www.w3.org/TR/html/syntax.html#void-elements
be8405795e09 plugin text syntaxes: handle empty content in cleanXHTML + don't use self-closing tags for non-void elements. Goffi <goffi@goffi.org> parents: 2782 diff changeset	102 VOID_ELEMENTS = (
be8405795e09 plugin text syntaxes: handle empty content in cleanXHTML + don't use self-closing tags for non-void elements. Goffi <goffi@goffi.org> parents: 2782 diff changeset	103 "area",
be8405795e09 plugin text syntaxes: handle empty content in cleanXHTML + don't use self-closing tags for non-void elements. Goffi <goffi@goffi.org> parents: 2782 diff changeset	104 "base",
be8405795e09 plugin text syntaxes: handle empty content in cleanXHTML + don't use self-closing tags for non-void elements. Goffi <goffi@goffi.org> parents: 2782 diff changeset	105 "br",
be8405795e09 plugin text syntaxes: handle empty content in cleanXHTML + don't use self-closing tags for non-void elements. Goffi <goffi@goffi.org> parents: 2782 diff changeset	106 "col",
be8405795e09 plugin text syntaxes: handle empty content in cleanXHTML + don't use self-closing tags for non-void elements. Goffi <goffi@goffi.org> parents: 2782 diff changeset	107 "embed",
be8405795e09 plugin text syntaxes: handle empty content in cleanXHTML + don't use self-closing tags for non-void elements. Goffi <goffi@goffi.org> parents: 2782 diff changeset	108 "hr",
be8405795e09 plugin text syntaxes: handle empty content in cleanXHTML + don't use self-closing tags for non-void elements. Goffi <goffi@goffi.org> parents: 2782 diff changeset	109 "img",
be8405795e09 plugin text syntaxes: handle empty content in cleanXHTML + don't use self-closing tags for non-void elements. Goffi <goffi@goffi.org> parents: 2782 diff changeset	110 "input",
be8405795e09 plugin text syntaxes: handle empty content in cleanXHTML + don't use self-closing tags for non-void elements. Goffi <goffi@goffi.org> parents: 2782 diff changeset	111 "keygen",
be8405795e09 plugin text syntaxes: handle empty content in cleanXHTML + don't use self-closing tags for non-void elements. Goffi <goffi@goffi.org> parents: 2782 diff changeset	112 "link",
be8405795e09 plugin text syntaxes: handle empty content in cleanXHTML + don't use self-closing tags for non-void elements. Goffi <goffi@goffi.org> parents: 2782 diff changeset	113 "menuitem",
be8405795e09 plugin text syntaxes: handle empty content in cleanXHTML + don't use self-closing tags for non-void elements. Goffi <goffi@goffi.org> parents: 2782 diff changeset	114 "meta",
be8405795e09 plugin text syntaxes: handle empty content in cleanXHTML + don't use self-closing tags for non-void elements. Goffi <goffi@goffi.org> parents: 2782 diff changeset	115 "param",
be8405795e09 plugin text syntaxes: handle empty content in cleanXHTML + don't use self-closing tags for non-void elements. Goffi <goffi@goffi.org> parents: 2782 diff changeset	116 "source",
be8405795e09 plugin text syntaxes: handle empty content in cleanXHTML + don't use self-closing tags for non-void elements. Goffi <goffi@goffi.org> parents: 2782 diff changeset	117 "track",
be8405795e09 plugin text syntaxes: handle empty content in cleanXHTML + don't use self-closing tags for non-void elements. Goffi <goffi@goffi.org> parents: 2782 diff changeset	118 "wbr")
be8405795e09 plugin text syntaxes: handle empty content in cleanXHTML + don't use self-closing tags for non-void elements. Goffi <goffi@goffi.org> parents: 2782 diff changeset	119
3693 0bbdc50aa405 plugin text syntaxes: remove `id` attributes and whitelist allowed classes: Goffi <goffi@goffi.org> parents: 3479 diff changeset	120 SAFE_ATTRS = html.defs.safe_attrs.union({"style", "poster", "controls"}) - {"id"}
0bbdc50aa405 plugin text syntaxes: remove `id` attributes and whitelist allowed classes: Goffi <goffi@goffi.org> parents: 3479 diff changeset	121 SAFE_CLASSES = {
0bbdc50aa405 plugin text syntaxes: remove `id` attributes and whitelist allowed classes: Goffi <goffi@goffi.org> parents: 3479 diff changeset	122 # those classes are used for code highlighting
0bbdc50aa405 plugin text syntaxes: remove `id` attributes and whitelist allowed classes: Goffi <goffi@goffi.org> parents: 3479 diff changeset	123 "bp", "c", "ch", "cm", "cp", "cpf", "cs", "dl", "err", "fm", "gd", "ge", "get", "gh",
0bbdc50aa405 plugin text syntaxes: remove `id` attributes and whitelist allowed classes: Goffi <goffi@goffi.org> parents: 3479 diff changeset	124 "gi", "go", "gp", "gr", "gs", "gt", "gu", "highlight", "hll", "il", "k", "kc", "kd",
0bbdc50aa405 plugin text syntaxes: remove `id` attributes and whitelist allowed classes: Goffi <goffi@goffi.org> parents: 3479 diff changeset	125 "kn", "kp", "kr", "kt", "m", "mb", "mf", "mh", "mi", "mo", "na", "nb", "nc", "nd",
0bbdc50aa405 plugin text syntaxes: remove `id` attributes and whitelist allowed classes: Goffi <goffi@goffi.org> parents: 3479 diff changeset	126 "ne", "nf", "ni", "nl", "nn", "no", "nt", "nv", "o", "ow", "s", "sa", "sb", "sc",
0bbdc50aa405 plugin text syntaxes: remove `id` attributes and whitelist allowed classes: Goffi <goffi@goffi.org> parents: 3479 diff changeset	127 "sd", "se", "sh", "si", "sr", "ss", "sx", "vc", "vg", "vi", "vm", "w", "write",
0bbdc50aa405 plugin text syntaxes: remove `id` attributes and whitelist allowed classes: Goffi <goffi@goffi.org> parents: 3479 diff changeset	128 }
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	129 STYLES_VALUES_REGEX = (
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	130 r"^("
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	131 + "\|".join(
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	132 [
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	133 "([a-z-]+)", # alphabetical names
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	134 "(#[0-9a-f]+)", # hex value
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	135 "(\d+(.\d+)? *(\|%\|em\|ex\|px\|in\|cm\|mm\|pt\|pc))", # values with units (or not)
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	136 "rgb$ ((\d+(.\d+)?), ){2}(\d+(.\d+)?) *$", # rgb function
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	137 "rgba$ ((\d+(.\d+)?), ){3}(\d+(.\d+)?) *$", # rgba function
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	138 ]
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	139 )
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	140 + ") *(!important)?$"
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	141 ) # we accept "!important" at the end
692 e98db42cd78c plugin text syntaxes: styles sanitisation Goffi <goffi@goffi.org> parents: 674 diff changeset	142 STYLES_ACCEPTED_VALUE = re.compile(STYLES_VALUES_REGEX)
665 6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	143
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	144 PLUGIN_INFO = {
2145 33c8c4973743 core (plugins): added missing contants + use of new constants in PLUGIN_INFO Goffi <goffi@goffi.org> parents: 2106 diff changeset	145 C.PI_NAME: "Text syntaxes",
2780 85d3240a400f plugin text syntaxes: changed import name to TEXT_SYNTAX (better with underscore for autocompletion) Goffi <goffi@goffi.org> parents: 2771 diff changeset	146 C.PI_IMPORT_NAME: "TEXT_SYNTAXES",
2145 33c8c4973743 core (plugins): added missing contants + use of new constants in PLUGIN_INFO Goffi <goffi@goffi.org> parents: 2106 diff changeset	147 C.PI_TYPE: "MISC",
3726 33d75cd3c371 plugin XEP-0060, XEP-0163, XEP-0277, text syntaxes: make those plugins usable with components Goffi <goffi@goffi.org> parents: 3709 diff changeset	148 C.PI_MODES: C.PLUG_MODE_BOTH,
2145 33c8c4973743 core (plugins): added missing contants + use of new constants in PLUGIN_INFO Goffi <goffi@goffi.org> parents: 2106 diff changeset	149 C.PI_PROTOCOLS: [],
33c8c4973743 core (plugins): added missing contants + use of new constants in PLUGIN_INFO Goffi <goffi@goffi.org> parents: 2106 diff changeset	150 C.PI_DEPENDENCIES: [],
33c8c4973743 core (plugins): added missing contants + use of new constants in PLUGIN_INFO Goffi <goffi@goffi.org> parents: 2106 diff changeset	151 C.PI_MAIN: "TextSyntaxes",
33c8c4973743 core (plugins): added missing contants + use of new constants in PLUGIN_INFO Goffi <goffi@goffi.org> parents: 2106 diff changeset	152 C.PI_HANDLER: "no",
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	153 C.PI_DESCRIPTION: _(
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	154 """Management of various text syntaxes (XHTML-IM, Markdown, etc)"""
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	155 ),
665 6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	156 }
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	157
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	158
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	159 class TextSyntaxes(object):
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	160 """ Text conversion class
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	161 XHTML utf-8 is used as intermediate language for conversions
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	162 """
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	163
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	164 OPT_DEFAULT = "DEFAULT"
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	165 OPT_HIDDEN = "HIDDEN"
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	166 OPT_NO_THREAD = "NO_THREAD"
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	167 SYNTAX_XHTML = _SYNTAX_XHTML
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	168 SYNTAX_MARKDOWN = "markdown"
832 c4b22aedb7d7 plugin groupblog, XEP-0071, XEP-0277, text_syntaxes: manage raw/rich/xhtml data for content/title: souliane <souliane@mailoo.org> parents: 811 diff changeset	169 SYNTAX_TEXT = "text"
2869 148d30147890 plugin text syntaxes: fixed default syntax Goffi <goffi@goffi.org> parents: 2786 diff changeset	170 # default_syntax must be lower case
2324 fe922e6fabd4 plugin text syntaxes: various improvments: Goffi <goffi@goffi.org> parents: 2145 diff changeset	171 default_syntax = SYNTAX_XHTML
665 6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	172
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	173
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	174 def __init__(self, host):
993 301b342c697a core: use of the new core.log module: Goffi <goffi@goffi.org> parents: 968 diff changeset	175 log.info(_("Text syntaxes plugin initialization"))
665 6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	176 self.host = host
3620 f568f304c982 plugin text syntaxes: remove side effect on init: Goffi <goffi@goffi.org> parents: 3479 diff changeset	177 self.syntaxes = {}
f568f304c982 plugin text syntaxes: remove side effect on init: Goffi <goffi@goffi.org> parents: 3479 diff changeset	178
f568f304c982 plugin text syntaxes: remove side effect on init: Goffi <goffi@goffi.org> parents: 3479 diff changeset	179 self.params = """
f568f304c982 plugin text syntaxes: remove side effect on init: Goffi <goffi@goffi.org> parents: 3479 diff changeset	180 <params>
f568f304c982 plugin text syntaxes: remove side effect on init: Goffi <goffi@goffi.org> parents: 3479 diff changeset	181 <individual>
f568f304c982 plugin text syntaxes: remove side effect on init: Goffi <goffi@goffi.org> parents: 3479 diff changeset	182 <category name="%(category_name)s" label="%(category_label)s">
f568f304c982 plugin text syntaxes: remove side effect on init: Goffi <goffi@goffi.org> parents: 3479 diff changeset	183 <param name="%(name)s" label="%(label)s" type="list" security="0">
f568f304c982 plugin text syntaxes: remove side effect on init: Goffi <goffi@goffi.org> parents: 3479 diff changeset	184 %(options)s
f568f304c982 plugin text syntaxes: remove side effect on init: Goffi <goffi@goffi.org> parents: 3479 diff changeset	185 </param>
f568f304c982 plugin text syntaxes: remove side effect on init: Goffi <goffi@goffi.org> parents: 3479 diff changeset	186 </category>
f568f304c982 plugin text syntaxes: remove side effect on init: Goffi <goffi@goffi.org> parents: 3479 diff changeset	187 </individual>
f568f304c982 plugin text syntaxes: remove side effect on init: Goffi <goffi@goffi.org> parents: 3479 diff changeset	188 </params>
f568f304c982 plugin text syntaxes: remove side effect on init: Goffi <goffi@goffi.org> parents: 3479 diff changeset	189 """
f568f304c982 plugin text syntaxes: remove side effect on init: Goffi <goffi@goffi.org> parents: 3479 diff changeset	190
f568f304c982 plugin text syntaxes: remove side effect on init: Goffi <goffi@goffi.org> parents: 3479 diff changeset	191 self.params_data = {
f568f304c982 plugin text syntaxes: remove side effect on init: Goffi <goffi@goffi.org> parents: 3479 diff changeset	192 "category_name": CATEGORY,
f568f304c982 plugin text syntaxes: remove side effect on init: Goffi <goffi@goffi.org> parents: 3479 diff changeset	193 "category_label": _(CATEGORY),
f568f304c982 plugin text syntaxes: remove side effect on init: Goffi <goffi@goffi.org> parents: 3479 diff changeset	194 "name": NAME,
f568f304c982 plugin text syntaxes: remove side effect on init: Goffi <goffi@goffi.org> parents: 3479 diff changeset	195 "label": _(NAME),
f568f304c982 plugin text syntaxes: remove side effect on init: Goffi <goffi@goffi.org> parents: 3479 diff changeset	196 "syntaxes": self.syntaxes,
f568f304c982 plugin text syntaxes: remove side effect on init: Goffi <goffi@goffi.org> parents: 3479 diff changeset	197 }
f568f304c982 plugin text syntaxes: remove side effect on init: Goffi <goffi@goffi.org> parents: 3479 diff changeset	198
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	199 self.addSyntax(
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	200 self.SYNTAX_XHTML,
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	201 lambda xhtml: defer.succeed(xhtml),
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	202 lambda xhtml: defer.succeed(xhtml),
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	203 TextSyntaxes.OPT_NO_THREAD,
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	204 )
1826 d80ccf4bf201 plugin blog import dotclear: this plugin import Dotclear 2 backups Goffi <goffi@goffi.org> parents: 1811 diff changeset	205 # TODO: text => XHTML should add <a/> to url like in frontends
d80ccf4bf201 plugin blog import dotclear: this plugin import Dotclear 2 backups Goffi <goffi@goffi.org> parents: 1811 diff changeset	206 # it's probably best to move sat_frontends.tools.strings to sat.tools.common or similar
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	207 self.addSyntax(
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	208 self.SYNTAX_TEXT,
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	209 lambda text: escape(text),
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	210 lambda xhtml: self._removeMarkups(xhtml),
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	211 [TextSyntaxes.OPT_HIDDEN],
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	212 )
665 6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	213 try:
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	214 import markdown, html2text
2873 e1207b8ad97c plugin text syntaxes: disable raw HTML parsing in mardown by default Goffi <goffi@goffi.org> parents: 2869 diff changeset	215 from markdown.extensions import Extension
e1207b8ad97c plugin text syntaxes: disable raw HTML parsing in mardown by default Goffi <goffi@goffi.org> parents: 2869 diff changeset	216
e1207b8ad97c plugin text syntaxes: disable raw HTML parsing in mardown by default Goffi <goffi@goffi.org> parents: 2869 diff changeset	217 # XXX: we disable raw HTML parsing by default, to avoid parsing error
e1207b8ad97c plugin text syntaxes: disable raw HTML parsing in mardown by default Goffi <goffi@goffi.org> parents: 2869 diff changeset	218 # when the user is not aware of markdown and HTML
e1207b8ad97c plugin text syntaxes: disable raw HTML parsing in mardown by default Goffi <goffi@goffi.org> parents: 2869 diff changeset	219 class EscapeHTML(Extension):
e1207b8ad97c plugin text syntaxes: disable raw HTML parsing in mardown by default Goffi <goffi@goffi.org> parents: 2869 diff changeset	220 def extendMarkdown(self, md):
e1207b8ad97c plugin text syntaxes: disable raw HTML parsing in mardown by default Goffi <goffi@goffi.org> parents: 2869 diff changeset	221 md.preprocessors.deregister('html_block')
e1207b8ad97c plugin text syntaxes: disable raw HTML parsing in mardown by default Goffi <goffi@goffi.org> parents: 2869 diff changeset	222 md.inlinePatterns.deregister('html')
841 831f208b4ea3 plugin text_syntaxes: html2text was breaking the long URLs souliane <souliane@mailoo.org> parents: 836 diff changeset	223
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	224 def _html2text(html, baseurl=""):
841 831f208b4ea3 plugin text_syntaxes: html2text was breaking the long URLs souliane <souliane@mailoo.org> parents: 836 diff changeset	225 h = html2text.HTML2Text(baseurl=baseurl)
831f208b4ea3 plugin text_syntaxes: html2text was breaking the long URLs souliane <souliane@mailoo.org> parents: 836 diff changeset	226 h.body_width = 0 # do not truncate the lines, it breaks the long URLs
831f208b4ea3 plugin text_syntaxes: html2text was breaking the long URLs souliane <souliane@mailoo.org> parents: 836 diff changeset	227 return h.handle(html)
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	228
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	229 self.addSyntax(
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	230 self.SYNTAX_MARKDOWN,
2878 a02ad4bc0a6d plugin text syntaxes: activated useful markdown extensions: Goffi <goffi@goffi.org> parents: 2873 diff changeset	231 partial(markdown.markdown,
a02ad4bc0a6d plugin text syntaxes: activated useful markdown extensions: Goffi <goffi@goffi.org> parents: 2873 diff changeset	232 extensions=[
a02ad4bc0a6d plugin text syntaxes: activated useful markdown extensions: Goffi <goffi@goffi.org> parents: 2873 diff changeset	233 EscapeHTML(),
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2878 diff changeset	234 'nl2br',
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2878 diff changeset	235 'codehilite',
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2878 diff changeset	236 'fenced_code',
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2878 diff changeset	237 'sane_lists',
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2878 diff changeset	238 'tables',
2878 a02ad4bc0a6d plugin text syntaxes: activated useful markdown extensions: Goffi <goffi@goffi.org> parents: 2873 diff changeset	239 ],
a02ad4bc0a6d plugin text syntaxes: activated useful markdown extensions: Goffi <goffi@goffi.org> parents: 2873 diff changeset	240 extension_configs = {
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2878 diff changeset	241 "codehilite": {
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2878 diff changeset	242 "css_class": "highlight",
2878 a02ad4bc0a6d plugin text syntaxes: activated useful markdown extensions: Goffi <goffi@goffi.org> parents: 2873 diff changeset	243 }
a02ad4bc0a6d plugin text syntaxes: activated useful markdown extensions: Goffi <goffi@goffi.org> parents: 2873 diff changeset	244 }),
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	245 _html2text,
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	246 [TextSyntaxes.OPT_DEFAULT],
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	247 )
665 6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	248 except ImportError:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2878 diff changeset	249 log.warning("markdown or html2text not found, can't use Markdown syntax")
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	250 log.info(
3693 0bbdc50aa405 plugin text syntaxes: remove `id` attributes and whitelist allowed classes: Goffi <goffi@goffi.org> parents: 3479 diff changeset	251 "You can download/install them from https://pythonhosted.org/Markdown/ "
0bbdc50aa405 plugin text syntaxes: remove `id` attributes and whitelist allowed classes: Goffi <goffi@goffi.org> parents: 3479 diff changeset	252 "and https://github.com/Alir3z4/html2text/"
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	253 )
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	254 host.bridge.addMethod(
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	255 "syntaxConvert",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	256 ".plugin",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	257 in_sign="sssbs",
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	258 out_sign="s",
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2878 diff changeset	259 async_=True,
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	260 method=self.convert,
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	261 )
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	262 host.bridge.addMethod(
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	263 "syntaxGet", ".plugin", in_sign="s", out_sign="s", method=self.getSyntax
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	264 )
2782 b17e6fa1e607 core (XMLUI): new XHTMLBox widget: Goffi <goffi@goffi.org> parents: 2781 diff changeset	265 if xml_tools.cleanXHTML is None:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2878 diff changeset	266 log.debug("Installing cleaning method")
2782 b17e6fa1e607 core (XMLUI): new XHTMLBox widget: Goffi <goffi@goffi.org> parents: 2781 diff changeset	267 xml_tools.cleanXHTML = self.cleanXHTML
665 6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	268
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	269 def _updateParamOptions(self):
3620 f568f304c982 plugin text syntaxes: remove side effect on init: Goffi <goffi@goffi.org> parents: 3479 diff changeset	270 data_synt = self.syntaxes
2324 fe922e6fabd4 plugin text syntaxes: various improvments: Goffi <goffi@goffi.org> parents: 2145 diff changeset	271 default_synt = TextSyntaxes.default_syntax
665 6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	272 syntaxes = []
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	273
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2878 diff changeset	274 for syntax in list(data_synt.keys()):
665 6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	275 flags = data_synt[syntax]["flags"]
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	276 if TextSyntaxes.OPT_HIDDEN not in flags:
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	277 syntaxes.append(syntax)
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	278
1805 3c40fa0dcd7a pluging text syntaxes: various minor improvments: Goffi <goffi@goffi.org> parents: 1803 diff changeset	279 syntaxes.sort(key=lambda synt: synt.lower())
665 6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	280 options = []
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	281
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	282 for syntax in syntaxes:
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	283 selected = 'selected="true"' if syntax == default_synt else ""
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2878 diff changeset	284 options.append('<option value="%s" %s/>' % (syntax, selected))
665 6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	285
3620 f568f304c982 plugin text syntaxes: remove side effect on init: Goffi <goffi@goffi.org> parents: 3479 diff changeset	286 self.params_data["options"] = "\n".join(options)
f568f304c982 plugin text syntaxes: remove side effect on init: Goffi <goffi@goffi.org> parents: 3479 diff changeset	287 self.host.memory.updateParams(self.params % self.params_data)
665 6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	288
702 a25db3fe3959 plugin XEP-0071: rich messages management for sendMessage Goffi <goffi@goffi.org> parents: 699 diff changeset	289 def getCurrentSyntax(self, profile):
665 6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	290 """ Return the selected syntax for the given profile
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	291
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	292 @param profile: %(doc_profile)s
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	293 @return: profile selected syntax
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	294 """
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	295 return self.host.memory.getParamA(NAME, CATEGORY, profile_key=profile)
665 6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	296
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2878 diff changeset	297 def _logError(self, failure, action="converting syntax"):
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	298 log.error(
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2878 diff changeset	299 "Error while {action}: {failure}".format(action=action, failure=failure)
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	300 )
2106 5874da3811b7 plugin text syntaxes: log error on cleanXHTML failure Goffi <goffi@goffi.org> parents: 1934 diff changeset	301 return failure
5874da3811b7 plugin text syntaxes: log error on cleanXHTML failure Goffi <goffi@goffi.org> parents: 1934 diff changeset	302
3693 0bbdc50aa405 plugin text syntaxes: remove `id` attributes and whitelist allowed classes: Goffi <goffi@goffi.org> parents: 3479 diff changeset	303 def cleanStyle(self, styles_raw: str) -> str:
2781 816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	304 """"Clean unsafe CSS styles
816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	305
816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	306 Remove styles not in the whitelist, or where the value doesn't match the regex
3693 0bbdc50aa405 plugin text syntaxes: remove `id` attributes and whitelist allowed classes: Goffi <goffi@goffi.org> parents: 3479 diff changeset	307 @param styles_raw: CSS styles
0bbdc50aa405 plugin text syntaxes: remove `id` attributes and whitelist allowed classes: Goffi <goffi@goffi.org> parents: 3479 diff changeset	308 @return: cleaned styles
2781 816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	309 """
3693 0bbdc50aa405 plugin text syntaxes: remove `id` attributes and whitelist allowed classes: Goffi <goffi@goffi.org> parents: 3479 diff changeset	310 styles: List[str] = styles_raw.split(";")
2781 816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	311 cleaned_styles = []
816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	312 for style in styles:
816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	313 try:
816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	314 key, value = style.split(":")
816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	315 except ValueError:
816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	316 continue
816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	317 key = key.lower().strip()
816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	318 if key not in STYLES_WHITELIST:
816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	319 continue
816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	320 value = value.lower().strip()
816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	321 if not STYLES_ACCEPTED_VALUE.match(value):
816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	322 continue
816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	323 if value == "none":
816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	324 continue
816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	325 cleaned_styles.append((key, value))
816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	326 return "; ".join(
816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	327 ["%s: %s" % (key_, value_) for key_, value_ in cleaned_styles]
816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	328 )
816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	329
3693 0bbdc50aa405 plugin text syntaxes: remove `id` attributes and whitelist allowed classes: Goffi <goffi@goffi.org> parents: 3479 diff changeset	330 def cleanClasses(self, classes_raw: str) -> str:
0bbdc50aa405 plugin text syntaxes: remove `id` attributes and whitelist allowed classes: Goffi <goffi@goffi.org> parents: 3479 diff changeset	331 """Remove any non whitelisted class
0bbdc50aa405 plugin text syntaxes: remove `id` attributes and whitelist allowed classes: Goffi <goffi@goffi.org> parents: 3479 diff changeset	332
0bbdc50aa405 plugin text syntaxes: remove `id` attributes and whitelist allowed classes: Goffi <goffi@goffi.org> parents: 3479 diff changeset	333 @param classes_raw: classes set on an element
0bbdc50aa405 plugin text syntaxes: remove `id` attributes and whitelist allowed classes: Goffi <goffi@goffi.org> parents: 3479 diff changeset	334 @return: remaining classes (can be empty string)
0bbdc50aa405 plugin text syntaxes: remove `id` attributes and whitelist allowed classes: Goffi <goffi@goffi.org> parents: 3479 diff changeset	335 """
0bbdc50aa405 plugin text syntaxes: remove `id` attributes and whitelist allowed classes: Goffi <goffi@goffi.org> parents: 3479 diff changeset	336 return " ".join(SAFE_CLASSES.intersection(classes_raw.split()))
0bbdc50aa405 plugin text syntaxes: remove `id` attributes and whitelist allowed classes: Goffi <goffi@goffi.org> parents: 3479 diff changeset	337
1805 3c40fa0dcd7a pluging text syntaxes: various minor improvments: Goffi <goffi@goffi.org> parents: 1803 diff changeset	338 def cleanXHTML(self, xhtml):
2781 816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	339 """Clean XHTML text by removing potentially dangerous/malicious parts
816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	340
816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	341 @param xhtml(unicode, lxml.etree._Element): raw HTML/XHTML text to clean
816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	342 @return (unicode): cleaned XHTML
665 6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	343 """
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	344
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2878 diff changeset	345 if isinstance(xhtml, str):
2786 be8405795e09 plugin text syntaxes: handle empty content in cleanXHTML + don't use self-closing tags for non-void elements. Goffi <goffi@goffi.org> parents: 2782 diff changeset	346 try:
be8405795e09 plugin text syntaxes: handle empty content in cleanXHTML + don't use self-closing tags for non-void elements. Goffi <goffi@goffi.org> parents: 2782 diff changeset	347 xhtml_elt = html.fromstring(xhtml)
be8405795e09 plugin text syntaxes: handle empty content in cleanXHTML + don't use self-closing tags for non-void elements. Goffi <goffi@goffi.org> parents: 2782 diff changeset	348 except etree.ParserError as e:
be8405795e09 plugin text syntaxes: handle empty content in cleanXHTML + don't use self-closing tags for non-void elements. Goffi <goffi@goffi.org> parents: 2782 diff changeset	349 if not xhtml.strip():
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2878 diff changeset	350 return ""
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2878 diff changeset	351 log.error("Can't clean XHTML: {xhtml}".format(xhtml=xhtml))
2786 be8405795e09 plugin text syntaxes: handle empty content in cleanXHTML + don't use self-closing tags for non-void elements. Goffi <goffi@goffi.org> parents: 2782 diff changeset	352 raise e
2781 816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	353 elif isinstance(xhtml, html.HtmlElement):
816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	354 xhtml_elt = xhtml
816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	355 else:
816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	356 log.error("Only strings and HtmlElements can be cleaned")
816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	357 raise exceptions.DataError
816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	358 cleaner = clean.Cleaner(
816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	359 style=False, add_nofollow=False, safe_attrs=SAFE_ATTRS
816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	360 )
816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	361 xhtml_elt = cleaner.clean_html(xhtml_elt)
816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	362 for elt in xhtml_elt.xpath("//*[@style]"):
816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	363 elt.set("style", self.cleanStyle(elt.get("style")))
3693 0bbdc50aa405 plugin text syntaxes: remove `id` attributes and whitelist allowed classes: Goffi <goffi@goffi.org> parents: 3479 diff changeset	364 for elt in xhtml_elt.xpath("//*[@class]"):
0bbdc50aa405 plugin text syntaxes: remove `id` attributes and whitelist allowed classes: Goffi <goffi@goffi.org> parents: 3479 diff changeset	365 elt.set("class", self.cleanClasses(elt.get("class")))
2786 be8405795e09 plugin text syntaxes: handle empty content in cleanXHTML + don't use self-closing tags for non-void elements. Goffi <goffi@goffi.org> parents: 2782 diff changeset	366 # we remove self-closing elements for non-void elements
be8405795e09 plugin text syntaxes: handle empty content in cleanXHTML + don't use self-closing tags for non-void elements. Goffi <goffi@goffi.org> parents: 2782 diff changeset	367 for element in xhtml_elt.iter(tag=etree.Element):
be8405795e09 plugin text syntaxes: handle empty content in cleanXHTML + don't use self-closing tags for non-void elements. Goffi <goffi@goffi.org> parents: 2782 diff changeset	368 if not element.text:
be8405795e09 plugin text syntaxes: handle empty content in cleanXHTML + don't use self-closing tags for non-void elements. Goffi <goffi@goffi.org> parents: 2782 diff changeset	369 if element.tag in VOID_ELEMENTS:
be8405795e09 plugin text syntaxes: handle empty content in cleanXHTML + don't use self-closing tags for non-void elements. Goffi <goffi@goffi.org> parents: 2782 diff changeset	370 element.text = None
be8405795e09 plugin text syntaxes: handle empty content in cleanXHTML + don't use self-closing tags for non-void elements. Goffi <goffi@goffi.org> parents: 2782 diff changeset	371 else:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2878 diff changeset	372 element.text = ''
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2878 diff changeset	373 return html.tostring(xhtml_elt, encoding=str, method="xml")
665 6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	374
3040 fee60f17ebac jp: jp asyncio port: Goffi <goffi@goffi.org> parents: 3028 diff changeset	375 def convert(self, text, syntax_from, syntax_to=_SYNTAX_XHTML, safe=True,
fee60f17ebac jp: jp asyncio port: Goffi <goffi@goffi.org> parents: 3028 diff changeset	376 profile=None):
1803 14a97a5fe1c0 plugin text syntaxes: a non blocking syntax callback can now return a unicode directly instead of a Deferred Goffi <goffi@goffi.org> parents: 1766 diff changeset	377 """Convert a text between two syntaxes
14a97a5fe1c0 plugin text syntaxes: a non blocking syntax callback can now return a unicode directly instead of a Deferred Goffi <goffi@goffi.org> parents: 1766 diff changeset	378
665 6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	379 @param text: text to convert
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	380 @param syntax_from: source syntax (e.g. "markdown")
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	381 @param syntax_to: dest syntax (e.g.: "XHTML")
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	382 @param safe: clean resulting XHTML to avoid malicious code if True
2781 816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	383 @param profile: needed only when syntax_from or syntax_to is set to
816be0a23877 plugin text syntaxes: cleanStyle is an independent method, cleanXHTML is now blocking (no need to launch thread for that) Goffi <goffi@goffi.org> parents: 2780 diff changeset	384 _SYNTAX_CURRENT
1803 14a97a5fe1c0 plugin text syntaxes: a non blocking syntax callback can now return a unicode directly instead of a Deferred Goffi <goffi@goffi.org> parents: 1766 diff changeset	385 @return(unicode): converted text
14a97a5fe1c0 plugin text syntaxes: a non blocking syntax callback can now return a unicode directly instead of a Deferred Goffi <goffi@goffi.org> parents: 1766 diff changeset	386 """
1805 3c40fa0dcd7a pluging text syntaxes: various minor improvments: Goffi <goffi@goffi.org> parents: 1803 diff changeset	387 # FIXME: convert should be abled to handle domish.Element directly
3c40fa0dcd7a pluging text syntaxes: various minor improvments: Goffi <goffi@goffi.org> parents: 1803 diff changeset	388 # when dealing with XHTML
3c40fa0dcd7a pluging text syntaxes: various minor improvments: Goffi <goffi@goffi.org> parents: 1803 diff changeset	389 # TODO: a way for parser to return parsing errors/warnings
665 6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	390
744 312a2842b2b8 plugins text-syntaxes: added a default value to use the current user syntax in convert souliane <souliane@mailoo.org> parents: 705 diff changeset	391 if syntax_from == _SYNTAX_CURRENT:
312a2842b2b8 plugins text-syntaxes: added a default value to use the current user syntax in convert souliane <souliane@mailoo.org> parents: 705 diff changeset	392 syntax_from = self.getCurrentSyntax(profile)
2324 fe922e6fabd4 plugin text syntaxes: various improvments: Goffi <goffi@goffi.org> parents: 2145 diff changeset	393 else:
fe922e6fabd4 plugin text syntaxes: various improvments: Goffi <goffi@goffi.org> parents: 2145 diff changeset	394 syntax_from = syntax_from.lower().strip()
744 312a2842b2b8 plugins text-syntaxes: added a default value to use the current user syntax in convert souliane <souliane@mailoo.org> parents: 705 diff changeset	395 if syntax_to == _SYNTAX_CURRENT:
312a2842b2b8 plugins text-syntaxes: added a default value to use the current user syntax in convert souliane <souliane@mailoo.org> parents: 705 diff changeset	396 syntax_to = self.getCurrentSyntax(profile)
2324 fe922e6fabd4 plugin text syntaxes: various improvments: Goffi <goffi@goffi.org> parents: 2145 diff changeset	397 else:
fe922e6fabd4 plugin text syntaxes: various improvments: Goffi <goffi@goffi.org> parents: 2145 diff changeset	398 syntax_to = syntax_to.lower().strip()
3620 f568f304c982 plugin text syntaxes: remove side effect on init: Goffi <goffi@goffi.org> parents: 3479 diff changeset	399 syntaxes = self.syntaxes
665 6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	400 if syntax_from not in syntaxes:
2324 fe922e6fabd4 plugin text syntaxes: various improvments: Goffi <goffi@goffi.org> parents: 2145 diff changeset	401 raise exceptions.NotFound(syntax_from)
665 6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	402 if syntax_to not in syntaxes:
2324 fe922e6fabd4 plugin text syntaxes: various improvments: Goffi <goffi@goffi.org> parents: 2145 diff changeset	403 raise exceptions.NotFound(syntax_to)
665 6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	404 d = None
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	405
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	406 if TextSyntaxes.OPT_NO_THREAD in syntaxes[syntax_from]["flags"]:
1803 14a97a5fe1c0 plugin text syntaxes: a non blocking syntax callback can now return a unicode directly instead of a Deferred Goffi <goffi@goffi.org> parents: 1766 diff changeset	407 d = defer.maybeDeferred(syntaxes[syntax_from]["to"], text)
665 6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	408 else:
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	409 d = deferToThread(syntaxes[syntax_from]["to"], text)
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	410
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	411 # TODO: keep only body element and change it to a div here ?
665 6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	412
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	413 if safe:
1805 3c40fa0dcd7a pluging text syntaxes: various minor improvments: Goffi <goffi@goffi.org> parents: 1803 diff changeset	414 d.addCallback(self.cleanXHTML)
665 6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	415
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	416 if TextSyntaxes.OPT_NO_THREAD in syntaxes[syntax_to]["flags"]:
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	417 d.addCallback(syntaxes[syntax_to]["from"])
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	418 else:
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	419 d.addCallback(lambda xhtml: deferToThread(syntaxes[syntax_to]["from"], xhtml))
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	420
836 2cc0201b4613 plugin text_syntaxes: rstrip the conversion result to avoid new lines systematically added by converters (e.g. html2text do this) souliane <souliane@mailoo.org> parents: 832 diff changeset	421 # converters can add new lines that disturb the microblog change detection
2cc0201b4613 plugin text_syntaxes: rstrip the conversion result to avoid new lines systematically added by converters (e.g. html2text do this) souliane <souliane@mailoo.org> parents: 832 diff changeset	422 d.addCallback(lambda text: text.rstrip())
665 6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	423 return d
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	424
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	425 def addSyntax(self, name, to_xhtml_cb, from_xhtml_cb, flags=None):
1803 14a97a5fe1c0 plugin text syntaxes: a non blocking syntax callback can now return a unicode directly instead of a Deferred Goffi <goffi@goffi.org> parents: 1766 diff changeset	426 """Add a new syntax to the manager
14a97a5fe1c0 plugin text syntaxes: a non blocking syntax callback can now return a unicode directly instead of a Deferred Goffi <goffi@goffi.org> parents: 1766 diff changeset	427
665 6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	428 @param name: unique name of the syntax
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	429 @param to_xhtml_cb: callback to convert from syntax to XHTML
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	430 @param from_xhtml_cb: callback to convert from XHTML to syntax
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	431 @param flags: set of optional flags, can be:
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	432 TextSyntaxes.OPT_DEFAULT: use as the default syntax (replace former one)
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	433 TextSyntaxes.OPT_HIDDEN: do not show in parameters
1803 14a97a5fe1c0 plugin text syntaxes: a non blocking syntax callback can now return a unicode directly instead of a Deferred Goffi <goffi@goffi.org> parents: 1766 diff changeset	434 TextSyntaxes.OPT_NO_THREAD: do not defer to thread when converting (the callback may then return a deferred)
665 6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	435 """
1805 3c40fa0dcd7a pluging text syntaxes: various minor improvments: Goffi <goffi@goffi.org> parents: 1803 diff changeset	436 flags = flags if flags is not None else []
665 6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	437 if TextSyntaxes.OPT_HIDDEN in flags and TextSyntaxes.OPT_DEFAULT in flags:
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	438 raise ValueError(
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2878 diff changeset	439 "{} and {} are mutually exclusive".format(
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	440 TextSyntaxes.OPT_HIDDEN, TextSyntaxes.OPT_DEFAULT
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	441 )
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	442 )
665 6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	443
3620 f568f304c982 plugin text syntaxes: remove side effect on init: Goffi <goffi@goffi.org> parents: 3479 diff changeset	444 syntaxes = self.syntaxes
2324 fe922e6fabd4 plugin text syntaxes: various improvments: Goffi <goffi@goffi.org> parents: 2145 diff changeset	445 key = name.lower().strip()
fe922e6fabd4 plugin text syntaxes: various improvments: Goffi <goffi@goffi.org> parents: 2145 diff changeset	446 if key in syntaxes:
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	447 raise exceptions.ConflictError(
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2878 diff changeset	448 "This syntax key already exists: {}".format(key)
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	449 )
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	450 syntaxes[key] = {
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	451 "name": name,
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	452 "to": to_xhtml_cb,
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	453 "from": from_xhtml_cb,
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	454 "flags": flags,
56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	455 }
665 6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	456 if TextSyntaxes.OPT_DEFAULT in flags:
2869 148d30147890 plugin text syntaxes: fixed default syntax Goffi <goffi@goffi.org> parents: 2786 diff changeset	457 TextSyntaxes.default_syntax = key
665 6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	458
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	459 self._updateParamOptions()
6a64e0a759e6 plugin text syntaxes: this plugin manage rich text syntaxes conversions and cleaning. Goffi <goffi@goffi.org> parents: diff changeset	460
2324 fe922e6fabd4 plugin text syntaxes: various improvments: Goffi <goffi@goffi.org> parents: 2145 diff changeset	461 def getSyntax(self, name):
fe922e6fabd4 plugin text syntaxes: various improvments: Goffi <goffi@goffi.org> parents: 2145 diff changeset	462 """get syntax key corresponding to a name
fe922e6fabd4 plugin text syntaxes: various improvments: Goffi <goffi@goffi.org> parents: 2145 diff changeset	463
fe922e6fabd4 plugin text syntaxes: various improvments: Goffi <goffi@goffi.org> parents: 2145 diff changeset	464 @raise exceptions.NotFound: syntax doesn't exist
832 c4b22aedb7d7 plugin groupblog, XEP-0071, XEP-0277, text_syntaxes: manage raw/rich/xhtml data for content/title: souliane <souliane@mailoo.org> parents: 811 diff changeset	465 """
2324 fe922e6fabd4 plugin text syntaxes: various improvments: Goffi <goffi@goffi.org> parents: 2145 diff changeset	466 key = name.lower().strip()
fe922e6fabd4 plugin text syntaxes: various improvments: Goffi <goffi@goffi.org> parents: 2145 diff changeset	467 if key in self.syntaxes:
fe922e6fabd4 plugin text syntaxes: various improvments: Goffi <goffi@goffi.org> parents: 2145 diff changeset	468 return key
fe922e6fabd4 plugin text syntaxes: various improvments: Goffi <goffi@goffi.org> parents: 2145 diff changeset	469 raise exceptions.NotFound
fe922e6fabd4 plugin text syntaxes: various improvments: Goffi <goffi@goffi.org> parents: 2145 diff changeset	470
fe922e6fabd4 plugin text syntaxes: various improvments: Goffi <goffi@goffi.org> parents: 2145 diff changeset	471 def _removeMarkups(self, xhtml):
fe922e6fabd4 plugin text syntaxes: various improvments: Goffi <goffi@goffi.org> parents: 2145 diff changeset	472 """Remove XHTML markups from the given string.
fe922e6fabd4 plugin text syntaxes: various improvments: Goffi <goffi@goffi.org> parents: 2145 diff changeset	473
832 c4b22aedb7d7 plugin groupblog, XEP-0071, XEP-0277, text_syntaxes: manage raw/rich/xhtml data for content/title: souliane <souliane@mailoo.org> parents: 811 diff changeset	474 @param xhtml: the XHTML string to be cleaned
c4b22aedb7d7 plugin groupblog, XEP-0071, XEP-0277, text_syntaxes: manage raw/rich/xhtml data for content/title: souliane <souliane@mailoo.org> parents: 811 diff changeset	475 @return: the cleaned string
c4b22aedb7d7 plugin groupblog, XEP-0071, XEP-0277, text_syntaxes: manage raw/rich/xhtml data for content/title: souliane <souliane@mailoo.org> parents: 811 diff changeset	476 """
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	477 cleaner = clean.Cleaner(kill_tags=["style"])
832 c4b22aedb7d7 plugin groupblog, XEP-0071, XEP-0277, text_syntaxes: manage raw/rich/xhtml data for content/title: souliane <souliane@mailoo.org> parents: 811 diff changeset	478 cleaned = cleaner.clean_html(html.fromstring(xhtml))
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2878 diff changeset	479 return html.tostring(cleaned, encoding=str, method="text")

Mercurial > libervia-backend

annotate sat/plugins/plugin_misc_text_syntaxes.py @ 4002:5245b675f7ad