libervia-backend: sat/memory/sqlite.py annotate

annotate sat/memory/sqlite.py @ 3696:52ee22d78e18

plugin XEP-0384: fix encryption for user own devices: our own jid is added automatically by OMEMO library, but during the generation of the stanza, our list of jids was used to get the keys (and thus our own jid and devices keys was missing). fix 402

author	Goffi <goffi@goffi.org>
date	Fri, 22 Oct 2021 18:46:51 +0200
parents	0ce37cf3fbf4
children

rev	line source
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1 #!/usr/bin/env python3
3137 559a625a236b fixed shebangs Goffi <goffi@goffi.org> parents: 3136 diff changeset	2
412 62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	3
609 84a6e83157c2 fixed licences in docstrings (they are now in comments) Goffi <goffi@goffi.org> parents: 593 diff changeset	4 # SAT: a jabber client
3479 be6d91572633 date update Goffi <goffi@goffi.org> parents: 3317 diff changeset	5 # Copyright (C) 2009-2021 Jérôme Poisson (goffi@goffi.org)
412 62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	6
609 84a6e83157c2 fixed licences in docstrings (they are now in comments) Goffi <goffi@goffi.org> parents: 593 diff changeset	7 # This program is free software: you can redistribute it and/or modify
84a6e83157c2 fixed licences in docstrings (they are now in comments) Goffi <goffi@goffi.org> parents: 593 diff changeset	8 # it under the terms of the GNU Affero General Public License as published by
84a6e83157c2 fixed licences in docstrings (they are now in comments) Goffi <goffi@goffi.org> parents: 593 diff changeset	9 # the Free Software Foundation, either version 3 of the License, or
84a6e83157c2 fixed licences in docstrings (they are now in comments) Goffi <goffi@goffi.org> parents: 593 diff changeset	10 # (at your option) any later version.
412 62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	11
609 84a6e83157c2 fixed licences in docstrings (they are now in comments) Goffi <goffi@goffi.org> parents: 593 diff changeset	12 # This program is distributed in the hope that it will be useful,
84a6e83157c2 fixed licences in docstrings (they are now in comments) Goffi <goffi@goffi.org> parents: 593 diff changeset	13 # but WITHOUT ANY WARRANTY; without even the implied warranty of
84a6e83157c2 fixed licences in docstrings (they are now in comments) Goffi <goffi@goffi.org> parents: 593 diff changeset	14 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
84a6e83157c2 fixed licences in docstrings (they are now in comments) Goffi <goffi@goffi.org> parents: 593 diff changeset	15 # GNU Affero General Public License for more details.
412 62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	16
609 84a6e83157c2 fixed licences in docstrings (they are now in comments) Goffi <goffi@goffi.org> parents: 593 diff changeset	17 # You should have received a copy of the GNU Affero General Public License
84a6e83157c2 fixed licences in docstrings (they are now in comments) Goffi <goffi@goffi.org> parents: 593 diff changeset	18 # along with this program. If not, see <http://www.gnu.org/licenses/>.
412 62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	19
771 bfabeedbf32e core: i18n refactoring: Goffi <goffi@goffi.org> parents: 681 diff changeset	20 from sat.core.i18n import _
1030 15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	21 from sat.core.constants import Const as C
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	22 from sat.core import exceptions
993 301b342c697a core: use of the new core.log module: Goffi <goffi@goffi.org> parents: 936 diff changeset	23 from sat.core.log import getLogger
1030 15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	24 from sat.memory.crypto import BlockCipher, PasswordHasher
1045 65fffdcb97f1 memory: auto-update configuration file with libervia's passphrase when migrating the database souliane <souliane@mailoo.org> parents: 1030 diff changeset	25 from sat.tools.config import fixConfigOption
412 62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	26 from twisted.enterprise import adbapi
62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	27 from twisted.internet import defer
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	28 from twisted.words.protocols.jabber import jid
1961 c73e08094a95 memory (sqlite): better handling of IntegrityError Goffi <goffi@goffi.org> parents: 1955 diff changeset	29 from twisted.python import failure
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	30 from collections import OrderedDict
2964 17c61d09a85b memory (sqlite): exit with an error if current db version is higher than version expected in current code Goffi <goffi@goffi.org> parents: 2928 diff changeset	31 import sys
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	32 import re
412 62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	33 import os.path
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	34 import pickle as pickle
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	35 import hashlib
1961 c73e08094a95 memory (sqlite): better handling of IntegrityError Goffi <goffi@goffi.org> parents: 1955 diff changeset	36 import sqlite3
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	37 import json
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	38
2964 17c61d09a85b memory (sqlite): exit with an error if current db version is higher than version expected in current code Goffi <goffi@goffi.org> parents: 2928 diff changeset	39 log = getLogger(__name__)
17c61d09a85b memory (sqlite): exit with an error if current db version is higher than version expected in current code Goffi <goffi@goffi.org> parents: 2928 diff changeset	40
3288 780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	41 CURRENT_DB_VERSION = 9
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	42
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	43 # XXX: DATABASE schemas are used in the following way:
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	44 # - 'current' key is for the actual database schema, for a new base
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	45 # - x(int) is for update needed between x-1 and x. All number are needed between y and z to do an update
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	46 # e.g.: if CURRENT_DB_VERSION is 6, 'current' is the actuel DB, and to update from version 3, numbers 4, 5 and 6 are needed
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	47 # a 'current' data dict can contains the keys:
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	48 # - 'CREATE': it contains an Ordered dict with table to create as keys, and a len 2 tuple as value, where value[0] are the columns definitions and value[1] are the table constraints
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	49 # - 'INSERT': it contains an Ordered dict with table where values have to be inserted, and many tuples containing values to insert in the order of the rows (#TODO: manage named columns)
2787 298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	50 # - 'INDEX':
1030 15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	51 # an update data dict (the ones with a number) can contains the keys 'create', 'delete', 'cols create', 'cols delete', 'cols modify', 'insert' or 'specific'. See Updater.generateUpdateData for more infos. This method can be used to autogenerate update_data, to ease the work of the developers.
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	52 # TODO: indexes need to be improved
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	53
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	54 DATABASE_SCHEMAS = {
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	55 "current": {'CREATE': OrderedDict((
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	56 ('profiles', (("id INTEGER PRIMARY KEY ASC", "name TEXT"),
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	57 ("UNIQUE (name)",))),
2143 c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	58 ('components', (("profile_id INTEGER PRIMARY KEY", "entry_point TEXT NOT NULL"),
c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	59 ("FOREIGN KEY(profile_id) REFERENCES profiles(id) ON DELETE CASCADE",))),
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	60 ('message_types', (("type TEXT PRIMARY KEY",),
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	61 ())),
2699 310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	62 ('history', (("uid TEXT PRIMARY KEY", "stanza_id TEXT", "update_uid TEXT", "profile_id INTEGER", "source TEXT", "dest TEXT", "source_res TEXT", "dest_res TEXT",
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	63 "timestamp DATETIME NOT NULL", "received_timestamp DATETIME", # XXX: timestamp is the time when the message was emitted. If received time stamp is not NULL, the message was delayed and timestamp is the declared value (and received_timestamp the time of reception)
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	64 "type TEXT", "extra BLOB"),
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	65 ("FOREIGN KEY(profile_id) REFERENCES profiles(id) ON DELETE CASCADE", "FOREIGN KEY(type) REFERENCES message_types(type)",
2716 06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	66 "UNIQUE (profile_id, stanza_id, source, dest)" # avoid storing 2 times the same message
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	67 ))),
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	68 ('message', (("id INTEGER PRIMARY KEY ASC", "history_uid INTEGER", "message TEXT", "language TEXT"),
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	69 ("FOREIGN KEY(history_uid) REFERENCES history(uid) ON DELETE CASCADE",))),
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	70 ('subject', (("id INTEGER PRIMARY KEY ASC", "history_uid INTEGER", "subject TEXT", "language TEXT"),
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	71 ("FOREIGN KEY(history_uid) REFERENCES history(uid) ON DELETE CASCADE",))),
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	72 ('thread', (("id INTEGER PRIMARY KEY ASC", "history_uid INTEGER", "thread_id TEXT", "parent_id TEXT"),("FOREIGN KEY(history_uid) REFERENCES history(uid) ON DELETE CASCADE",))),
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	73 ('param_gen', (("category TEXT", "name TEXT", "value TEXT"),
2787 298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	74 ("PRIMARY KEY (category, name)",))),
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	75 ('param_ind', (("category TEXT", "name TEXT", "profile_id INTEGER", "value TEXT"),
2787 298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	76 ("PRIMARY KEY (profile_id, category, name)", "FOREIGN KEY(profile_id) REFERENCES profiles(id) ON DELETE CASCADE"))),
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	77 ('private_gen', (("namespace TEXT", "key TEXT", "value TEXT"),
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	78 ("PRIMARY KEY (namespace, key)",))),
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	79 ('private_ind', (("namespace TEXT", "key TEXT", "profile_id INTEGER", "value TEXT"),
2787 298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	80 ("PRIMARY KEY (profile_id, namespace, key)", "FOREIGN KEY(profile_id) REFERENCES profiles(id) ON DELETE CASCADE"))),
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	81 ('private_gen_bin', (("namespace TEXT", "key TEXT", "value BLOB"),
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	82 ("PRIMARY KEY (namespace, key)",))),
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	83 ('private_ind_bin', (("namespace TEXT", "key TEXT", "profile_id INTEGER", "value BLOB"),
2787 298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	84 ("PRIMARY KEY (profile_id, namespace, key)", "FOREIGN KEY(profile_id) REFERENCES profiles(id) ON DELETE CASCADE"))),
3288 780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	85 ('files', (("id TEXT NOT NULL", "public_id TEXT", "version TEXT NOT NULL",
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	86 "parent TEXT NOT NULL",
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	87 "type TEXT CHECK(type in ('{file}', '{directory}')) NOT NULL DEFAULT '{file}'".format(
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	88 file=C.FILE_TYPE_FILE, directory=C.FILE_TYPE_DIRECTORY),
2507 4c45df43ea44 core (memory/sqlite): renamed column for files from hash to file_hash: Goffi <goffi@goffi.org> parents: 2500 diff changeset	89 "file_hash TEXT", "hash_algo TEXT", "name TEXT NOT NULL", "size INTEGER",
3288 780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	90 "namespace TEXT", "media_type TEXT", "media_subtype TEXT",
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	91 "created DATETIME NOT NULL", "modified DATETIME",
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	92 "owner TEXT", "access TEXT", "extra TEXT", "profile_id INTEGER"),
3288 780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	93 ("PRIMARY KEY (id, version)", "FOREIGN KEY(profile_id) REFERENCES profiles(id) ON DELETE CASCADE",
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	94 "UNIQUE (public_id)"))),
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	95 )),
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	96 'INSERT': OrderedDict((
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	97 ('message_types', (("'chat'",),
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	98 ("'error'",),
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	99 ("'groupchat'",),
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	100 ("'headline'",),
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	101 ("'normal'",),
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	102 ("'info'",) # info is not standard, but used to keep track of info like join/leave in a MUC
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	103 )),
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	104 )),
2787 298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	105 'INDEX': (('history', (('profile_id', 'timestamp'),
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	106 ('profile_id', 'received_timestamp'))),
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	107 ('message', ('history_uid',)),
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	108 ('subject', ('history_uid',)),
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	109 ('thread', ('history_uid',)),
3288 780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	110 ('files', (('profile_id', 'owner', 'media_type', 'media_subtype'),
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	111 ('profile_id', 'owner', 'parent'))),
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	112 )
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	113 },
3288 780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	114 9: {'specific': 'update_v9'
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	115 },
2787 298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	116 8: {'specific': 'update_v8'
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	117 },
2716 06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	118 7: {'specific': 'update_v7'
06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	119 },
2699 310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	120 6: {'cols create': {'history': ('stanza_id TEXT',)},
310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	121 },
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	122 5: {'create': {'files': (("id TEXT NOT NULL", "version TEXT NOT NULL", "parent TEXT NOT NULL",
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	123 "type TEXT CHECK(type in ('{file}', '{directory}')) NOT NULL DEFAULT '{file}'".format(
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	124 file=C.FILE_TYPE_FILE, directory=C.FILE_TYPE_DIRECTORY),
2507 4c45df43ea44 core (memory/sqlite): renamed column for files from hash to file_hash: Goffi <goffi@goffi.org> parents: 2500 diff changeset	125 "file_hash TEXT", "hash_algo TEXT", "name TEXT NOT NULL", "size INTEGER",
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	126 "namespace TEXT", "mime_type TEXT",
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	127 "created DATETIME NOT NULL", "modified DATETIME",
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	128 "owner TEXT", "access TEXT", "extra TEXT", "profile_id INTEGER"),
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	129 ("PRIMARY KEY (id, version)", "FOREIGN KEY(profile_id) REFERENCES profiles(id) ON DELETE CASCADE"))},
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	130 },
2143 c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	131 4: {'create': {'components': (('profile_id INTEGER PRIMARY KEY', 'entry_point TEXT NOT NULL'), ('FOREIGN KEY(profile_id) REFERENCES profiles(id) ON DELETE CASCADE',))}
c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	132 },
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	133 3: {'specific': 'update_v3'
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	134 },
1030 15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	135 2: {'specific': 'update2raw_v2'
15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	136 },
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	137 1: {'cols create': {'history': ('extra BLOB',)},
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	138 },
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	139 }
412 62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	140
2699 310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	141 NOT_IN_EXTRA = ('stanza_id', 'received_timestamp', 'update_uid') # keys which are in message data extra but not stored in sqlite's extra field
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	142 # this is specific to this sqlite storage and for now only used for received_timestamp
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	143 # because this value is stored in a separate field
592 e5a875a3311b Fix pep8 support in src/memory. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 588 diff changeset	144
1961 c73e08094a95 memory (sqlite): better handling of IntegrityError Goffi <goffi@goffi.org> parents: 1955 diff changeset	145
c73e08094a95 memory (sqlite): better handling of IntegrityError Goffi <goffi@goffi.org> parents: 1955 diff changeset	146 class ConnectionPool(adbapi.ConnectionPool):
c73e08094a95 memory (sqlite): better handling of IntegrityError Goffi <goffi@goffi.org> parents: 1955 diff changeset	147 def _runQuery(self, trans, args, *kw):
2357 fa43e285df1d core (memory/sqlite): better stability: Goffi <goffi@goffi.org> parents: 2209 diff changeset	148 retry = kw.pop('query_retry', 6)
1961 c73e08094a95 memory (sqlite): better handling of IntegrityError Goffi <goffi@goffi.org> parents: 1955 diff changeset	149 try:
c73e08094a95 memory (sqlite): better handling of IntegrityError Goffi <goffi@goffi.org> parents: 1955 diff changeset	150 trans.execute(args, *kw)
c73e08094a95 memory (sqlite): better handling of IntegrityError Goffi <goffi@goffi.org> parents: 1955 diff changeset	151 except sqlite3.IntegrityError as e:
2747 6487b8855c9a memory (sqlite): minor comment move Goffi <goffi@goffi.org> parents: 2746 diff changeset	152 # Workaround to avoid IntegrityError causing (i)pdb to be
6487b8855c9a memory (sqlite): minor comment move Goffi <goffi@goffi.org> parents: 2746 diff changeset	153 # launched in debug mode
1961 c73e08094a95 memory (sqlite): better handling of IntegrityError Goffi <goffi@goffi.org> parents: 1955 diff changeset	154 raise failure.Failure(e)
2357 fa43e285df1d core (memory/sqlite): better stability: Goffi <goffi@goffi.org> parents: 2209 diff changeset	155 except Exception as e:
fa43e285df1d core (memory/sqlite): better stability: Goffi <goffi@goffi.org> parents: 2209 diff changeset	156 # FIXME: in case of error, we retry a couple of times
fa43e285df1d core (memory/sqlite): better stability: Goffi <goffi@goffi.org> parents: 2209 diff changeset	157 # this is a workaround, we need to move to better
fa43e285df1d core (memory/sqlite): better stability: Goffi <goffi@goffi.org> parents: 2209 diff changeset	158 # Sqlite integration, probably with high level library
fa43e285df1d core (memory/sqlite): better stability: Goffi <goffi@goffi.org> parents: 2209 diff changeset	159 retry -= 1
fa43e285df1d core (memory/sqlite): better stability: Goffi <goffi@goffi.org> parents: 2209 diff changeset	160 if retry == 0:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	161 log.error(_('too many db tries, we abandon! Error message: {msg}\n'
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	162 'query was {query}'
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	163 .format(msg=e, query=' '.join([str(a) for a in args]))))
2357 fa43e285df1d core (memory/sqlite): better stability: Goffi <goffi@goffi.org> parents: 2209 diff changeset	164 raise e
2746 1e2f0856c845 memory (sqlite): retry _runInteraction in the same way as for _runQuery, this improve reliability Goffi <goffi@goffi.org> parents: 2735 diff changeset	165 log.warning(
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	166 _('exception while running query, retrying ({try_}): {msg}').format(
2357 fa43e285df1d core (memory/sqlite): better stability: Goffi <goffi@goffi.org> parents: 2209 diff changeset	167 try_ = 6 - retry,
fa43e285df1d core (memory/sqlite): better stability: Goffi <goffi@goffi.org> parents: 2209 diff changeset	168 msg = e))
fa43e285df1d core (memory/sqlite): better stability: Goffi <goffi@goffi.org> parents: 2209 diff changeset	169 kw['query_retry'] = retry
fa43e285df1d core (memory/sqlite): better stability: Goffi <goffi@goffi.org> parents: 2209 diff changeset	170 return self._runQuery(trans, args, *kw)
1961 c73e08094a95 memory (sqlite): better handling of IntegrityError Goffi <goffi@goffi.org> parents: 1955 diff changeset	171 return trans.fetchall()
c73e08094a95 memory (sqlite): better handling of IntegrityError Goffi <goffi@goffi.org> parents: 1955 diff changeset	172
2746 1e2f0856c845 memory (sqlite): retry _runInteraction in the same way as for _runQuery, this improve reliability Goffi <goffi@goffi.org> parents: 2735 diff changeset	173 def _runInteraction(self, interaction, args, *kw):
1e2f0856c845 memory (sqlite): retry _runInteraction in the same way as for _runQuery, this improve reliability Goffi <goffi@goffi.org> parents: 2735 diff changeset	174 # sometimes interaction may fail while committing in _runInteraction
1e2f0856c845 memory (sqlite): retry _runInteraction in the same way as for _runQuery, this improve reliability Goffi <goffi@goffi.org> parents: 2735 diff changeset	175 # and it may be due to a db lock. So we work around it in a similar way
1e2f0856c845 memory (sqlite): retry _runInteraction in the same way as for _runQuery, this improve reliability Goffi <goffi@goffi.org> parents: 2735 diff changeset	176 # as for _runQuery but with only 3 tries
1e2f0856c845 memory (sqlite): retry _runInteraction in the same way as for _runQuery, this improve reliability Goffi <goffi@goffi.org> parents: 2735 diff changeset	177 retry = kw.pop('interaction_retry', 4)
1e2f0856c845 memory (sqlite): retry _runInteraction in the same way as for _runQuery, this improve reliability Goffi <goffi@goffi.org> parents: 2735 diff changeset	178 try:
1e2f0856c845 memory (sqlite): retry _runInteraction in the same way as for _runQuery, this improve reliability Goffi <goffi@goffi.org> parents: 2735 diff changeset	179 return adbapi.ConnectionPool._runInteraction(self, interaction, args, *kw)
1e2f0856c845 memory (sqlite): retry _runInteraction in the same way as for _runQuery, this improve reliability Goffi <goffi@goffi.org> parents: 2735 diff changeset	180 except Exception as e:
1e2f0856c845 memory (sqlite): retry _runInteraction in the same way as for _runQuery, this improve reliability Goffi <goffi@goffi.org> parents: 2735 diff changeset	181 retry -= 1
1e2f0856c845 memory (sqlite): retry _runInteraction in the same way as for _runQuery, this improve reliability Goffi <goffi@goffi.org> parents: 2735 diff changeset	182 if retry == 0:
1e2f0856c845 memory (sqlite): retry _runInteraction in the same way as for _runQuery, this improve reliability Goffi <goffi@goffi.org> parents: 2735 diff changeset	183 log.error(
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	184 _('too many interaction tries, we abandon! Error message: {msg}\n'
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	185 'interaction method was: {interaction}\n'
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	186 'interaction arguments were: {args}'
2919 e4715a609d75 memory (sqlite): minor log improvment Goffi <goffi@goffi.org> parents: 2918 diff changeset	187 .format(msg=e, interaction=interaction,
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	188 args=', '.join([str(a) for a in args]))))
2746 1e2f0856c845 memory (sqlite): retry _runInteraction in the same way as for _runQuery, this improve reliability Goffi <goffi@goffi.org> parents: 2735 diff changeset	189 raise e
1e2f0856c845 memory (sqlite): retry _runInteraction in the same way as for _runQuery, this improve reliability Goffi <goffi@goffi.org> parents: 2735 diff changeset	190 log.warning(
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	191 _('exception while running interaction, retrying ({try_}): {msg}')
2746 1e2f0856c845 memory (sqlite): retry _runInteraction in the same way as for _runQuery, this improve reliability Goffi <goffi@goffi.org> parents: 2735 diff changeset	192 .format(try_ = 4 - retry, msg = e))
1e2f0856c845 memory (sqlite): retry _runInteraction in the same way as for _runQuery, this improve reliability Goffi <goffi@goffi.org> parents: 2735 diff changeset	193 kw['interaction_retry'] = retry
1e2f0856c845 memory (sqlite): retry _runInteraction in the same way as for _runQuery, this improve reliability Goffi <goffi@goffi.org> parents: 2735 diff changeset	194 return self._runInteraction(interaction, args, *kw)
1e2f0856c845 memory (sqlite): retry _runInteraction in the same way as for _runQuery, this improve reliability Goffi <goffi@goffi.org> parents: 2735 diff changeset	195
1961 c73e08094a95 memory (sqlite): better handling of IntegrityError Goffi <goffi@goffi.org> parents: 1955 diff changeset	196
588 beaf6bec2fcd Remove every old-style class. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 587 diff changeset	197 class SqliteStorage(object):
412 62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	198 """This class manage storage with Sqlite database"""
62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	199
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	200 def __init__(self, db_filename, sat_version):
412 62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	201 """Connect to the given database
2143 c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	202
c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	203 @param db_filename: full path to the Sqlite database
c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	204 """
3120 0c29155ac68b core: backend autoconnection: Goffi <goffi@goffi.org> parents: 3079 diff changeset	205 # triggered when memory is fully initialised and ready
0c29155ac68b core: backend autoconnection: Goffi <goffi@goffi.org> parents: 3079 diff changeset	206 self.initialized = defer.Deferred()
0c29155ac68b core: backend autoconnection: Goffi <goffi@goffi.org> parents: 3079 diff changeset	207 # we keep cache for the profiles (key: profile name, value: profile id)
0c29155ac68b core: backend autoconnection: Goffi <goffi@goffi.org> parents: 3079 diff changeset	208 self.profiles = {}
412 62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	209
993 301b342c697a core: use of the new core.log module: Goffi <goffi@goffi.org> parents: 936 diff changeset	210 log.info(_("Connecting database"))
592 e5a875a3311b Fix pep8 support in src/memory. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 588 diff changeset	211 new_base = not os.path.exists(db_filename) # do we have to create the database ?
936 6404df5305e3 memory: be sure that local_dir exists before creating a new database souliane <souliane@mailoo.org> parents: 934 diff changeset	212 if new_base: # the dir may not exist if it's not the XDG recommended one
6404df5305e3 memory: be sure that local_dir exists before creating a new database souliane <souliane@mailoo.org> parents: 934 diff changeset	213 dir_ = os.path.dirname(db_filename)
6404df5305e3 memory: be sure that local_dir exists before creating a new database souliane <souliane@mailoo.org> parents: 934 diff changeset	214 if not os.path.exists(dir_):
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	215 os.makedirs(dir_, 0o700)
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	216
2357 fa43e285df1d core (memory/sqlite): better stability: Goffi <goffi@goffi.org> parents: 2209 diff changeset	217 def foreignKeysOn(sqlite):
fa43e285df1d core (memory/sqlite): better stability: Goffi <goffi@goffi.org> parents: 2209 diff changeset	218 sqlite.execute('PRAGMA foreign_keys = ON')
fa43e285df1d core (memory/sqlite): better stability: Goffi <goffi@goffi.org> parents: 2209 diff changeset	219
fa43e285df1d core (memory/sqlite): better stability: Goffi <goffi@goffi.org> parents: 2209 diff changeset	220 self.dbpool = ConnectionPool("sqlite3", db_filename, cp_openfun=foreignKeysOn, check_same_thread=False, timeout=15)
624 70988f08d0ad core: fixed bad database creation on first run in sqlite storage Goffi <goffi@goffi.org> parents: 609 diff changeset	221
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	222 def getNewBaseSql():
993 301b342c697a core: use of the new core.log module: Goffi <goffi@goffi.org> parents: 936 diff changeset	223 log.info(_("The database is new, creating the tables"))
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	224 database_creation = ["PRAGMA user_version=%d" % CURRENT_DB_VERSION]
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	225 database_creation.extend(Updater.createData2Raw(DATABASE_SCHEMAS['current']['CREATE']))
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	226 database_creation.extend(Updater.insertData2Raw(DATABASE_SCHEMAS['current']['INSERT']))
2787 298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	227 database_creation.extend(Updater.indexData2Raw(DATABASE_SCHEMAS['current']['INDEX']))
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	228 return database_creation
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	229
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	230 def getUpdateSql():
2716 06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	231 updater = Updater(self, sat_version)
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	232 return updater.checkUpdates()
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	233
2357 fa43e285df1d core (memory/sqlite): better stability: Goffi <goffi@goffi.org> parents: 2209 diff changeset	234 # init_defer is the initialisation deferred, initialisation is ok when all its callbacks have been done
fa43e285df1d core (memory/sqlite): better stability: Goffi <goffi@goffi.org> parents: 2209 diff changeset	235
fa43e285df1d core (memory/sqlite): better stability: Goffi <goffi@goffi.org> parents: 2209 diff changeset	236 init_defer = defer.succeed(None)
fa43e285df1d core (memory/sqlite): better stability: Goffi <goffi@goffi.org> parents: 2209 diff changeset	237
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	238 init_defer.addCallback(lambda ignore: getNewBaseSql() if new_base else getUpdateSql())
2716 06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	239 init_defer.addCallback(self.commitStatements)
412 62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	240
62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	241 def fillProfileCache(ignore):
3120 0c29155ac68b core: backend autoconnection: Goffi <goffi@goffi.org> parents: 3079 diff changeset	242 return self.dbpool.runQuery("SELECT profile_id, entry_point FROM components").addCallback(self._cacheComponentsAndProfiles)
587 952322b1d490 Remove trailing whitespaces. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 572 diff changeset	243
624 70988f08d0ad core: fixed bad database creation on first run in sqlite storage Goffi <goffi@goffi.org> parents: 609 diff changeset	244 init_defer.addCallback(fillProfileCache)
3120 0c29155ac68b core: backend autoconnection: Goffi <goffi@goffi.org> parents: 3079 diff changeset	245 init_defer.chainDeferred(self.initialized)
412 62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	246
2716 06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	247 def commitStatements(self, statements):
06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	248
06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	249 if statements is None:
06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	250 return defer.succeed(None)
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	251 log.debug("\n===== COMMITTING STATEMENTS =====\n%s\n============\n\n" % '\n'.join(statements))
2716 06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	252 d = self.dbpool.runInteraction(self._updateDb, tuple(statements))
06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	253 return d
06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	254
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	255 def _updateDb(self, interaction, statements):
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	256 for statement in statements:
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	257 interaction.execute(statement)
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	258
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	259 ## Profiles
2143 c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	260
c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	261 def _cacheComponentsAndProfiles(self, components_result):
c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	262 """Get components results and send requests profiles
c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	263
c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	264 they will be both put in cache in _profilesCache
c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	265 """
c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	266 return self.dbpool.runQuery("SELECT name,id FROM profiles").addCallback(
c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	267 self._cacheComponentsAndProfiles2, components_result)
c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	268
c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	269 def _cacheComponentsAndProfiles2(self, profiles_result, components):
412 62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	270 """Fill the profiles cache
2143 c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	271
c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	272 @param profiles_result: result of the sql profiles query
c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	273 """
c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	274 self.components = dict(components)
412 62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	275 for profile in profiles_result:
592 e5a875a3311b Fix pep8 support in src/memory. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 588 diff changeset	276 name, id_ = profile
e5a875a3311b Fix pep8 support in src/memory. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 588 diff changeset	277 self.profiles[name] = id_
587 952322b1d490 Remove trailing whitespaces. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 572 diff changeset	278
412 62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	279 def getProfilesList(self):
62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	280 """"Return list of all registered profiles"""
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	281 return list(self.profiles.keys())
412 62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	282
62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	283 def hasProfile(self, profile_name):
62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	284 """return True if profile_name exists
2143 c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	285
c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	286 @param profile_name: name of the profile to check
c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	287 """
592 e5a875a3311b Fix pep8 support in src/memory. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 588 diff changeset	288 return profile_name in self.profiles
587 952322b1d490 Remove trailing whitespaces. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 572 diff changeset	289
2143 c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	290 def profileIsComponent(self, profile_name):
c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	291 try:
c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	292 return self.profiles[profile_name] in self.components
c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	293 except KeyError:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	294 raise exceptions.NotFound("the requested profile doesn't exists")
2143 c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	295
2144 1d3f73e065e1 core, jp: component handling + client handling refactoring: Goffi <goffi@goffi.org> parents: 2143 diff changeset	296 def getEntryPoint(self, profile_name):
1d3f73e065e1 core, jp: component handling + client handling refactoring: Goffi <goffi@goffi.org> parents: 2143 diff changeset	297 try:
1d3f73e065e1 core, jp: component handling + client handling refactoring: Goffi <goffi@goffi.org> parents: 2143 diff changeset	298 return self.components[self.profiles[profile_name]]
1d3f73e065e1 core, jp: component handling + client handling refactoring: Goffi <goffi@goffi.org> parents: 2143 diff changeset	299 except KeyError:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	300 raise exceptions.NotFound("the requested profile doesn't exists or is not a component")
2144 1d3f73e065e1 core, jp: component handling + client handling refactoring: Goffi <goffi@goffi.org> parents: 2143 diff changeset	301
1d3f73e065e1 core, jp: component handling + client handling refactoring: Goffi <goffi@goffi.org> parents: 2143 diff changeset	302 def createProfile(self, name, component=None):
412 62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	303 """Create a new profile
2143 c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	304
2144 1d3f73e065e1 core, jp: component handling + client handling refactoring: Goffi <goffi@goffi.org> parents: 2143 diff changeset	305 @param name(unicode): name of the profile
1d3f73e065e1 core, jp: component handling + client handling refactoring: Goffi <goffi@goffi.org> parents: 2143 diff changeset	306 @param component(None, unicode): if not None, must point to a component entry point
2143 c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	307 @return: deferred triggered once profile is actually created
c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	308 """
592 e5a875a3311b Fix pep8 support in src/memory. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 588 diff changeset	309
412 62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	310 def getProfileId(ignore):
592 e5a875a3311b Fix pep8 support in src/memory. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 588 diff changeset	311 return self.dbpool.runQuery("SELECT (id) FROM profiles WHERE name = ?", (name, ))
587 952322b1d490 Remove trailing whitespaces. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 572 diff changeset	312
2144 1d3f73e065e1 core, jp: component handling + client handling refactoring: Goffi <goffi@goffi.org> parents: 2143 diff changeset	313 def setComponent(profile_id):
1d3f73e065e1 core, jp: component handling + client handling refactoring: Goffi <goffi@goffi.org> parents: 2143 diff changeset	314 id_ = profile_id[0][0]
1d3f73e065e1 core, jp: component handling + client handling refactoring: Goffi <goffi@goffi.org> parents: 2143 diff changeset	315 d_comp = self.dbpool.runQuery("INSERT INTO components(profile_id, entry_point) VALUES (?, ?)", (id_, component))
2765 378188abe941 misc: replaced all "dummy" by the more conventional and readable "__" ("_" being used for gettext) Goffi <goffi@goffi.org> parents: 2747 diff changeset	316 d_comp.addCallback(lambda __: profile_id)
2144 1d3f73e065e1 core, jp: component handling + client handling refactoring: Goffi <goffi@goffi.org> parents: 2143 diff changeset	317 return d_comp
1d3f73e065e1 core, jp: component handling + client handling refactoring: Goffi <goffi@goffi.org> parents: 2143 diff changeset	318
412 62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	319 def profile_created(profile_id):
2144 1d3f73e065e1 core, jp: component handling + client handling refactoring: Goffi <goffi@goffi.org> parents: 2143 diff changeset	320 id_= profile_id[0][0]
1d3f73e065e1 core, jp: component handling + client handling refactoring: Goffi <goffi@goffi.org> parents: 2143 diff changeset	321 self.profiles[name] = id_ # we synchronise the cache
587 952322b1d490 Remove trailing whitespaces. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 572 diff changeset	322
592 e5a875a3311b Fix pep8 support in src/memory. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 588 diff changeset	323 d = self.dbpool.runQuery("INSERT INTO profiles(name) VALUES (?)", (name, ))
412 62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	324 d.addCallback(getProfileId)
2144 1d3f73e065e1 core, jp: component handling + client handling refactoring: Goffi <goffi@goffi.org> parents: 2143 diff changeset	325 if component is not None:
1d3f73e065e1 core, jp: component handling + client handling refactoring: Goffi <goffi@goffi.org> parents: 2143 diff changeset	326 d.addCallback(setComponent)
412 62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	327 d.addCallback(profile_created)
62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	328 return d
587 952322b1d490 Remove trailing whitespaces. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 572 diff changeset	329
420 acd908528ef7 core: profile creation/deletion through database Goffi <goffi@goffi.org> parents: 416 diff changeset	330 def deleteProfile(self, name):
acd908528ef7 core: profile creation/deletion through database Goffi <goffi@goffi.org> parents: 416 diff changeset	331 """Delete profile
2143 c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	332
420 acd908528ef7 core: profile creation/deletion through database Goffi <goffi@goffi.org> parents: 416 diff changeset	333 @param name: name of the profile
2143 c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	334 @return: deferred triggered once profile is actually deleted
c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	335 """
1961 c73e08094a95 memory (sqlite): better handling of IntegrityError Goffi <goffi@goffi.org> parents: 1955 diff changeset	336 def deletionError(failure_):
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	337 log.error(_("Can't delete profile [%s]") % name)
1961 c73e08094a95 memory (sqlite): better handling of IntegrityError Goffi <goffi@goffi.org> parents: 1955 diff changeset	338 return failure_
592 e5a875a3311b Fix pep8 support in src/memory. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 588 diff changeset	339
934 34dd9287dfe5 plugin account: bug fix profile deletion from the database + unsubscribe the contacts souliane <souliane@mailoo.org> parents: 853 diff changeset	340 def delete(txn):
1705 3765e10ec52f core (sqlite storage): do explicit delete to profile linked tables for deleteProfile Goffi <goffi@goffi.org> parents: 1455 diff changeset	341 profile_id = self.profiles.pop(name)
934 34dd9287dfe5 plugin account: bug fix profile deletion from the database + unsubscribe the contacts souliane <souliane@mailoo.org> parents: 853 diff changeset	342 txn.execute("DELETE FROM profiles WHERE name = ?", (name,))
1705 3765e10ec52f core (sqlite storage): do explicit delete to profile linked tables for deleteProfile Goffi <goffi@goffi.org> parents: 1455 diff changeset	343 # FIXME: the following queries should be done by the ON DELETE CASCADE
3765e10ec52f core (sqlite storage): do explicit delete to profile linked tables for deleteProfile Goffi <goffi@goffi.org> parents: 1455 diff changeset	344 # but it seems they are not, so we explicitly do them by security
3765e10ec52f core (sqlite storage): do explicit delete to profile linked tables for deleteProfile Goffi <goffi@goffi.org> parents: 1455 diff changeset	345 # this need more investigation
3765e10ec52f core (sqlite storage): do explicit delete to profile linked tables for deleteProfile Goffi <goffi@goffi.org> parents: 1455 diff changeset	346 txn.execute("DELETE FROM history WHERE profile_id = ?", (profile_id,))
3765e10ec52f core (sqlite storage): do explicit delete to profile linked tables for deleteProfile Goffi <goffi@goffi.org> parents: 1455 diff changeset	347 txn.execute("DELETE FROM param_ind WHERE profile_id = ?", (profile_id,))
3765e10ec52f core (sqlite storage): do explicit delete to profile linked tables for deleteProfile Goffi <goffi@goffi.org> parents: 1455 diff changeset	348 txn.execute("DELETE FROM private_ind WHERE profile_id = ?", (profile_id,))
3765e10ec52f core (sqlite storage): do explicit delete to profile linked tables for deleteProfile Goffi <goffi@goffi.org> parents: 1455 diff changeset	349 txn.execute("DELETE FROM private_ind_bin WHERE profile_id = ?", (profile_id,))
2143 c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	350 txn.execute("DELETE FROM components WHERE profile_id = ?", (profile_id,))
934 34dd9287dfe5 plugin account: bug fix profile deletion from the database + unsubscribe the contacts souliane <souliane@mailoo.org> parents: 853 diff changeset	351 return None
34dd9287dfe5 plugin account: bug fix profile deletion from the database + unsubscribe the contacts souliane <souliane@mailoo.org> parents: 853 diff changeset	352
34dd9287dfe5 plugin account: bug fix profile deletion from the database + unsubscribe the contacts souliane <souliane@mailoo.org> parents: 853 diff changeset	353 d = self.dbpool.runInteraction(delete)
993 301b342c697a core: use of the new core.log module: Goffi <goffi@goffi.org> parents: 936 diff changeset	354 d.addCallback(lambda ignore: log.info(_("Profile [%s] deleted") % name))
593 70bae685d05c core: added forgotten errback in sqlite's deleteProfile Goffi <goffi@goffi.org> parents: 592 diff changeset	355 d.addErrback(deletionError)
420 acd908528ef7 core: profile creation/deletion through database Goffi <goffi@goffi.org> parents: 416 diff changeset	356 return d
acd908528ef7 core: profile creation/deletion through database Goffi <goffi@goffi.org> parents: 416 diff changeset	357
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	358 ## Params
412 62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	359 def loadGenParams(self, params_gen):
62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	360 """Load general parameters
2143 c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	361
412 62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	362 @param params_gen: dictionary to fill
2143 c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	363 @return: deferred
c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	364 """
592 e5a875a3311b Fix pep8 support in src/memory. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 588 diff changeset	365
412 62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	366 def fillParams(result):
413 dd4caab17008 core: - individual parameters managed through sqlite Goffi <goffi@goffi.org> parents: 412 diff changeset	367 for param in result:
592 e5a875a3311b Fix pep8 support in src/memory. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 588 diff changeset	368 category, name, value = param
413 dd4caab17008 core: - individual parameters managed through sqlite Goffi <goffi@goffi.org> parents: 412 diff changeset	369 params_gen[(category, name)] = value
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	370 log.debug(_("loading general parameters from database"))
412 62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	371 return self.dbpool.runQuery("SELECT category,name,value FROM param_gen").addCallback(fillParams)
62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	372
62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	373 def loadIndParams(self, params_ind, profile):
432 31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory) Goffi <goffi@goffi.org> parents: 425 diff changeset	374 """Load individual parameters
2143 c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	375
412 62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	376 @param params_ind: dictionary to fill
62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	377 @param profile: a profile which must exist
2143 c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	378 @return: deferred
c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	379 """
592 e5a875a3311b Fix pep8 support in src/memory. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 588 diff changeset	380
412 62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	381 def fillParams(result):
413 dd4caab17008 core: - individual parameters managed through sqlite Goffi <goffi@goffi.org> parents: 412 diff changeset	382 for param in result:
592 e5a875a3311b Fix pep8 support in src/memory. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 588 diff changeset	383 category, name, value = param
423 6c20c76abdcc backend: - bridge async D-Bus method now automatically manage callback and errback, we just have to return a deferred Goffi <goffi@goffi.org> parents: 420 diff changeset	384 params_ind[(category, name)] = value
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	385 log.debug(_("loading individual parameters from database"))
592 e5a875a3311b Fix pep8 support in src/memory. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 588 diff changeset	386 d = self.dbpool.runQuery("SELECT category,name,value FROM param_ind WHERE profile_id=?", (self.profiles[profile], ))
412 62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	387 d.addCallback(fillParams)
62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	388 return d
62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	389
413 dd4caab17008 core: - individual parameters managed through sqlite Goffi <goffi@goffi.org> parents: 412 diff changeset	390 def getIndParam(self, category, name, profile):
dd4caab17008 core: - individual parameters managed through sqlite Goffi <goffi@goffi.org> parents: 412 diff changeset	391 """Ask database for the value of one specific individual parameter
2143 c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	392
413 dd4caab17008 core: - individual parameters managed through sqlite Goffi <goffi@goffi.org> parents: 412 diff changeset	393 @param category: category of the parameter
dd4caab17008 core: - individual parameters managed through sqlite Goffi <goffi@goffi.org> parents: 412 diff changeset	394 @param name: name of the parameter
dd4caab17008 core: - individual parameters managed through sqlite Goffi <goffi@goffi.org> parents: 412 diff changeset	395 @param profile: %(doc_profile)s
2143 c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	396 @return: deferred
c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	397 """
3120 0c29155ac68b core: backend autoconnection: Goffi <goffi@goffi.org> parents: 3079 diff changeset	398 d = self.dbpool.runQuery(
0c29155ac68b core: backend autoconnection: Goffi <goffi@goffi.org> parents: 3079 diff changeset	399 "SELECT value FROM param_ind WHERE category=? AND name=? AND profile_id=?",
0c29155ac68b core: backend autoconnection: Goffi <goffi@goffi.org> parents: 3079 diff changeset	400 (category, name, self.profiles[profile]))
413 dd4caab17008 core: - individual parameters managed through sqlite Goffi <goffi@goffi.org> parents: 412 diff changeset	401 d.addCallback(self.__getFirstResult)
dd4caab17008 core: - individual parameters managed through sqlite Goffi <goffi@goffi.org> parents: 412 diff changeset	402 return d
dd4caab17008 core: - individual parameters managed through sqlite Goffi <goffi@goffi.org> parents: 412 diff changeset	403
3120 0c29155ac68b core: backend autoconnection: Goffi <goffi@goffi.org> parents: 3079 diff changeset	404 async def getIndParamValues(self, category, name):
0c29155ac68b core: backend autoconnection: Goffi <goffi@goffi.org> parents: 3079 diff changeset	405 """Ask database for the individual values of a parameter for all profiles
0c29155ac68b core: backend autoconnection: Goffi <goffi@goffi.org> parents: 3079 diff changeset	406
0c29155ac68b core: backend autoconnection: Goffi <goffi@goffi.org> parents: 3079 diff changeset	407 @param category: category of the parameter
0c29155ac68b core: backend autoconnection: Goffi <goffi@goffi.org> parents: 3079 diff changeset	408 @param name: name of the parameter
0c29155ac68b core: backend autoconnection: Goffi <goffi@goffi.org> parents: 3079 diff changeset	409 @return dict: profile => value map
0c29155ac68b core: backend autoconnection: Goffi <goffi@goffi.org> parents: 3079 diff changeset	410 """
0c29155ac68b core: backend autoconnection: Goffi <goffi@goffi.org> parents: 3079 diff changeset	411 result = await self.dbpool.runQuery(
0c29155ac68b core: backend autoconnection: Goffi <goffi@goffi.org> parents: 3079 diff changeset	412 "SELECT profiles.name, param_ind.value FROM param_ind JOIN profiles ON "
0c29155ac68b core: backend autoconnection: Goffi <goffi@goffi.org> parents: 3079 diff changeset	413 "param_ind.profile_id = profiles.id WHERE param_ind.category=? "
0c29155ac68b core: backend autoconnection: Goffi <goffi@goffi.org> parents: 3079 diff changeset	414 "and param_ind.name=?",
0c29155ac68b core: backend autoconnection: Goffi <goffi@goffi.org> parents: 3079 diff changeset	415 (category, name))
0c29155ac68b core: backend autoconnection: Goffi <goffi@goffi.org> parents: 3079 diff changeset	416 return dict(result)
0c29155ac68b core: backend autoconnection: Goffi <goffi@goffi.org> parents: 3079 diff changeset	417
412 62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	418 def setGenParam(self, category, name, value):
62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	419 """Save the general parameters in database
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	420
412 62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	421 @param category: category of the parameter
62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	422 @param name: name of the parameter
62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	423 @param value: value to set
62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	424 @return: deferred"""
62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	425 d = self.dbpool.runQuery("REPLACE INTO param_gen(category,name,value) VALUES (?,?,?)", (category, name, value))
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	426 d.addErrback(lambda ignore: log.error(_("Can't set general parameter (%(category)s/%(name)s) in database" % {"category": category, "name": name})))
412 62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	427 return d
62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	428
62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	429 def setIndParam(self, category, name, value, profile):
432 31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory) Goffi <goffi@goffi.org> parents: 425 diff changeset	430 """Save the individual parameters in database
2143 c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	431
412 62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	432 @param category: category of the parameter
62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	433 @param name: name of the parameter
62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	434 @param value: value to set
62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	435 @param profile: a profile which must exist
2143 c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	436 @return: deferred
c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	437 """
412 62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	438 d = self.dbpool.runQuery("REPLACE INTO param_ind(category,name,profile_id,value) VALUES (?,?,?,?)", (category, name, self.profiles[profile], value))
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	439 d.addErrback(lambda ignore: log.error(_("Can't set individual parameter (%(category)s/%(name)s) for [%(profile)s] in database" % {"category": category, "name": name, "profile": profile})))
412 62b17854254e database integration: first draft Goffi <goffi@goffi.org> parents: diff changeset	440 return d
413 dd4caab17008 core: - individual parameters managed through sqlite Goffi <goffi@goffi.org> parents: 412 diff changeset	441
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	442 ## History
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	443
2765 378188abe941 misc: replaced all "dummy" by the more conventional and readable "__" ("_" being used for gettext) Goffi <goffi@goffi.org> parents: 2747 diff changeset	444 def _addToHistoryCb(self, __, data):
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	445 # Message metadata were successfuly added to history
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	446 # now we can add message and subject
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	447 uid = data['uid']
3013 860c550028d6 memory (sqlite): properly wait for messages to be writen in database: Goffi <goffi@goffi.org> parents: 2989 diff changeset	448 d_list = []
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	449 for key in ('message', 'subject'):
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	450 for lang, value in data[key].items():
3191 7255286a298a memory (sqlite): don't store empty messages Goffi <goffi@goffi.org> parents: 3160 diff changeset	451 if not value.strip():
7255286a298a memory (sqlite): don't store empty messages Goffi <goffi@goffi.org> parents: 3160 diff changeset	452 # no need to store empty messages
7255286a298a memory (sqlite): don't store empty messages Goffi <goffi@goffi.org> parents: 3160 diff changeset	453 continue
3013 860c550028d6 memory (sqlite): properly wait for messages to be writen in database: Goffi <goffi@goffi.org> parents: 2989 diff changeset	454 d = self.dbpool.runQuery(
860c550028d6 memory (sqlite): properly wait for messages to be writen in database: Goffi <goffi@goffi.org> parents: 2989 diff changeset	455 "INSERT INTO {key}(history_uid, {key}, language) VALUES (?,?,?)"
860c550028d6 memory (sqlite): properly wait for messages to be writen in database: Goffi <goffi@goffi.org> parents: 2989 diff changeset	456 .format(key=key),
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	457 (uid, value, lang or None))
3013 860c550028d6 memory (sqlite): properly wait for messages to be writen in database: Goffi <goffi@goffi.org> parents: 2989 diff changeset	458 d.addErrback(lambda __: log.error(
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	459 _("Can't save following {key} in history (uid: {uid}, lang:{lang}):"
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	460 " {value}").format(
3013 860c550028d6 memory (sqlite): properly wait for messages to be writen in database: Goffi <goffi@goffi.org> parents: 2989 diff changeset	461 key=key, uid=uid, lang=lang, value=value)))
860c550028d6 memory (sqlite): properly wait for messages to be writen in database: Goffi <goffi@goffi.org> parents: 2989 diff changeset	462 d_list.append(d)
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	463 try:
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	464 thread = data['extra']['thread']
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	465 except KeyError:
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	466 pass
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	467 else:
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	468 thread_parent = data['extra'].get('thread_parent')
3013 860c550028d6 memory (sqlite): properly wait for messages to be writen in database: Goffi <goffi@goffi.org> parents: 2989 diff changeset	469 d = self.dbpool.runQuery(
860c550028d6 memory (sqlite): properly wait for messages to be writen in database: Goffi <goffi@goffi.org> parents: 2989 diff changeset	470 "INSERT INTO thread(history_uid, thread_id, parent_id) VALUES (?,?,?)",
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	471 (uid, thread, thread_parent))
3013 860c550028d6 memory (sqlite): properly wait for messages to be writen in database: Goffi <goffi@goffi.org> parents: 2989 diff changeset	472 d.addErrback(lambda __: log.error(
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	473 _("Can't save following thread in history (uid: {uid}): thread: "
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	474 "{thread}), parent:{parent}").format(
3013 860c550028d6 memory (sqlite): properly wait for messages to be writen in database: Goffi <goffi@goffi.org> parents: 2989 diff changeset	475 uid=uid, thread=thread, parent=thread_parent)))
860c550028d6 memory (sqlite): properly wait for messages to be writen in database: Goffi <goffi@goffi.org> parents: 2989 diff changeset	476 d_list.append(d)
860c550028d6 memory (sqlite): properly wait for messages to be writen in database: Goffi <goffi@goffi.org> parents: 2989 diff changeset	477 return defer.DeferredList(d_list)
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	478
1961 c73e08094a95 memory (sqlite): better handling of IntegrityError Goffi <goffi@goffi.org> parents: 1955 diff changeset	479 def _addToHistoryEb(self, failure_, data):
c73e08094a95 memory (sqlite): better handling of IntegrityError Goffi <goffi@goffi.org> parents: 1955 diff changeset	480 failure_.trap(sqlite3.IntegrityError)
c73e08094a95 memory (sqlite): better handling of IntegrityError Goffi <goffi@goffi.org> parents: 1955 diff changeset	481 sqlite_msg = failure_.value.args[0]
c73e08094a95 memory (sqlite): better handling of IntegrityError Goffi <goffi@goffi.org> parents: 1955 diff changeset	482 if "UNIQUE constraint failed" in sqlite_msg:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	483 log.debug("message {} is already in history, not storing it again"
3013 860c550028d6 memory (sqlite): properly wait for messages to be writen in database: Goffi <goffi@goffi.org> parents: 2989 diff changeset	484 .format(data['uid']))
1961 c73e08094a95 memory (sqlite): better handling of IntegrityError Goffi <goffi@goffi.org> parents: 1955 diff changeset	485 if 'received_timestamp' not in data:
3013 860c550028d6 memory (sqlite): properly wait for messages to be writen in database: Goffi <goffi@goffi.org> parents: 2989 diff changeset	486 log.warning(
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	487 "duplicate message is not delayed, this is maybe a bug: data={}"
3013 860c550028d6 memory (sqlite): properly wait for messages to be writen in database: Goffi <goffi@goffi.org> parents: 2989 diff changeset	488 .format(data))
1961 c73e08094a95 memory (sqlite): better handling of IntegrityError Goffi <goffi@goffi.org> parents: 1955 diff changeset	489 # we cancel message to avoid sending duplicate message to frontends
c73e08094a95 memory (sqlite): better handling of IntegrityError Goffi <goffi@goffi.org> parents: 1955 diff changeset	490 raise failure.Failure(exceptions.CancelError("Cancelled duplicated message"))
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	491 else:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	492 log.error("Can't store message in history: {}".format(failure_))
1961 c73e08094a95 memory (sqlite): better handling of IntegrityError Goffi <goffi@goffi.org> parents: 1955 diff changeset	493
c73e08094a95 memory (sqlite): better handling of IntegrityError Goffi <goffi@goffi.org> parents: 1955 diff changeset	494 def _logHistoryError(self, failure_, from_jid, to_jid, data):
c73e08094a95 memory (sqlite): better handling of IntegrityError Goffi <goffi@goffi.org> parents: 1955 diff changeset	495 if failure_.check(exceptions.CancelError):
c73e08094a95 memory (sqlite): better handling of IntegrityError Goffi <goffi@goffi.org> parents: 1955 diff changeset	496 # we propagate CancelError to avoid sending message to frontends
c73e08094a95 memory (sqlite): better handling of IntegrityError Goffi <goffi@goffi.org> parents: 1955 diff changeset	497 raise failure_
3013 860c550028d6 memory (sqlite): properly wait for messages to be writen in database: Goffi <goffi@goffi.org> parents: 2989 diff changeset	498 log.error(_(
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	499 "Can't save following message in history: from [{from_jid}] to [{to_jid}] "
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	500 "(uid: {uid})")
3013 860c550028d6 memory (sqlite): properly wait for messages to be writen in database: Goffi <goffi@goffi.org> parents: 2989 diff changeset	501 .format(from_jid=from_jid.full(), to_jid=to_jid.full(), uid=data['uid']))
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	502
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	503 def addToHistory(self, data, profile):
425 e4e9187e3b5b backend, bridge: asynchronous history Goffi <goffi@goffi.org> parents: 423 diff changeset	504 """Store a new message in history
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	505
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	506 @param data(dict): message data as build by SatMessageProtocol.onMessage
425 e4e9187e3b5b backend, bridge: asynchronous history Goffi <goffi@goffi.org> parents: 423 diff changeset	507 """
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	508 extra = pickle.dumps({k: v for k, v in data['extra'].items()
3013 860c550028d6 memory (sqlite): properly wait for messages to be writen in database: Goffi <goffi@goffi.org> parents: 2989 diff changeset	509 if k not in NOT_IN_EXTRA}, 0)
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	510 from_jid = data['from']
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	511 to_jid = data['to']
3013 860c550028d6 memory (sqlite): properly wait for messages to be writen in database: Goffi <goffi@goffi.org> parents: 2989 diff changeset	512 d = self.dbpool.runQuery(
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	513 "INSERT INTO history(uid, stanza_id, update_uid, profile_id, source, dest, "
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	514 "source_res, dest_res, timestamp, received_timestamp, type, extra) VALUES "
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	515 "(?,?,?,?,?,?,?,?,?,?,?,?)",
3013 860c550028d6 memory (sqlite): properly wait for messages to be writen in database: Goffi <goffi@goffi.org> parents: 2989 diff changeset	516 (data['uid'], data['extra'].get('stanza_id'), data['extra'].get('update_uid'),
860c550028d6 memory (sqlite): properly wait for messages to be writen in database: Goffi <goffi@goffi.org> parents: 2989 diff changeset	517 self.profiles[profile], data['from'].userhost(), to_jid.userhost(),
860c550028d6 memory (sqlite): properly wait for messages to be writen in database: Goffi <goffi@goffi.org> parents: 2989 diff changeset	518 from_jid.resource, to_jid.resource, data['timestamp'],
860c550028d6 memory (sqlite): properly wait for messages to be writen in database: Goffi <goffi@goffi.org> parents: 2989 diff changeset	519 data.get('received_timestamp'), data['type'], sqlite3.Binary(extra)))
860c550028d6 memory (sqlite): properly wait for messages to be writen in database: Goffi <goffi@goffi.org> parents: 2989 diff changeset	520 d.addCallbacks(self._addToHistoryCb,
860c550028d6 memory (sqlite): properly wait for messages to be writen in database: Goffi <goffi@goffi.org> parents: 2989 diff changeset	521 self._addToHistoryEb,
860c550028d6 memory (sqlite): properly wait for messages to be writen in database: Goffi <goffi@goffi.org> parents: 2989 diff changeset	522 callbackArgs=[data],
860c550028d6 memory (sqlite): properly wait for messages to be writen in database: Goffi <goffi@goffi.org> parents: 2989 diff changeset	523 errbackArgs=[data])
1961 c73e08094a95 memory (sqlite): better handling of IntegrityError Goffi <goffi@goffi.org> parents: 1955 diff changeset	524 d.addErrback(self._logHistoryError, from_jid, to_jid, data)
425 e4e9187e3b5b backend, bridge: asynchronous history Goffi <goffi@goffi.org> parents: 423 diff changeset	525 return d
e4e9187e3b5b backend, bridge: asynchronous history Goffi <goffi@goffi.org> parents: 423 diff changeset	526
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	527 def sqliteHistoryToList(self, query_result):
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	528 """Get SQL query result and return a list of message data dicts"""
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	529 result = []
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	530 current = {'uid': None}
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	531 for row in reversed(query_result):
3013 860c550028d6 memory (sqlite): properly wait for messages to be writen in database: Goffi <goffi@goffi.org> parents: 2989 diff changeset	532 (uid, stanza_id, update_uid, source, dest, source_res, dest_res, timestamp,
860c550028d6 memory (sqlite): properly wait for messages to be writen in database: Goffi <goffi@goffi.org> parents: 2989 diff changeset	533 received_timestamp, type_, extra, message, message_lang, subject,
860c550028d6 memory (sqlite): properly wait for messages to be writen in database: Goffi <goffi@goffi.org> parents: 2989 diff changeset	534 subject_lang, thread, thread_parent) = row
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	535 if uid != current['uid']:
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	536 # new message
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	537 try:
3545 31cbcdd096a2 memory (sqlite): work around encoding issues with pickle Goffi <goffi@goffi.org> parents: 3535 diff changeset	538 extra = self._load_pickle(extra or b"")
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	539 except EOFError:
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	540 extra = {}
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	541 current = {
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	542 'from': "%s/%s" % (source, source_res) if source_res else source,
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	543 'to': "%s/%s" % (dest, dest_res) if dest_res else dest,
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	544 'uid': uid,
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	545 'message': {},
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	546 'subject': {},
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	547 'type': type_,
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	548 'extra': extra,
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	549 'timestamp': timestamp,
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	550 }
2699 310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	551 if stanza_id is not None:
310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	552 current['extra']['stanza_id'] = stanza_id
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	553 if update_uid is not None:
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	554 current['extra']['update_uid'] = update_uid
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	555 if received_timestamp is not None:
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	556 current['extra']['received_timestamp'] = str(received_timestamp)
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	557 result.append(current)
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	558
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	559 if message is not None:
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	560 current['message'][message_lang or ''] = message
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	561
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	562 if subject is not None:
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	563 current['subject'][subject_lang or ''] = subject
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	564
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	565 if thread is not None:
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	566 current_extra = current['extra']
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	567 current_extra['thread'] = thread
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	568 if thread_parent is not None:
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	569 current_extra['thread_parent'] = thread_parent
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	570 else:
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	571 if thread_parent is not None:
3013 860c550028d6 memory (sqlite): properly wait for messages to be writen in database: Goffi <goffi@goffi.org> parents: 2989 diff changeset	572 log.error(
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	573 "Database inconsistency: thread parent without thread (uid: "
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	574 "{uid}, thread_parent: {parent})"
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	575 .format(uid=uid, parent=thread_parent))
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	576
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	577 return result
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	578
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	579 def listDict2listTuple(self, messages_data):
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	580 """Return a list of tuple as used in bridge from a list of messages data"""
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	581 ret = []
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	582 for m in messages_data:
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	583 ret.append((m['uid'], m['timestamp'], m['from'], m['to'], m['message'], m['subject'], m['type'], m['extra']))
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	584 return ret
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	585
2013 b536dd121da1 backend (memory), frontends: improved history filtering: Goffi <goffi@goffi.org> parents: 1962 diff changeset	586 def historyGet(self, from_jid, to_jid, limit=None, between=True, filters=None, profile=None):
1222 e6e0ea4dc835 memory: add Parameter "Chat history limit" souliane <souliane@mailoo.org> parents: 1221 diff changeset	587 """Retrieve messages in history
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	588
1222 e6e0ea4dc835 memory: add Parameter "Chat history limit" souliane <souliane@mailoo.org> parents: 1221 diff changeset	589 @param from_jid (JID): source JID (full, or bare for catchall)
e6e0ea4dc835 memory: add Parameter "Chat history limit" souliane <souliane@mailoo.org> parents: 1221 diff changeset	590 @param to_jid (JID): dest JID (full, or bare for catchall)
e6e0ea4dc835 memory: add Parameter "Chat history limit" souliane <souliane@mailoo.org> parents: 1221 diff changeset	591 @param limit (int): maximum number of messages to get:
e6e0ea4dc835 memory: add Parameter "Chat history limit" souliane <souliane@mailoo.org> parents: 1221 diff changeset	592 - 0 for no message (returns the empty list)
e6e0ea4dc835 memory: add Parameter "Chat history limit" souliane <souliane@mailoo.org> parents: 1221 diff changeset	593 - None for unlimited
e6e0ea4dc835 memory: add Parameter "Chat history limit" souliane <souliane@mailoo.org> parents: 1221 diff changeset	594 @param between (bool): confound source and dest (ignore the direction)
2699 310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	595 @param filters (dict[unicode, unicode]): pattern to filter the history results
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	596 @param profile (unicode): %(doc_profile)s
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	597 @return: list of tuple as in [messageNew]
425 e4e9187e3b5b backend, bridge: asynchronous history Goffi <goffi@goffi.org> parents: 423 diff changeset	598 """
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	599 assert profile
2013 b536dd121da1 backend (memory), frontends: improved history filtering: Goffi <goffi@goffi.org> parents: 1962 diff changeset	600 if filters is None:
b536dd121da1 backend (memory), frontends: improved history filtering: Goffi <goffi@goffi.org> parents: 1962 diff changeset	601 filters = {}
1221 5e5661ab5c81 memory: getHistory with limit=0 now returns an empty list, use limit=None to return all messages souliane <souliane@mailoo.org> parents: 1099 diff changeset	602 if limit == 0:
5e5661ab5c81 memory: getHistory with limit=0 now returns an empty list, use limit=None to return all messages souliane <souliane@mailoo.org> parents: 1099 diff changeset	603 return defer.succeed([])
592 e5a875a3311b Fix pep8 support in src/memory. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 588 diff changeset	604
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	605 query_parts = ["SELECT uid, stanza_id, update_uid, source, dest, source_res, dest_res, timestamp, received_timestamp,\
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	606 type, extra, message, message.language, subject, subject.language, thread_id, thread.parent_id\
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	607 FROM history LEFT JOIN message ON history.uid = message.history_uid\
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	608 LEFT JOIN subject ON history.uid=subject.history_uid\
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	609 LEFT JOIN thread ON history.uid=thread.history_uid\
2699 310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	610 WHERE profile_id=?"] # FIXME: not sure if it's the best request, messages and subjects can appear several times here
538 2c4016921403 core, frontends, bridgen plugins: fixed methods which were unproperly managing multi-profiles Goffi <goffi@goffi.org> parents: 512 diff changeset	611 values = [self.profiles[profile]]
425 e4e9187e3b5b backend, bridge: asynchronous history Goffi <goffi@goffi.org> parents: 423 diff changeset	612
2699 310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	613 def test_jid(type_, jid_):
310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	614 values.append(jid_.userhost())
310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	615 if jid_.resource:
310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	616 values.append(jid_.resource)
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	617 return '({type_}=? AND {type_}_res=?)'.format(type_=type_)
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	618 return '{type_}=?'.format(type_=type_)
587 952322b1d490 Remove trailing whitespaces. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 572 diff changeset	619
2699 310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	620 if not from_jid and not to_jid:
310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	621 # not jid specified, we want all one2one communications
310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	622 pass
310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	623 elif between:
310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	624 if not from_jid or not to_jid:
310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	625 # we only have one jid specified, we check all messages
310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	626 # from or to this jid
310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	627 jid_ = from_jid or to_jid
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	628 query_parts.append("AND ({source} OR {dest})".format(
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	629 source=test_jid('source', jid_),
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	630 dest=test_jid('dest' , jid_)))
2699 310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	631 else:
310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	632 # we have 2 jids specified, we check all communications between
310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	633 # those 2 jids
310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	634 query_parts.append(
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	635 "AND (({source_from} AND {dest_to}) "
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	636 "OR ({source_to} AND {dest_from}))".format(
2699 310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	637 source_from=test_jid('source', from_jid),
310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	638 dest_to=test_jid('dest', to_jid),
310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	639 source_to=test_jid('source', to_jid),
310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	640 dest_from=test_jid('dest', from_jid)))
425 e4e9187e3b5b backend, bridge: asynchronous history Goffi <goffi@goffi.org> parents: 423 diff changeset	641 else:
2699 310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	642 # we want one communication in specific direction (from somebody or
310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	643 # to somebody).
310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	644 q = []
310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	645 if from_jid is not None:
310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	646 q.append(test_jid('source', from_jid))
310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	647 if to_jid is not None:
310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	648 q.append(test_jid('dest', to_jid))
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	649 query_parts.append("AND " + " AND ".join(q))
2699 310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	650
2013 b536dd121da1 backend (memory), frontends: improved history filtering: Goffi <goffi@goffi.org> parents: 1962 diff changeset	651 if filters:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	652 if 'timestamp_start' in filters:
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	653 query_parts.append("AND timestamp>= ?")
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	654 values.append(float(filters['timestamp_start']))
3079 f8cc88c773c8 core (memory/sqlite): added `before_uid` filter to retrieve history before a message UID Goffi <goffi@goffi.org> parents: 3028 diff changeset	655 if 'before_uid' in filters:
f8cc88c773c8 core (memory/sqlite): added `before_uid` filter to retrieve history before a message UID Goffi <goffi@goffi.org> parents: 3028 diff changeset	656 query_parts.append("AND history.rowid<(select rowid from history where uid=?)")
f8cc88c773c8 core (memory/sqlite): added `before_uid` filter to retrieve history before a message UID Goffi <goffi@goffi.org> parents: 3028 diff changeset	657 values.append(filters['before_uid'])
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	658 if 'body' in filters:
2013 b536dd121da1 backend (memory), frontends: improved history filtering: Goffi <goffi@goffi.org> parents: 1962 diff changeset	659 # TODO: use REGEXP (function to be defined) instead of GLOB: https://www.sqlite.org/lang_expr.html
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	660 query_parts.append("AND message LIKE ?")
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	661 values.append("%{}%".format(filters['body']))
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	662 if 'search' in filters:
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	663 query_parts.append("AND (message LIKE ? OR source_res LIKE ?)")
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	664 values.extend(["%{}%".format(filters['search'])] * 2)
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	665 if 'types' in filters:
2013 b536dd121da1 backend (memory), frontends: improved history filtering: Goffi <goffi@goffi.org> parents: 1962 diff changeset	666 types = filters['types'].split()
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	667 query_parts.append("AND type IN ({})".format(','.join("?"*len(types))))
2013 b536dd121da1 backend (memory), frontends: improved history filtering: Goffi <goffi@goffi.org> parents: 1962 diff changeset	668 values.extend(types)
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	669 if 'not_types' in filters:
2013 b536dd121da1 backend (memory), frontends: improved history filtering: Goffi <goffi@goffi.org> parents: 1962 diff changeset	670 types = filters['not_types'].split()
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	671 query_parts.append("AND type NOT IN ({})".format(','.join("?"*len(types))))
2013 b536dd121da1 backend (memory), frontends: improved history filtering: Goffi <goffi@goffi.org> parents: 1962 diff changeset	672 values.extend(types)
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	673 if 'last_stanza_id' in filters:
2699 310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	674 # this request get the last message with a "stanza_id" that we
310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	675 # have in history. This is mainly used to retrieve messages sent
310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	676 # while we were offline, using MAM (XEP-0313).
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	677 if (filters['last_stanza_id'] is not True
2699 310e41bd6666 core (memory/sqlite): added stanza_id: Goffi <goffi@goffi.org> parents: 2562 diff changeset	678 or limit != 1):
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	679 raise ValueError("Unexpected values for last_stanza_id filter")
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	680 query_parts.append("AND stanza_id IS NOT NULL")
2013 b536dd121da1 backend (memory), frontends: improved history filtering: Goffi <goffi@goffi.org> parents: 1962 diff changeset	681
2715 b35c84ea73cf plugin XEP-0045: MAM implementation for MUC Goffi <goffi@goffi.org> parents: 2711 diff changeset	682
2989 c13333fcde5e memory (sqlite): fixed order for last_stanza_id: Goffi <goffi@goffi.org> parents: 2964 diff changeset	683 # timestamp may be identical for 2 close messages (specially when delay is
c13333fcde5e memory (sqlite): fixed order for last_stanza_id: Goffi <goffi@goffi.org> parents: 2964 diff changeset	684 # used) that's why we order ties by received_timestamp
c13333fcde5e memory (sqlite): fixed order for last_stanza_id: Goffi <goffi@goffi.org> parents: 2964 diff changeset	685 # We'll reverse the order in sqliteHistoryToList
c13333fcde5e memory (sqlite): fixed order for last_stanza_id: Goffi <goffi@goffi.org> parents: 2964 diff changeset	686 # we use DESC here so LIMIT keep the last messages
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	687 query_parts.append("ORDER BY timestamp DESC, history.received_timestamp DESC")
1221 5e5661ab5c81 memory: getHistory with limit=0 now returns an empty list, use limit=None to return all messages souliane <souliane@mailoo.org> parents: 1099 diff changeset	688 if limit is not None:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	689 query_parts.append("LIMIT ?")
425 e4e9187e3b5b backend, bridge: asynchronous history Goffi <goffi@goffi.org> parents: 423 diff changeset	690 values.append(limit)
587 952322b1d490 Remove trailing whitespaces. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 572 diff changeset	691
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	692 d = self.dbpool.runQuery(" ".join(query_parts), values)
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	693 d.addCallback(self.sqliteHistoryToList)
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	694 d.addCallback(self.listDict2listTuple)
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	695 return d
425 e4e9187e3b5b backend, bridge: asynchronous history Goffi <goffi@goffi.org> parents: 423 diff changeset	696
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	697 ## Private values
2143 c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	698
2182 087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	699 def _privateDataEb(self, failure_, operation, namespace, key=None, profile=None):
087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	700 """generic errback for data queries"""
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	701 log.error(_("Can't {operation} data in database for namespace {namespace}{and_key}{for_profile}: {msg}").format(
2182 087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	702 operation = operation,
087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	703 namespace = namespace,
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	704 and_key = (" and key " + key) if key is not None else "",
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	705 for_profile = (' [' + profile + ']') if profile is not None else '',
2182 087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	706 msg = failure_))
592 e5a875a3311b Fix pep8 support in src/memory. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 588 diff changeset	707
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	708 def _load_pickle(self, v):
3551 0ce37cf3fbf4 memory(sqlite): fix encoding when loading pickled data Goffi <goffi@goffi.org> parents: 3545 diff changeset	709 # FIXME: workaround for Python 3 port, some pickled data are bytes while other are strings
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	710 try:
3551 0ce37cf3fbf4 memory(sqlite): fix encoding when loading pickled data Goffi <goffi@goffi.org> parents: 3545 diff changeset	711 return pickle.loads(v, encoding="utf-8")
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	712 except TypeError:
3551 0ce37cf3fbf4 memory(sqlite): fix encoding when loading pickled data Goffi <goffi@goffi.org> parents: 3545 diff changeset	713 data = pickle.loads(v.encode('utf-8'), encoding="utf-8")
0ce37cf3fbf4 memory(sqlite): fix encoding when loading pickled data Goffi <goffi@goffi.org> parents: 3545 diff changeset	714 log.debug(f"encoding issue in pickled data: {data}")
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	715 return data
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	716
2182 087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	717 def _generateDataDict(self, query_result, binary):
087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	718 if binary:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	719 return {k: self._load_pickle(v) for k,v in query_result}
2182 087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	720 else:
087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	721 return dict(query_result)
087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	722
087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	723 def _getPrivateTable(self, binary, profile):
087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	724 """Get table to use for private values"""
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	725 table = ['private']
425 e4e9187e3b5b backend, bridge: asynchronous history Goffi <goffi@goffi.org> parents: 423 diff changeset	726
2182 087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	727 if profile is None:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	728 table.append('gen')
2182 087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	729 else:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	730 table.append('ind')
2182 087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	731
087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	732 if binary:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	733 table.append('bin')
2182 087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	734
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	735 return '_'.join(table)
2143 c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	736
2182 087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	737 def getPrivates(self, namespace, keys=None, binary=False, profile=None):
087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	738 """Get private value(s) from databases
592 e5a875a3311b Fix pep8 support in src/memory. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 588 diff changeset	739
2182 087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	740 @param namespace(unicode): namespace of the values
087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	741 @param keys(iterable, None): keys of the values to get
087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	742 None to get all keys/values
087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	743 @param binary(bool): True to deserialise binary values
087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	744 @param profile(unicode, None): profile to use for individual values
087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	745 None to use general values
087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	746 @return (dict[unicode, object]): gotten keys/values
087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	747 """
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	748 log.debug(_("getting {type}{binary} private values from database for namespace {namespace}{keys}".format(
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	749 type = "general" if profile is None else "individual",
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	750 binary = " binary" if binary else "",
2182 087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	751 namespace = namespace,
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	752 keys = " with keys {}".format(", ".join(keys)) if keys is not None else "")))
2182 087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	753 table = self._getPrivateTable(binary, profile)
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	754 query_parts = ["SELECT key,value FROM", table, "WHERE namespace=?"]
2182 087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	755 args = [namespace]
432 31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory) Goffi <goffi@goffi.org> parents: 425 diff changeset	756
2182 087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	757 if keys is not None:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	758 placeholders = ','.join(len(keys) * '?')
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	759 query_parts.append('AND key IN (' + placeholders + ')')
2209 ea41cf1e6d29 memory (persistent, sqlite): fixed getPrivates bug with keys arguments and its use in LazyPersistentBinaryDict Goffi <goffi@goffi.org> parents: 2182 diff changeset	760 args.extend(keys)
2143 c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	761
2182 087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	762 if profile is not None:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	763 query_parts.append('AND profile_id=?')
2182 087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	764 args.append(self.profiles[profile])
087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	765
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	766 d = self.dbpool.runQuery(" ".join(query_parts), args)
2182 087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	767 d.addCallback(self._generateDataDict, binary)
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	768 d.addErrback(self._privateDataEb, "get", namespace, profile=profile)
432 31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory) Goffi <goffi@goffi.org> parents: 425 diff changeset	769 return d
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory) Goffi <goffi@goffi.org> parents: 425 diff changeset	770
2182 087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	771 def setPrivateValue(self, namespace, key, value, binary=False, profile=None):
087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	772 """Set a private value in database
087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	773
087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	774 @param namespace(unicode): namespace of the values
087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	775 @param key(unicode): key of the value to set
087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	776 @param value(object): value to set
087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	777 @param binary(bool): True if it's a binary values
087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	778 binary values need to be serialised, used for everything but strings
087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	779 @param profile(unicode, None): profile to use for individual value
087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	780 if None, it's a general value
087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	781 """
087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	782 table = self._getPrivateTable(binary, profile)
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	783 query_values_names = ['namespace', 'key', 'value']
2182 087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	784 query_values = [namespace, key]
2143 c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	785
2182 087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	786 if binary:
087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	787 value = sqlite3.Binary(pickle.dumps(value, 0))
087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	788
087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	789 query_values.append(value)
087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	790
087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	791 if profile is not None:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	792 query_values_names.append('profile_id')
2182 087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	793 query_values.append(self.profiles[profile])
087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	794
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	795 query_parts = ["REPLACE INTO", table, '(', ','.join(query_values_names), ')',
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	796 "VALUES (", ",".join('?'*len(query_values_names)), ')']
2182 087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	797
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	798 d = self.dbpool.runQuery(" ".join(query_parts), query_values)
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	799 d.addErrback(self._privateDataEb, "set", namespace, key, profile=profile)
432 31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory) Goffi <goffi@goffi.org> parents: 425 diff changeset	800 return d
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory) Goffi <goffi@goffi.org> parents: 425 diff changeset	801
2182 087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	802 def delPrivateValue(self, namespace, key, binary=False, profile=None):
087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	803 """Delete private value from database
2143 c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	804
432 31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory) Goffi <goffi@goffi.org> parents: 425 diff changeset	805 @param category: category of the privateeter
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory) Goffi <goffi@goffi.org> parents: 425 diff changeset	806 @param key: key of the private value
2182 087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	807 @param binary(bool): True if it's a binary values
087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	808 @param profile(unicode, None): profile to use for individual value
087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	809 if None, it's a general value
2143 c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	810 """
2182 087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	811 table = self._getPrivateTable(binary, profile)
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	812 query_parts = ["DELETE FROM", table, "WHERE namespace=? AND key=?"]
2182 087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	813 args = [namespace, key]
087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	814 if profile is not None:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	815 query_parts.append("AND profile_id=?")
2182 087eec4c6c07 memory (persistent, sqlite): better private values handling + new LazyPersistentBinaryDict: Goffi <goffi@goffi.org> parents: 2144 diff changeset	816 args.append(self.profiles[profile])
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	817 d = self.dbpool.runQuery(" ".join(query_parts), args)
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	818 d.addErrback(self._privateDataEb, "delete", namespace, key, profile=profile)
432 31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory) Goffi <goffi@goffi.org> parents: 425 diff changeset	819 return d
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory) Goffi <goffi@goffi.org> parents: 425 diff changeset	820
2735 ba74914277cf core (memory/sqlite): new delPrivateNamespace method to delete all data of a specific namespace for a profile. Goffi <goffi@goffi.org> parents: 2722 diff changeset	821 def delPrivateNamespace(self, namespace, binary=False, profile=None):
ba74914277cf core (memory/sqlite): new delPrivateNamespace method to delete all data of a specific namespace for a profile. Goffi <goffi@goffi.org> parents: 2722 diff changeset	822 """Delete all data from a private namespace
ba74914277cf core (memory/sqlite): new delPrivateNamespace method to delete all data of a specific namespace for a profile. Goffi <goffi@goffi.org> parents: 2722 diff changeset	823
ba74914277cf core (memory/sqlite): new delPrivateNamespace method to delete all data of a specific namespace for a profile. Goffi <goffi@goffi.org> parents: 2722 diff changeset	824 Be really cautious when you use this method, as all data with given namespace are
ba74914277cf core (memory/sqlite): new delPrivateNamespace method to delete all data of a specific namespace for a profile. Goffi <goffi@goffi.org> parents: 2722 diff changeset	825 removed.
ba74914277cf core (memory/sqlite): new delPrivateNamespace method to delete all data of a specific namespace for a profile. Goffi <goffi@goffi.org> parents: 2722 diff changeset	826 Params are the same as for delPrivateValue
ba74914277cf core (memory/sqlite): new delPrivateNamespace method to delete all data of a specific namespace for a profile. Goffi <goffi@goffi.org> parents: 2722 diff changeset	827 """
ba74914277cf core (memory/sqlite): new delPrivateNamespace method to delete all data of a specific namespace for a profile. Goffi <goffi@goffi.org> parents: 2722 diff changeset	828 table = self._getPrivateTable(binary, profile)
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	829 query_parts = ["DELETE FROM", table, "WHERE namespace=?"]
2735 ba74914277cf core (memory/sqlite): new delPrivateNamespace method to delete all data of a specific namespace for a profile. Goffi <goffi@goffi.org> parents: 2722 diff changeset	830 args = [namespace]
ba74914277cf core (memory/sqlite): new delPrivateNamespace method to delete all data of a specific namespace for a profile. Goffi <goffi@goffi.org> parents: 2722 diff changeset	831 if profile is not None:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	832 query_parts.append("AND profile_id=?")
2735 ba74914277cf core (memory/sqlite): new delPrivateNamespace method to delete all data of a specific namespace for a profile. Goffi <goffi@goffi.org> parents: 2722 diff changeset	833 args.append(self.profiles[profile])
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	834 d = self.dbpool.runQuery(" ".join(query_parts), args)
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	835 d.addErrback(self._privateDataEb, "delete namespace", namespace, profile=profile)
2735 ba74914277cf core (memory/sqlite): new delPrivateNamespace method to delete all data of a specific namespace for a profile. Goffi <goffi@goffi.org> parents: 2722 diff changeset	836 return d
ba74914277cf core (memory/sqlite): new delPrivateNamespace method to delete all data of a specific namespace for a profile. Goffi <goffi@goffi.org> parents: 2722 diff changeset	837
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	838 ## Files
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	839
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	840 @defer.inlineCallbacks
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	841 def getFiles(self, client, file_id=None, version='', parent=None, type_=None,
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	842 file_hash=None, hash_algo=None, name=None, namespace=None, mime_type=None,
3288 780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	843 public_id=None, owner=None, access=None, projection=None, unique=False):
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	844 """retrieve files with with given filters
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	845
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	846 @param file_id(unicode, None): id of the file
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	847 None to ignore
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	848 @param version(unicode, None): version of the file
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	849 None to ignore
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	850 empty string to look for current version
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	851 @param parent(unicode, None): id of the directory containing the files
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	852 None to ignore
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	853 empty string to look for root files/directories
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	854 @param projection(list[unicode], None): name of columns to retrieve
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	855 None to retrieve all
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	856 @param unique(bool): if True will remove duplicates
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	857 other params are the same as for [setFile]
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	858 @return (list[dict]): files corresponding to filters
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	859 """
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	860 query_parts = ["SELECT"]
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	861 if unique:
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	862 query_parts.append('DISTINCT')
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	863 if projection is None:
2507 4c45df43ea44 core (memory/sqlite): renamed column for files from hash to file_hash: Goffi <goffi@goffi.org> parents: 2500 diff changeset	864 projection = ['id', 'version', 'parent', 'type', 'file_hash', 'hash_algo', 'name',
3288 780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	865 'size', 'namespace', 'media_type', 'media_subtype', 'public_id', 'created', 'modified', 'owner',
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	866 'access', 'extra']
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	867 query_parts.append(','.join(projection))
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	868 query_parts.append("FROM files WHERE")
3288 780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	869 if client is not None:
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	870 filters = ['profile_id=?']
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	871 args = [self.profiles[client.profile]]
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	872 else:
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	873 if public_id is None:
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	874 raise exceptions.InternalError(
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	875 "client can only be omitted when public_id is set")
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	876 filters = []
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	877 args = []
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	878
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	879 if file_id is not None:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	880 filters.append('id=?')
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	881 args.append(file_id)
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	882 if version is not None:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	883 filters.append('version=?')
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	884 args.append(version)
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	885 if parent is not None:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	886 filters.append('parent=?')
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	887 args.append(parent)
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	888 if type_ is not None:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	889 filters.append('type=?')
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	890 args.append(type_)
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	891 if file_hash is not None:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	892 filters.append('file_hash=?')
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	893 args.append(file_hash)
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	894 if hash_algo is not None:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	895 filters.append('hash_algo=?')
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	896 args.append(hash_algo)
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	897 if name is not None:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	898 filters.append('name=?')
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	899 args.append(name)
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	900 if namespace is not None:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	901 filters.append('namespace=?')
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	902 args.append(namespace)
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	903 if mime_type is not None:
3288 780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	904 if '/' in mime_type:
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	905 filters.extend('media_type=?', 'media_subtype=?')
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	906 args.extend(mime_type.split('/', 1))
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	907 else:
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	908 filters.append('media_type=?')
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	909 args.append(mime_type)
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	910 if public_id is not None:
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	911 filters.append('public_id=?')
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	912 args.append(public_id)
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	913 if owner is not None:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	914 filters.append('owner=?')
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	915 args.append(owner.full())
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	916 if access is not None:
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	917 raise NotImplementedError('Access check is not implemented yet')
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	918 # a JSON comparison is needed here
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	919
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	920 filters = ' AND '.join(filters)
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	921 query_parts.append(filters)
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	922 query = ' '.join(query_parts)
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	923
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	924 result = yield self.dbpool.runQuery(query, args)
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	925 files_data = [dict(list(zip(projection, row))) for row in result]
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	926 to_parse = {'access', 'extra'}.intersection(projection)
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	927 to_filter = {'owner'}.intersection(projection)
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	928 if to_parse or to_filter:
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	929 for file_data in files_data:
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	930 for key in to_parse:
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	931 value = file_data[key]
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	932 file_data[key] = {} if value is None else json.loads(value)
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	933 owner = file_data.get('owner')
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	934 if owner is not None:
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	935 file_data['owner'] = jid.JID(owner)
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	936 defer.returnValue(files_data)
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	937
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	938 def setFile(self, client, name, file_id, version='', parent=None, type_=C.FILE_TYPE_FILE,
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	939 file_hash=None, hash_algo=None, size=None, namespace=None, mime_type=None,
3288 780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	940 public_id=None, created=None, modified=None, owner=None, access=None, extra=None):
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	941 """set a file metadata
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	942
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	943 @param client(SatXMPPClient): client owning the file
3288 780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	944 @param name(str): name of the file (must not contain "/")
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	945 @param file_id(str): unique id of the file
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	946 @param version(str): version of this file
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	947 @param parent(str): id of the directory containing this file
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	948 None if it is a root file/directory
3288 780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	949 @param type_(str): one of:
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	950 - file
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	951 - directory
3288 780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	952 @param file_hash(str): unique hash of the payload
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	953 @param hash_algo(str): algorithm used for hashing the file (usually sha-256)
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	954 @param size(int): size in bytes
3288 780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	955 @param namespace(str, None): identifier (human readable is better) to group files
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	956 for instance, namespace could be used to group files in a specific photo album
3288 780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	957 @param mime_type(str): media type of the file, or None if not known/guessed
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	958 @param public_id(str): ID used to server the file publicly via HTTP
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	959 @param created(int): UNIX time of creation
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	960 @param modified(int,None): UNIX time of last modification, or None to use created date
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	961 @param owner(jid.JID, None): jid of the owner of the file (mainly useful for component)
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	962 @param access(dict, None): serialisable dictionary with access rules. See [memory.memory] for details
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	963 @param extra(dict, None): serialisable dictionary of any extra data
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	964 will be encoded to json in database
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	965 """
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	966 if extra is not None:
3288 780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	967 assert isinstance(extra, dict)
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	968
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	969 if mime_type is None:
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	970 media_type = media_subtype = None
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	971 elif '/' in mime_type:
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	972 media_type, media_subtype = mime_type.split('/', 1)
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	973 else:
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	974 media_type, media_subtype = mime_type, None
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	975
2507 4c45df43ea44 core (memory/sqlite): renamed column for files from hash to file_hash: Goffi <goffi@goffi.org> parents: 2500 diff changeset	976 query = ('INSERT INTO files(id, version, parent, type, file_hash, hash_algo, name, size, namespace, '
3288 780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	977 'media_type, media_subtype, public_id, created, modified, owner, access, extra, profile_id) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)')
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	978 d = self.dbpool.runQuery(query, (file_id, version.strip(), parent, type_,
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	979 file_hash, hash_algo,
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	980 name, size, namespace,
3288 780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	981 media_type, media_subtype, public_id, created, modified,
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	982 owner.full() if owner is not None else None,
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	983 json.dumps(access) if access else None,
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	984 json.dumps(extra) if extra else None,
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	985 self.profiles[client.profile]))
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	986 d.addErrback(lambda failure: log.error(_("Can't save file metadata for [{profile}]: {reason}".format(profile=client.profile, reason=failure))))
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	987 return d
898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	988
3523 30779935c0aa core (memory, sqlite): new `fileGetUsedSpace` method Goffi <goffi@goffi.org> parents: 3479 diff changeset	989 async def fileGetUsedSpace(self, client, owner):
30779935c0aa core (memory, sqlite): new `fileGetUsedSpace` method Goffi <goffi@goffi.org> parents: 3479 diff changeset	990 """Get space used by owner of file"""
30779935c0aa core (memory, sqlite): new `fileGetUsedSpace` method Goffi <goffi@goffi.org> parents: 3479 diff changeset	991 query = "SELECT SUM(size) FROM files WHERE owner=? AND type='file' AND profile_id=?"
30779935c0aa core (memory, sqlite): new `fileGetUsedSpace` method Goffi <goffi@goffi.org> parents: 3479 diff changeset	992 ret = await self.dbpool.runQuery(
30779935c0aa core (memory, sqlite): new `fileGetUsedSpace` method Goffi <goffi@goffi.org> parents: 3479 diff changeset	993 query,
30779935c0aa core (memory, sqlite): new `fileGetUsedSpace` method Goffi <goffi@goffi.org> parents: 3479 diff changeset	994 (owner.userhost(), self.profiles[client.profile])
30779935c0aa core (memory, sqlite): new `fileGetUsedSpace` method Goffi <goffi@goffi.org> parents: 3479 diff changeset	995 )
3535 67aebefb44fc core (memory/sqlite): return 0 instead of None when user has no file yet Goffi <goffi@goffi.org> parents: 3523 diff changeset	996 return ret[0][0] or 0
3523 30779935c0aa core (memory, sqlite): new `fileGetUsedSpace` method Goffi <goffi@goffi.org> parents: 3479 diff changeset	997
2526 35d591086974 core (memory, sqlite): added fileUpdate method to update "extra" and "access" avoiding race condition Goffi <goffi@goffi.org> parents: 2507 diff changeset	998 def _fileUpdate(self, cursor, file_id, column, update_cb):
35d591086974 core (memory, sqlite): added fileUpdate method to update "extra" and "access" avoiding race condition Goffi <goffi@goffi.org> parents: 2507 diff changeset	999 query = 'SELECT {column} FROM files where id=?'.format(column=column)
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1000 for i in range(5):
2526 35d591086974 core (memory, sqlite): added fileUpdate method to update "extra" and "access" avoiding race condition Goffi <goffi@goffi.org> parents: 2507 diff changeset	1001 cursor.execute(query, [file_id])
35d591086974 core (memory, sqlite): added fileUpdate method to update "extra" and "access" avoiding race condition Goffi <goffi@goffi.org> parents: 2507 diff changeset	1002 try:
35d591086974 core (memory, sqlite): added fileUpdate method to update "extra" and "access" avoiding race condition Goffi <goffi@goffi.org> parents: 2507 diff changeset	1003 older_value_raw = cursor.fetchone()[0]
35d591086974 core (memory, sqlite): added fileUpdate method to update "extra" and "access" avoiding race condition Goffi <goffi@goffi.org> parents: 2507 diff changeset	1004 except TypeError:
35d591086974 core (memory, sqlite): added fileUpdate method to update "extra" and "access" avoiding race condition Goffi <goffi@goffi.org> parents: 2507 diff changeset	1005 raise exceptions.NotFound
2918 21cf8395616c memory (sqlite): fixed fileUpdate when original value is not set (NULL) Goffi <goffi@goffi.org> parents: 2895 diff changeset	1006 if older_value_raw is None:
21cf8395616c memory (sqlite): fixed fileUpdate when original value is not set (NULL) Goffi <goffi@goffi.org> parents: 2895 diff changeset	1007 value = {}
21cf8395616c memory (sqlite): fixed fileUpdate when original value is not set (NULL) Goffi <goffi@goffi.org> parents: 2895 diff changeset	1008 else:
21cf8395616c memory (sqlite): fixed fileUpdate when original value is not set (NULL) Goffi <goffi@goffi.org> parents: 2895 diff changeset	1009 value = json.loads(older_value_raw)
2526 35d591086974 core (memory, sqlite): added fileUpdate method to update "extra" and "access" avoiding race condition Goffi <goffi@goffi.org> parents: 2507 diff changeset	1010 update_cb(value)
35d591086974 core (memory, sqlite): added fileUpdate method to update "extra" and "access" avoiding race condition Goffi <goffi@goffi.org> parents: 2507 diff changeset	1011 value_raw = json.dumps(value)
2918 21cf8395616c memory (sqlite): fixed fileUpdate when original value is not set (NULL) Goffi <goffi@goffi.org> parents: 2895 diff changeset	1012 if older_value_raw is None:
21cf8395616c memory (sqlite): fixed fileUpdate when original value is not set (NULL) Goffi <goffi@goffi.org> parents: 2895 diff changeset	1013 update_query = 'UPDATE files SET {column}=? WHERE id=? AND {column} is NULL'.format(column=column)
21cf8395616c memory (sqlite): fixed fileUpdate when original value is not set (NULL) Goffi <goffi@goffi.org> parents: 2895 diff changeset	1014 update_args = (value_raw, file_id)
21cf8395616c memory (sqlite): fixed fileUpdate when original value is not set (NULL) Goffi <goffi@goffi.org> parents: 2895 diff changeset	1015 else:
21cf8395616c memory (sqlite): fixed fileUpdate when original value is not set (NULL) Goffi <goffi@goffi.org> parents: 2895 diff changeset	1016 update_query = 'UPDATE files SET {column}=? WHERE id=? AND {column}=?'.format(column=column)
21cf8395616c memory (sqlite): fixed fileUpdate when original value is not set (NULL) Goffi <goffi@goffi.org> parents: 2895 diff changeset	1017 update_args = (value_raw, file_id, older_value_raw)
2526 35d591086974 core (memory, sqlite): added fileUpdate method to update "extra" and "access" avoiding race condition Goffi <goffi@goffi.org> parents: 2507 diff changeset	1018 try:
35d591086974 core (memory, sqlite): added fileUpdate method to update "extra" and "access" avoiding race condition Goffi <goffi@goffi.org> parents: 2507 diff changeset	1019 cursor.execute(update_query, update_args)
35d591086974 core (memory, sqlite): added fileUpdate method to update "extra" and "access" avoiding race condition Goffi <goffi@goffi.org> parents: 2507 diff changeset	1020 except sqlite3.Error:
35d591086974 core (memory, sqlite): added fileUpdate method to update "extra" and "access" avoiding race condition Goffi <goffi@goffi.org> parents: 2507 diff changeset	1021 pass
35d591086974 core (memory, sqlite): added fileUpdate method to update "extra" and "access" avoiding race condition Goffi <goffi@goffi.org> parents: 2507 diff changeset	1022 else:
35d591086974 core (memory, sqlite): added fileUpdate method to update "extra" and "access" avoiding race condition Goffi <goffi@goffi.org> parents: 2507 diff changeset	1023 if cursor.rowcount == 1:
35d591086974 core (memory, sqlite): added fileUpdate method to update "extra" and "access" avoiding race condition Goffi <goffi@goffi.org> parents: 2507 diff changeset	1024 break;
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1025 log.warning(_("table not updated, probably due to race condition, trying again ({tries})").format(tries=i+1))
2526 35d591086974 core (memory, sqlite): added fileUpdate method to update "extra" and "access" avoiding race condition Goffi <goffi@goffi.org> parents: 2507 diff changeset	1026 else:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1027 log.error(_("Can't update file table"))
2526 35d591086974 core (memory, sqlite): added fileUpdate method to update "extra" and "access" avoiding race condition Goffi <goffi@goffi.org> parents: 2507 diff changeset	1028
35d591086974 core (memory, sqlite): added fileUpdate method to update "extra" and "access" avoiding race condition Goffi <goffi@goffi.org> parents: 2507 diff changeset	1029 def fileUpdate(self, file_id, column, update_cb):
2928 c0f6fd75af5f core (memory, memory/sqlite): implemented fileDelete Goffi <goffi@goffi.org> parents: 2919 diff changeset	1030 """Update a column value using a method to avoid race conditions
2526 35d591086974 core (memory, sqlite): added fileUpdate method to update "extra" and "access" avoiding race condition Goffi <goffi@goffi.org> parents: 2507 diff changeset	1031
35d591086974 core (memory, sqlite): added fileUpdate method to update "extra" and "access" avoiding race condition Goffi <goffi@goffi.org> parents: 2507 diff changeset	1032 the older value will be retrieved from database, then update_cb will be applied
35d591086974 core (memory, sqlite): added fileUpdate method to update "extra" and "access" avoiding race condition Goffi <goffi@goffi.org> parents: 2507 diff changeset	1033 to update it, and file will be updated checking that older value has not been changed meanwhile
35d591086974 core (memory, sqlite): added fileUpdate method to update "extra" and "access" avoiding race condition Goffi <goffi@goffi.org> parents: 2507 diff changeset	1034 by an other user. If it has changed, it tries again a couple of times before failing
35d591086974 core (memory, sqlite): added fileUpdate method to update "extra" and "access" avoiding race condition Goffi <goffi@goffi.org> parents: 2507 diff changeset	1035 @param column(str): column name (only "access" or "extra" are allowed)
35d591086974 core (memory, sqlite): added fileUpdate method to update "extra" and "access" avoiding race condition Goffi <goffi@goffi.org> parents: 2507 diff changeset	1036 @param update_cb(callable): method to update the value of the colum
35d591086974 core (memory, sqlite): added fileUpdate method to update "extra" and "access" avoiding race condition Goffi <goffi@goffi.org> parents: 2507 diff changeset	1037 the method will take older value as argument, and must update it in place
35d591086974 core (memory, sqlite): added fileUpdate method to update "extra" and "access" avoiding race condition Goffi <goffi@goffi.org> parents: 2507 diff changeset	1038 update_cb must not care about serialization,
35d591086974 core (memory, sqlite): added fileUpdate method to update "extra" and "access" avoiding race condition Goffi <goffi@goffi.org> parents: 2507 diff changeset	1039 it get the deserialized data (i.e. a Python object) directly
35d591086974 core (memory, sqlite): added fileUpdate method to update "extra" and "access" avoiding race condition Goffi <goffi@goffi.org> parents: 2507 diff changeset	1040 Note that the callable must be thread-safe
35d591086974 core (memory, sqlite): added fileUpdate method to update "extra" and "access" avoiding race condition Goffi <goffi@goffi.org> parents: 2507 diff changeset	1041 @raise exceptions.NotFound: there is not file with this id
35d591086974 core (memory, sqlite): added fileUpdate method to update "extra" and "access" avoiding race condition Goffi <goffi@goffi.org> parents: 2507 diff changeset	1042 """
35d591086974 core (memory, sqlite): added fileUpdate method to update "extra" and "access" avoiding race condition Goffi <goffi@goffi.org> parents: 2507 diff changeset	1043 if column not in ('access', 'extra'):
35d591086974 core (memory, sqlite): added fileUpdate method to update "extra" and "access" avoiding race condition Goffi <goffi@goffi.org> parents: 2507 diff changeset	1044 raise exceptions.InternalError('bad column name')
35d591086974 core (memory, sqlite): added fileUpdate method to update "extra" and "access" avoiding race condition Goffi <goffi@goffi.org> parents: 2507 diff changeset	1045 return self.dbpool.runInteraction(self._fileUpdate, file_id, column, update_cb)
35d591086974 core (memory, sqlite): added fileUpdate method to update "extra" and "access" avoiding race condition Goffi <goffi@goffi.org> parents: 2507 diff changeset	1046
2928 c0f6fd75af5f core (memory, memory/sqlite): implemented fileDelete Goffi <goffi@goffi.org> parents: 2919 diff changeset	1047 def fileDelete(self, file_id):
c0f6fd75af5f core (memory, memory/sqlite): implemented fileDelete Goffi <goffi@goffi.org> parents: 2919 diff changeset	1048 """Delete file metadata from the database
c0f6fd75af5f core (memory, memory/sqlite): implemented fileDelete Goffi <goffi@goffi.org> parents: 2919 diff changeset	1049
c0f6fd75af5f core (memory, memory/sqlite): implemented fileDelete Goffi <goffi@goffi.org> parents: 2919 diff changeset	1050 @param file_id(unicode): id of the file to delete
c0f6fd75af5f core (memory, memory/sqlite): implemented fileDelete Goffi <goffi@goffi.org> parents: 2919 diff changeset	1051 NOTE: file itself must still be removed, this method only handle metadata in
c0f6fd75af5f core (memory, memory/sqlite): implemented fileDelete Goffi <goffi@goffi.org> parents: 2919 diff changeset	1052 database
c0f6fd75af5f core (memory, memory/sqlite): implemented fileDelete Goffi <goffi@goffi.org> parents: 2919 diff changeset	1053 """
c0f6fd75af5f core (memory, memory/sqlite): implemented fileDelete Goffi <goffi@goffi.org> parents: 2919 diff changeset	1054 return self.dbpool.runQuery("DELETE FROM files WHERE id = ?", (file_id,))
c0f6fd75af5f core (memory, memory/sqlite): implemented fileDelete Goffi <goffi@goffi.org> parents: 2919 diff changeset	1055
413 dd4caab17008 core: - individual parameters managed through sqlite Goffi <goffi@goffi.org> parents: 412 diff changeset	1056 ##Helper methods##
dd4caab17008 core: - individual parameters managed through sqlite Goffi <goffi@goffi.org> parents: 412 diff changeset	1057
dd4caab17008 core: - individual parameters managed through sqlite Goffi <goffi@goffi.org> parents: 412 diff changeset	1058 def __getFirstResult(self, result):
dd4caab17008 core: - individual parameters managed through sqlite Goffi <goffi@goffi.org> parents: 412 diff changeset	1059 """Return the first result of a database query
dd4caab17008 core: - individual parameters managed through sqlite Goffi <goffi@goffi.org> parents: 412 diff changeset	1060 Useful when we are looking for one specific value"""
416 32dc8b18c2ae core: param loading/purging on profile connection/disconnection Goffi <goffi@goffi.org> parents: 413 diff changeset	1061 return None if not result else result[0][0]
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1062
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1063
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1064 class Updater(object):
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	1065 stmnt_regex = re.compile(r"[\w/' ]+(?:$.?$)?[^,]")
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1066 clean_regex = re.compile(r"^ +\|(?<= ) +\|(?<=,) +\| +$")
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1067 CREATE_SQL = "CREATE TABLE %s (%s)"
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1068 INSERT_SQL = "INSERT INTO %s VALUES (%s)"
2787 298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1069 INDEX_SQL = "CREATE INDEX %s ON %s(%s)"
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1070 DROP_SQL = "DROP TABLE %s"
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1071 ALTER_SQL = "ALTER TABLE %s ADD COLUMN %s"
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1072 RENAME_TABLE_SQL = "ALTER TABLE %s RENAME TO %s"
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1073
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1074 CONSTRAINTS = ('PRIMARY', 'UNIQUE', 'CHECK', 'FOREIGN')
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1075 TMP_TABLE = "tmp_sat_update"
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1076
2716 06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1077 def __init__(self, sqlite_storage, sat_version):
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1078 self._sat_version = sat_version
2716 06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1079 self.sqlite_storage = sqlite_storage
06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1080
06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1081 @property
06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1082 def dbpool(self):
06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1083 return self.sqlite_storage.dbpool
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1084
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1085 def getLocalVersion(self):
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1086 """ Get local database version
2143 c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	1087
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1088 @return: version (int)
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1089 """
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1090 return self.dbpool.runQuery("PRAGMA user_version").addCallback(lambda ret: int(ret[0][0]))
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1091
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1092 def _setLocalVersion(self, version):
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1093 """ Set local database version
2143 c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	1094
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1095 @param version: version (int)
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1096 @return: deferred
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1097 """
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1098 return self.dbpool.runOperation("PRAGMA user_version=%d" % version)
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1099
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1100 def getLocalSchema(self):
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1101 """ return raw local schema
2143 c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	1102
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1103 @return: list of strings with CREATE sql statements for local database
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1104 """
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1105 d = self.dbpool.runQuery("select sql from sqlite_master where type = 'table'")
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1106 d.addCallback(lambda result: [row[0] for row in result])
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1107 return d
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1108
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1109 @defer.inlineCallbacks
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1110 def checkUpdates(self):
1030 15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1111 """ Check if a database schema/content update is needed, according to DATABASE_SCHEMAS
2143 c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	1112
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1113 @return: deferred which fire a list of SQL update statements, or None if no update is needed
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1114 """
2787 298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1115 # TODO: only "table" type (i.e. "CREATE" statements) is checked,
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1116 # "index" should be checked too.
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1117 # This may be not relevant is we move to a higher level library (alchimia?)
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1118 local_version = yield self.getLocalVersion()
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1119 raw_local_sch = yield self.getLocalSchema()
2500 898b6e1fdc7a memory (sqlite): files handling: Goffi <goffi@goffi.org> parents: 2483 diff changeset	1120
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1121 local_sch = self.rawStatements2data(raw_local_sch)
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1122 current_sch = DATABASE_SCHEMAS['current']['CREATE']
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1123 local_hash = self.statementHash(local_sch)
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1124 current_hash = self.statementHash(current_sch)
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1125
1030 15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1126 # Force the update if the schemas are unchanged but a specific update is needed
15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1127 force_update = local_hash == current_hash and local_version < CURRENT_DB_VERSION \
2787 298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1128 and {'index', 'specific'}.intersection(DATABASE_SCHEMAS[CURRENT_DB_VERSION])
1030 15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1129
15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1130 if local_hash == current_hash and not force_update:
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1131 if local_version != CURRENT_DB_VERSION:
993 301b342c697a core: use of the new core.log module: Goffi <goffi@goffi.org> parents: 936 diff changeset	1132 log.warning(_("Your local schema is up-to-date, but database versions mismatch, fixing it..."))
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1133 yield self._setLocalVersion(CURRENT_DB_VERSION)
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1134 else:
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1135 # an update is needed
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1136
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1137 if local_version == CURRENT_DB_VERSION:
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1138 # Database mismatch and we have the latest version
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1139 if self._sat_version.endswith('D'):
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1140 # we are in a development version
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1141 update_data = self.generateUpdateData(local_sch, current_sch, False)
993 301b342c697a core: use of the new core.log module: Goffi <goffi@goffi.org> parents: 936 diff changeset	1142 log.warning(_("There is a schema mismatch, but as we are on a dev version, database will be updated"))
1030 15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1143 update_raw = yield self.update2raw(update_data, True)
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1144 defer.returnValue(update_raw)
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1145 else:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1146 log.error(_("schema version is up-to-date, but local schema differ from expected current schema"))
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1147 update_data = self.generateUpdateData(local_sch, current_sch, True)
1030 15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1148 update_raw = yield self.update2raw(update_data)
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1149 log.warning(_("Here are the commands that should fix the situation, use at your own risk (do a backup before modifying database), you can go to SàT's MUC room at sat@chat.jabberfr.org for help\n### SQL###\n%s\n### END SQL ###\n") % '\n'.join("%s;" % statement for statement in update_raw))
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1150 raise exceptions.DatabaseError("Database mismatch")
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1151 else:
2964 17c61d09a85b memory (sqlite): exit with an error if current db version is higher than version expected in current code Goffi <goffi@goffi.org> parents: 2928 diff changeset	1152 if local_version > CURRENT_DB_VERSION:
17c61d09a85b memory (sqlite): exit with an error if current db version is higher than version expected in current code Goffi <goffi@goffi.org> parents: 2928 diff changeset	1153 log.error(_(
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1154 "You database version is higher than the one used in this SàT "
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1155 "version, are you using several version at the same time? We "
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1156 "can't run SàT with this database."))
2964 17c61d09a85b memory (sqlite): exit with an error if current db version is higher than version expected in current code Goffi <goffi@goffi.org> parents: 2928 diff changeset	1157 sys.exit(1)
17c61d09a85b memory (sqlite): exit with an error if current db version is higher than version expected in current code Goffi <goffi@goffi.org> parents: 2928 diff changeset	1158
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1159 # Database is not up-to-date, we'll do the update
1030 15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1160 if force_update:
15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1161 log.info(_("Database content needs a specific processing, local database will be updated"))
15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1162 else:
15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1163 log.info(_("Database schema has changed, local database will be updated"))
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1164 update_raw = []
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1165 for version in range(local_version + 1, CURRENT_DB_VERSION + 1):
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1166 try:
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1167 update_data = DATABASE_SCHEMAS[version]
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1168 except KeyError:
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1169 raise exceptions.InternalError("Missing update definition (version %d)" % version)
2720 453a12ff6f51 core (memory/sqlite): do not call commitStatements if there is nothing to commit in checkUpdates Goffi <goffi@goffi.org> parents: 2716 diff changeset	1170 if "specific" in update_data and update_raw:
2716 06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1171 # if we have a specific, we must commit current statements
06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1172 # because a specific may modify database itself, and the database
06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1173 # must be in the expected state of the previous version.
06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1174 yield self.sqlite_storage.commitStatements(update_raw)
06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1175 del update_raw[:]
1030 15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1176 update_raw_step = yield self.update2raw(update_data)
2716 06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1177 if update_raw_step is not None:
06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1178 # can be None with specifics
06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1179 update_raw.extend(update_raw_step)
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1180 update_raw.append("PRAGMA user_version=%d" % CURRENT_DB_VERSION)
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1181 defer.returnValue(update_raw)
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1182
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1183 @staticmethod
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1184 def createData2Raw(data):
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1185 """ Generate SQL statements from statements data
2143 c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	1186
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1187 @param data: dictionary with table as key, and statements data in tuples as value
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1188 @return: list of strings with raw statements
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1189 """
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1190 ret = []
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1191 for table in data:
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1192 defs, constraints = data[table]
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1193 assert isinstance(defs, tuple)
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1194 assert isinstance(constraints, tuple)
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1195 ret.append(Updater.CREATE_SQL % (table, ', '.join(defs + constraints)))
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1196 return ret
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1197
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1198 @staticmethod
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1199 def insertData2Raw(data):
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1200 """ Generate SQL statements from statements data
2143 c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	1201
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1202 @param data: dictionary with table as key, and statements data in tuples as value
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1203 @return: list of strings with raw statements
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1204 """
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1205 ret = []
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1206 for table in data:
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1207 values_tuple = data[table]
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1208 assert isinstance(values_tuple, tuple)
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1209 for values in values_tuple:
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1210 assert isinstance(values, tuple)
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1211 ret.append(Updater.INSERT_SQL % (table, ', '.join(values)))
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1212 return ret
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1213
2787 298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1214 @staticmethod
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1215 def indexData2Raw(data):
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1216 """ Generate SQL statements from statements data
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1217
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1218 @param data: dictionary with table as key, and statements data in tuples as value
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1219 @return: list of strings with raw statements
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1220 """
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1221 ret = []
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1222 assert isinstance(data, tuple)
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1223 for table, col_data in data:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1224 assert isinstance(table, str)
2787 298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1225 assert isinstance(col_data, tuple)
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1226 for cols in col_data:
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1227 if isinstance(cols, tuple):
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1228 assert all([isinstance(c, str) for c in cols])
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1229 indexed_cols = ','.join(cols)
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1230 elif isinstance(cols, str):
2787 298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1231 indexed_cols = cols
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1232 else:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1233 raise exceptions.InternalError("unexpected index columns value")
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1234 index_name = table + '__' + indexed_cols.replace(',', '_')
2787 298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1235 ret.append(Updater.INDEX_SQL % (index_name, table, indexed_cols))
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1236 return ret
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1237
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1238 def statementHash(self, data):
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1239 """ Generate hash of template data
2143 c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	1240
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1241 useful to compare schemas
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1242 @param data: dictionary of "CREATE" statement, with tables names as key,
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1243 and tuples of (col_defs, constraints) as values
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1244 @return: hash as string
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1245 """
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1246 hash_ = hashlib.sha1()
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1247 tables = list(data.keys())
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1248 tables.sort()
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1249
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1250 def stmnts2str(stmts):
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1251 return ','.join([self.clean_regex.sub('',stmt) for stmt in sorted(stmts)])
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1252
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1253 for table in tables:
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1254 col_defs, col_constr = data[table]
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1255 hash_.update(
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1256 ("%s:%s:%s" % (table, stmnts2str(col_defs), stmnts2str(col_constr)))
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1257 .encode('utf-8'))
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1258 return hash_.digest()
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1259
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1260 def rawStatements2data(self, raw_statements):
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1261 """ separate "CREATE" statements into dictionary/tuples data
2143 c3cac21157d4 memory (sqlite): introduced component table, schema updated: Goffi <goffi@goffi.org> parents: 2013 diff changeset	1262
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1263 @param raw_statements: list of CREATE statements as strings
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1264 @return: dictionary with table names as key, and a (col_defs, constraints) tuple
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1265 """
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1266 schema_dict = {}
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1267 for create_statement in raw_statements:
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1268 if not create_statement.startswith("CREATE TABLE "):
993 301b342c697a core: use of the new core.log module: Goffi <goffi@goffi.org> parents: 936 diff changeset	1269 log.warning("Unexpected statement, ignoring it")
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1270 continue
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1271 _create_statement = create_statement[13:]
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1272 table, raw_col_stats = _create_statement.split(' ',1)
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1273 if raw_col_stats[0] != '(' or raw_col_stats[-1] != ')':
993 301b342c697a core: use of the new core.log module: Goffi <goffi@goffi.org> parents: 936 diff changeset	1274 log.warning("Unexpected statement structure, ignoring it")
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1275 continue
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1276 col_stats = [stmt.strip() for stmt in self.stmnt_regex.findall(raw_col_stats[1:-1])]
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1277 col_defs = []
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1278 constraints = []
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1279 for col_stat in col_stats:
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1280 name = col_stat.split(' ',1)[0]
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1281 if name in self.CONSTRAINTS:
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1282 constraints.append(col_stat)
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1283 else:
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1284 col_defs.append(col_stat)
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1285 schema_dict[table] = (tuple(col_defs), tuple(constraints))
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1286 return schema_dict
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1287
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1288 def generateUpdateData(self, old_data, new_data, modify=False):
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1289 """ Generate data for automatic update between two schema data
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1290
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1291 @param old_data: data of the former schema (which must be updated)
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1292 @param new_data: data of the current schema
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1293 @param modify: if True, always use "cols modify" table, else try to ALTER tables
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1294 @return: update data, a dictionary with:
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1295 - 'create': dictionary of tables to create
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1296 - 'delete': tuple of tables to delete
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1297 - 'cols create': dictionary of columns to create (table as key, tuple of columns to create as value)
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1298 - 'cols delete': dictionary of columns to delete (table as key, tuple of columns to delete as value)
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1299 - 'cols modify': dictionary of columns to modify (table as key, tuple of old columns to transfert as value). With this table, a new table will be created, and content from the old table will be transfered to it, only cols specified in the tuple will be transfered.
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1300 """
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1301
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1302 create_tables_data = {}
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1303 create_cols_data = {}
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1304 modify_cols_data = {}
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1305 delete_cols_data = {}
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1306 old_tables = set(old_data.keys())
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1307 new_tables = set(new_data.keys())
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1308
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1309 def getChanges(set_olds, set_news):
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1310 to_create = set_news.difference(set_olds)
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1311 to_delete = set_olds.difference(set_news)
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1312 to_check = set_news.intersection(set_olds)
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1313 return tuple(to_create), tuple(to_delete), tuple(to_check)
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1314
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1315 tables_to_create, tables_to_delete, tables_to_check = getChanges(old_tables, new_tables)
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1316
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1317 for table in tables_to_create:
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1318 create_tables_data[table] = new_data[table]
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1319
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1320 for table in tables_to_check:
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1321 old_col_defs, old_constraints = old_data[table]
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1322 new_col_defs, new_constraints = new_data[table]
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1323 for obj in old_col_defs, old_constraints, new_col_defs, new_constraints:
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1324 if not isinstance(obj, tuple):
993 301b342c697a core: use of the new core.log module: Goffi <goffi@goffi.org> parents: 936 diff changeset	1325 raise exceptions.InternalError("Columns definitions must be tuples")
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1326 defs_create, defs_delete, ignore = getChanges(set(old_col_defs), set(new_col_defs))
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1327 constraints_create, constraints_delete, ignore = getChanges(set(old_constraints), set(new_constraints))
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1328 created_col_names = set([name.split(' ',1)[0] for name in defs_create])
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1329 deleted_col_names = set([name.split(' ',1)[0] for name in defs_delete])
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1330 if (created_col_names.intersection(deleted_col_names or constraints_create or constraints_delete) or
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1331 (modify and (defs_create or constraints_create or defs_delete or constraints_delete))):
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1332 # we have modified columns, we need to transfer table
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1333 # we determinate which columns are in both schema so we can transfer them
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1334 old_names = set([name.split(' ',1)[0] for name in old_col_defs])
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1335 new_names = set([name.split(' ',1)[0] for name in new_col_defs])
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1336 modify_cols_data[table] = tuple(old_names.intersection(new_names));
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1337 else:
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1338 if defs_create:
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1339 create_cols_data[table] = (defs_create)
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1340 if defs_delete or constraints_delete:
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1341 delete_cols_data[table] = (defs_delete)
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1342
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1343 return {'create': create_tables_data,
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1344 'delete': tables_to_delete,
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1345 'cols create': create_cols_data,
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1346 'cols delete': delete_cols_data,
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1347 'cols modify': modify_cols_data
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1348 }
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1349
1030 15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1350 @defer.inlineCallbacks
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1351 def update2raw(self, update, dev_version=False):
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1352 """ Transform update data to raw SQLite statements
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1353
1455 4fb3280c4568 tmp (wokkel.rsm): use of super instead of direct call in PubSubRequest Goffi <goffi@goffi.org> parents: 1409 diff changeset	1354 @param update: update data as returned by generateUpdateData
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1355 @param dev_version: if True, update will be done in dev mode: no deletion will be done, instead a message will be shown. This prevent accidental lost of data while working on the code/database.
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1356 @return: list of string with SQL statements needed to update the base
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1357 """
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1358 ret = self.createData2Raw(update.get('create', {}))
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1359 drop = []
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1360 for table in update.get('delete', tuple()):
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1361 drop.append(self.DROP_SQL % table)
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1362 if dev_version:
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1363 if drop:
993 301b342c697a core: use of the new core.log module: Goffi <goffi@goffi.org> parents: 936 diff changeset	1364 log.info("Dev version, SQL NOT EXECUTED:\n--\n%s\n--\n" % "\n".join(drop))
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1365 else:
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1366 ret.extend(drop)
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1367
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1368 cols_create = update.get('cols create', {})
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1369 for table in cols_create:
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1370 for col_def in cols_create[table]:
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1371 ret.append(self.ALTER_SQL % (table, col_def))
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1372
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1373 cols_delete = update.get('cols delete', {})
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1374 for table in cols_delete:
993 301b342c697a core: use of the new core.log module: Goffi <goffi@goffi.org> parents: 936 diff changeset	1375 log.info("Following columns in table [%s] are not needed anymore, but are kept for dev version: %s" % (table, ", ".join(cols_delete[table])))
853 c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1376
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1377 cols_modify = update.get('cols modify', {})
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1378 for table in cols_modify:
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1379 ret.append(self.RENAME_TABLE_SQL % (table, self.TMP_TABLE))
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1380 main, extra = DATABASE_SCHEMAS['current']['CREATE'][table]
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1381 ret.append(self.CREATE_SQL % (table, ', '.join(main + extra)))
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1382 common_cols = ', '.join(cols_modify[table])
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1383 ret.append("INSERT INTO %s (%s) SELECT %s FROM %s" % (table, common_cols, common_cols, self.TMP_TABLE))
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1384 ret.append(self.DROP_SQL % self.TMP_TABLE)
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1385
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1386 insert = update.get('insert', {})
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1387 ret.extend(self.insertData2Raw(insert))
c2f6ada7858f core (sqlite): automatic database update: Goffi <goffi@goffi.org> parents: 839 diff changeset	1388
2787 298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1389 index = update.get('index', tuple())
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1390 ret.extend(self.indexData2Raw(index))
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1391
1030 15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1392 specific = update.get('specific', None)
15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1393 if specific:
15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1394 cmds = yield getattr(self, specific)()
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1395 ret.extend(cmds or [])
1030 15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1396 defer.returnValue(ret)
15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1397
3288 780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1398 def update_v9(self):
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1399 """Update database from v8 to v9
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1400
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1401 (public_id on file with UNIQUE constraint, files indexes fix, media_type split)
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1402 """
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1403 # we have to do a specific update because we can't set UNIQUE constraint when adding a column
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1404 # (see https://sqlite.org/lang_altertable.html#alter_table_add_column)
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1405 log.info("Database update to v9")
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1406
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1407 create = {
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1408 'files': (("id TEXT NOT NULL", "public_id TEXT", "version TEXT NOT NULL",
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1409 "parent TEXT NOT NULL",
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1410 "type TEXT CHECK(type in ('{file}', '{directory}')) NOT NULL DEFAULT '{file}'".format(
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1411 file=C.FILE_TYPE_FILE, directory=C.FILE_TYPE_DIRECTORY),
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1412 "file_hash TEXT", "hash_algo TEXT", "name TEXT NOT NULL", "size INTEGER",
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1413 "namespace TEXT", "media_type TEXT", "media_subtype TEXT",
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1414 "created DATETIME NOT NULL", "modified DATETIME",
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1415 "owner TEXT", "access TEXT", "extra TEXT", "profile_id INTEGER"),
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1416 ("PRIMARY KEY (id, version)", "FOREIGN KEY(profile_id) REFERENCES profiles(id) ON DELETE CASCADE",
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1417 "UNIQUE (public_id)")),
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1418
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1419 }
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1420 index = tuple({'files': (('profile_id', 'owner', 'media_type', 'media_subtype'),
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1421 ('profile_id', 'owner', 'parent'))}.items())
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1422 # XXX: Sqlite doc recommends to do the other way around (i.e. create new table,
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1423 # copy, drop old table then rename), but the RENAME would then add
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1424 # "IF NOT EXISTS" which breaks the (admittely fragile) schema comparison.
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1425 # TODO: rework sqlite update management, don't try to automatically detect
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1426 # update, the database version is now enough.
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1427 statements = ["ALTER TABLE files RENAME TO files_old"]
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1428 statements.extend(Updater.createData2Raw(create))
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1429 cols = ','.join([col_stmt.split()[0] for col_stmt in create['files'][0] if "public_id" not in col_stmt])
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1430 old_cols = cols[:]
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1431 # we need to split mime_type to the new media_type and media_subtype
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1432 old_cols = old_cols.replace(
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1433 'media_type,media_subtype',
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1434 "substr(mime_type, 0, instr(mime_type,'/')),substr(mime_type, instr(mime_type,'/')+1)"
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1435 )
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1436 statements.extend([
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1437 f"INSERT INTO files({cols}) SELECT {old_cols} FROM files_old",
3317 83f25da66bec core (memory): files are now public by default except for those in root directory: Goffi <goffi@goffi.org> parents: 3288 diff changeset	1438 "DROP TABLE files_old",
3288 780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1439 ])
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1440 statements.extend(Updater.indexData2Raw(index))
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1441 return statements
780fb8dd07ef core (memory/sqlite): new database schema (v9): Goffi <goffi@goffi.org> parents: 3191 diff changeset	1442
2787 298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1443 def update_v8(self):
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1444 """Update database from v7 to v8 (primary keys order changes + indexes)"""
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1445 log.info("Database update to v8")
2787 298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1446 statements = ["PRAGMA foreign_keys = OFF"]
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1447
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1448 # here is a copy of create and index data, we can't use "current" table
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1449 # because it may change in a future version, which would break the update
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1450 # when doing v8
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1451 create = {
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1452 'param_gen': (
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1453 ("category TEXT", "name TEXT", "value TEXT"),
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1454 ("PRIMARY KEY (category, name)",)),
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1455 'param_ind': (
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1456 ("category TEXT", "name TEXT", "profile_id INTEGER", "value TEXT"),
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1457 ("PRIMARY KEY (profile_id, category, name)", "FOREIGN KEY(profile_id) REFERENCES profiles(id) ON DELETE CASCADE")),
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1458 'private_ind': (
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1459 ("namespace TEXT", "key TEXT", "profile_id INTEGER", "value TEXT"),
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1460 ("PRIMARY KEY (profile_id, namespace, key)", "FOREIGN KEY(profile_id) REFERENCES profiles(id) ON DELETE CASCADE")),
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1461 'private_ind_bin': (
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1462 ("namespace TEXT", "key TEXT", "profile_id INTEGER", "value BLOB"),
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1463 ("PRIMARY KEY (profile_id, namespace, key)", "FOREIGN KEY(profile_id) REFERENCES profiles(id) ON DELETE CASCADE")),
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1464 }
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1465 index = (
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1466 ('history', (('profile_id', 'timestamp'),
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1467 ('profile_id', 'received_timestamp'))),
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1468 ('message', ('history_uid',)),
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1469 ('subject', ('history_uid',)),
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1470 ('thread', ('history_uid',)),
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1471 ('files', ('profile_id', 'mime_type', 'owner', 'parent')))
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1472
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1473 for table in ('param_gen', 'param_ind', 'private_ind', 'private_ind_bin'):
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1474 statements.append("ALTER TABLE {0} RENAME TO {0}_old".format(table))
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1475 schema = {table: create[table]}
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1476 cols = [d.split()[0] for d in schema[table][0]]
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1477 statements.extend(Updater.createData2Raw(schema))
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1478 statements.append("INSERT INTO {table}({cols}) "
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1479 "SELECT {cols} FROM {table}_old".format(
2787 298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1480 table=table,
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1481 cols=','.join(cols)))
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1482 statements.append("DROP TABLE {}_old".format(table))
2787 298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1483
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1484 statements.extend(Updater.indexData2Raw(index))
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1485 statements.append("PRAGMA foreign_keys = ON")
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1486 return statements
298408833ec2 memory (sqlite): optimizations Goffi <goffi@goffi.org> parents: 2771 diff changeset	1487
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1488 @defer.inlineCallbacks
2716 06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1489 def update_v7(self):
06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1490 """Update database from v6 to v7 (history unique constraint change)"""
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1491 log.info("Database update to v7, this may be long depending on your history "
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1492 "size, please be patient.")
2721 4aaa47f62d8d core (memory/sqlite): fixed v7 update performance issue: Goffi <goffi@goffi.org> parents: 2720 diff changeset	1493
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1494 log.info("Some cleaning first")
2721 4aaa47f62d8d core (memory/sqlite): fixed v7 update performance issue: Goffi <goffi@goffi.org> parents: 2720 diff changeset	1495 # we need to fix duplicate stanza_id, as it can result in conflicts with the new schema
4aaa47f62d8d core (memory/sqlite): fixed v7 update performance issue: Goffi <goffi@goffi.org> parents: 2720 diff changeset	1496 # normally database should not contain any, but better safe than sorry.
4aaa47f62d8d core (memory/sqlite): fixed v7 update performance issue: Goffi <goffi@goffi.org> parents: 2720 diff changeset	1497 rows = yield self.dbpool.runQuery(
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1498 "SELECT stanza_id, COUNT(*) as c FROM history WHERE stanza_id is not NULL "
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1499 "GROUP BY stanza_id HAVING c>1")
2721 4aaa47f62d8d core (memory/sqlite): fixed v7 update performance issue: Goffi <goffi@goffi.org> parents: 2720 diff changeset	1500 if rows:
4aaa47f62d8d core (memory/sqlite): fixed v7 update performance issue: Goffi <goffi@goffi.org> parents: 2720 diff changeset	1501 count = sum([r[1] for r in rows]) - len(rows)
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1502 log.info("{count} duplicate stanzas found, cleaning".format(count=count))
2721 4aaa47f62d8d core (memory/sqlite): fixed v7 update performance issue: Goffi <goffi@goffi.org> parents: 2720 diff changeset	1503 for stanza_id, count in rows:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1504 log.info("cleaning duplicate stanza {stanza_id}".format(stanza_id=stanza_id))
2721 4aaa47f62d8d core (memory/sqlite): fixed v7 update performance issue: Goffi <goffi@goffi.org> parents: 2720 diff changeset	1505 row_uids = yield self.dbpool.runQuery(
4aaa47f62d8d core (memory/sqlite): fixed v7 update performance issue: Goffi <goffi@goffi.org> parents: 2720 diff changeset	1506 "SELECT uid FROM history WHERE stanza_id = ? LIMIT ?",
4aaa47f62d8d core (memory/sqlite): fixed v7 update performance issue: Goffi <goffi@goffi.org> parents: 2720 diff changeset	1507 (stanza_id, count-1))
4aaa47f62d8d core (memory/sqlite): fixed v7 update performance issue: Goffi <goffi@goffi.org> parents: 2720 diff changeset	1508 uids = [r[0] for r in row_uids]
4aaa47f62d8d core (memory/sqlite): fixed v7 update performance issue: Goffi <goffi@goffi.org> parents: 2720 diff changeset	1509 yield self.dbpool.runQuery(
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1510 "DELETE FROM history WHERE uid IN ({})".format(",".join("?"*len(uids))),
2721 4aaa47f62d8d core (memory/sqlite): fixed v7 update performance issue: Goffi <goffi@goffi.org> parents: 2720 diff changeset	1511 uids)
4aaa47f62d8d core (memory/sqlite): fixed v7 update performance issue: Goffi <goffi@goffi.org> parents: 2720 diff changeset	1512
2722 14e1db0c1383 core (memory/sqlite): further improvments in update_v7: Goffi <goffi@goffi.org> parents: 2721 diff changeset	1513 def deleteInfo(txn):
14e1db0c1383 core (memory/sqlite): further improvments in update_v7: Goffi <goffi@goffi.org> parents: 2721 diff changeset	1514 # with foreign_keys on, the delete takes ages, so we deactivate it here
14e1db0c1383 core (memory/sqlite): further improvments in update_v7: Goffi <goffi@goffi.org> parents: 2721 diff changeset	1515 # the time to delete info messages from history.
14e1db0c1383 core (memory/sqlite): further improvments in update_v7: Goffi <goffi@goffi.org> parents: 2721 diff changeset	1516 txn.execute("PRAGMA foreign_keys = OFF")
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1517 txn.execute("DELETE FROM message WHERE history_uid IN (SELECT uid FROM history WHERE "
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1518 "type='info')")
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1519 txn.execute("DELETE FROM subject WHERE history_uid IN (SELECT uid FROM history WHERE "
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1520 "type='info')")
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1521 txn.execute("DELETE FROM thread WHERE history_uid IN (SELECT uid FROM history WHERE "
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1522 "type='info')")
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1523 txn.execute("DELETE FROM message WHERE history_uid IN (SELECT uid FROM history WHERE "
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1524 "type='info')")
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1525 txn.execute("DELETE FROM history WHERE type='info'")
2722 14e1db0c1383 core (memory/sqlite): further improvments in update_v7: Goffi <goffi@goffi.org> parents: 2721 diff changeset	1526 # not sure that is is necessary to reactivate here, but in doubt…
14e1db0c1383 core (memory/sqlite): further improvments in update_v7: Goffi <goffi@goffi.org> parents: 2721 diff changeset	1527 txn.execute("PRAGMA foreign_keys = ON")
14e1db0c1383 core (memory/sqlite): further improvments in update_v7: Goffi <goffi@goffi.org> parents: 2721 diff changeset	1528
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1529 log.info('Deleting "info" messages (this can take a while)')
2722 14e1db0c1383 core (memory/sqlite): further improvments in update_v7: Goffi <goffi@goffi.org> parents: 2721 diff changeset	1530 yield self.dbpool.runInteraction(deleteInfo)
14e1db0c1383 core (memory/sqlite): further improvments in update_v7: Goffi <goffi@goffi.org> parents: 2721 diff changeset	1531
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1532 log.info("Cleaning done")
2721 4aaa47f62d8d core (memory/sqlite): fixed v7 update performance issue: Goffi <goffi@goffi.org> parents: 2720 diff changeset	1533
2716 06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1534 # we have to rename table we will replace
06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1535 # tables referencing history need to be replaced to, else reference would
06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1536 # be to the old table (which will be dropped at the end). This buggy behaviour
06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1537 # seems to be fixed in new version of Sqlite
06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1538 yield self.dbpool.runQuery("ALTER TABLE history RENAME TO history_old")
06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1539 yield self.dbpool.runQuery("ALTER TABLE message RENAME TO message_old")
06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1540 yield self.dbpool.runQuery("ALTER TABLE subject RENAME TO subject_old")
06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1541 yield self.dbpool.runQuery("ALTER TABLE thread RENAME TO thread_old")
06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1542
06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1543 # history
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1544 query = ("CREATE TABLE history (uid TEXT PRIMARY KEY, stanza_id TEXT, "
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1545 "update_uid TEXT, profile_id INTEGER, source TEXT, dest TEXT, "
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1546 "source_res TEXT, dest_res TEXT, timestamp DATETIME NOT NULL, "
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1547 "received_timestamp DATETIME, type TEXT, extra BLOB, "
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1548 "FOREIGN KEY(profile_id) REFERENCES profiles(id) ON DELETE CASCADE, "
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1549 "FOREIGN KEY(type) REFERENCES message_types(type), "
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1550 "UNIQUE (profile_id, stanza_id, source, dest))")
2716 06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1551 yield self.dbpool.runQuery(query)
06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1552
06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1553 # message
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1554 query = ("CREATE TABLE message (id INTEGER PRIMARY KEY ASC, history_uid INTEGER"
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1555 ", message TEXT, language TEXT, FOREIGN KEY(history_uid) REFERENCES "
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1556 "history(uid) ON DELETE CASCADE)")
2716 06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1557 yield self.dbpool.runQuery(query)
06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1558
06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1559 # subject
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1560 query = ("CREATE TABLE subject (id INTEGER PRIMARY KEY ASC, history_uid INTEGER"
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1561 ", subject TEXT, language TEXT, FOREIGN KEY(history_uid) REFERENCES "
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1562 "history(uid) ON DELETE CASCADE)")
2716 06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1563 yield self.dbpool.runQuery(query)
06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1564
06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1565 # thread
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1566 query = ("CREATE TABLE thread (id INTEGER PRIMARY KEY ASC, history_uid INTEGER"
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1567 ", thread_id TEXT, parent_id TEXT, FOREIGN KEY(history_uid) REFERENCES "
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1568 "history(uid) ON DELETE CASCADE)")
2716 06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1569 yield self.dbpool.runQuery(query)
06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1570
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1571 log.info("Now transfering old data to new tables, please be patient.")
2716 06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1572
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1573 log.info("\nTransfering table history")
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1574 query = ("INSERT INTO history (uid, stanza_id, update_uid, profile_id, source, "
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1575 "dest, source_res, dest_res, timestamp, received_timestamp, type, extra"
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1576 ") SELECT uid, stanza_id, update_uid, profile_id, source, dest, "
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1577 "source_res, dest_res, timestamp, received_timestamp, type, extra "
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1578 "FROM history_old")
2716 06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1579 yield self.dbpool.runQuery(query)
06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1580
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1581 log.info("\nTransfering table message")
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1582 query = ("INSERT INTO message (id, history_uid, message, language) SELECT id, "
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1583 "history_uid, message, language FROM message_old")
2716 06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1584 yield self.dbpool.runQuery(query)
06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1585
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1586 log.info("\nTransfering table subject")
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1587 query = ("INSERT INTO subject (id, history_uid, subject, language) SELECT id, "
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1588 "history_uid, subject, language FROM subject_old")
2716 06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1589 yield self.dbpool.runQuery(query)
06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1590
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1591 log.info("\nTransfering table thread")
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1592 query = ("INSERT INTO thread (id, history_uid, thread_id, parent_id) SELECT id"
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1593 ", history_uid, thread_id, parent_id FROM thread_old")
2716 06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1594 yield self.dbpool.runQuery(query)
06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1595
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1596 log.info("\nRemoving old tables")
2721 4aaa47f62d8d core (memory/sqlite): fixed v7 update performance issue: Goffi <goffi@goffi.org> parents: 2720 diff changeset	1597 # because of foreign keys, tables referencing history_old
4aaa47f62d8d core (memory/sqlite): fixed v7 update performance issue: Goffi <goffi@goffi.org> parents: 2720 diff changeset	1598 # must be deleted first
4aaa47f62d8d core (memory/sqlite): fixed v7 update performance issue: Goffi <goffi@goffi.org> parents: 2720 diff changeset	1599 yield self.dbpool.runQuery("DROP TABLE thread_old")
4aaa47f62d8d core (memory/sqlite): fixed v7 update performance issue: Goffi <goffi@goffi.org> parents: 2720 diff changeset	1600 yield self.dbpool.runQuery("DROP TABLE subject_old")
2716 06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1601 yield self.dbpool.runQuery("DROP TABLE message_old")
2721 4aaa47f62d8d core (memory/sqlite): fixed v7 update performance issue: Goffi <goffi@goffi.org> parents: 2720 diff changeset	1602 yield self.dbpool.runQuery("DROP TABLE history_old")
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1603 log.info("\nReducing database size (this can take a while)")
2722 14e1db0c1383 core (memory/sqlite): further improvments in update_v7: Goffi <goffi@goffi.org> parents: 2721 diff changeset	1604 yield self.dbpool.runQuery("VACUUM")
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1605 log.info("Database update done :)")
2716 06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1606
06160b529da6 core (memory/sqlite): changed history constraint Goffi <goffi@goffi.org> parents: 2715 diff changeset	1607 @defer.inlineCallbacks
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1608 def update_v3(self):
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1609 """Update database from v2 to v3 (message refactoring)"""
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1610 # XXX: this update do all the messages in one huge transaction
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1611 # this is really memory consuming, but was OK on a reasonably
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1612 # big database for tests. If issues are happening, we can cut it
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1613 # in smaller transactions using LIMIT and by deleting already updated
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1614 # messages
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1615 log.info("Database update to v3, this may take a while")
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1616
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1617 # we need to fix duplicate timestamp, as it can result in conflicts with the new schema
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1618 rows = yield self.dbpool.runQuery("SELECT timestamp, COUNT(*) as c FROM history GROUP BY timestamp HAVING c>1")
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1619 if rows:
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1620 log.info("fixing duplicate timestamp")
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1621 fixed = []
2765 378188abe941 misc: replaced all "dummy" by the more conventional and readable "__" ("_" being used for gettext) Goffi <goffi@goffi.org> parents: 2747 diff changeset	1622 for timestamp, __ in rows:
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1623 ids_rows = yield self.dbpool.runQuery("SELECT id from history where timestamp=?", (timestamp,))
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1624 for idx, (id_,) in enumerate(ids_rows):
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1625 fixed.append(id_)
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1626 yield self.dbpool.runQuery("UPDATE history SET timestamp=? WHERE id=?", (float(timestamp) + idx * 0.001, id_))
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1627 log.info("fixed messages with ids {}".format(', '.join([str(id_) for id_ in fixed])))
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1628
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1629 def historySchema(txn):
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1630 log.info("History schema update")
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1631 txn.execute("ALTER TABLE history RENAME TO tmp_sat_update")
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1632 txn.execute("CREATE TABLE history (uid TEXT PRIMARY KEY, update_uid TEXT, profile_id INTEGER, source TEXT, dest TEXT, source_res TEXT, dest_res TEXT, timestamp DATETIME NOT NULL, received_timestamp DATETIME, type TEXT, extra BLOB, FOREIGN KEY(profile_id) REFERENCES profiles(id) ON DELETE CASCADE, FOREIGN KEY(type) REFERENCES message_types(type), UNIQUE (profile_id, timestamp, source, dest, source_res, dest_res))")
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1633 txn.execute("INSERT INTO history (uid, profile_id, source, dest, source_res, dest_res, timestamp, type, extra) SELECT id, profile_id, source, dest, source_res, dest_res, timestamp, type, extra FROM tmp_sat_update")
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1634
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1635 yield self.dbpool.runInteraction(historySchema)
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1636
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1637 def newTables(txn):
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1638 log.info("Creating new tables")
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1639 txn.execute("CREATE TABLE message (id INTEGER PRIMARY KEY ASC, history_uid INTEGER, message TEXT, language TEXT, FOREIGN KEY(history_uid) REFERENCES history(uid) ON DELETE CASCADE)")
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1640 txn.execute("CREATE TABLE thread (id INTEGER PRIMARY KEY ASC, history_uid INTEGER, thread_id TEXT, parent_id TEXT, FOREIGN KEY(history_uid) REFERENCES history(uid) ON DELETE CASCADE)")
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1641 txn.execute("CREATE TABLE subject (id INTEGER PRIMARY KEY ASC, history_uid INTEGER, subject TEXT, language TEXT, FOREIGN KEY(history_uid) REFERENCES history(uid) ON DELETE CASCADE)")
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1642
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1643 yield self.dbpool.runInteraction(newTables)
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1644
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1645 log.info("inserting new message type")
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1646 yield self.dbpool.runQuery("INSERT INTO message_types VALUES (?)", ('info',))
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1647
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1648 log.info("messages update")
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1649 rows = yield self.dbpool.runQuery("SELECT id, timestamp, message, extra FROM tmp_sat_update")
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1650 total = len(rows)
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1651
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1652 def updateHistory(txn, queries):
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1653 for query, args in iter(queries):
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1654 txn.execute(query, args)
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1655
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1656 queries = []
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1657 for idx, row in enumerate(rows, 1):
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1658 if idx % 1000 == 0 or total - idx == 0:
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1659 log.info("preparing message {}/{}".format(idx, total))
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1660 id_, timestamp, message, extra = row
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1661 try:
3545 31cbcdd096a2 memory (sqlite): work around encoding issues with pickle Goffi <goffi@goffi.org> parents: 3535 diff changeset	1662 extra = self._load_pickle(extra or b"")
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1663 except EOFError:
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1664 extra = {}
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1665 except Exception:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1666 log.warning("Can't handle extra data for message id {}, ignoring it".format(id_))
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1667 extra = {}
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1668
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1669 queries.append(("INSERT INTO message(history_uid, message) VALUES (?,?)", (id_, message)))
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1670
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1671 try:
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1672 subject = extra.pop('subject')
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1673 except KeyError:
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1674 pass
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1675 else:
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1676 try:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1677 subject = subject
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1678 except UnicodeEncodeError:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1679 log.warning("Error while decoding subject, ignoring it")
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1680 del extra['subject']
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1681 else:
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1682 queries.append(("INSERT INTO subject(history_uid, subject) VALUES (?,?)", (id_, subject)))
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1683
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1684 received_timestamp = extra.pop('timestamp', None)
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1685 try:
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1686 del extra['archive']
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1687 except KeyError:
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1688 # archive was not used
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1689 pass
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1690
1962 a45235d8dc93 memory (sqlite): fixed handling of extra (pickled data) by using sqlite3.Binary Goffi <goffi@goffi.org> parents: 1961 diff changeset	1691 queries.append(("UPDATE history SET received_timestamp=?,extra=? WHERE uid=?",(id_, received_timestamp, sqlite3.Binary(pickle.dumps(extra, 0)))))
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1692
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1693 yield self.dbpool.runInteraction(updateHistory, queries)
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1694
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1695 log.info("Dropping temporary table")
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1696 yield self.dbpool.runQuery("DROP TABLE tmp_sat_update")
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1697 log.info("Database update finished :)")
633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1698
1030 15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1699 def update2raw_v2(self):
15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1700 """Update the database from v1 to v2 (add passwords encryptions):
1955 633b5c21aefd backend, frontend: messages refactoring (huge commit, not finished): Goffi <goffi@goffi.org> parents: 1934 diff changeset	1701
1030 15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1702 - the XMPP password value is re-used for the profile password (new parameter)
15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1703 - the profile password is stored hashed
15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1704 - the XMPP password is stored encrypted, with the profile password as key
15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1705 - as there are no other stored passwords yet, it is enough, otherwise we
15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1706 would need to encrypt the other passwords as it's done for XMPP password
15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1707 """
15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1708 xmpp_pass_path = ('Connection', 'Password')
15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1709
15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1710 def encrypt_values(values):
15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1711 ret = []
15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1712 list_ = []
15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1713
1065 9378c80e3408 memory (sqlite): fixes upgrade to database v2 souliane <souliane@mailoo.org> parents: 1047 diff changeset	1714 def prepare_queries(result, xmpp_password):
9378c80e3408 memory (sqlite): fixes upgrade to database v2 souliane <souliane@mailoo.org> parents: 1047 diff changeset	1715 try:
9378c80e3408 memory (sqlite): fixes upgrade to database v2 souliane <souliane@mailoo.org> parents: 1047 diff changeset	1716 id_ = result[0][0]
9378c80e3408 memory (sqlite): fixes upgrade to database v2 souliane <souliane@mailoo.org> parents: 1047 diff changeset	1717 except IndexError:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 3013 diff changeset	1718 log.error("Profile of id %d is referenced in 'param_ind' but it doesn't exist!" % profile_id)
1099 1baa116501fa memory (sqlite): fixes upgrade Deferred issue souliane <souliane@mailoo.org> parents: 1065 diff changeset	1719 return defer.succeed(None)
1065 9378c80e3408 memory (sqlite): fixes upgrade to database v2 souliane <souliane@mailoo.org> parents: 1047 diff changeset	1720
9378c80e3408 memory (sqlite): fixes upgrade to database v2 souliane <souliane@mailoo.org> parents: 1047 diff changeset	1721 sat_password = xmpp_password
3160 330a5f1d9eea core (memory/crypto): replaced `PyCrypto` by `cryptography`: Goffi <goffi@goffi.org> parents: 3137 diff changeset	1722 sat_cipher = PasswordHasher.hash(sat_password)
1065 9378c80e3408 memory (sqlite): fixes upgrade to database v2 souliane <souliane@mailoo.org> parents: 1047 diff changeset	1723 personal_key = BlockCipher.getRandomKey(base64=True)
3160 330a5f1d9eea core (memory/crypto): replaced `PyCrypto` by `cryptography`: Goffi <goffi@goffi.org> parents: 3137 diff changeset	1724 personal_cipher = BlockCipher.encrypt(sat_password, personal_key)
330a5f1d9eea core (memory/crypto): replaced `PyCrypto` by `cryptography`: Goffi <goffi@goffi.org> parents: 3137 diff changeset	1725 xmpp_cipher = BlockCipher.encrypt(personal_key, xmpp_password)
1030 15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1726
3160 330a5f1d9eea core (memory/crypto): replaced `PyCrypto` by `cryptography`: Goffi <goffi@goffi.org> parents: 3137 diff changeset	1727 ret.append("INSERT INTO param_ind(category,name,profile_id,value) VALUES ('%s','%s',%s,'%s')" %
330a5f1d9eea core (memory/crypto): replaced `PyCrypto` by `cryptography`: Goffi <goffi@goffi.org> parents: 3137 diff changeset	1728 (C.PROFILE_PASS_PATH[0], C.PROFILE_PASS_PATH[1], id_, sat_cipher))
1030 15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1729
3160 330a5f1d9eea core (memory/crypto): replaced `PyCrypto` by `cryptography`: Goffi <goffi@goffi.org> parents: 3137 diff changeset	1730 ret.append("INSERT INTO private_ind(namespace,key,profile_id,value) VALUES ('%s','%s',%s,'%s')" %
330a5f1d9eea core (memory/crypto): replaced `PyCrypto` by `cryptography`: Goffi <goffi@goffi.org> parents: 3137 diff changeset	1731 (C.MEMORY_CRYPTO_NAMESPACE, C.MEMORY_CRYPTO_KEY, id_, personal_cipher))
1030 15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1732
3160 330a5f1d9eea core (memory/crypto): replaced `PyCrypto` by `cryptography`: Goffi <goffi@goffi.org> parents: 3137 diff changeset	1733 ret.append("REPLACE INTO param_ind(category,name,profile_id,value) VALUES ('%s','%s',%s,'%s')" %
330a5f1d9eea core (memory/crypto): replaced `PyCrypto` by `cryptography`: Goffi <goffi@goffi.org> parents: 3137 diff changeset	1734 (xmpp_pass_path[0], xmpp_pass_path[1], id_, xmpp_cipher))
1030 15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1735
15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1736
1065 9378c80e3408 memory (sqlite): fixes upgrade to database v2 souliane <souliane@mailoo.org> parents: 1047 diff changeset	1737 for profile_id, xmpp_password in values:
1030 15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1738 d = self.dbpool.runQuery("SELECT id FROM profiles WHERE id=?", (profile_id,))
1065 9378c80e3408 memory (sqlite): fixes upgrade to database v2 souliane <souliane@mailoo.org> parents: 1047 diff changeset	1739 d.addCallback(prepare_queries, xmpp_password)
1030 15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1740 list_.append(d)
15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1741
15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1742 d_list = defer.DeferredList(list_)
2765 378188abe941 misc: replaced all "dummy" by the more conventional and readable "__" ("_" being used for gettext) Goffi <goffi@goffi.org> parents: 2747 diff changeset	1743 d_list.addCallback(lambda __: ret)
1030 15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1744 return d_list
15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1745
1045 65fffdcb97f1 memory: auto-update configuration file with libervia's passphrase when migrating the database souliane <souliane@mailoo.org> parents: 1030 diff changeset	1746 def updateLiberviaConf(values):
65fffdcb97f1 memory: auto-update configuration file with libervia's passphrase when migrating the database souliane <souliane@mailoo.org> parents: 1030 diff changeset	1747 try:
65fffdcb97f1 memory: auto-update configuration file with libervia's passphrase when migrating the database souliane <souliane@mailoo.org> parents: 1030 diff changeset	1748 profile_id = values[0][0]
65fffdcb97f1 memory: auto-update configuration file with libervia's passphrase when migrating the database souliane <souliane@mailoo.org> parents: 1030 diff changeset	1749 except IndexError:
65fffdcb97f1 memory: auto-update configuration file with libervia's passphrase when migrating the database souliane <souliane@mailoo.org> parents: 1030 diff changeset	1750 return # no profile called "libervia"
65fffdcb97f1 memory: auto-update configuration file with libervia's passphrase when migrating the database souliane <souliane@mailoo.org> parents: 1030 diff changeset	1751
65fffdcb97f1 memory: auto-update configuration file with libervia's passphrase when migrating the database souliane <souliane@mailoo.org> parents: 1030 diff changeset	1752 def cb(selected):
65fffdcb97f1 memory: auto-update configuration file with libervia's passphrase when migrating the database souliane <souliane@mailoo.org> parents: 1030 diff changeset	1753 try:
65fffdcb97f1 memory: auto-update configuration file with libervia's passphrase when migrating the database souliane <souliane@mailoo.org> parents: 1030 diff changeset	1754 password = selected[0][0]
65fffdcb97f1 memory: auto-update configuration file with libervia's passphrase when migrating the database souliane <souliane@mailoo.org> parents: 1030 diff changeset	1755 except IndexError:
65fffdcb97f1 memory: auto-update configuration file with libervia's passphrase when migrating the database souliane <souliane@mailoo.org> parents: 1030 diff changeset	1756 log.error("Libervia profile exists but no password is set! Update Libervia configuration will be skipped.")
65fffdcb97f1 memory: auto-update configuration file with libervia's passphrase when migrating the database souliane <souliane@mailoo.org> parents: 1030 diff changeset	1757 return
65fffdcb97f1 memory: auto-update configuration file with libervia's passphrase when migrating the database souliane <souliane@mailoo.org> parents: 1030 diff changeset	1758 fixConfigOption('libervia', 'passphrase', password, False)
65fffdcb97f1 memory: auto-update configuration file with libervia's passphrase when migrating the database souliane <souliane@mailoo.org> parents: 1030 diff changeset	1759 d = self.dbpool.runQuery("SELECT value FROM param_ind WHERE category=? AND name=? AND profile_id=?", xmpp_pass_path + (profile_id,))
1047 5f7f913c05ac memory: synchronize the upgrade to database v2 to be sure libervia's profile password is read to be written in the config file before its encryption souliane <souliane@mailoo.org> parents: 1045 diff changeset	1760 return d.addCallback(cb)
1045 65fffdcb97f1 memory: auto-update configuration file with libervia's passphrase when migrating the database souliane <souliane@mailoo.org> parents: 1030 diff changeset	1761
65fffdcb97f1 memory: auto-update configuration file with libervia's passphrase when migrating the database souliane <souliane@mailoo.org> parents: 1030 diff changeset	1762 d = self.dbpool.runQuery("SELECT id FROM profiles WHERE name='libervia'")
65fffdcb97f1 memory: auto-update configuration file with libervia's passphrase when migrating the database souliane <souliane@mailoo.org> parents: 1030 diff changeset	1763 d.addCallback(updateLiberviaConf)
2765 378188abe941 misc: replaced all "dummy" by the more conventional and readable "__" ("_" being used for gettext) Goffi <goffi@goffi.org> parents: 2747 diff changeset	1764 d.addCallback(lambda __: self.dbpool.runQuery("SELECT profile_id,value FROM param_ind WHERE category=? AND name=?", xmpp_pass_path))
1030 15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1765 d.addCallback(encrypt_values)
15f43b54d697 core, memory, bridge: added profile password + password encryption: souliane <souliane@mailoo.org> parents: 1019 diff changeset	1766 return d

Mercurial > libervia-backend

annotate sat/memory/sqlite.py @ 3696:52ee22d78e18