annotate src/memory/sqlite.py @ 1005:b4af31a8a4f2

core (logs): added formatting, name filter and outputs management: - formatting is inspired from, and use when possible, standard logging. "message", "levelname", and "name" are the only format managed, depending on backend more can be managed (standard backend formats are specified in official python logging doc) - name filter use regular expressions. It's possible to log only plugins with SAT_LOG_LOGGER="^sat.plugins". To log only XEPs 96 & 65, we can use SAT_LOG_LOGGER='(xep_0095|xep_0065)' - output management use a particular syntax: - output handler are name with "//", so far there are "//default" (most of time stderr), "//memory" and "//file" - options can be specified in parenthesis, e.g. "//memory(50)" mean a 50 lines memory buffer (50 is the current default, so that's equivalent to "//memory") - several handlers can be specified: "//file(/tmp/sat.log)//default" will use the default logging + a the /tmp/sat.log file - if there is only one handler, it use the file handler: "/tmp/sat.log" is the same as "//file(/tmp/sat.log)" - not finished, need more work for twisted and basic backends
author Goffi <goffi@goffi.org>
date Mon, 05 May 2014 18:58:34 +0200
parents 301b342c697a
children 6a16ec17a458
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
412
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
1 #!/usr/bin/python
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
2 # -*- coding: utf-8 -*-
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
3
609
84a6e83157c2 fixed licences in docstrings (they are now in comments)
Goffi <goffi@goffi.org>
parents: 593
diff changeset
4 # SAT: a jabber client
811
1fe00f0c9a91 dates update
Goffi <goffi@goffi.org>
parents: 771
diff changeset
5 # Copyright (C) 2009, 2010, 2011, 2012, 2013, 2014 Jérôme Poisson (goffi@goffi.org)
412
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
6
609
84a6e83157c2 fixed licences in docstrings (they are now in comments)
Goffi <goffi@goffi.org>
parents: 593
diff changeset
7 # This program is free software: you can redistribute it and/or modify
84a6e83157c2 fixed licences in docstrings (they are now in comments)
Goffi <goffi@goffi.org>
parents: 593
diff changeset
8 # it under the terms of the GNU Affero General Public License as published by
84a6e83157c2 fixed licences in docstrings (they are now in comments)
Goffi <goffi@goffi.org>
parents: 593
diff changeset
9 # the Free Software Foundation, either version 3 of the License, or
84a6e83157c2 fixed licences in docstrings (they are now in comments)
Goffi <goffi@goffi.org>
parents: 593
diff changeset
10 # (at your option) any later version.
412
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
11
609
84a6e83157c2 fixed licences in docstrings (they are now in comments)
Goffi <goffi@goffi.org>
parents: 593
diff changeset
12 # This program is distributed in the hope that it will be useful,
84a6e83157c2 fixed licences in docstrings (they are now in comments)
Goffi <goffi@goffi.org>
parents: 593
diff changeset
13 # but WITHOUT ANY WARRANTY; without even the implied warranty of
84a6e83157c2 fixed licences in docstrings (they are now in comments)
Goffi <goffi@goffi.org>
parents: 593
diff changeset
14 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
84a6e83157c2 fixed licences in docstrings (they are now in comments)
Goffi <goffi@goffi.org>
parents: 593
diff changeset
15 # GNU Affero General Public License for more details.
412
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
16
609
84a6e83157c2 fixed licences in docstrings (they are now in comments)
Goffi <goffi@goffi.org>
parents: 593
diff changeset
17 # You should have received a copy of the GNU Affero General Public License
84a6e83157c2 fixed licences in docstrings (they are now in comments)
Goffi <goffi@goffi.org>
parents: 593
diff changeset
18 # along with this program. If not, see <http://www.gnu.org/licenses/>.
412
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
19
771
bfabeedbf32e core: i18n refactoring:
Goffi <goffi@goffi.org>
parents: 681
diff changeset
20 from sat.core.i18n import _
853
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
21 from sat.core import exceptions
993
301b342c697a core: use of the new core.log module:
Goffi <goffi@goffi.org>
parents: 936
diff changeset
22 from sat.core.log import getLogger
301b342c697a core: use of the new core.log module:
Goffi <goffi@goffi.org>
parents: 936
diff changeset
23 log = getLogger(__name__)
412
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
24 from twisted.enterprise import adbapi
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
25 from twisted.internet import defer
853
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
26 from collections import OrderedDict
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
27 from time import time
853
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
28 import re
412
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
29 import os.path
432
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
30 import cPickle as pickle
853
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
31 import hashlib
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
32
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
33 CURRENT_DB_VERSION = 1
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
34
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
35 # XXX: DATABASE schemas are used in the following way:
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
36 # - 'current' key is for the actual database schema, for a new base
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
37 # - x(int) is for update needed between x-1 and x. All number are needed between y and z to do an update
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
38 # e.g.: if CURRENT_DB_VERSION is 6, 'current' is the actuel DB, and to update from version 3, numbers 4, 5 and 6 are needed
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
39 # a 'current' data dict can contains the keys:
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
40 # - 'CREATE': it contains an Ordered dict with table to create as keys, and a len 2 tuple as value, where value[0] are the columns definitions and value[1] are the table constraints
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
41 # - 'INSERT': it contains an Ordered dict with table where values have to be inserted, and many tuples containing values to insert in the order of the rows (#TODO: manage named columns)
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
42 # an update data dict (the ones with a number) can contains the keys 'create', 'delete', 'cols create', 'cols delete', 'cols modify' or 'insert'. See Updater.generateUpdateData for more infos. This metho can be used to autogenerate update_data, to ease the work of the developers.
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
43
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
44 DATABASE_SCHEMAS = {
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
45 "current": {'CREATE': OrderedDict((
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
46 ('profiles', (("id INTEGER PRIMARY KEY ASC", "name TEXT"),
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
47 ("UNIQUE (name)",))),
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
48 ('message_types', (("type TEXT PRIMARY KEY",),
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
49 tuple())),
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
50 ('history', (("id INTEGER PRIMARY KEY ASC", "profile_id INTEGER", "source TEXT", "dest TEXT", "source_res TEXT", "dest_res TEXT", "timestamp DATETIME", "message TEXT", "type TEXT", "extra BLOB"),
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
51 ("FOREIGN KEY(profile_id) REFERENCES profiles(id) ON DELETE CASCADE", "FOREIGN KEY(type) REFERENCES message_types(type)"))),
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
52 ('param_gen', (("category TEXT", "name TEXT", "value TEXT"),
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
53 ("PRIMARY KEY (category,name)",))),
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
54 ('param_ind', (("category TEXT", "name TEXT", "profile_id INTEGER", "value TEXT"),
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
55 ("PRIMARY KEY (category,name,profile_id)", "FOREIGN KEY(profile_id) REFERENCES profiles(id) ON DELETE CASCADE"))),
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
56 ('private_gen', (("namespace TEXT", "key TEXT", "value TEXT"),
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
57 ("PRIMARY KEY (namespace, key)",))),
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
58 ('private_ind', (("namespace TEXT", "key TEXT", "profile_id INTEGER", "value TEXT"),
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
59 ("PRIMARY KEY (namespace, key, profile_id)", "FOREIGN KEY(profile_id) REFERENCES profiles(id) ON DELETE CASCADE"))),
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
60 ('private_gen_bin', (("namespace TEXT", "key TEXT", "value BLOB"),
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
61 ("PRIMARY KEY (namespace, key)",))),
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
62
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
63 ('private_ind_bin', (("namespace TEXT", "key TEXT", "profile_id INTEGER", "value BLOB"),
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
64 ("PRIMARY KEY (namespace, key, profile_id)", "FOREIGN KEY(profile_id) REFERENCES profiles(id) ON DELETE CASCADE")))
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
65 )),
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
66 'INSERT': OrderedDict((
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
67 ('message_types', (("'chat'",), ("'error'",), ("'groupchat'",), ("'headline'",), ("'normal'",))),
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
68 )),
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
69 },
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
70 1: {'cols create': {'history': ('extra BLOB',)}
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
71 },
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
72 }
412
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
73
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
74
588
beaf6bec2fcd Remove every old-style class.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 587
diff changeset
75 class SqliteStorage(object):
412
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
76 """This class manage storage with Sqlite database"""
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
77
853
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
78 def __init__(self, db_filename, sat_version):
412
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
79 """Connect to the given database
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
80 @param db_filename: full path to the Sqlite database"""
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
81 self.initialized = defer.Deferred() # triggered when memory is fully initialised and ready
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
82 self.profiles = {} # we keep cache for the profiles (key: profile name, value: profile id)
412
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
83
993
301b342c697a core: use of the new core.log module:
Goffi <goffi@goffi.org>
parents: 936
diff changeset
84 log.info(_("Connecting database"))
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
85 new_base = not os.path.exists(db_filename) # do we have to create the database ?
936
6404df5305e3 memory: be sure that local_dir exists before creating a new database
souliane <souliane@mailoo.org>
parents: 934
diff changeset
86 if new_base: # the dir may not exist if it's not the XDG recommended one
6404df5305e3 memory: be sure that local_dir exists before creating a new database
souliane <souliane@mailoo.org>
parents: 934
diff changeset
87 dir_ = os.path.dirname(db_filename)
6404df5305e3 memory: be sure that local_dir exists before creating a new database
souliane <souliane@mailoo.org>
parents: 934
diff changeset
88 if not os.path.exists(dir_):
6404df5305e3 memory: be sure that local_dir exists before creating a new database
souliane <souliane@mailoo.org>
parents: 934
diff changeset
89 os.makedirs(dir_, 0700)
412
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
90 self.dbpool = adbapi.ConnectionPool("sqlite3", db_filename, check_same_thread=False)
853
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
91
624
70988f08d0ad core: fixed bad database creation on first run in sqlite storage
Goffi <goffi@goffi.org>
parents: 609
diff changeset
92 # init_defer is the initialisation deferred, initialisation is ok when all its callbacks have been done
934
34dd9287dfe5 plugin account: bug fix profile deletion from the database + unsubscribe the contacts
souliane <souliane@mailoo.org>
parents: 853
diff changeset
93 # XXX: foreign_keys activation doesn't seem to work, probably because of the multi-threading
34dd9287dfe5 plugin account: bug fix profile deletion from the database + unsubscribe the contacts
souliane <souliane@mailoo.org>
parents: 853
diff changeset
94 # All the requests that need to use this feature should be run with runInteraction instead,
34dd9287dfe5 plugin account: bug fix profile deletion from the database + unsubscribe the contacts
souliane <souliane@mailoo.org>
parents: 853
diff changeset
95 # so you can set the PRAGMA as it is done in self.deleteProfile
993
301b342c697a core: use of the new core.log module:
Goffi <goffi@goffi.org>
parents: 936
diff changeset
96 init_defer = self.dbpool.runOperation("PRAGMA foreign_keys = ON").addErrback(lambda x: log.error(_("Can't activate foreign keys")))
624
70988f08d0ad core: fixed bad database creation on first run in sqlite storage
Goffi <goffi@goffi.org>
parents: 609
diff changeset
97
853
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
98 def getNewBaseSql():
993
301b342c697a core: use of the new core.log module:
Goffi <goffi@goffi.org>
parents: 936
diff changeset
99 log.info(_("The database is new, creating the tables"))
853
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
100 database_creation = ["PRAGMA user_version=%d" % CURRENT_DB_VERSION]
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
101 database_creation.extend(Updater.createData2Raw(DATABASE_SCHEMAS['current']['CREATE']))
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
102 database_creation.extend(Updater.insertData2Raw(DATABASE_SCHEMAS['current']['INSERT']))
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
103 return database_creation
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
104
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
105 def getUpdateSql():
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
106 updater = Updater(self.dbpool, sat_version)
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
107 return updater.checkUpdates()
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
108
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
109 def commitStatements(statements):
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
110
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
111 if statements is None:
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
112 return defer.succeed(None)
993
301b342c697a core: use of the new core.log module:
Goffi <goffi@goffi.org>
parents: 936
diff changeset
113 log.debug("===== COMMITING STATEMENTS =====\n%s\n============\n\n" % '\n'.join(statements))
853
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
114 d = self.dbpool.runInteraction(self._updateDb, tuple(statements))
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
115 return d
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
116
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
117 init_defer.addCallback(lambda ignore: getNewBaseSql() if new_base else getUpdateSql())
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
118 init_defer.addCallback(commitStatements)
412
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
119
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
120 def fillProfileCache(ignore):
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
121 d = self.dbpool.runQuery("SELECT name,id FROM profiles").addCallback(self._profilesCache)
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
122 d.chainDeferred(self.initialized)
587
952322b1d490 Remove trailing whitespaces.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 572
diff changeset
123
624
70988f08d0ad core: fixed bad database creation on first run in sqlite storage
Goffi <goffi@goffi.org>
parents: 609
diff changeset
124 init_defer.addCallback(fillProfileCache)
412
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
125
853
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
126 def _updateDb(self, interaction, statements):
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
127 for statement in statements:
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
128 interaction.execute(statement)
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
129
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
130
412
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
131 #Profiles
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
132 def _profilesCache(self, profiles_result):
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
133 """Fill the profiles cache
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
134 @param profiles_result: result of the sql profiles query"""
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
135 for profile in profiles_result:
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
136 name, id_ = profile
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
137 self.profiles[name] = id_
587
952322b1d490 Remove trailing whitespaces.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 572
diff changeset
138
412
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
139 def getProfilesList(self):
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
140 """"Return list of all registered profiles"""
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
141 return self.profiles.keys()
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
142
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
143 def hasProfile(self, profile_name):
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
144 """return True if profile_name exists
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
145 @param profile_name: name of the profile to check"""
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
146 return profile_name in self.profiles
587
952322b1d490 Remove trailing whitespaces.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 572
diff changeset
147
412
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
148 def createProfile(self, name):
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
149 """Create a new profile
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
150 @param name: name of the profile
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
151 @return: deferred triggered once profile is actually created"""
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
152
412
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
153 def getProfileId(ignore):
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
154 return self.dbpool.runQuery("SELECT (id) FROM profiles WHERE name = ?", (name, ))
587
952322b1d490 Remove trailing whitespaces.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 572
diff changeset
155
412
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
156 def profile_created(profile_id):
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
157 _id = profile_id[0][0]
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
158 self.profiles[name] = _id # we synchronise the cache
587
952322b1d490 Remove trailing whitespaces.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 572
diff changeset
159
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
160 d = self.dbpool.runQuery("INSERT INTO profiles(name) VALUES (?)", (name, ))
412
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
161 d.addCallback(getProfileId)
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
162 d.addCallback(profile_created)
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
163 return d
587
952322b1d490 Remove trailing whitespaces.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 572
diff changeset
164
420
acd908528ef7 core: profile creation/deletion through database
Goffi <goffi@goffi.org>
parents: 416
diff changeset
165 def deleteProfile(self, name):
acd908528ef7 core: profile creation/deletion through database
Goffi <goffi@goffi.org>
parents: 416
diff changeset
166 """Delete profile
acd908528ef7 core: profile creation/deletion through database
Goffi <goffi@goffi.org>
parents: 416
diff changeset
167 @param name: name of the profile
acd908528ef7 core: profile creation/deletion through database
Goffi <goffi@goffi.org>
parents: 416
diff changeset
168 @return: deferred triggered once profile is actually deleted"""
acd908528ef7 core: profile creation/deletion through database
Goffi <goffi@goffi.org>
parents: 416
diff changeset
169 def deletionError(failure):
993
301b342c697a core: use of the new core.log module:
Goffi <goffi@goffi.org>
parents: 936
diff changeset
170 log.error(_("Can't delete profile [%s]") % name)
420
acd908528ef7 core: profile creation/deletion through database
Goffi <goffi@goffi.org>
parents: 416
diff changeset
171 return failure
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
172
934
34dd9287dfe5 plugin account: bug fix profile deletion from the database + unsubscribe the contacts
souliane <souliane@mailoo.org>
parents: 853
diff changeset
173 def delete(txn):
34dd9287dfe5 plugin account: bug fix profile deletion from the database + unsubscribe the contacts
souliane <souliane@mailoo.org>
parents: 853
diff changeset
174 del self.profiles[name]
34dd9287dfe5 plugin account: bug fix profile deletion from the database + unsubscribe the contacts
souliane <souliane@mailoo.org>
parents: 853
diff changeset
175 txn.execute("PRAGMA foreign_keys = ON")
34dd9287dfe5 plugin account: bug fix profile deletion from the database + unsubscribe the contacts
souliane <souliane@mailoo.org>
parents: 853
diff changeset
176 txn.execute("DELETE FROM profiles WHERE name = ?", (name,))
34dd9287dfe5 plugin account: bug fix profile deletion from the database + unsubscribe the contacts
souliane <souliane@mailoo.org>
parents: 853
diff changeset
177 return None
34dd9287dfe5 plugin account: bug fix profile deletion from the database + unsubscribe the contacts
souliane <souliane@mailoo.org>
parents: 853
diff changeset
178
34dd9287dfe5 plugin account: bug fix profile deletion from the database + unsubscribe the contacts
souliane <souliane@mailoo.org>
parents: 853
diff changeset
179 d = self.dbpool.runInteraction(delete)
993
301b342c697a core: use of the new core.log module:
Goffi <goffi@goffi.org>
parents: 936
diff changeset
180 d.addCallback(lambda ignore: log.info(_("Profile [%s] deleted") % name))
593
70bae685d05c core: added forgotten errback in sqlite's deleteProfile
Goffi <goffi@goffi.org>
parents: 592
diff changeset
181 d.addErrback(deletionError)
420
acd908528ef7 core: profile creation/deletion through database
Goffi <goffi@goffi.org>
parents: 416
diff changeset
182 return d
acd908528ef7 core: profile creation/deletion through database
Goffi <goffi@goffi.org>
parents: 416
diff changeset
183
412
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
184 #Params
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
185 def loadGenParams(self, params_gen):
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
186 """Load general parameters
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
187 @param params_gen: dictionary to fill
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
188 @return: deferred"""
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
189
412
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
190 def fillParams(result):
413
dd4caab17008 core: - individual parameters managed through sqlite
Goffi <goffi@goffi.org>
parents: 412
diff changeset
191 for param in result:
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
192 category, name, value = param
413
dd4caab17008 core: - individual parameters managed through sqlite
Goffi <goffi@goffi.org>
parents: 412
diff changeset
193 params_gen[(category, name)] = value
993
301b342c697a core: use of the new core.log module:
Goffi <goffi@goffi.org>
parents: 936
diff changeset
194 log.debug(_("loading general parameters from database"))
412
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
195 return self.dbpool.runQuery("SELECT category,name,value FROM param_gen").addCallback(fillParams)
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
196
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
197 def loadIndParams(self, params_ind, profile):
432
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
198 """Load individual parameters
412
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
199 @param params_ind: dictionary to fill
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
200 @param profile: a profile which *must* exist
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
201 @return: deferred"""
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
202
412
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
203 def fillParams(result):
413
dd4caab17008 core: - individual parameters managed through sqlite
Goffi <goffi@goffi.org>
parents: 412
diff changeset
204 for param in result:
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
205 category, name, value = param
423
6c20c76abdcc backend: - bridge async D-Bus method now automatically manage callback and errback, we just have to return a deferred
Goffi <goffi@goffi.org>
parents: 420
diff changeset
206 params_ind[(category, name)] = value
993
301b342c697a core: use of the new core.log module:
Goffi <goffi@goffi.org>
parents: 936
diff changeset
207 log.debug(_("loading individual parameters from database"))
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
208 d = self.dbpool.runQuery("SELECT category,name,value FROM param_ind WHERE profile_id=?", (self.profiles[profile], ))
412
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
209 d.addCallback(fillParams)
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
210 return d
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
211
413
dd4caab17008 core: - individual parameters managed through sqlite
Goffi <goffi@goffi.org>
parents: 412
diff changeset
212 def getIndParam(self, category, name, profile):
dd4caab17008 core: - individual parameters managed through sqlite
Goffi <goffi@goffi.org>
parents: 412
diff changeset
213 """Ask database for the value of one specific individual parameter
dd4caab17008 core: - individual parameters managed through sqlite
Goffi <goffi@goffi.org>
parents: 412
diff changeset
214 @param category: category of the parameter
dd4caab17008 core: - individual parameters managed through sqlite
Goffi <goffi@goffi.org>
parents: 412
diff changeset
215 @param name: name of the parameter
dd4caab17008 core: - individual parameters managed through sqlite
Goffi <goffi@goffi.org>
parents: 412
diff changeset
216 @param profile: %(doc_profile)s
dd4caab17008 core: - individual parameters managed through sqlite
Goffi <goffi@goffi.org>
parents: 412
diff changeset
217 @return: deferred"""
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
218 d = self.dbpool.runQuery("SELECT value FROM param_ind WHERE category=? AND name=? AND profile_id=?", (category, name, self.profiles[profile]))
413
dd4caab17008 core: - individual parameters managed through sqlite
Goffi <goffi@goffi.org>
parents: 412
diff changeset
219 d.addCallback(self.__getFirstResult)
dd4caab17008 core: - individual parameters managed through sqlite
Goffi <goffi@goffi.org>
parents: 412
diff changeset
220 return d
dd4caab17008 core: - individual parameters managed through sqlite
Goffi <goffi@goffi.org>
parents: 412
diff changeset
221
412
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
222 def setGenParam(self, category, name, value):
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
223 """Save the general parameters in database
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
224 @param category: category of the parameter
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
225 @param name: name of the parameter
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
226 @param value: value to set
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
227 @return: deferred"""
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
228 d = self.dbpool.runQuery("REPLACE INTO param_gen(category,name,value) VALUES (?,?,?)", (category, name, value))
993
301b342c697a core: use of the new core.log module:
Goffi <goffi@goffi.org>
parents: 936
diff changeset
229 d.addErrback(lambda ignore: log.error(_("Can't set general parameter (%(category)s/%(name)s) in database" % {"category": category, "name": name})))
412
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
230 return d
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
231
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
232 def setIndParam(self, category, name, value, profile):
432
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
233 """Save the individual parameters in database
412
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
234 @param category: category of the parameter
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
235 @param name: name of the parameter
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
236 @param value: value to set
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
237 @param profile: a profile which *must* exist
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
238 @return: deferred"""
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
239 d = self.dbpool.runQuery("REPLACE INTO param_ind(category,name,profile_id,value) VALUES (?,?,?,?)", (category, name, self.profiles[profile], value))
993
301b342c697a core: use of the new core.log module:
Goffi <goffi@goffi.org>
parents: 936
diff changeset
240 d.addErrback(lambda ignore: log.error(_("Can't set individual parameter (%(category)s/%(name)s) for [%(profile)s] in database" % {"category": category, "name": name, "profile": profile})))
412
62b17854254e database integration: first draft
Goffi <goffi@goffi.org>
parents:
diff changeset
241 return d
413
dd4caab17008 core: - individual parameters managed through sqlite
Goffi <goffi@goffi.org>
parents: 412
diff changeset
242
425
e4e9187e3b5b backend, bridge: asynchronous history
Goffi <goffi@goffi.org>
parents: 423
diff changeset
243 #History
669
ffb716804580 core, bridge: extra parameter is saved in history:
Goffi <goffi@goffi.org>
parents: 624
diff changeset
244 def addToHistory(self, from_jid, to_jid, message, _type='chat', extra=None, timestamp=None, profile=None):
425
e4e9187e3b5b backend, bridge: asynchronous history
Goffi <goffi@goffi.org>
parents: 423
diff changeset
245 """Store a new message in history
e4e9187e3b5b backend, bridge: asynchronous history
Goffi <goffi@goffi.org>
parents: 423
diff changeset
246 @param from_jid: full source JID
e4e9187e3b5b backend, bridge: asynchronous history
Goffi <goffi@goffi.org>
parents: 423
diff changeset
247 @param to_jid: full dest JID
e4e9187e3b5b backend, bridge: asynchronous history
Goffi <goffi@goffi.org>
parents: 423
diff changeset
248 @param message: message
512
862c0d6ab974 core, bridge, quick_frontend: MUC private messages history management:
Goffi <goffi@goffi.org>
parents: 480
diff changeset
249 @param _type: message type (see RFC 6121 §5.2.2)
669
ffb716804580 core, bridge: extra parameter is saved in history:
Goffi <goffi@goffi.org>
parents: 624
diff changeset
250 @param extra: dictionary (keys and values are unicode) of extra message data
425
e4e9187e3b5b backend, bridge: asynchronous history
Goffi <goffi@goffi.org>
parents: 423
diff changeset
251 @param timestamp: timestamp in seconds since epoch, or None to use current time
e4e9187e3b5b backend, bridge: asynchronous history
Goffi <goffi@goffi.org>
parents: 423
diff changeset
252 """
538
2c4016921403 core, frontends, bridgen plugins: fixed methods which were unproperly managing multi-profiles
Goffi <goffi@goffi.org>
parents: 512
diff changeset
253 assert(profile)
669
ffb716804580 core, bridge: extra parameter is saved in history:
Goffi <goffi@goffi.org>
parents: 624
diff changeset
254 if extra is None:
ffb716804580 core, bridge: extra parameter is saved in history:
Goffi <goffi@goffi.org>
parents: 624
diff changeset
255 extra = {}
839
f8681a7fd834 memory (sqlite): the result of pickle.dumps in addToHistory must be encoded for the special characters to be stored.
souliane <souliane@mailoo.org>
parents: 811
diff changeset
256 extra_ = pickle.dumps({k: v.encode('utf-8') for k, v in extra.items()}, 0).decode('utf-8')
669
ffb716804580 core, bridge: extra parameter is saved in history:
Goffi <goffi@goffi.org>
parents: 624
diff changeset
257 d = self.dbpool.runQuery("INSERT INTO history(source, source_res, dest, dest_res, timestamp, message, type, extra, profile_id) VALUES (?,?,?,?,?,?,?,?,?)",
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
258 (from_jid.userhost(), from_jid.resource, to_jid.userhost(), to_jid.resource, timestamp or time(),
839
f8681a7fd834 memory (sqlite): the result of pickle.dumps in addToHistory must be encoded for the special characters to be stored.
souliane <souliane@mailoo.org>
parents: 811
diff changeset
259 message, _type, extra_, self.profiles[profile]))
993
301b342c697a core: use of the new core.log module:
Goffi <goffi@goffi.org>
parents: 936
diff changeset
260 d.addErrback(lambda ignore: log.error(_("Can't save following message in history: from [%(from_jid)s] to [%(to_jid)s] ==> [%(message)s]" %
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
261 {"from_jid": from_jid.full(), "to_jid": to_jid.full(), "message": message})))
425
e4e9187e3b5b backend, bridge: asynchronous history
Goffi <goffi@goffi.org>
parents: 423
diff changeset
262 return d
e4e9187e3b5b backend, bridge: asynchronous history
Goffi <goffi@goffi.org>
parents: 423
diff changeset
263
538
2c4016921403 core, frontends, bridgen plugins: fixed methods which were unproperly managing multi-profiles
Goffi <goffi@goffi.org>
parents: 512
diff changeset
264 def getHistory(self, from_jid, to_jid, limit=0, between=True, profile=None):
425
e4e9187e3b5b backend, bridge: asynchronous history
Goffi <goffi@goffi.org>
parents: 423
diff changeset
265 """Store a new message in history
e4e9187e3b5b backend, bridge: asynchronous history
Goffi <goffi@goffi.org>
parents: 423
diff changeset
266 @param from_jid: source JID (full, or bare for catchall
e4e9187e3b5b backend, bridge: asynchronous history
Goffi <goffi@goffi.org>
parents: 423
diff changeset
267 @param to_jid: dest JID (full, or bare for catchall
e4e9187e3b5b backend, bridge: asynchronous history
Goffi <goffi@goffi.org>
parents: 423
diff changeset
268 @param size: maximum number of messages to get, or 0 for unlimited
e4e9187e3b5b backend, bridge: asynchronous history
Goffi <goffi@goffi.org>
parents: 423
diff changeset
269 """
538
2c4016921403 core, frontends, bridgen plugins: fixed methods which were unproperly managing multi-profiles
Goffi <goffi@goffi.org>
parents: 512
diff changeset
270 assert(profile)
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
271
669
ffb716804580 core, bridge: extra parameter is saved in history:
Goffi <goffi@goffi.org>
parents: 624
diff changeset
272 def sqliteToList(query_result):
448
17c7e48bf68f core: - history management improved
Goffi <goffi@goffi.org>
parents: 441
diff changeset
273 query_result.reverse()
17c7e48bf68f core: - history management improved
Goffi <goffi@goffi.org>
parents: 441
diff changeset
274 result = []
17c7e48bf68f core: - history management improved
Goffi <goffi@goffi.org>
parents: 441
diff changeset
275 for row in query_result:
669
ffb716804580 core, bridge: extra parameter is saved in history:
Goffi <goffi@goffi.org>
parents: 624
diff changeset
276 timestamp, source, source_res, dest, dest_res, message, type_, extra_raw = row
ffb716804580 core, bridge: extra parameter is saved in history:
Goffi <goffi@goffi.org>
parents: 624
diff changeset
277 try:
ffb716804580 core, bridge: extra parameter is saved in history:
Goffi <goffi@goffi.org>
parents: 624
diff changeset
278 extra = pickle.loads(str(extra_raw or ""))
ffb716804580 core, bridge: extra parameter is saved in history:
Goffi <goffi@goffi.org>
parents: 624
diff changeset
279 except EOFError:
ffb716804580 core, bridge: extra parameter is saved in history:
Goffi <goffi@goffi.org>
parents: 624
diff changeset
280 extra = {}
448
17c7e48bf68f core: - history management improved
Goffi <goffi@goffi.org>
parents: 441
diff changeset
281 result.append((timestamp, "%s/%s" % (source, source_res) if source_res else source,
425
e4e9187e3b5b backend, bridge: asynchronous history
Goffi <goffi@goffi.org>
parents: 423
diff changeset
282 "%s/%s" % (dest, dest_res) if dest_res else dest,
669
ffb716804580 core, bridge: extra parameter is saved in history:
Goffi <goffi@goffi.org>
parents: 624
diff changeset
283 message, type_, extra))
448
17c7e48bf68f core: - history management improved
Goffi <goffi@goffi.org>
parents: 441
diff changeset
284 return result
512
862c0d6ab974 core, bridge, quick_frontend: MUC private messages history management:
Goffi <goffi@goffi.org>
parents: 480
diff changeset
285
669
ffb716804580 core, bridge: extra parameter is saved in history:
Goffi <goffi@goffi.org>
parents: 624
diff changeset
286 query_parts = ["SELECT timestamp, source, source_res, dest, dest_res, message, type, extra FROM history WHERE profile_id=? AND"]
538
2c4016921403 core, frontends, bridgen plugins: fixed methods which were unproperly managing multi-profiles
Goffi <goffi@goffi.org>
parents: 512
diff changeset
287 values = [self.profiles[profile]]
425
e4e9187e3b5b backend, bridge: asynchronous history
Goffi <goffi@goffi.org>
parents: 423
diff changeset
288
669
ffb716804580 core, bridge: extra parameter is saved in history:
Goffi <goffi@goffi.org>
parents: 624
diff changeset
289 def test_jid(type_, _jid):
512
862c0d6ab974 core, bridge, quick_frontend: MUC private messages history management:
Goffi <goffi@goffi.org>
parents: 480
diff changeset
290 values.append(_jid.userhost())
862c0d6ab974 core, bridge, quick_frontend: MUC private messages history management:
Goffi <goffi@goffi.org>
parents: 480
diff changeset
291 if _jid.resource:
862c0d6ab974 core, bridge, quick_frontend: MUC private messages history management:
Goffi <goffi@goffi.org>
parents: 480
diff changeset
292 values.append(_jid.resource)
669
ffb716804580 core, bridge: extra parameter is saved in history:
Goffi <goffi@goffi.org>
parents: 624
diff changeset
293 return '(%s=? AND %s_res=?)' % (type_, type_)
ffb716804580 core, bridge: extra parameter is saved in history:
Goffi <goffi@goffi.org>
parents: 624
diff changeset
294 return '%s=?' % (type_, )
587
952322b1d490 Remove trailing whitespaces.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 572
diff changeset
295
425
e4e9187e3b5b backend, bridge: asynchronous history
Goffi <goffi@goffi.org>
parents: 423
diff changeset
296 if between:
681
66633443bcd7 memory: fixed getHistory behavior when parameter between=True:
souliane <souliane@mailoo.org>
parents: 669
diff changeset
297 query_parts.append("((%s AND %s) OR (%s AND %s))" % (test_jid('source', from_jid),
66633443bcd7 memory: fixed getHistory behavior when parameter between=True:
souliane <souliane@mailoo.org>
parents: 669
diff changeset
298 test_jid('dest', to_jid),
66633443bcd7 memory: fixed getHistory behavior when parameter between=True:
souliane <souliane@mailoo.org>
parents: 669
diff changeset
299 test_jid('source', to_jid),
66633443bcd7 memory: fixed getHistory behavior when parameter between=True:
souliane <souliane@mailoo.org>
parents: 669
diff changeset
300 test_jid('dest', from_jid)))
425
e4e9187e3b5b backend, bridge: asynchronous history
Goffi <goffi@goffi.org>
parents: 423
diff changeset
301 else:
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
302 query_parts.append("%s AND %s" % (test_jid('source', from_jid),
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
303 test_jid('dest', to_jid)))
448
17c7e48bf68f core: - history management improved
Goffi <goffi@goffi.org>
parents: 441
diff changeset
304
17c7e48bf68f core: - history management improved
Goffi <goffi@goffi.org>
parents: 441
diff changeset
305 query_parts.append("ORDER BY timestamp DESC")
17c7e48bf68f core: - history management improved
Goffi <goffi@goffi.org>
parents: 441
diff changeset
306
425
e4e9187e3b5b backend, bridge: asynchronous history
Goffi <goffi@goffi.org>
parents: 423
diff changeset
307 if limit:
e4e9187e3b5b backend, bridge: asynchronous history
Goffi <goffi@goffi.org>
parents: 423
diff changeset
308 query_parts.append("LIMIT ?")
e4e9187e3b5b backend, bridge: asynchronous history
Goffi <goffi@goffi.org>
parents: 423
diff changeset
309 values.append(limit)
587
952322b1d490 Remove trailing whitespaces.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 572
diff changeset
310
425
e4e9187e3b5b backend, bridge: asynchronous history
Goffi <goffi@goffi.org>
parents: 423
diff changeset
311 d = self.dbpool.runQuery(" ".join(query_parts), values)
669
ffb716804580 core, bridge: extra parameter is saved in history:
Goffi <goffi@goffi.org>
parents: 624
diff changeset
312 return d.addCallback(sqliteToList)
425
e4e9187e3b5b backend, bridge: asynchronous history
Goffi <goffi@goffi.org>
parents: 423
diff changeset
313
432
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
314 #Private values
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
315 def loadGenPrivates(self, private_gen, namespace):
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
316 """Load general private values
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
317 @param private_gen: dictionary to fill
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
318 @param namespace: namespace of the values
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
319 @return: deferred"""
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
320
432
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
321 def fillPrivates(result):
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
322 for private in result:
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
323 key, value = private
432
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
324 private_gen[key] = value
993
301b342c697a core: use of the new core.log module:
Goffi <goffi@goffi.org>
parents: 936
diff changeset
325 log.debug(_("loading general private values [namespace: %s] from database") % (namespace, ))
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
326 d = self.dbpool.runQuery("SELECT key,value FROM private_gen WHERE namespace=?", (namespace, )).addCallback(fillPrivates)
993
301b342c697a core: use of the new core.log module:
Goffi <goffi@goffi.org>
parents: 936
diff changeset
327 return d.addErrback(lambda x: log.debug(_("No data present in database for namespace %s") % namespace))
425
e4e9187e3b5b backend, bridge: asynchronous history
Goffi <goffi@goffi.org>
parents: 423
diff changeset
328
432
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
329 def loadIndPrivates(self, private_ind, namespace, profile):
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
330 """Load individual private values
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
331 @param privates_ind: dictionary to fill
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
332 @param namespace: namespace of the values
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
333 @param profile: a profile which *must* exist
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
334 @return: deferred"""
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
335
432
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
336 def fillPrivates(result):
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
337 for private in result:
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
338 key, value = private
432
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
339 private_ind[key] = value
993
301b342c697a core: use of the new core.log module:
Goffi <goffi@goffi.org>
parents: 936
diff changeset
340 log.debug(_("loading individual private values [namespace: %s] from database") % (namespace, ))
432
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
341 d = self.dbpool.runQuery("SELECT key,value FROM private_ind WHERE namespace=? AND profile_id=?", (namespace, self.profiles[profile]))
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
342 d.addCallback(fillPrivates)
993
301b342c697a core: use of the new core.log module:
Goffi <goffi@goffi.org>
parents: 936
diff changeset
343 return d.addErrback(lambda x: log.debug(_("No data present in database for namespace %s") % namespace))
432
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
344
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
345 def setGenPrivate(self, namespace, key, value):
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
346 """Save the general private value in database
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
347 @param category: category of the privateeter
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
348 @param key: key of the private value
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
349 @param value: value to set
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
350 @return: deferred"""
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
351 d = self.dbpool.runQuery("REPLACE INTO private_gen(namespace,key,value) VALUES (?,?,?)", (namespace, key, value))
993
301b342c697a core: use of the new core.log module:
Goffi <goffi@goffi.org>
parents: 936
diff changeset
352 d.addErrback(lambda ignore: log.error(_("Can't set general private value (%(key)s) [namespace:%(namespace)s] in database" %
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
353 {"namespace": namespace, "key": key})))
432
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
354 return d
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
355
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
356 def setIndPrivate(self, namespace, key, value, profile):
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
357 """Save the individual private value in database
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
358 @param namespace: namespace of the value
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
359 @param key: key of the private value
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
360 @param value: value to set
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
361 @param profile: a profile which *must* exist
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
362 @return: deferred"""
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
363 d = self.dbpool.runQuery("REPLACE INTO private_ind(namespace,key,profile_id,value) VALUES (?,?,?,?)", (namespace, key, self.profiles[profile], value))
993
301b342c697a core: use of the new core.log module:
Goffi <goffi@goffi.org>
parents: 936
diff changeset
364 d.addErrback(lambda ignore: log.error(_("Can't set individual private value (%(key)s) [namespace: %(namespace)s] for [%(profile)s] in database" %
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
365 {"namespace": namespace, "key": key, "profile": profile})))
432
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
366 return d
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
367
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
368 def delGenPrivate(self, namespace, key):
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
369 """Delete the general private value from database
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
370 @param category: category of the privateeter
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
371 @param key: key of the private value
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
372 @return: deferred"""
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
373 d = self.dbpool.runQuery("DELETE FROM private_gen WHERE namespace=? AND key=?", (namespace, key))
993
301b342c697a core: use of the new core.log module:
Goffi <goffi@goffi.org>
parents: 936
diff changeset
374 d.addErrback(lambda ignore: log.error(_("Can't delete general private value (%(key)s) [namespace:%(namespace)s] in database" %
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
375 {"namespace": namespace, "key": key})))
432
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
376 return d
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
377
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
378 def delIndPrivate(self, namespace, key, profile):
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
379 """Delete the individual private value from database
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
380 @param namespace: namespace of the value
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
381 @param key: key of the private value
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
382 @param profile: a profile which *must* exist
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
383 @return: deferred"""
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
384 d = self.dbpool.runQuery("DELETE FROM private_ind WHERE namespace=? AND key=? AND profile=?)", (namespace, key, self.profiles[profile]))
993
301b342c697a core: use of the new core.log module:
Goffi <goffi@goffi.org>
parents: 936
diff changeset
385 d.addErrback(lambda ignore: log.error(_("Can't delete individual private value (%(key)s) [namespace: %(namespace)s] for [%(profile)s] in database" %
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
386 {"namespace": namespace, "key": key, "profile": profile})))
432
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
387 return d
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
388
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
389 def loadGenPrivatesBinary(self, private_gen, namespace):
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
390 """Load general private binary values
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
391 @param private_gen: dictionary to fill
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
392 @param namespace: namespace of the values
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
393 @return: deferred"""
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
394
432
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
395 def fillPrivates(result):
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
396 for private in result:
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
397 key, value = private
436
5e9d28ca5109 core: sqlite persistentBinaryDict storage fix
Goffi <goffi@goffi.org>
parents: 432
diff changeset
398 private_gen[key] = pickle.loads(str(value))
993
301b342c697a core: use of the new core.log module:
Goffi <goffi@goffi.org>
parents: 936
diff changeset
399 log.debug(_("loading general private binary values [namespace: %s] from database") % (namespace, ))
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
400 d = self.dbpool.runQuery("SELECT key,value FROM private_gen_bin WHERE namespace=?", (namespace, )).addCallback(fillPrivates)
993
301b342c697a core: use of the new core.log module:
Goffi <goffi@goffi.org>
parents: 936
diff changeset
401 return d.addErrback(lambda x: log.debug(_("No binary data present in database for namespace %s") % namespace))
432
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
402
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
403 def loadIndPrivatesBinary(self, private_ind, namespace, profile):
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
404 """Load individual private binary values
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
405 @param privates_ind: dictionary to fill
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
406 @param namespace: namespace of the values
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
407 @param profile: a profile which *must* exist
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
408 @return: deferred"""
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
409
432
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
410 def fillPrivates(result):
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
411 for private in result:
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
412 key, value = private
436
5e9d28ca5109 core: sqlite persistentBinaryDict storage fix
Goffi <goffi@goffi.org>
parents: 432
diff changeset
413 private_ind[key] = pickle.loads(str(value))
993
301b342c697a core: use of the new core.log module:
Goffi <goffi@goffi.org>
parents: 936
diff changeset
414 log.debug(_("loading individual private binary values [namespace: %s] from database") % (namespace, ))
432
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
415 d = self.dbpool.runQuery("SELECT key,value FROM private_ind_bin WHERE namespace=? AND profile_id=?", (namespace, self.profiles[profile]))
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
416 d.addCallback(fillPrivates)
993
301b342c697a core: use of the new core.log module:
Goffi <goffi@goffi.org>
parents: 936
diff changeset
417 return d.addErrback(lambda x: log.debug(_("No binary data present in database for namespace %s") % namespace))
432
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
418
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
419 def setGenPrivateBinary(self, namespace, key, value):
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
420 """Save the general private binary value in database
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
421 @param category: category of the privateeter
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
422 @param key: key of the private value
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
423 @param value: value to set
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
424 @return: deferred"""
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
425 d = self.dbpool.runQuery("REPLACE INTO private_gen_bin(namespace,key,value) VALUES (?,?,?)", (namespace, key, pickle.dumps(value, 0)))
993
301b342c697a core: use of the new core.log module:
Goffi <goffi@goffi.org>
parents: 936
diff changeset
426 d.addErrback(lambda ignore: log.error(_("Can't set general private binary value (%(key)s) [namespace:%(namespace)s] in database" %
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
427 {"namespace": namespace, "key": key})))
432
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
428 return d
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
429
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
430 def setIndPrivateBinary(self, namespace, key, value, profile):
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
431 """Save the individual private binary value in database
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
432 @param namespace: namespace of the value
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
433 @param key: key of the private value
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
434 @param value: value to set
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
435 @param profile: a profile which *must* exist
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
436 @return: deferred"""
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
437 d = self.dbpool.runQuery("REPLACE INTO private_ind_bin(namespace,key,profile_id,value) VALUES (?,?,?,?)", (namespace, key, self.profiles[profile], pickle.dumps(value, 0)))
993
301b342c697a core: use of the new core.log module:
Goffi <goffi@goffi.org>
parents: 936
diff changeset
438 d.addErrback(lambda ignore: log.error(_("Can't set individual binary private value (%(key)s) [namespace: %(namespace)s] for [%(profile)s] in database" %
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
439 {"namespace": namespace, "key": key, "profile": profile})))
432
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
440 return d
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
441
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
442 def delGenPrivateBinary(self, namespace, key):
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
443 """Delete the general private binary value from database
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
444 @param category: category of the privateeter
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
445 @param key: key of the private value
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
446 @return: deferred"""
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
447 d = self.dbpool.runQuery("DELETE FROM private_gen_bin WHERE namespace=? AND key=?", (namespace, key))
993
301b342c697a core: use of the new core.log module:
Goffi <goffi@goffi.org>
parents: 936
diff changeset
448 d.addErrback(lambda ignore: log.error(_("Can't delete general private binary value (%(key)s) [namespace:%(namespace)s] in database" %
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
449 {"namespace": namespace, "key": key})))
432
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
450 return d
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
451
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
452 def delIndPrivateBinary(self, namespace, key, profile):
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
453 """Delete the individual private binary value from database
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
454 @param namespace: namespace of the value
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
455 @param key: key of the private value
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
456 @param profile: a profile which *must* exist
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
457 @return: deferred"""
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
458 d = self.dbpool.runQuery("DELETE FROM private_ind_bin WHERE namespace=? AND key=? AND profile=?)", (namespace, key, self.profiles[profile]))
993
301b342c697a core: use of the new core.log module:
Goffi <goffi@goffi.org>
parents: 936
diff changeset
459 d.addErrback(lambda ignore: log.error(_("Can't delete individual private binary value (%(key)s) [namespace: %(namespace)s] for [%(profile)s] in database" %
592
e5a875a3311b Fix pep8 support in src/memory.
Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
parents: 588
diff changeset
460 {"namespace": namespace, "key": key, "profile": profile})))
432
31e8c48b5f5d core: - memory refactoring (moved memory.py and sqlite.py from tools to memory)
Goffi <goffi@goffi.org>
parents: 425
diff changeset
461 return d
413
dd4caab17008 core: - individual parameters managed through sqlite
Goffi <goffi@goffi.org>
parents: 412
diff changeset
462 ##Helper methods##
dd4caab17008 core: - individual parameters managed through sqlite
Goffi <goffi@goffi.org>
parents: 412
diff changeset
463
dd4caab17008 core: - individual parameters managed through sqlite
Goffi <goffi@goffi.org>
parents: 412
diff changeset
464 def __getFirstResult(self, result):
dd4caab17008 core: - individual parameters managed through sqlite
Goffi <goffi@goffi.org>
parents: 412
diff changeset
465 """Return the first result of a database query
dd4caab17008 core: - individual parameters managed through sqlite
Goffi <goffi@goffi.org>
parents: 412
diff changeset
466 Useful when we are looking for one specific value"""
416
32dc8b18c2ae core: param loading/purging on profile connection/disconnection
Goffi <goffi@goffi.org>
parents: 413
diff changeset
467 return None if not result else result[0][0]
853
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
468
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
469
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
470 class Updater(object):
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
471 stmnt_regex = re.compile(r"(?:[\w ]+(?:\([\w, ]+\))?)+")
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
472 clean_regex = re.compile(r"^ +|(?<= ) +|(?<=,) +| +$")
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
473 CREATE_SQL = "CREATE TABLE %s (%s)"
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
474 INSERT_SQL = "INSERT INTO %s VALUES (%s)"
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
475 DROP_SQL = "DROP TABLE %s"
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
476 ALTER_SQL = "ALTER TABLE %s ADD COLUMN %s"
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
477 RENAME_TABLE_SQL = "ALTER TABLE %s RENAME TO %s"
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
478
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
479 CONSTRAINTS = ('PRIMARY', 'UNIQUE', 'CHECK', 'FOREIGN')
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
480 TMP_TABLE = "tmp_sat_update"
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
481
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
482 def __init__(self, dbpool, sat_version):
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
483 self._sat_version = sat_version
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
484 self.dbpool = dbpool
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
485
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
486 def getLocalVersion(self):
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
487 """ Get local database version
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
488 @return: version (int)
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
489
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
490 """
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
491 return self.dbpool.runQuery("PRAGMA user_version").addCallback(lambda ret: int(ret[0][0]))
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
492
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
493 def _setLocalVersion(self, version):
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
494 """ Set local database version
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
495 @param version: version (int)
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
496 @return: deferred
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
497
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
498 """
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
499 return self.dbpool.runOperation("PRAGMA user_version=%d" % version)
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
500
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
501 def getLocalSchema(self):
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
502 """ return raw local schema
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
503 @return: list of strings with CREATE sql statements for local database
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
504
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
505 """
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
506 d = self.dbpool.runQuery("select sql from sqlite_master where type = 'table'")
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
507 d.addCallback(lambda result: [row[0] for row in result])
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
508 return d
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
509
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
510 @defer.inlineCallbacks
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
511 def checkUpdates(self):
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
512 """ Check is database schema update is needed, according to DATABASE_SCHEMAS
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
513 @return: deferred which fire a list of SQL update statements, or None if no update is needed
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
514
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
515 """
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
516 local_version = yield self.getLocalVersion()
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
517 raw_local_sch = yield self.getLocalSchema()
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
518 local_sch = self.rawStatements2data(raw_local_sch)
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
519 current_sch = DATABASE_SCHEMAS['current']['CREATE']
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
520 local_hash = self.statementHash(local_sch)
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
521 current_hash = self.statementHash(current_sch)
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
522
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
523 if local_hash == current_hash:
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
524 if local_version != CURRENT_DB_VERSION:
993
301b342c697a core: use of the new core.log module:
Goffi <goffi@goffi.org>
parents: 936
diff changeset
525 log.warning(_("Your local schema is up-to-date, but database versions mismatch, fixing it..."))
853
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
526 yield self._setLocalVersion(CURRENT_DB_VERSION)
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
527 else:
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
528 # an update is needed
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
529
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
530 if local_version == CURRENT_DB_VERSION:
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
531 # Database mismatch and we have the latest version
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
532 if self._sat_version.endswith('D'):
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
533 # we are in a development version
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
534 update_data = self.generateUpdateData(local_sch, current_sch, False)
993
301b342c697a core: use of the new core.log module:
Goffi <goffi@goffi.org>
parents: 936
diff changeset
535 log.warning(_("There is a schema mismatch, but as we are on a dev version, database will be updated"))
853
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
536 update_raw = self.update2raw(update_data, True)
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
537 defer.returnValue(update_raw)
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
538 else:
993
301b342c697a core: use of the new core.log module:
Goffi <goffi@goffi.org>
parents: 936
diff changeset
539 log.error(_(u"schema version is up-to-date, but local schema differ from expected current schema"))
853
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
540 update_data = self.generateUpdateData(local_sch, current_sch, True)
993
301b342c697a core: use of the new core.log module:
Goffi <goffi@goffi.org>
parents: 936
diff changeset
541 log.warning(_(u"Here are the commands that should fix the situation, use at your own risk (do a backup before modifying database), you can go to SàT's MUC room at sat@chat.jabberfr.org for help\n### SQL###\n%s\n### END SQL ###\n") % u'\n'.join(("%s;" % statement for statement in self.update2raw(update_data))))
853
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
542 raise exceptions.DatabaseError("Database mismatch")
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
543 else:
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
544 # Database is not up-to-date, we'll do the update
993
301b342c697a core: use of the new core.log module:
Goffi <goffi@goffi.org>
parents: 936
diff changeset
545 log.info(_("Database schema has changed, local database will be updated"))
853
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
546 update_raw = []
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
547 for version in xrange(local_version+1, CURRENT_DB_VERSION+1):
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
548 try:
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
549 update_data = DATABASE_SCHEMAS[version]
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
550 except KeyError:
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
551 raise exceptions.InternalError("Missing update definition (version %d)" % version)
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
552 update_raw.extend(self.update2raw(update_data))
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
553 update_raw.append("PRAGMA user_version=%d" % CURRENT_DB_VERSION)
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
554 defer.returnValue(update_raw)
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
555
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
556 @staticmethod
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
557 def createData2Raw(data):
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
558 """ Generate SQL statements from statements data
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
559 @param data: dictionary with table as key, and statements data in tuples as value
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
560 @return: list of strings with raw statements
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
561
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
562 """
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
563 ret = []
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
564 for table in data:
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
565 defs, constraints = data[table]
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
566 assert isinstance(defs, tuple)
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
567 assert isinstance(constraints, tuple)
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
568 ret.append(Updater.CREATE_SQL % (table, ', '.join(defs + constraints)))
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
569 return ret
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
570
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
571 @staticmethod
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
572 def insertData2Raw(data):
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
573 """ Generate SQL statements from statements data
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
574 @param data: dictionary with table as key, and statements data in tuples as value
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
575 @return: list of strings with raw statements
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
576
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
577 """
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
578 ret = []
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
579 for table in data:
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
580 values_tuple = data[table]
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
581 assert isinstance(values_tuple, tuple)
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
582 for values in values_tuple:
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
583 assert isinstance(values, tuple)
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
584 ret.append(Updater.INSERT_SQL % (table, ', '.join(values)))
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
585 return ret
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
586
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
587 def statementHash(self, data):
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
588 """ Generate hash of template data
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
589 useful to compare schemas
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
590
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
591 @param data: dictionary of "CREATE" statement, with tables names as key,
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
592 and tuples of (col_defs, constraints) as values
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
593 @return: hash as string
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
594 """
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
595 hash_ = hashlib.sha1()
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
596 tables = data.keys()
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
597 tables.sort()
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
598
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
599 def stmnts2str(stmts):
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
600 return ','.join([self.clean_regex.sub('',stmt) for stmt in sorted(stmts)])
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
601
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
602 for table in tables:
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
603 col_defs, col_constr = data[table]
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
604 hash_.update("%s:%s:%s" % (table, stmnts2str(col_defs), stmnts2str(col_constr)))
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
605 return hash_.digest()
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
606
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
607 def rawStatements2data(self, raw_statements):
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
608 """ separate "CREATE" statements into dictionary/tuples data
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
609 @param raw_statements: list of CREATE statements as strings
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
610 @return: dictionary with table names as key, and a (col_defs, constraints) tuple
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
611
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
612 """
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
613 schema_dict = {}
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
614 for create_statement in raw_statements:
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
615 if not create_statement.startswith("CREATE TABLE "):
993
301b342c697a core: use of the new core.log module:
Goffi <goffi@goffi.org>
parents: 936
diff changeset
616 log.warning("Unexpected statement, ignoring it")
853
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
617 continue
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
618 _create_statement = create_statement[13:]
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
619 table, raw_col_stats = _create_statement.split(' ',1)
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
620 if raw_col_stats[0] != '(' or raw_col_stats[-1] != ')':
993
301b342c697a core: use of the new core.log module:
Goffi <goffi@goffi.org>
parents: 936
diff changeset
621 log.warning("Unexpected statement structure, ignoring it")
853
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
622 continue
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
623 col_stats = [stmt.strip() for stmt in self.stmnt_regex.findall(raw_col_stats[1:-1])]
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
624 col_defs = []
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
625 constraints = []
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
626 for col_stat in col_stats:
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
627 name = col_stat.split(' ',1)[0]
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
628 if name in self.CONSTRAINTS:
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
629 constraints.append(col_stat)
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
630 else:
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
631 col_defs.append(col_stat)
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
632 schema_dict[table] = (tuple(col_defs), tuple(constraints))
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
633 return schema_dict
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
634
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
635 def generateUpdateData(self, old_data, new_data, modify=False):
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
636 """ Generate data for automatic update between two schema data
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
637 @param old_data: data of the former schema (which must be updated)
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
638 @param new_data: data of the current schema
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
639 @param modify: if True, always use "cols modify" table, else try to ALTER tables
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
640 @return: update data, a dictionary with:
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
641 - 'create': dictionary of tables to create
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
642 - 'delete': tuple of tables to delete
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
643 - 'cols create': dictionary of columns to create (table as key, tuple of columns to create as value)
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
644 - 'cols delete': dictionary of columns to delete (table as key, tuple of columns to delete as value)
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
645 - 'cols modify': dictionary of columns to modify (table as key, tuple of old columns to transfert as value). With this table, a new table will be created, and content from the old table will be transfered to it, only cols specified in the tuple will be transfered.
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
646
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
647 """
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
648
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
649 create_tables_data = {}
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
650 create_cols_data = {}
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
651 modify_cols_data = {}
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
652 delete_cols_data = {}
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
653 old_tables = set(old_data.keys())
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
654 new_tables = set(new_data.keys())
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
655
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
656 def getChanges(set_olds, set_news):
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
657 to_create = set_news.difference(set_olds)
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
658 to_delete = set_olds.difference(set_news)
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
659 to_check = set_news.intersection(set_olds)
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
660 return tuple(to_create), tuple(to_delete), tuple(to_check)
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
661
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
662 tables_to_create, tables_to_delete, tables_to_check = getChanges(old_tables, new_tables)
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
663
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
664 for table in tables_to_create:
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
665 create_tables_data[table] = new_data[table]
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
666
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
667 for table in tables_to_check:
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
668 old_col_defs, old_constraints = old_data[table]
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
669 new_col_defs, new_constraints = new_data[table]
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
670 for obj in old_col_defs, old_constraints, new_col_defs, new_constraints:
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
671 if not isinstance(obj, tuple):
993
301b342c697a core: use of the new core.log module:
Goffi <goffi@goffi.org>
parents: 936
diff changeset
672 raise exceptions.InternalError("Columns definitions must be tuples")
853
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
673 defs_create, defs_delete, ignore = getChanges(set(old_col_defs), set(new_col_defs))
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
674 constraints_create, constraints_delete, ignore = getChanges(set(old_constraints), set(new_constraints))
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
675 created_col_names = set([name.split(' ',1)[0] for name in defs_create])
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
676 deleted_col_names = set([name.split(' ',1)[0] for name in defs_delete])
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
677 if (created_col_names.intersection(deleted_col_names or constraints_create or constraints_delete) or
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
678 (modify and (defs_create or constraints_create or defs_delete or constraints_delete))):
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
679 # we have modified columns, we need to transfer table
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
680 # we determinate which columns are in both schema so we can transfer them
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
681 old_names = set([name.split(' ',1)[0] for name in old_col_defs])
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
682 new_names = set([name.split(' ',1)[0] for name in new_col_defs])
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
683 modify_cols_data[table] = tuple(old_names.intersection(new_names));
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
684 else:
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
685 if defs_create:
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
686 create_cols_data[table] = (defs_create)
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
687 if defs_delete or constraints_delete:
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
688 delete_cols_data[table] = (defs_delete)
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
689
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
690 return {'create': create_tables_data,
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
691 'delete': tables_to_delete,
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
692 'cols create': create_cols_data,
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
693 'cols delete': delete_cols_data,
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
694 'cols modify': modify_cols_data
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
695 }
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
696
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
697 def update2raw(self, update, dev_version=False):
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
698 """ Transform update data to raw SQLite statements
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
699 @param upadte: update data as returned by generateUpdateData
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
700 @param dev_version: if True, update will be done in dev mode: no deletion will be done, instead a message will be shown. This prevent accidental lost of data while working on the code/database.
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
701 @return: list of string with SQL statements needed to update the base
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
702
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
703 """
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
704 ret = self.createData2Raw(update.get('create', {}))
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
705 drop = []
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
706 for table in update.get('delete', tuple()):
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
707 drop.append(self.DROP_SQL % table)
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
708 if dev_version:
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
709 if drop:
993
301b342c697a core: use of the new core.log module:
Goffi <goffi@goffi.org>
parents: 936
diff changeset
710 log.info("Dev version, SQL NOT EXECUTED:\n--\n%s\n--\n" % "\n".join(drop))
853
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
711 else:
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
712 ret.extend(drop)
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
713
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
714 cols_create = update.get('cols create', {})
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
715 for table in cols_create:
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
716 for col_def in cols_create[table]:
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
717 ret.append(self.ALTER_SQL % (table, col_def))
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
718
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
719 cols_delete = update.get('cols delete', {})
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
720 for table in cols_delete:
993
301b342c697a core: use of the new core.log module:
Goffi <goffi@goffi.org>
parents: 936
diff changeset
721 log.info("Following columns in table [%s] are not needed anymore, but are kept for dev version: %s" % (table, ", ".join(cols_delete[table])))
853
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
722
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
723 cols_modify = update.get('cols modify', {})
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
724 for table in cols_modify:
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
725 ret.append(self.RENAME_TABLE_SQL % (table, self.TMP_TABLE))
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
726 main, extra = DATABASE_SCHEMAS['current']['CREATE'][table]
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
727 ret.append(self.CREATE_SQL % (table, ', '.join(main + extra)))
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
728 common_cols = ', '.join(cols_modify[table])
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
729 ret.append("INSERT INTO %s (%s) SELECT %s FROM %s" % (table, common_cols, common_cols, self.TMP_TABLE))
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
730 ret.append(self.DROP_SQL % self.TMP_TABLE)
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
731
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
732 insert = update.get('insert', {})
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
733 ret.extend(self.insertData2Raw(insert))
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
734
c2f6ada7858f core (sqlite): automatic database update:
Goffi <goffi@goffi.org>
parents: 839
diff changeset
735 return ret