Mercurial > prosody-modules

annotate mod_storage_appendmap/mod_storage_appendmap.lua @ 4542:fb4a50bf60f1

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
mod_prometheus: Invoke stats collection if in 'manual' mode Since 10d13e0554f9 a special value for statistics_interval "manual" exists, where a module is expected to invoke processing in connection to collection of stats. This makes internal collection and exporting to Prometheus happens at the same time with no chance of timers getting out of sync.
author Kim Alvefur <zash@zash.se>
date Tue, 13 Apr 2021 23:53:53 +0200
parents 6a7b7cb7148e
children e44b868cc575
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
2061
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
1 local dump = require "util.serialization".serialize;
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
2 local load = require "util.envload".envloadfile;
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
3 local dm = require "core.storagemanager".olddm;
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
4
2796
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
5 local REMOVE = {}; -- Special value for removing keys
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
6
2061
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
7 local driver = {};
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
8
2796
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
9
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
10 local keywords = {
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
11 ["do"] = true; ["and"] = true; ["else"] = true; ["break"] = true;
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
12 ["if"] = true; ["end"] = true; ["goto"] = true; ["false"] = true;
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
13 ["in"] = true; ["for"] = true; ["then"] = true; ["local"] = true;
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
14 ["or"] = true; ["nil"] = true; ["true"] = true; ["until"] = true;
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
15 ["elseif"] = true; ["function"] = true; ["not"] = true;
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
16 ["repeat"] = true; ["return"] = true; ["while"] = true;
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
17
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
18 -- _ENV is not technically a keyword but we need to treat it as such
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
19 ["_ENV"] = true;
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
20 };
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
21
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
22 local function is_usable_identifier(s)
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
23 return type(s) == "string" and not keywords[s] and s:find("^[%a_][%w_]*$");
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
24 end
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
25
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
26 local function serialize_key(key)
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
27 if is_usable_identifier(key) then
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
28 return key;
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
29 else
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
30 return "_ENV[" .. dump(key) .. "]";
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
31 end
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
32 end
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
33
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
34 local function serialize_value(value)
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
35 if value == REMOVE then
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
36 return "nil";
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
37 else
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
38 return dump(value);
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
39 end
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
40 end
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
41
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
42 local function serialize_pair(key, value)
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
43 key = serialize_key(key);
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
44 value = serialize_value(value);
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
45 return key .. " = " .. value .. ";\n";
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
46 end
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
47
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
48 local function serialize_map(keyvalues)
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
49 local keys, values = {}, {};
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
50 for key, value in pairs(keyvalues) do
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
51 key = serialize_key(key);
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
52 value = serialize_value(value);
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
53 table.insert(keys, key);
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
54 table.insert(values, value);
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
55 end
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
56 return table.concat(keys, ", ") .. " = " .. table.concat(values, ", ") .. ";\n";
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
57 end
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
58
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
59 local map = { remove = REMOVE };
2061
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
60 local map_mt = { __index = map };
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
61
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
62 function map:get(user, key)
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
63 module:log("debug", "map:get(%s, %s)", tostring(user), tostring(key))
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
64 local filename = dm.getpath(user, module.host, self.store, "map");
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
65 module:log("debug", "File is %s", filename);
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
66 local env = {};
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
67 if _VERSION == "Lua 5.1" then -- HACK
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
68 env._ENV = env; -- HACK
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
69 end -- SO MANY HACKS
2635
4548c3d685b4 mod_storage_appendmap: Successfully return nothing on ENOENT
Kim Alvefur <zash@zash.se>
parents: 2431
diff changeset
70 local chunk, err, errno = load(filename, env);
4548c3d685b4 mod_storage_appendmap: Successfully return nothing on ENOENT
Kim Alvefur <zash@zash.se>
parents: 2431
diff changeset
71 if not chunk then if errno == 2 then return end return chunk, err; end
2061
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
72 local ok, err = pcall(chunk);
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
73 if not ok then return ok, err; end
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
74 if _VERSION == "Lua 5.1" then -- HACK
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
75 env._ENV = nil; -- HACK
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
76 end -- HACKS EVERYWHERE
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
77 if key == nil then
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
78 return env;
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
79 end
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
80 return env[key];
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
81 end
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
82
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
83 function map:set_keys(user, keyvalues)
2796
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
84 local data = serialize_map(keyvalues);
2061
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
85 return dm.append_raw(user, module.host, self.store, "map", data);
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
86 end
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
87
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
88 function map:set(user, key, value)
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
89 if _VERSION == "Lua 5.1" then
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
90 assert(key ~= "_ENV", "'_ENV' is a restricted key");
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
91 end
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
92 if key == nil then
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
93 local filename = dm.getpath(user, module.host, self.store, "map");
2796
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
94 return os.remove(filename);
2061
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
95 end
2796
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
96 local data = serialize_pair(key, value);
2061
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
97 return dm.append_raw(user, module.host, self.store, "map", data);
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
98 end
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
99
2796
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
100 local keyval = { remove = REMOVE };
2061
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
101 local keyval_mt = { __index = keyval };
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
102
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
103 function keyval:get(user)
2796
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
104 return map.get(self, user, nil);
2061
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
105 end
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
106
2796
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
107 function keyval:set(user, keyvalues)
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
108 local data = serialize_map(keyvalues);
6a7b7cb7148e mod_storage_appendmap: Factor out serialization into reusable functions
Kim Alvefur <zash@zash.se>
parents: 2635
diff changeset
109 return dm.store_raw(user, module.host, self.store, "map", data);
2062
8f7083b980cf mod_storage_appendmap: Fix keyvalue:set()
Kim Alvefur <zash@zash.se>
parents: 2061
diff changeset
110 end
2061
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
111
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
112 -- TODO some kind of periodic compaction thing?
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
113 function map:_compact(user)
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
114 local data = self:get(user);
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
115 return keyval.set(self, user, data);
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
116 end
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
117
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
118 function driver:open(store, typ)
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
119 if typ == "map" then
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
120 return setmetatable({ store = store, }, map_mt);
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
121 elseif typ == nil or typ == "keyval" then
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
122 return setmetatable({ store = store, }, keyval_mt);
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
123 end
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
124 return nil, "unsupported-store";
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
125 end
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
126
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
127 module:provides("storage", driver);
b84284144e21 mod_storage_appendmap: Experimental storage module optimized for map stores
Kim Alvefur <zash@zash.se>
parents:
diff changeset
128