Mercurial > prosody-modules

annotate mod_migrate/mod_migrate.lua @ 3772:22f02716819f

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
mod_s2s_keepalive: Isolate source host of pings The incoming_s2s table is not restricted to the current virtualhost so this prevents opening more connections than what's needed. Also prevents useless double sending of one whitespace per local host.
author Kim Alvefur <zash@zash.se>
date Mon, 23 Dec 2019 01:18:02 +0100
parents 611ac62c5b63
children c9e1eee6a948
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1791
8df071457dee mod_migrate: Provides a prosodyctl mod_migrate command for copying data between storage backends
Kim Alvefur <zash@zash.se>
parents:
diff changeset
1 -- mod_migrate
8df071457dee mod_migrate: Provides a prosodyctl mod_migrate command for copying data between storage backends
Kim Alvefur <zash@zash.se>
parents:
diff changeset
2
3584
611ac62c5b63 mod_migrate: Try to use per storage user listing where available
Kim Alvefur <zash@zash.se>
parents: 2816
diff changeset
3 local unpack = table.unpack or unpack;
1791
8df071457dee mod_migrate: Provides a prosodyctl mod_migrate command for copying data between storage backends
Kim Alvefur <zash@zash.se>
parents:
diff changeset
4 local sm = require"core.storagemanager";
8df071457dee mod_migrate: Provides a prosodyctl mod_migrate command for copying data between storage backends
Kim Alvefur <zash@zash.se>
parents:
diff changeset
5 local um = require"core.usermanager";
8df071457dee mod_migrate: Provides a prosodyctl mod_migrate command for copying data between storage backends
Kim Alvefur <zash@zash.se>
parents:
diff changeset
6
3584
611ac62c5b63 mod_migrate: Try to use per storage user listing where available
Kim Alvefur <zash@zash.se>
parents: 2816
diff changeset
7 local function users(store, host)
611ac62c5b63 mod_migrate: Try to use per storage user listing where available
Kim Alvefur <zash@zash.se>
parents: 2816
diff changeset
8 if store.users then
611ac62c5b63 mod_migrate: Try to use per storage user listing where available
Kim Alvefur <zash@zash.se>
parents: 2816
diff changeset
9 return store:users();
611ac62c5b63 mod_migrate: Try to use per storage user listing where available
Kim Alvefur <zash@zash.se>
parents: 2816
diff changeset
10 else
611ac62c5b63 mod_migrate: Try to use per storage user listing where available
Kim Alvefur <zash@zash.se>
parents: 2816
diff changeset
11 return um.users(host);
611ac62c5b63 mod_migrate: Try to use per storage user listing where available
Kim Alvefur <zash@zash.se>
parents: 2816
diff changeset
12 end
611ac62c5b63 mod_migrate: Try to use per storage user listing where available
Kim Alvefur <zash@zash.se>
parents: 2816
diff changeset
13 end
611ac62c5b63 mod_migrate: Try to use per storage user listing where available
Kim Alvefur <zash@zash.se>
parents: 2816
diff changeset
14
1791
8df071457dee mod_migrate: Provides a prosodyctl mod_migrate command for copying data between storage backends
Kim Alvefur <zash@zash.se>
parents:
diff changeset
15 function module.command(arg)
2816
cd9831b27746 mod_migrate: Remove unused locals
Kim Alvefur <zash@zash.se>
parents: 2812
diff changeset
16 local host, source_stores, migrate_to = unpack(arg);
1791
8df071457dee mod_migrate: Provides a prosodyctl mod_migrate command for copying data between storage backends
Kim Alvefur <zash@zash.se>
parents:
diff changeset
17 if not migrate_to then
1813
f02f52a2ee11 mod_migrate: Add support for migrating archives
Kim Alvefur <zash@zash.se>
parents: 1812
diff changeset
18 return print("Usage: prosodyctl mod_migrate example.com <source-store>[-<store-type>] <target-driver> [users]*");
1791
8df071457dee mod_migrate: Provides a prosodyctl mod_migrate command for copying data between storage backends
Kim Alvefur <zash@zash.se>
parents:
diff changeset
19 end
2139
17408ddd34b0 mod_migrate: Check that the hostname is known, or yell and abort (thanks Link Mauve)
Kim Alvefur <zash@zash.se>
parents: 2064
diff changeset
20 if not prosody.hosts[host] then
17408ddd34b0 mod_migrate: Check that the hostname is known, or yell and abort (thanks Link Mauve)
Kim Alvefur <zash@zash.se>
parents: 2064
diff changeset
21 return print(("The host %q is not know by Prosody."):format(host));
17408ddd34b0 mod_migrate: Check that the hostname is known, or yell and abort (thanks Link Mauve)
Kim Alvefur <zash@zash.se>
parents: 2064
diff changeset
22 end
1791
8df071457dee mod_migrate: Provides a prosodyctl mod_migrate command for copying data between storage backends
Kim Alvefur <zash@zash.se>
parents:
diff changeset
23 sm.initialize_host(host);
8df071457dee mod_migrate: Provides a prosodyctl mod_migrate command for copying data between storage backends
Kim Alvefur <zash@zash.se>
parents:
diff changeset
24 um.initialize_host(host);
8df071457dee mod_migrate: Provides a prosodyctl mod_migrate command for copying data between storage backends
Kim Alvefur <zash@zash.se>
parents:
diff changeset
25 local module = module:context(host);
1985
7821a6986e68 mod_migrate: Support migrating multiple stores
Kim Alvefur <zash@zash.se>
parents: 1813
diff changeset
26 for source_store in source_stores:gmatch("[^,]+") do
7821a6986e68 mod_migrate: Support migrating multiple stores
Kim Alvefur <zash@zash.se>
parents: 1813
diff changeset
27 local store_type = source_store:match("%-(%a+)$");
7821a6986e68 mod_migrate: Support migrating multiple stores
Kim Alvefur <zash@zash.se>
parents: 1813
diff changeset
28 if store_type then
7821a6986e68 mod_migrate: Support migrating multiple stores
Kim Alvefur <zash@zash.se>
parents: 1813
diff changeset
29 source_store = source_store:sub(1, -2-#store_type);
7821a6986e68 mod_migrate: Support migrating multiple stores
Kim Alvefur <zash@zash.se>
parents: 1813
diff changeset
30 end
7821a6986e68 mod_migrate: Support migrating multiple stores
Kim Alvefur <zash@zash.se>
parents: 1813
diff changeset
31 local storage = module:open_store(source_store, store_type);
7821a6986e68 mod_migrate: Support migrating multiple stores
Kim Alvefur <zash@zash.se>
parents: 1813
diff changeset
32 local target = assert(sm.load_driver(host, migrate_to));
7821a6986e68 mod_migrate: Support migrating multiple stores
Kim Alvefur <zash@zash.se>
parents: 1813
diff changeset
33 target = assert(target:open(source_store, store_type));
1813
f02f52a2ee11 mod_migrate: Add support for migrating archives
Kim Alvefur <zash@zash.se>
parents: 1812
diff changeset
34
1985
7821a6986e68 mod_migrate: Support migrating multiple stores
Kim Alvefur <zash@zash.se>
parents: 1813
diff changeset
35 local function migrate_user(username)
2028
530bf8ef2e55 mod_migrate: Include debug data of what store is being migrated
Kim Alvefur <zash@zash.se>
parents: 1985
diff changeset
36 module:log("info", "Migrating %s data for %s", source_store, username);
1985
7821a6986e68 mod_migrate: Support migrating multiple stores
Kim Alvefur <zash@zash.se>
parents: 1813
diff changeset
37 local data, err = storage:get(username);
2064
2a82c55762a7 mod_migrate: Log read and write errors instead of raising error
Kim Alvefur <zash@zash.se>
parents: 2028
diff changeset
38 if not data and err then
2a82c55762a7 mod_migrate: Log read and write errors instead of raising error
Kim Alvefur <zash@zash.se>
parents: 2028
diff changeset
39 module:log("error", "Could not read data: %s", err);
2a82c55762a7 mod_migrate: Log read and write errors instead of raising error
Kim Alvefur <zash@zash.se>
parents: 2028
diff changeset
40 else
2a82c55762a7 mod_migrate: Log read and write errors instead of raising error
Kim Alvefur <zash@zash.se>
parents: 2028
diff changeset
41 local ok, err = target:set(username, data);
2a82c55762a7 mod_migrate: Log read and write errors instead of raising error
Kim Alvefur <zash@zash.se>
parents: 2028
diff changeset
42 if not ok then
2a82c55762a7 mod_migrate: Log read and write errors instead of raising error
Kim Alvefur <zash@zash.se>
parents: 2028
diff changeset
43 module:log("error", "Could not write data: %s", err);
2a82c55762a7 mod_migrate: Log read and write errors instead of raising error
Kim Alvefur <zash@zash.se>
parents: 2028
diff changeset
44 end
2a82c55762a7 mod_migrate: Log read and write errors instead of raising error
Kim Alvefur <zash@zash.se>
parents: 2028
diff changeset
45 end
1985
7821a6986e68 mod_migrate: Support migrating multiple stores
Kim Alvefur <zash@zash.se>
parents: 1813
diff changeset
46 end
1791
8df071457dee mod_migrate: Provides a prosodyctl mod_migrate command for copying data between storage backends
Kim Alvefur <zash@zash.se>
parents:
diff changeset
47
1985
7821a6986e68 mod_migrate: Support migrating multiple stores
Kim Alvefur <zash@zash.se>
parents: 1813
diff changeset
48 if store_type == "archive" then
7821a6986e68 mod_migrate: Support migrating multiple stores
Kim Alvefur <zash@zash.se>
parents: 1813
diff changeset
49 function migrate_user(username)
2028
530bf8ef2e55 mod_migrate: Include debug data of what store is being migrated
Kim Alvefur <zash@zash.se>
parents: 1985
diff changeset
50 module:log("info", "Migrating %s archive items for %s", source_store, username);
1985
7821a6986e68 mod_migrate: Support migrating multiple stores
Kim Alvefur <zash@zash.se>
parents: 1813
diff changeset
51 local count, errs = 0, 0;
7821a6986e68 mod_migrate: Support migrating multiple stores
Kim Alvefur <zash@zash.se>
parents: 1813
diff changeset
52 for id, item, when, with in storage:find(username) do
7821a6986e68 mod_migrate: Support migrating multiple stores
Kim Alvefur <zash@zash.se>
parents: 1813
diff changeset
53 local ok, err = target:append(username, id, item, when, with);
7821a6986e68 mod_migrate: Support migrating multiple stores
Kim Alvefur <zash@zash.se>
parents: 1813
diff changeset
54 if ok then
7821a6986e68 mod_migrate: Support migrating multiple stores
Kim Alvefur <zash@zash.se>
parents: 1813
diff changeset
55 count = count + 1;
7821a6986e68 mod_migrate: Support migrating multiple stores
Kim Alvefur <zash@zash.se>
parents: 1813
diff changeset
56 else
7821a6986e68 mod_migrate: Support migrating multiple stores
Kim Alvefur <zash@zash.se>
parents: 1813
diff changeset
57 module:log("warn", "Error: %s", err);
7821a6986e68 mod_migrate: Support migrating multiple stores
Kim Alvefur <zash@zash.se>
parents: 1813
diff changeset
58 errs = errs + 1;
7821a6986e68 mod_migrate: Support migrating multiple stores
Kim Alvefur <zash@zash.se>
parents: 1813
diff changeset
59 end
7821a6986e68 mod_migrate: Support migrating multiple stores
Kim Alvefur <zash@zash.se>
parents: 1813
diff changeset
60 if ( count + errs ) % 100 == 0 then
7821a6986e68 mod_migrate: Support migrating multiple stores
Kim Alvefur <zash@zash.se>
parents: 1813
diff changeset
61 module:log("info", "%d items migrated, %d errors", count, errs);
7821a6986e68 mod_migrate: Support migrating multiple stores
Kim Alvefur <zash@zash.se>
parents: 1813
diff changeset
62 end
1813
f02f52a2ee11 mod_migrate: Add support for migrating archives
Kim Alvefur <zash@zash.se>
parents: 1812
diff changeset
63 end
1985
7821a6986e68 mod_migrate: Support migrating multiple stores
Kim Alvefur <zash@zash.se>
parents: 1813
diff changeset
64 module:log("info", "%d items migrated, %d errors", count, errs);
1813
f02f52a2ee11 mod_migrate: Add support for migrating archives
Kim Alvefur <zash@zash.se>
parents: 1812
diff changeset
65 end
f02f52a2ee11 mod_migrate: Add support for migrating archives
Kim Alvefur <zash@zash.se>
parents: 1812
diff changeset
66 end
f02f52a2ee11 mod_migrate: Add support for migrating archives
Kim Alvefur <zash@zash.se>
parents: 1812
diff changeset
67
1985
7821a6986e68 mod_migrate: Support migrating multiple stores
Kim Alvefur <zash@zash.se>
parents: 1813
diff changeset
68 if arg[4] then
7821a6986e68 mod_migrate: Support migrating multiple stores
Kim Alvefur <zash@zash.se>
parents: 1813
diff changeset
69 for i = 4, #arg do
7821a6986e68 mod_migrate: Support migrating multiple stores
Kim Alvefur <zash@zash.se>
parents: 1813
diff changeset
70 migrate_user(arg[i]);
7821a6986e68 mod_migrate: Support migrating multiple stores
Kim Alvefur <zash@zash.se>
parents: 1813
diff changeset
71 end
7821a6986e68 mod_migrate: Support migrating multiple stores
Kim Alvefur <zash@zash.se>
parents: 1813
diff changeset
72 else
2812
fd4a5ba12fa2 mod_migrate: Catch failure to use user listing API and suggest the alternative
Kim Alvefur <zash@zash.se>
parents: 2139
diff changeset
73 xpcall(function()
3584
611ac62c5b63 mod_migrate: Try to use per storage user listing where available
Kim Alvefur <zash@zash.se>
parents: 2816
diff changeset
74 for user in users(storage, host) do
2812
fd4a5ba12fa2 mod_migrate: Catch failure to use user listing API and suggest the alternative
Kim Alvefur <zash@zash.se>
parents: 2139
diff changeset
75 migrate_user(user);
fd4a5ba12fa2 mod_migrate: Catch failure to use user listing API and suggest the alternative
Kim Alvefur <zash@zash.se>
parents: 2139
diff changeset
76 end
fd4a5ba12fa2 mod_migrate: Catch failure to use user listing API and suggest the alternative
Kim Alvefur <zash@zash.se>
parents: 2139
diff changeset
77 end,
fd4a5ba12fa2 mod_migrate: Catch failure to use user listing API and suggest the alternative
Kim Alvefur <zash@zash.se>
parents: 2139
diff changeset
78 function (err)
fd4a5ba12fa2 mod_migrate: Catch failure to use user listing API and suggest the alternative
Kim Alvefur <zash@zash.se>
parents: 2139
diff changeset
79 module:log("error", "Could not list users, you'll have to supply them as arguments");
fd4a5ba12fa2 mod_migrate: Catch failure to use user listing API and suggest the alternative
Kim Alvefur <zash@zash.se>
parents: 2139
diff changeset
80 module:log("error", "The error was: %s", err);
fd4a5ba12fa2 mod_migrate: Catch failure to use user listing API and suggest the alternative
Kim Alvefur <zash@zash.se>
parents: 2139
diff changeset
81 end);
1791
8df071457dee mod_migrate: Provides a prosodyctl mod_migrate command for copying data between storage backends
Kim Alvefur <zash@zash.se>
parents:
diff changeset
82 end
8df071457dee mod_migrate: Provides a prosodyctl mod_migrate command for copying data between storage backends
Kim Alvefur <zash@zash.se>
parents:
diff changeset
83 end
8df071457dee mod_migrate: Provides a prosodyctl mod_migrate command for copying data between storage backends
Kim Alvefur <zash@zash.se>
parents:
diff changeset
84 end