Mercurial > prosody-modules
annotate mod_storage_xmlarchive/README.markdown @ 4095:de2390d6bbe4
mod_http_libjs: Add compatibility with Prosody 0.11.x
author | Matthew Wild <mwild1@gmail.com> |
---|---|
date | Fri, 11 Sep 2020 13:53:00 +0100 |
parents | dddcf094af5c |
children | 591c643d55b2 |
rev | line source |
---|---|
1803
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1782
diff
changeset
|
1 --- |
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1782
diff
changeset
|
2 labels: |
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1782
diff
changeset
|
3 - 'Stage-Beta' |
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1782
diff
changeset
|
4 - 'Type-Storage' |
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1782
diff
changeset
|
5 - ArchiveStorage |
2359
33e85070dc60
mod_storage_xmlarchive: Expand the summary a little bit
Kim Alvefur <zash@zash.se>
parents:
2291
diff
changeset
|
6 summary: XML file based archive storage |
2817
f052b62c653c
mod_storage_xmlarchive/README: Minor tweak
Kim Alvefur <zash@zash.se>
parents:
2815
diff
changeset
|
7 --- |
1782 | 8 |
1803
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1782
diff
changeset
|
9 Introduction |
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1782
diff
changeset
|
10 ============ |
1782 | 11 |
1803
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1782
diff
changeset
|
12 This module implements stanza archives using files, similar to the |
3434
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2855
diff
changeset
|
13 default "internal" storage. Unlike "internal", it saves messages in two |
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2855
diff
changeset
|
14 files per day (and per user), one containing metadata and one containing |
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2855
diff
changeset
|
15 the actual messages in XML format (hence the name). |
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2855
diff
changeset
|
16 |
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2855
diff
changeset
|
17 Splitting data per day improves performance for larger archives as it |
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2855
diff
changeset
|
18 does not have to look through data from other days. |
1782 | 19 |
1940
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
20 Configuration |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
21 ============= |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
22 |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
23 To use this with [mod\_mam] add this to your config: |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
24 |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
25 ``` lua |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
26 storage = { |
3451
7b97abf26612
mod_storage_xmlarchive/README: Change example to match Prosodys mod_mam
Kim Alvefur <zash@zash.se>
parents:
3434
diff
changeset
|
27 archive = "xmlarchive" |
1940
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
28 } |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
29 ``` |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
30 |
1969
e63dba236a2a
mod_storage_xmlarchive: Use datamanager.append_raw (had that code duplicated here)
Kim Alvefur <zash@zash.se>
parents:
1940
diff
changeset
|
31 To use it with [mod\_mam\_muc] or [mod\_http\_muc\_log]: |
1940
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
32 |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
33 ``` lua |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
34 storage = { |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
35 muc_log = "xmlarchive" |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
36 } |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
37 ``` |
1782 | 38 |
1969
e63dba236a2a
mod_storage_xmlarchive: Use datamanager.append_raw (had that code duplicated here)
Kim Alvefur <zash@zash.se>
parents:
1940
diff
changeset
|
39 Refer to [Prosodys data storage documentation][doc:storage] for more |
e63dba236a2a
mod_storage_xmlarchive: Use datamanager.append_raw (had that code duplicated here)
Kim Alvefur <zash@zash.se>
parents:
1940
diff
changeset
|
40 information. |
1803
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1782
diff
changeset
|
41 |
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1782
diff
changeset
|
42 Note that this module does not implement the "keyval" storage method and |
1940
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
43 can't be used by anything other than archives. |
1782 | 44 |
1803
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1782
diff
changeset
|
45 Compatibility |
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1782
diff
changeset
|
46 ============= |
1782 | 47 |
2291
1eeda34d668b
mod_storage_xmlarchive/README: Should work in 0.9 now
Kim Alvefur <zash@zash.se>
parents:
2048
diff
changeset
|
48 ------ --------------- |
3987
dddcf094af5c
mod_storage_xmlarchive: Update compat table
Kim Alvefur <zash@zash.se>
parents:
3460
diff
changeset
|
49 trunk Should work |
dddcf094af5c
mod_storage_xmlarchive: Update compat table
Kim Alvefur <zash@zash.se>
parents:
3460
diff
changeset
|
50 0.11 Works |
1940
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
51 0.10 Works |
2291
1eeda34d668b
mod_storage_xmlarchive/README: Should work in 0.9 now
Kim Alvefur <zash@zash.se>
parents:
2048
diff
changeset
|
52 0.9 Should work |
1940
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
53 0.8 Does not work |
2291
1eeda34d668b
mod_storage_xmlarchive/README: Should work in 0.9 now
Kim Alvefur <zash@zash.se>
parents:
2048
diff
changeset
|
54 ------ --------------- |
2815
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
55 |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
56 Conversion to or from internal storage |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
57 -------------------------------------- |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
58 |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
59 This module stores data in a way that overlaps with the more recent |
2817
f052b62c653c
mod_storage_xmlarchive/README: Minor tweak
Kim Alvefur <zash@zash.se>
parents:
2815
diff
changeset
|
60 archive support in `mod_storage_internal`, meaning e.g. [mod_migrate] |
2815
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
61 will not be able to cleanly convert to or from the `xmlarchive` format. |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
62 |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
63 To mitigate this, an migration command has been added to |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
64 `mod_storage_xmlarchive`: |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
65 |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
66 ``` bash |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
67 prosodyctl mod_storage_xmlarchive convert $DIR internal $STORE $JID |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
68 ``` |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
69 |
2818 | 70 Where `$DIR` is `to` or `from`, `$STORE` is e.g. `archive` or `archive2` |
2815
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
71 for MAM and `muc_log` for MUC logs. Finally, `$JID` is the JID of the |
2855
7713cd4fff8f
mod_storage_xmlarchive/README: Fix typo
Kim Alvefur <zash@zash.se>
parents:
2854
diff
changeset
|
72 user or MUC room to be migrated, which can be repeated. |
2854
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
73 |
3459
471855552f49
mod_storage_xmlarchive/README: Add reminder about backups
Kim Alvefur <zash@zash.se>
parents:
3451
diff
changeset
|
74 ::: {.alert .alert-danger} |
471855552f49
mod_storage_xmlarchive/README: Add reminder about backups
Kim Alvefur <zash@zash.se>
parents:
3451
diff
changeset
|
75 Since this is a destructive command, don't forget to backup your data |
471855552f49
mod_storage_xmlarchive/README: Add reminder about backups
Kim Alvefur <zash@zash.se>
parents:
3451
diff
changeset
|
76 first. |
3460
f25b0623b276
mod_storage_xmlarchive/README: And don't have prosody running at the same time
Kim Alvefur <zash@zash.se>
parents:
3459
diff
changeset
|
77 |
f25b0623b276
mod_storage_xmlarchive/README: And don't have prosody running at the same time
Kim Alvefur <zash@zash.se>
parents:
3459
diff
changeset
|
78 Prosody should *not* be running while converting data. |
3459
471855552f49
mod_storage_xmlarchive/README: Add reminder about backups
Kim Alvefur <zash@zash.se>
parents:
3451
diff
changeset
|
79 ::: |
471855552f49
mod_storage_xmlarchive/README: Add reminder about backups
Kim Alvefur <zash@zash.se>
parents:
3451
diff
changeset
|
80 |
471855552f49
mod_storage_xmlarchive/README: Add reminder about backups
Kim Alvefur <zash@zash.se>
parents:
3451
diff
changeset
|
81 |
2854
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
82 Data structure |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
83 ============== |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
84 |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
85 Data is split in three kinds of files and messages are grouped by day. |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
86 Prosodys `util.datamanager` is used, so all special characters in these |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
87 filenames are escaped and reside under `hostname/store` in Prosodys Data |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
88 directory, commonly `/var/lib/prosody`. |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
89 |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
90 `username.list` |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
91 : A list of dates in `YYYY-MM-DD` format. |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
92 |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
93 `username@YYYY-MM-DD.list` |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
94 : Index containing metadata for messages stored on that day. |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
95 |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
96 `username@YYYY-MM-DD.xml` |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
97 : Messages in textual XML format, separated by newlines. |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
98 |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
99 This makes it fairly simple and fast to find messages by timestamp. |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
100 Queries that are not time based, but limited to a specific contact may |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
101 be expensive as potentially the entire archive will be read. |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
102 |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
103 Each archive ID is of the form `YYYY-MM-DD-random`, making lookups by |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
104 archive id just as simple as time based queries. |
3434
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2855
diff
changeset
|
105 |
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2855
diff
changeset
|
106 ## Limitations |
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2855
diff
changeset
|
107 |
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2855
diff
changeset
|
108 - Only XML stanzas can be stored. |
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2855
diff
changeset
|
109 - The deletion method only supports removing entire days at a time. |