Mercurial > prosody-modules

annotate mod_storage_xmlarchive/README.markdown @ 3456:b85622b577ad

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
mod_cache_c2s_caps: Unset the caps_cache also on errors due to the result validation.
author Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
date Tue, 22 Jan 2019 15:39:35 +0100
parents 7b97abf26612
children 471855552f49
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1803
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1782
diff changeset
1 ---
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1782
diff changeset
2 labels:
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1782
diff changeset
3 - 'Stage-Beta'
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1782
diff changeset
4 - 'Type-Storage'
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1782
diff changeset
5 - ArchiveStorage
2359
33e85070dc60 mod_storage_xmlarchive: Expand the summary a little bit
Kim Alvefur <zash@zash.se>
parents: 2291
diff changeset
6 summary: XML file based archive storage
2817
f052b62c653c mod_storage_xmlarchive/README: Minor tweak
Kim Alvefur <zash@zash.se>
parents: 2815
diff changeset
7 ---
1782
29f3d6b7ad16 Import wiki pages
Kim Alvefur <zash@zash.se>
parents:
diff changeset
8
1803
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1782
diff changeset
9 Introduction
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1782
diff changeset
10 ============
1782
29f3d6b7ad16 Import wiki pages
Kim Alvefur <zash@zash.se>
parents:
diff changeset
11
1803
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1782
diff changeset
12 This module implements stanza archives using files, similar to the
3434
a6722a35f35e mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents: 2855
diff changeset
13 default "internal" storage. Unlike "internal", it saves messages in two
a6722a35f35e mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents: 2855
diff changeset
14 files per day (and per user), one containing metadata and one containing
a6722a35f35e mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents: 2855
diff changeset
15 the actual messages in XML format (hence the name).
a6722a35f35e mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents: 2855
diff changeset
16
a6722a35f35e mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents: 2855
diff changeset
17 Splitting data per day improves performance for larger archives as it
a6722a35f35e mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents: 2855
diff changeset
18 does not have to look through data from other days.
1782
29f3d6b7ad16 Import wiki pages
Kim Alvefur <zash@zash.se>
parents:
diff changeset
19
1940
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
20 Configuration
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
21 =============
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
22
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
23 To use this with [mod\_mam] add this to your config:
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
24
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
25 ``` lua
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
26 storage = {
3451
7b97abf26612 mod_storage_xmlarchive/README: Change example to match Prosodys mod_mam
Kim Alvefur <zash@zash.se>
parents: 3434
diff changeset
27 archive = "xmlarchive"
1940
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
28 }
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
29 ```
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
30
1969
e63dba236a2a mod_storage_xmlarchive: Use datamanager.append_raw (had that code duplicated here)
Kim Alvefur <zash@zash.se>
parents: 1940
diff changeset
31 To use it with [mod\_mam\_muc] or [mod\_http\_muc\_log]:
1940
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
32
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
33 ``` lua
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
34 storage = {
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
35 muc_log = "xmlarchive"
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
36 }
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
37 ```
1782
29f3d6b7ad16 Import wiki pages
Kim Alvefur <zash@zash.se>
parents:
diff changeset
38
1969
e63dba236a2a mod_storage_xmlarchive: Use datamanager.append_raw (had that code duplicated here)
Kim Alvefur <zash@zash.se>
parents: 1940
diff changeset
39 Refer to [Prosodys data storage documentation][doc:storage] for more
e63dba236a2a mod_storage_xmlarchive: Use datamanager.append_raw (had that code duplicated here)
Kim Alvefur <zash@zash.se>
parents: 1940
diff changeset
40 information.
1803
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1782
diff changeset
41
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1782
diff changeset
42 Note that this module does not implement the "keyval" storage method and
1940
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
43 can't be used by anything other than archives.
1782
29f3d6b7ad16 Import wiki pages
Kim Alvefur <zash@zash.se>
parents:
diff changeset
44
1803
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1782
diff changeset
45 Compatibility
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1782
diff changeset
46 =============
1782
29f3d6b7ad16 Import wiki pages
Kim Alvefur <zash@zash.se>
parents:
diff changeset
47
2291
1eeda34d668b mod_storage_xmlarchive/README: Should work in 0.9 now
Kim Alvefur <zash@zash.se>
parents: 2048
diff changeset
48 ------ ---------------
1940
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
49 0.10 Works
2291
1eeda34d668b mod_storage_xmlarchive/README: Should work in 0.9 now
Kim Alvefur <zash@zash.se>
parents: 2048
diff changeset
50 0.9 Should work
1940
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
51 0.8 Does not work
2291
1eeda34d668b mod_storage_xmlarchive/README: Should work in 0.9 now
Kim Alvefur <zash@zash.se>
parents: 2048
diff changeset
52 ------ ---------------
2815
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
53
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
54 Conversion to or from internal storage
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
55 --------------------------------------
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
56
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
57 This module stores data in a way that overlaps with the more recent
2817
f052b62c653c mod_storage_xmlarchive/README: Minor tweak
Kim Alvefur <zash@zash.se>
parents: 2815
diff changeset
58 archive support in `mod_storage_internal`, meaning e.g. [mod_migrate]
2815
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
59 will not be able to cleanly convert to or from the `xmlarchive` format.
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
60
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
61 To mitigate this, an migration command has been added to
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
62 `mod_storage_xmlarchive`:
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
63
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
64 ``` bash
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
65 prosodyctl mod_storage_xmlarchive convert $DIR internal $STORE $JID
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
66 ```
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
67
2818
88474dd1af48 Various READMEs: s/eg/e.g.g/
Kim Alvefur <zash@zash.se>
parents: 2817
diff changeset
68 Where `$DIR` is `to` or `from`, `$STORE` is e.g. `archive` or `archive2`
2815
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
69 for MAM and `muc_log` for MUC logs. Finally, `$JID` is the JID of the
2855
7713cd4fff8f mod_storage_xmlarchive/README: Fix typo
Kim Alvefur <zash@zash.se>
parents: 2854
diff changeset
70 user or MUC room to be migrated, which can be repeated.
2854
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
71
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
72 Data structure
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
73 ==============
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
74
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
75 Data is split in three kinds of files and messages are grouped by day.
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
76 Prosodys `util.datamanager` is used, so all special characters in these
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
77 filenames are escaped and reside under `hostname/store` in Prosodys Data
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
78 directory, commonly `/var/lib/prosody`.
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
79
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
80 `username.list`
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
81 : A list of dates in `YYYY-MM-DD` format.
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
82
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
83 `username@YYYY-MM-DD.list`
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
84 : Index containing metadata for messages stored on that day.
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
85
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
86 `username@YYYY-MM-DD.xml`
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
87 : Messages in textual XML format, separated by newlines.
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
88
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
89 This makes it fairly simple and fast to find messages by timestamp.
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
90 Queries that are not time based, but limited to a specific contact may
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
91 be expensive as potentially the entire archive will be read.
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
92
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
93 Each archive ID is of the form `YYYY-MM-DD-random`, making lookups by
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
94 archive id just as simple as time based queries.
3434
a6722a35f35e mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents: 2855
diff changeset
95
a6722a35f35e mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents: 2855
diff changeset
96 ## Limitations
a6722a35f35e mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents: 2855
diff changeset
97
a6722a35f35e mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents: 2855
diff changeset
98 - Only XML stanzas can be stored.
a6722a35f35e mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents: 2855
diff changeset
99 - The deletion method only supports removing entire days at a time.