prosody-modules: mod_storage_xmlarchive/README.markdown annotate

annotate mod_storage_xmlarchive/README.markdown @ 4515:2e33eeafe962

mod_muc_markers: Prevent any markers from reaching the archive, even if untracked Original intention was to leave alone things that this module isn't handling. However markers in archives are just problematic without more advanced logic about what is markable and what is not. It also requires a more advanced query in mod_muc_rai to determine the latest markable message instead of the latest archived message. I'd rather keep the "is archivable" and "is markable" definition the same for simplicity. I don't want to introduce yet another set of rules for no reason. No markers in MAM.

author	Matthew Wild <mwild1@gmail.com>
date	Mon, 22 Mar 2021 15:55:02 +0000
parents	dddcf094af5c
children	591c643d55b2

rev	line source
1803 4d73a1a6ba68 Convert all wiki pages to Markdown Kim Alvefur <zash@zash.se> parents: 1782 diff changeset	1 ---
4d73a1a6ba68 Convert all wiki pages to Markdown Kim Alvefur <zash@zash.se> parents: 1782 diff changeset	2 labels:
4d73a1a6ba68 Convert all wiki pages to Markdown Kim Alvefur <zash@zash.se> parents: 1782 diff changeset	3 - 'Stage-Beta'
4d73a1a6ba68 Convert all wiki pages to Markdown Kim Alvefur <zash@zash.se> parents: 1782 diff changeset	4 - 'Type-Storage'
4d73a1a6ba68 Convert all wiki pages to Markdown Kim Alvefur <zash@zash.se> parents: 1782 diff changeset	5 - ArchiveStorage
2359 33e85070dc60 mod_storage_xmlarchive: Expand the summary a little bit Kim Alvefur <zash@zash.se> parents: 2291 diff changeset	6 summary: XML file based archive storage
2817 f052b62c653c mod_storage_xmlarchive/README: Minor tweak Kim Alvefur <zash@zash.se> parents: 2815 diff changeset	7 ---
1782 29f3d6b7ad16 Import wiki pages Kim Alvefur <zash@zash.se> parents: diff changeset	8
1803 4d73a1a6ba68 Convert all wiki pages to Markdown Kim Alvefur <zash@zash.se> parents: 1782 diff changeset	9 Introduction
4d73a1a6ba68 Convert all wiki pages to Markdown Kim Alvefur <zash@zash.se> parents: 1782 diff changeset	10 ============
1782 29f3d6b7ad16 Import wiki pages Kim Alvefur <zash@zash.se> parents: diff changeset	11
1803 4d73a1a6ba68 Convert all wiki pages to Markdown Kim Alvefur <zash@zash.se> parents: 1782 diff changeset	12 This module implements stanza archives using files, similar to the
3434 a6722a35f35e mod_storage_xmlarchive: Expand description in README Kim Alvefur <zash@zash.se> parents: 2855 diff changeset	13 default "internal" storage. Unlike "internal", it saves messages in two
a6722a35f35e mod_storage_xmlarchive: Expand description in README Kim Alvefur <zash@zash.se> parents: 2855 diff changeset	14 files per day (and per user), one containing metadata and one containing
a6722a35f35e mod_storage_xmlarchive: Expand description in README Kim Alvefur <zash@zash.se> parents: 2855 diff changeset	15 the actual messages in XML format (hence the name).
a6722a35f35e mod_storage_xmlarchive: Expand description in README Kim Alvefur <zash@zash.se> parents: 2855 diff changeset	16
a6722a35f35e mod_storage_xmlarchive: Expand description in README Kim Alvefur <zash@zash.se> parents: 2855 diff changeset	17 Splitting data per day improves performance for larger archives as it
a6722a35f35e mod_storage_xmlarchive: Expand description in README Kim Alvefur <zash@zash.se> parents: 2855 diff changeset	18 does not have to look through data from other days.
1782 29f3d6b7ad16 Import wiki pages Kim Alvefur <zash@zash.se> parents: diff changeset	19
1940 453eee191e18 mod_storage_xmlarchive/README: Add example configuration Kim Alvefur <zash@zash.se> parents: 1803 diff changeset	20 Configuration
453eee191e18 mod_storage_xmlarchive/README: Add example configuration Kim Alvefur <zash@zash.se> parents: 1803 diff changeset	21 =============
453eee191e18 mod_storage_xmlarchive/README: Add example configuration Kim Alvefur <zash@zash.se> parents: 1803 diff changeset	22
453eee191e18 mod_storage_xmlarchive/README: Add example configuration Kim Alvefur <zash@zash.se> parents: 1803 diff changeset	23 To use this with [mod\_mam] add this to your config:
453eee191e18 mod_storage_xmlarchive/README: Add example configuration Kim Alvefur <zash@zash.se> parents: 1803 diff changeset	24
453eee191e18 mod_storage_xmlarchive/README: Add example configuration Kim Alvefur <zash@zash.se> parents: 1803 diff changeset	25 ``` lua
453eee191e18 mod_storage_xmlarchive/README: Add example configuration Kim Alvefur <zash@zash.se> parents: 1803 diff changeset	26 storage = {
3451 7b97abf26612 mod_storage_xmlarchive/README: Change example to match Prosodys mod_mam Kim Alvefur <zash@zash.se> parents: 3434 diff changeset	27 archive = "xmlarchive"
1940 453eee191e18 mod_storage_xmlarchive/README: Add example configuration Kim Alvefur <zash@zash.se> parents: 1803 diff changeset	28 }
453eee191e18 mod_storage_xmlarchive/README: Add example configuration Kim Alvefur <zash@zash.se> parents: 1803 diff changeset	29 ```
453eee191e18 mod_storage_xmlarchive/README: Add example configuration Kim Alvefur <zash@zash.se> parents: 1803 diff changeset	30
1969 e63dba236a2a mod_storage_xmlarchive: Use datamanager.append_raw (had that code duplicated here) Kim Alvefur <zash@zash.se> parents: 1940 diff changeset	31 To use it with [mod\_mam\_muc] or [mod\_http\_muc\_log]:
1940 453eee191e18 mod_storage_xmlarchive/README: Add example configuration Kim Alvefur <zash@zash.se> parents: 1803 diff changeset	32
453eee191e18 mod_storage_xmlarchive/README: Add example configuration Kim Alvefur <zash@zash.se> parents: 1803 diff changeset	33 ``` lua
453eee191e18 mod_storage_xmlarchive/README: Add example configuration Kim Alvefur <zash@zash.se> parents: 1803 diff changeset	34 storage = {
453eee191e18 mod_storage_xmlarchive/README: Add example configuration Kim Alvefur <zash@zash.se> parents: 1803 diff changeset	35 muc_log = "xmlarchive"
453eee191e18 mod_storage_xmlarchive/README: Add example configuration Kim Alvefur <zash@zash.se> parents: 1803 diff changeset	36 }
453eee191e18 mod_storage_xmlarchive/README: Add example configuration Kim Alvefur <zash@zash.se> parents: 1803 diff changeset	37 ```
1782 29f3d6b7ad16 Import wiki pages Kim Alvefur <zash@zash.se> parents: diff changeset	38
1969 e63dba236a2a mod_storage_xmlarchive: Use datamanager.append_raw (had that code duplicated here) Kim Alvefur <zash@zash.se> parents: 1940 diff changeset	39 Refer to [Prosodys data storage documentation][doc:storage] for more
e63dba236a2a mod_storage_xmlarchive: Use datamanager.append_raw (had that code duplicated here) Kim Alvefur <zash@zash.se> parents: 1940 diff changeset	40 information.
1803 4d73a1a6ba68 Convert all wiki pages to Markdown Kim Alvefur <zash@zash.se> parents: 1782 diff changeset	41
4d73a1a6ba68 Convert all wiki pages to Markdown Kim Alvefur <zash@zash.se> parents: 1782 diff changeset	42 Note that this module does not implement the "keyval" storage method and
1940 453eee191e18 mod_storage_xmlarchive/README: Add example configuration Kim Alvefur <zash@zash.se> parents: 1803 diff changeset	43 can't be used by anything other than archives.
1782 29f3d6b7ad16 Import wiki pages Kim Alvefur <zash@zash.se> parents: diff changeset	44
1803 4d73a1a6ba68 Convert all wiki pages to Markdown Kim Alvefur <zash@zash.se> parents: 1782 diff changeset	45 Compatibility
4d73a1a6ba68 Convert all wiki pages to Markdown Kim Alvefur <zash@zash.se> parents: 1782 diff changeset	46 =============
1782 29f3d6b7ad16 Import wiki pages Kim Alvefur <zash@zash.se> parents: diff changeset	47
2291 1eeda34d668b mod_storage_xmlarchive/README: Should work in 0.9 now Kim Alvefur <zash@zash.se> parents: 2048 diff changeset	48 ------ ---------------
3987 dddcf094af5c mod_storage_xmlarchive: Update compat table Kim Alvefur <zash@zash.se> parents: 3460 diff changeset	49 trunk Should work
dddcf094af5c mod_storage_xmlarchive: Update compat table Kim Alvefur <zash@zash.se> parents: 3460 diff changeset	50 0.11 Works
1940 453eee191e18 mod_storage_xmlarchive/README: Add example configuration Kim Alvefur <zash@zash.se> parents: 1803 diff changeset	51 0.10 Works
2291 1eeda34d668b mod_storage_xmlarchive/README: Should work in 0.9 now Kim Alvefur <zash@zash.se> parents: 2048 diff changeset	52 0.9 Should work
1940 453eee191e18 mod_storage_xmlarchive/README: Add example configuration Kim Alvefur <zash@zash.se> parents: 1803 diff changeset	53 0.8 Does not work
2291 1eeda34d668b mod_storage_xmlarchive/README: Should work in 0.9 now Kim Alvefur <zash@zash.se> parents: 2048 diff changeset	54 ------ ---------------
2815 d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format Kim Alvefur <zash@zash.se> parents: 2359 diff changeset	55
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format Kim Alvefur <zash@zash.se> parents: 2359 diff changeset	56 Conversion to or from internal storage
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format Kim Alvefur <zash@zash.se> parents: 2359 diff changeset	57 --------------------------------------
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format Kim Alvefur <zash@zash.se> parents: 2359 diff changeset	58
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format Kim Alvefur <zash@zash.se> parents: 2359 diff changeset	59 This module stores data in a way that overlaps with the more recent
2817 f052b62c653c mod_storage_xmlarchive/README: Minor tweak Kim Alvefur <zash@zash.se> parents: 2815 diff changeset	60 archive support in `mod_storage_internal`, meaning e.g. [mod_migrate]
2815 d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format Kim Alvefur <zash@zash.se> parents: 2359 diff changeset	61 will not be able to cleanly convert to or from the `xmlarchive` format.
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format Kim Alvefur <zash@zash.se> parents: 2359 diff changeset	62
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format Kim Alvefur <zash@zash.se> parents: 2359 diff changeset	63 To mitigate this, an migration command has been added to
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format Kim Alvefur <zash@zash.se> parents: 2359 diff changeset	64 `mod_storage_xmlarchive`:
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format Kim Alvefur <zash@zash.se> parents: 2359 diff changeset	65
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format Kim Alvefur <zash@zash.se> parents: 2359 diff changeset	66 ``` bash
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format Kim Alvefur <zash@zash.se> parents: 2359 diff changeset	67 prosodyctl mod_storage_xmlarchive convert $DIR internal $STORE $JID
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format Kim Alvefur <zash@zash.se> parents: 2359 diff changeset	68 ```
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format Kim Alvefur <zash@zash.se> parents: 2359 diff changeset	69
2818 88474dd1af48 Various READMEs: s/eg/e.g.g/ Kim Alvefur <zash@zash.se> parents: 2817 diff changeset	70 Where `$DIR` is `to` or `from`, `$STORE` is e.g. `archive` or `archive2`
2815 d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format Kim Alvefur <zash@zash.se> parents: 2359 diff changeset	71 for MAM and `muc_log` for MUC logs. Finally, `$JID` is the JID of the
2855 7713cd4fff8f mod_storage_xmlarchive/README: Fix typo Kim Alvefur <zash@zash.se> parents: 2854 diff changeset	72 user or MUC room to be migrated, which can be repeated.
2854 687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored Kim Alvefur <zash@zash.se> parents: 2818 diff changeset	73
3459 471855552f49 mod_storage_xmlarchive/README: Add reminder about backups Kim Alvefur <zash@zash.se> parents: 3451 diff changeset	74 ::: {.alert .alert-danger}
471855552f49 mod_storage_xmlarchive/README: Add reminder about backups Kim Alvefur <zash@zash.se> parents: 3451 diff changeset	75 Since this is a destructive command, don't forget to backup your data
471855552f49 mod_storage_xmlarchive/README: Add reminder about backups Kim Alvefur <zash@zash.se> parents: 3451 diff changeset	76 first.
3460 f25b0623b276 mod_storage_xmlarchive/README: And don't have prosody running at the same time Kim Alvefur <zash@zash.se> parents: 3459 diff changeset	77
f25b0623b276 mod_storage_xmlarchive/README: And don't have prosody running at the same time Kim Alvefur <zash@zash.se> parents: 3459 diff changeset	78 Prosody should not be running while converting data.
3459 471855552f49 mod_storage_xmlarchive/README: Add reminder about backups Kim Alvefur <zash@zash.se> parents: 3451 diff changeset	79 :::
471855552f49 mod_storage_xmlarchive/README: Add reminder about backups Kim Alvefur <zash@zash.se> parents: 3451 diff changeset	80
471855552f49 mod_storage_xmlarchive/README: Add reminder about backups Kim Alvefur <zash@zash.se> parents: 3451 diff changeset	81
2854 687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored Kim Alvefur <zash@zash.se> parents: 2818 diff changeset	82 Data structure
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored Kim Alvefur <zash@zash.se> parents: 2818 diff changeset	83 ==============
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored Kim Alvefur <zash@zash.se> parents: 2818 diff changeset	84
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored Kim Alvefur <zash@zash.se> parents: 2818 diff changeset	85 Data is split in three kinds of files and messages are grouped by day.
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored Kim Alvefur <zash@zash.se> parents: 2818 diff changeset	86 Prosodys `util.datamanager` is used, so all special characters in these
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored Kim Alvefur <zash@zash.se> parents: 2818 diff changeset	87 filenames are escaped and reside under `hostname/store` in Prosodys Data
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored Kim Alvefur <zash@zash.se> parents: 2818 diff changeset	88 directory, commonly `/var/lib/prosody`.
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored Kim Alvefur <zash@zash.se> parents: 2818 diff changeset	89
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored Kim Alvefur <zash@zash.se> parents: 2818 diff changeset	90 `username.list`
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored Kim Alvefur <zash@zash.se> parents: 2818 diff changeset	91 : A list of dates in `YYYY-MM-DD` format.
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored Kim Alvefur <zash@zash.se> parents: 2818 diff changeset	92
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored Kim Alvefur <zash@zash.se> parents: 2818 diff changeset	93 `username@YYYY-MM-DD.list`
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored Kim Alvefur <zash@zash.se> parents: 2818 diff changeset	94 : Index containing metadata for messages stored on that day.
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored Kim Alvefur <zash@zash.se> parents: 2818 diff changeset	95
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored Kim Alvefur <zash@zash.se> parents: 2818 diff changeset	96 `username@YYYY-MM-DD.xml`
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored Kim Alvefur <zash@zash.se> parents: 2818 diff changeset	97 : Messages in textual XML format, separated by newlines.
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored Kim Alvefur <zash@zash.se> parents: 2818 diff changeset	98
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored Kim Alvefur <zash@zash.se> parents: 2818 diff changeset	99 This makes it fairly simple and fast to find messages by timestamp.
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored Kim Alvefur <zash@zash.se> parents: 2818 diff changeset	100 Queries that are not time based, but limited to a specific contact may
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored Kim Alvefur <zash@zash.se> parents: 2818 diff changeset	101 be expensive as potentially the entire archive will be read.
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored Kim Alvefur <zash@zash.se> parents: 2818 diff changeset	102
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored Kim Alvefur <zash@zash.se> parents: 2818 diff changeset	103 Each archive ID is of the form `YYYY-MM-DD-random`, making lookups by
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored Kim Alvefur <zash@zash.se> parents: 2818 diff changeset	104 archive id just as simple as time based queries.
3434 a6722a35f35e mod_storage_xmlarchive: Expand description in README Kim Alvefur <zash@zash.se> parents: 2855 diff changeset	105
a6722a35f35e mod_storage_xmlarchive: Expand description in README Kim Alvefur <zash@zash.se> parents: 2855 diff changeset	106 ## Limitations
a6722a35f35e mod_storage_xmlarchive: Expand description in README Kim Alvefur <zash@zash.se> parents: 2855 diff changeset	107
a6722a35f35e mod_storage_xmlarchive: Expand description in README Kim Alvefur <zash@zash.se> parents: 2855 diff changeset	108 - Only XML stanzas can be stored.
a6722a35f35e mod_storage_xmlarchive: Expand description in README Kim Alvefur <zash@zash.se> parents: 2855 diff changeset	109 - The deletion method only supports removing entire days at a time.

Mercurial > prosody-modules

annotate mod_storage_xmlarchive/README.markdown @ 4515:2e33eeafe962