annotate sat/memory/cache.py @ 3415:814e118d9ef3

tests: end-2-end tests first draft: - e2e tests are launched inside the new docker e2e test environment - `run_e2e.py` launch the docker container, mount the current code base in it, launch the e2e tests and print report in real time - `conftest.py` are pytest fixtures managing many things such as account creation, fake files management, JSON or Domish.Element parsing, fake editor, etc. - `test_jp.py` are end-to-end test done with `jp`. `sh` library is used to make tests writting as user-friendly as possible. The `SAT_TEST_ENV_E2E` environment variable is checked, and tests will be skipped if it's not set.
author Goffi <goffi@goffi.org>
date Thu, 12 Nov 2020 14:53:16 +0100
parents fedec192a83f
children be6d91572633
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
3028
ab2696e34d29 Python 3 port:
Goffi <goffi@goffi.org>
parents: 2771
diff changeset
1 #!/usr/bin/env python3
3137
559a625a236b fixed shebangs
Goffi <goffi@goffi.org>
parents: 3136
diff changeset
2
2109
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
3
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
4 # SAT: a jabber client
3136
9d0df638c8b4 dates update
Goffi <goffi@goffi.org>
parents: 3028
diff changeset
5 # Copyright (C) 2009-2020 Jérôme Poisson (goffi@goffi.org)
2109
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
6
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
7 # This program is free software: you can redistribute it and/or modify
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
8 # it under the terms of the GNU Affero General Public License as published by
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
9 # the Free Software Foundation, either version 3 of the License, or
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
10 # (at your option) any later version.
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
11
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
12 # This program is distributed in the hope that it will be useful,
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
13 # but WITHOUT ANY WARRANTY; without even the implied warranty of
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
14 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
15 # GNU Affero General Public License for more details.
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
16
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
17 # You should have received a copy of the GNU Affero General Public License
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
18 # along with this program. If not, see <http://www.gnu.org/licenses/>.
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
19
3185
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
20 import pickle as pickle
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
21 import mimetypes
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
22 import time
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
23 from pathlib import Path
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
24 from sat.core.i18n import _
2109
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
25 from sat.core.log import getLogger
3185
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
26 from sat.core.constants import Const as C
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
27 from sat.core import exceptions
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
28 from sat.tools.common import regex
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
29
2624
56f94936df1e code style reformatting using black
Goffi <goffi@goffi.org>
parents: 2562
diff changeset
30
2109
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
31 log = getLogger(__name__)
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
32
2624
56f94936df1e code style reformatting using black
Goffi <goffi@goffi.org>
parents: 2562
diff changeset
33 DEFAULT_EXT = ".raw"
2509
d485e9416493 core (memory/cache): common cache:
Goffi <goffi@goffi.org>
parents: 2506
diff changeset
34
2109
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
35
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
36 class Cache(object):
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
37 """generic file caching"""
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
38
2509
d485e9416493 core (memory/cache): common cache:
Goffi <goffi@goffi.org>
parents: 2506
diff changeset
39 def __init__(self, host, profile):
d485e9416493 core (memory/cache): common cache:
Goffi <goffi@goffi.org>
parents: 2506
diff changeset
40 """
d485e9416493 core (memory/cache): common cache:
Goffi <goffi@goffi.org>
parents: 2506
diff changeset
41 @param profile(unicode, None): ame of the profile to set the cache for
d485e9416493 core (memory/cache): common cache:
Goffi <goffi@goffi.org>
parents: 2506
diff changeset
42 if None, the cache will be common for all profiles
d485e9416493 core (memory/cache): common cache:
Goffi <goffi@goffi.org>
parents: 2506
diff changeset
43 """
2109
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
44 self.profile = profile
2624
56f94936df1e code style reformatting using black
Goffi <goffi@goffi.org>
parents: 2562
diff changeset
45 path_elts = [host.memory.getConfig("", "local_dir"), C.CACHE_DIR]
2509
d485e9416493 core (memory/cache): common cache:
Goffi <goffi@goffi.org>
parents: 2506
diff changeset
46 if profile:
3028
ab2696e34d29 Python 3 port:
Goffi <goffi@goffi.org>
parents: 2771
diff changeset
47 path_elts.extend(["profiles", regex.pathEscape(profile)])
2509
d485e9416493 core (memory/cache): common cache:
Goffi <goffi@goffi.org>
parents: 2506
diff changeset
48 else:
3028
ab2696e34d29 Python 3 port:
Goffi <goffi@goffi.org>
parents: 2771
diff changeset
49 path_elts.append("common")
3185
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
50 self.cache_dir = Path(*path_elts)
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
51
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
52 self.cache_dir.mkdir(0o700, parents=True, exist_ok=True)
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
53 self.purge()
2509
d485e9416493 core (memory/cache): common cache:
Goffi <goffi@goffi.org>
parents: 2506
diff changeset
54
3185
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
55 def purge(self):
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
56 # remove expired files from cache
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
57 # TODO: this should not be called only on startup, but at regular interval
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
58 # (e.g. once a day)
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
59 purged = set()
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
60 # we sort files to have metadata files first
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
61 for cache_file in sorted(self.cache_dir.iterdir()):
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
62 if cache_file in purged:
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
63 continue
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
64 try:
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
65 with cache_file.open('rb') as f:
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
66 cache_data = pickle.load(f)
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
67 except IOError:
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
68 log.warning(
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
69 _("Can't read metadata file at {path}")
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
70 .format(path=cache_file))
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
71 continue
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
72 except (pickle.UnpicklingError, EOFError):
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
73 log.debug(f"File at {cache_file} is not a metadata file")
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
74 continue
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
75 try:
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
76 eol = cache_data['eol']
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
77 filename = cache_data['filename']
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
78 except KeyError:
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
79 log.warning(
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
80 _("Invalid cache metadata at {path}")
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
81 .format(path=cache_file))
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
82 continue
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
83
3209
f14eb24328d0 core (memory/cache): purge cache metadata when the referenced file doesn't exist
Goffi <goffi@goffi.org>
parents: 3198
diff changeset
84 filepath = self.getPath(filename)
f14eb24328d0 core (memory/cache): purge cache metadata when the referenced file doesn't exist
Goffi <goffi@goffi.org>
parents: 3198
diff changeset
85
f14eb24328d0 core (memory/cache): purge cache metadata when the referenced file doesn't exist
Goffi <goffi@goffi.org>
parents: 3198
diff changeset
86 if not filepath.exists():
f14eb24328d0 core (memory/cache): purge cache metadata when the referenced file doesn't exist
Goffi <goffi@goffi.org>
parents: 3198
diff changeset
87 log.warning(_(
f14eb24328d0 core (memory/cache): purge cache metadata when the referenced file doesn't exist
Goffi <goffi@goffi.org>
parents: 3198
diff changeset
88 "cache {cache_file!r} references an inexisting file: {filepath!r}"
f14eb24328d0 core (memory/cache): purge cache metadata when the referenced file doesn't exist
Goffi <goffi@goffi.org>
parents: 3198
diff changeset
89 ).format(cache_file=str(cache_file), filepath=str(filepath)))
f14eb24328d0 core (memory/cache): purge cache metadata when the referenced file doesn't exist
Goffi <goffi@goffi.org>
parents: 3198
diff changeset
90 log.debug("purging cache with missing file")
f14eb24328d0 core (memory/cache): purge cache metadata when the referenced file doesn't exist
Goffi <goffi@goffi.org>
parents: 3198
diff changeset
91 cache_file.unlink()
f14eb24328d0 core (memory/cache): purge cache metadata when the referenced file doesn't exist
Goffi <goffi@goffi.org>
parents: 3198
diff changeset
92 elif eol < time.time():
3185
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
93 log.debug(
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
94 "purging expired cache {filepath!r} (expired for {time}s)"
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
95 .format(filepath=str(filepath), time=int(time.time() - eol))
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
96 )
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
97 cache_file.unlink()
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
98 try:
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
99 filepath.unlink()
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
100 except FileNotFoundError:
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
101 log.warning(
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
102 _("following file is missing while purging cache: {path}")
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
103 .format(path=filepath)
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
104 )
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
105 purged.add(cache_file)
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
106 purged.add(filepath)
2109
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
107
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
108 def getPath(self, filename):
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
109 """return cached file URL
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
110
3185
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
111 @param filename(str): cached file name (cache data or actual file)
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
112 @return (Path): path to the cached file
2109
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
113 """
3028
ab2696e34d29 Python 3 port:
Goffi <goffi@goffi.org>
parents: 2771
diff changeset
114 if not filename or "/" in filename:
2624
56f94936df1e code style reformatting using black
Goffi <goffi@goffi.org>
parents: 2562
diff changeset
115 log.error(
3028
ab2696e34d29 Python 3 port:
Goffi <goffi@goffi.org>
parents: 2771
diff changeset
116 "invalid char found in file name, hack attempt? name:{}".format(filename)
2624
56f94936df1e code style reformatting using black
Goffi <goffi@goffi.org>
parents: 2562
diff changeset
117 )
3028
ab2696e34d29 Python 3 port:
Goffi <goffi@goffi.org>
parents: 2771
diff changeset
118 raise exceptions.DataError("Invalid char found")
3185
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
119 return self.cache_dir / filename
2109
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
120
3188
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved
Goffi <goffi@goffi.org>
parents: 3185
diff changeset
121 def getMetadata(self, uid, update_eol=True):
3185
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
122 """Retrieve metadata for cached data
2109
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
123
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
124 @param uid(unicode): unique identifier of file
3188
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved
Goffi <goffi@goffi.org>
parents: 3185
diff changeset
125 @param update_eol(bool): True if eol must extended
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved
Goffi <goffi@goffi.org>
parents: 3185
diff changeset
126 if True, max_age will be added to eol (only if it is not already expired)
2517
cd7a53c31eb6 core (memory/cache): new getMetadata method to retrieve metadata without opening the file
Goffi <goffi@goffi.org>
parents: 2509
diff changeset
127 @return (dict, None): metadata with following keys:
3188
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved
Goffi <goffi@goffi.org>
parents: 3185
diff changeset
128 see [cacheData] for data details, an additional "path" key is the full path to
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved
Goffi <goffi@goffi.org>
parents: 3185
diff changeset
129 cached file.
2109
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
130 None if file is not in cache (or cache is invalid)
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
131 """
2517
cd7a53c31eb6 core (memory/cache): new getMetadata method to retrieve metadata without opening the file
Goffi <goffi@goffi.org>
parents: 2509
diff changeset
132
2116
766dbbec56f2 core (memory/cache): geFilePath now return None when uid is empty
Goffi <goffi@goffi.org>
parents: 2109
diff changeset
133 uid = uid.strip()
766dbbec56f2 core (memory/cache): geFilePath now return None when uid is empty
Goffi <goffi@goffi.org>
parents: 2109
diff changeset
134 if not uid:
3028
ab2696e34d29 Python 3 port:
Goffi <goffi@goffi.org>
parents: 2771
diff changeset
135 raise exceptions.InternalError("uid must not be empty")
2109
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
136 cache_url = self.getPath(uid)
3188
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved
Goffi <goffi@goffi.org>
parents: 3185
diff changeset
137 if not cache_url.exists():
2109
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
138 return None
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
139
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
140 try:
3185
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
141 with cache_url.open("rb") as f:
2109
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
142 cache_data = pickle.load(f)
3188
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved
Goffi <goffi@goffi.org>
parents: 3185
diff changeset
143 except IOError as e:
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved
Goffi <goffi@goffi.org>
parents: 3185
diff changeset
144 log.warning(f"can't read cache at {cache_url}: {e}")
2109
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
145 return None
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
146 except pickle.UnpicklingError:
3188
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved
Goffi <goffi@goffi.org>
parents: 3185
diff changeset
147 log.warning(f"invalid cache found at {cache_url}")
2109
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
148 return None
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
149
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
150 try:
2624
56f94936df1e code style reformatting using black
Goffi <goffi@goffi.org>
parents: 2562
diff changeset
151 eol = cache_data["eol"]
2109
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
152 except KeyError:
3028
ab2696e34d29 Python 3 port:
Goffi <goffi@goffi.org>
parents: 2771
diff changeset
153 log.warning("no End Of Life found for cached file {}".format(uid))
2109
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
154 eol = 0
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
155 if eol < time.time():
2624
56f94936df1e code style reformatting using black
Goffi <goffi@goffi.org>
parents: 2562
diff changeset
156 log.debug(
3028
ab2696e34d29 Python 3 port:
Goffi <goffi@goffi.org>
parents: 2771
diff changeset
157 "removing expired cache (expired for {}s)".format(time.time() - eol)
2624
56f94936df1e code style reformatting using black
Goffi <goffi@goffi.org>
parents: 2562
diff changeset
158 )
2109
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
159 return None
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
160
3188
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved
Goffi <goffi@goffi.org>
parents: 3185
diff changeset
161 if update_eol:
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved
Goffi <goffi@goffi.org>
parents: 3185
diff changeset
162 try:
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved
Goffi <goffi@goffi.org>
parents: 3185
diff changeset
163 max_age = cache_data["max_age"]
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved
Goffi <goffi@goffi.org>
parents: 3185
diff changeset
164 except KeyError:
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved
Goffi <goffi@goffi.org>
parents: 3185
diff changeset
165 log.warning(f"no max_age found for cache at {cache_url}, using default")
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved
Goffi <goffi@goffi.org>
parents: 3185
diff changeset
166 max_age = cache_data["max_age"] = C.DEFAULT_MAX_AGE
3198
08151c103636 core (memory/cache): added some metadata:
Goffi <goffi@goffi.org>
parents: 3188
diff changeset
167 now = int(time.time())
08151c103636 core (memory/cache): added some metadata:
Goffi <goffi@goffi.org>
parents: 3188
diff changeset
168 cache_data["last_access"] = now
08151c103636 core (memory/cache): added some metadata:
Goffi <goffi@goffi.org>
parents: 3188
diff changeset
169 cache_data["eol"] = now + max_age
3188
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved
Goffi <goffi@goffi.org>
parents: 3185
diff changeset
170 with cache_url.open("wb") as f:
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved
Goffi <goffi@goffi.org>
parents: 3185
diff changeset
171 pickle.dump(cache_data, f, protocol=2)
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved
Goffi <goffi@goffi.org>
parents: 3185
diff changeset
172
2624
56f94936df1e code style reformatting using black
Goffi <goffi@goffi.org>
parents: 2562
diff changeset
173 cache_data["path"] = self.getPath(cache_data["filename"])
2517
cd7a53c31eb6 core (memory/cache): new getMetadata method to retrieve metadata without opening the file
Goffi <goffi@goffi.org>
parents: 2509
diff changeset
174 return cache_data
cd7a53c31eb6 core (memory/cache): new getMetadata method to retrieve metadata without opening the file
Goffi <goffi@goffi.org>
parents: 2509
diff changeset
175
cd7a53c31eb6 core (memory/cache): new getMetadata method to retrieve metadata without opening the file
Goffi <goffi@goffi.org>
parents: 2509
diff changeset
176 def getFilePath(self, uid):
3185
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
177 """Retrieve absolute path to file
2517
cd7a53c31eb6 core (memory/cache): new getMetadata method to retrieve metadata without opening the file
Goffi <goffi@goffi.org>
parents: 2509
diff changeset
178
cd7a53c31eb6 core (memory/cache): new getMetadata method to retrieve metadata without opening the file
Goffi <goffi@goffi.org>
parents: 2509
diff changeset
179 @param uid(unicode): unique identifier of file
cd7a53c31eb6 core (memory/cache): new getMetadata method to retrieve metadata without opening the file
Goffi <goffi@goffi.org>
parents: 2509
diff changeset
180 @return (unicode, None): absolute path to cached file
cd7a53c31eb6 core (memory/cache): new getMetadata method to retrieve metadata without opening the file
Goffi <goffi@goffi.org>
parents: 2509
diff changeset
181 None if file is not in cache (or cache is invalid)
cd7a53c31eb6 core (memory/cache): new getMetadata method to retrieve metadata without opening the file
Goffi <goffi@goffi.org>
parents: 2509
diff changeset
182 """
cd7a53c31eb6 core (memory/cache): new getMetadata method to retrieve metadata without opening the file
Goffi <goffi@goffi.org>
parents: 2509
diff changeset
183 metadata = self.getMetadata(uid)
cd7a53c31eb6 core (memory/cache): new getMetadata method to retrieve metadata without opening the file
Goffi <goffi@goffi.org>
parents: 2509
diff changeset
184 if metadata is not None:
2624
56f94936df1e code style reformatting using black
Goffi <goffi@goffi.org>
parents: 2562
diff changeset
185 return metadata["path"]
2109
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
186
3210
fedec192a83f core (memory/cache): new removeFromCache method
Goffi <goffi@goffi.org>
parents: 3209
diff changeset
187 def removeFromCache(self, uid, metadata=None):
fedec192a83f core (memory/cache): new removeFromCache method
Goffi <goffi@goffi.org>
parents: 3209
diff changeset
188 """Remove data from cache
fedec192a83f core (memory/cache): new removeFromCache method
Goffi <goffi@goffi.org>
parents: 3209
diff changeset
189
fedec192a83f core (memory/cache): new removeFromCache method
Goffi <goffi@goffi.org>
parents: 3209
diff changeset
190 @param uid(unicode): unique identifier cache file
fedec192a83f core (memory/cache): new removeFromCache method
Goffi <goffi@goffi.org>
parents: 3209
diff changeset
191 """
fedec192a83f core (memory/cache): new removeFromCache method
Goffi <goffi@goffi.org>
parents: 3209
diff changeset
192 cache_data = self.getMetadata(uid, update_eol=False)
fedec192a83f core (memory/cache): new removeFromCache method
Goffi <goffi@goffi.org>
parents: 3209
diff changeset
193 if cache_data is None:
fedec192a83f core (memory/cache): new removeFromCache method
Goffi <goffi@goffi.org>
parents: 3209
diff changeset
194 log.debug(f"cache with uid {uid!r} has already expired or been removed")
fedec192a83f core (memory/cache): new removeFromCache method
Goffi <goffi@goffi.org>
parents: 3209
diff changeset
195 return
fedec192a83f core (memory/cache): new removeFromCache method
Goffi <goffi@goffi.org>
parents: 3209
diff changeset
196
fedec192a83f core (memory/cache): new removeFromCache method
Goffi <goffi@goffi.org>
parents: 3209
diff changeset
197 try:
fedec192a83f core (memory/cache): new removeFromCache method
Goffi <goffi@goffi.org>
parents: 3209
diff changeset
198 filename = cache_data['filename']
fedec192a83f core (memory/cache): new removeFromCache method
Goffi <goffi@goffi.org>
parents: 3209
diff changeset
199 except KeyError:
fedec192a83f core (memory/cache): new removeFromCache method
Goffi <goffi@goffi.org>
parents: 3209
diff changeset
200 log.warning(_("missing filename for cache {uid!r}") .format(uid=uid))
fedec192a83f core (memory/cache): new removeFromCache method
Goffi <goffi@goffi.org>
parents: 3209
diff changeset
201 else:
fedec192a83f core (memory/cache): new removeFromCache method
Goffi <goffi@goffi.org>
parents: 3209
diff changeset
202 filepath = self.getPath(filename)
fedec192a83f core (memory/cache): new removeFromCache method
Goffi <goffi@goffi.org>
parents: 3209
diff changeset
203 try:
fedec192a83f core (memory/cache): new removeFromCache method
Goffi <goffi@goffi.org>
parents: 3209
diff changeset
204 filepath.unlink()
fedec192a83f core (memory/cache): new removeFromCache method
Goffi <goffi@goffi.org>
parents: 3209
diff changeset
205 except FileNotFoundError:
fedec192a83f core (memory/cache): new removeFromCache method
Goffi <goffi@goffi.org>
parents: 3209
diff changeset
206 log.warning(
fedec192a83f core (memory/cache): new removeFromCache method
Goffi <goffi@goffi.org>
parents: 3209
diff changeset
207 _("missing file referenced in cache {uid!r}: {filename}")
fedec192a83f core (memory/cache): new removeFromCache method
Goffi <goffi@goffi.org>
parents: 3209
diff changeset
208 .format(uid=uid, filename=filename)
fedec192a83f core (memory/cache): new removeFromCache method
Goffi <goffi@goffi.org>
parents: 3209
diff changeset
209 )
fedec192a83f core (memory/cache): new removeFromCache method
Goffi <goffi@goffi.org>
parents: 3209
diff changeset
210
fedec192a83f core (memory/cache): new removeFromCache method
Goffi <goffi@goffi.org>
parents: 3209
diff changeset
211 cache_file = self.getPath(uid)
fedec192a83f core (memory/cache): new removeFromCache method
Goffi <goffi@goffi.org>
parents: 3209
diff changeset
212 cache_file.unlink()
fedec192a83f core (memory/cache): new removeFromCache method
Goffi <goffi@goffi.org>
parents: 3209
diff changeset
213 log.debug(f"cache with uid {uid!r} has been removed")
fedec192a83f core (memory/cache): new removeFromCache method
Goffi <goffi@goffi.org>
parents: 3209
diff changeset
214
2509
d485e9416493 core (memory/cache): common cache:
Goffi <goffi@goffi.org>
parents: 2506
diff changeset
215 def cacheData(self, source, uid, mime_type=None, max_age=None, filename=None):
2109
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
216 """create cache metadata and file object to use for actual data
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
217
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
218 @param source(unicode): source of the cache (should be plugin's import_name)
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
219 @param uid(unicode): an identifier of the file which must be unique
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
220 @param mime_type(unicode): MIME type of the file to cache
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
221 it will be used notably to guess file extension
3198
08151c103636 core (memory/cache): added some metadata:
Goffi <goffi@goffi.org>
parents: 3188
diff changeset
222 It may be autogenerated if filename is specified
2109
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
223 @param max_age(int, None): maximum age in seconds
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
224 the cache metadata will have an "eol" (end of life)
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
225 None to use default value
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
226 0 to ignore cache (file will be re-downloaded on each access)
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
227 @param filename: if not None, will be used as filename
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
228 else one will be generated from uid and guessed extension
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
229 @return(file): file object opened in write mode
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
230 you have to close it yourself (hint: use with statement)
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
231 """
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
232 cache_url = self.getPath(uid)
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
233 if filename is None:
2509
d485e9416493 core (memory/cache): common cache:
Goffi <goffi@goffi.org>
parents: 2506
diff changeset
234 if mime_type:
d485e9416493 core (memory/cache): common cache:
Goffi <goffi@goffi.org>
parents: 2506
diff changeset
235 ext = mimetypes.guess_extension(mime_type, strict=False)
d485e9416493 core (memory/cache): common cache:
Goffi <goffi@goffi.org>
parents: 2506
diff changeset
236 if ext is None:
2624
56f94936df1e code style reformatting using black
Goffi <goffi@goffi.org>
parents: 2562
diff changeset
237 log.warning(
3028
ab2696e34d29 Python 3 port:
Goffi <goffi@goffi.org>
parents: 2771
diff changeset
238 "can't find extension for MIME type {}".format(mime_type)
2624
56f94936df1e code style reformatting using black
Goffi <goffi@goffi.org>
parents: 2562
diff changeset
239 )
2509
d485e9416493 core (memory/cache): common cache:
Goffi <goffi@goffi.org>
parents: 2506
diff changeset
240 ext = DEFAULT_EXT
3028
ab2696e34d29 Python 3 port:
Goffi <goffi@goffi.org>
parents: 2771
diff changeset
241 elif ext == ".jpe":
ab2696e34d29 Python 3 port:
Goffi <goffi@goffi.org>
parents: 2771
diff changeset
242 ext = ".jpg"
2509
d485e9416493 core (memory/cache): common cache:
Goffi <goffi@goffi.org>
parents: 2506
diff changeset
243 else:
d485e9416493 core (memory/cache): common cache:
Goffi <goffi@goffi.org>
parents: 2506
diff changeset
244 ext = DEFAULT_EXT
d485e9416493 core (memory/cache): common cache:
Goffi <goffi@goffi.org>
parents: 2506
diff changeset
245 mime_type = None
2109
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
246 filename = uid + ext
3198
08151c103636 core (memory/cache): added some metadata:
Goffi <goffi@goffi.org>
parents: 3188
diff changeset
247 elif mime_type is None:
08151c103636 core (memory/cache): added some metadata:
Goffi <goffi@goffi.org>
parents: 3188
diff changeset
248 # we have filename but not MIME type, we try to guess the later
08151c103636 core (memory/cache): added some metadata:
Goffi <goffi@goffi.org>
parents: 3188
diff changeset
249 mime_type = mimetypes.guess_type(filename, strict=False)[0]
2109
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
250 if max_age is None:
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
251 max_age = C.DEFAULT_MAX_AGE
3198
08151c103636 core (memory/cache): added some metadata:
Goffi <goffi@goffi.org>
parents: 3188
diff changeset
252 now = int(time.time())
2624
56f94936df1e code style reformatting using black
Goffi <goffi@goffi.org>
parents: 2562
diff changeset
253 cache_data = {
3028
ab2696e34d29 Python 3 port:
Goffi <goffi@goffi.org>
parents: 2771
diff changeset
254 "source": source,
ab2696e34d29 Python 3 port:
Goffi <goffi@goffi.org>
parents: 2771
diff changeset
255 "filename": filename,
3198
08151c103636 core (memory/cache): added some metadata:
Goffi <goffi@goffi.org>
parents: 3188
diff changeset
256 "creation": now,
08151c103636 core (memory/cache): added some metadata:
Goffi <goffi@goffi.org>
parents: 3188
diff changeset
257 "eol": now + max_age,
3188
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved
Goffi <goffi@goffi.org>
parents: 3185
diff changeset
258 # we also store max_age for updating eol
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved
Goffi <goffi@goffi.org>
parents: 3185
diff changeset
259 "max_age": max_age,
3028
ab2696e34d29 Python 3 port:
Goffi <goffi@goffi.org>
parents: 2771
diff changeset
260 "mime_type": mime_type,
2624
56f94936df1e code style reformatting using black
Goffi <goffi@goffi.org>
parents: 2562
diff changeset
261 }
2109
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
262 file_path = self.getPath(filename)
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
263
2624
56f94936df1e code style reformatting using black
Goffi <goffi@goffi.org>
parents: 2562
diff changeset
264 with open(cache_url, "wb") as f:
2109
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
265 pickle.dump(cache_data, f, protocol=2)
85f3e12e984d core (memory/cache): file caching handling, first draft:
Goffi <goffi@goffi.org>
parents:
diff changeset
266
3185
554b3b632378 memory (cache): purge + pathlib:
Goffi <goffi@goffi.org>
parents: 3137
diff changeset
267 return file_path.open("wb")