libervia-backend: sat/memory/cache.py annotate

annotate sat/memory/cache.py @ 3415:814e118d9ef3

tests: end-2-end tests first draft: - e2e tests are launched inside the new docker e2e test environment - `run_e2e.py` launch the docker container, mount the current code base in it, launch the e2e tests and print report in real time - `conftest.py` are pytest fixtures managing many things such as account creation, fake files management, JSON or Domish.Element parsing, fake editor, etc. - `test_jp.py` are end-to-end test done with `jp`. `sh` library is used to make tests writting as user-friendly as possible. The `SAT_TEST_ENV_E2E` environment variable is checked, and tests will be skipped if it's not set.

author	Goffi <goffi@goffi.org>
date	Thu, 12 Nov 2020 14:53:16 +0100
parents	fedec192a83f
children	be6d91572633

rev	line source
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2771 diff changeset	1 #!/usr/bin/env python3
3137 559a625a236b fixed shebangs Goffi <goffi@goffi.org> parents: 3136 diff changeset	2
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	3
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	4 # SAT: a jabber client
3136 9d0df638c8b4 dates update Goffi <goffi@goffi.org> parents: 3028 diff changeset	5 # Copyright (C) 2009-2020 Jérôme Poisson (goffi@goffi.org)
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	6
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	7 # This program is free software: you can redistribute it and/or modify
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	8 # it under the terms of the GNU Affero General Public License as published by
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	9 # the Free Software Foundation, either version 3 of the License, or
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	10 # (at your option) any later version.
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	11
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	12 # This program is distributed in the hope that it will be useful,
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	13 # but WITHOUT ANY WARRANTY; without even the implied warranty of
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	14 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	15 # GNU Affero General Public License for more details.
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	16
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	17 # You should have received a copy of the GNU Affero General Public License
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	18 # along with this program. If not, see <http://www.gnu.org/licenses/>.
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	19
3185 554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	20 import pickle as pickle
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	21 import mimetypes
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	22 import time
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	23 from pathlib import Path
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	24 from sat.core.i18n import _
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	25 from sat.core.log import getLogger
3185 554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	26 from sat.core.constants import Const as C
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	27 from sat.core import exceptions
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	28 from sat.tools.common import regex
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	29
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	30
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	31 log = getLogger(__name__)
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	32
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	33 DEFAULT_EXT = ".raw"
2509 d485e9416493 core (memory/cache): common cache: Goffi <goffi@goffi.org> parents: 2506 diff changeset	34
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	35
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	36 class Cache(object):
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	37 """generic file caching"""
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	38
2509 d485e9416493 core (memory/cache): common cache: Goffi <goffi@goffi.org> parents: 2506 diff changeset	39 def __init__(self, host, profile):
d485e9416493 core (memory/cache): common cache: Goffi <goffi@goffi.org> parents: 2506 diff changeset	40 """
d485e9416493 core (memory/cache): common cache: Goffi <goffi@goffi.org> parents: 2506 diff changeset	41 @param profile(unicode, None): ame of the profile to set the cache for
d485e9416493 core (memory/cache): common cache: Goffi <goffi@goffi.org> parents: 2506 diff changeset	42 if None, the cache will be common for all profiles
d485e9416493 core (memory/cache): common cache: Goffi <goffi@goffi.org> parents: 2506 diff changeset	43 """
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	44 self.profile = profile
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	45 path_elts = [host.memory.getConfig("", "local_dir"), C.CACHE_DIR]
2509 d485e9416493 core (memory/cache): common cache: Goffi <goffi@goffi.org> parents: 2506 diff changeset	46 if profile:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2771 diff changeset	47 path_elts.extend(["profiles", regex.pathEscape(profile)])
2509 d485e9416493 core (memory/cache): common cache: Goffi <goffi@goffi.org> parents: 2506 diff changeset	48 else:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2771 diff changeset	49 path_elts.append("common")
3185 554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	50 self.cache_dir = Path(*path_elts)
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	51
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	52 self.cache_dir.mkdir(0o700, parents=True, exist_ok=True)
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	53 self.purge()
2509 d485e9416493 core (memory/cache): common cache: Goffi <goffi@goffi.org> parents: 2506 diff changeset	54
3185 554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	55 def purge(self):
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	56 # remove expired files from cache
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	57 # TODO: this should not be called only on startup, but at regular interval
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	58 # (e.g. once a day)
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	59 purged = set()
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	60 # we sort files to have metadata files first
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	61 for cache_file in sorted(self.cache_dir.iterdir()):
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	62 if cache_file in purged:
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	63 continue
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	64 try:
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	65 with cache_file.open('rb') as f:
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	66 cache_data = pickle.load(f)
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	67 except IOError:
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	68 log.warning(
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	69 _("Can't read metadata file at {path}")
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	70 .format(path=cache_file))
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	71 continue
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	72 except (pickle.UnpicklingError, EOFError):
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	73 log.debug(f"File at {cache_file} is not a metadata file")
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	74 continue
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	75 try:
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	76 eol = cache_data['eol']
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	77 filename = cache_data['filename']
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	78 except KeyError:
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	79 log.warning(
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	80 _("Invalid cache metadata at {path}")
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	81 .format(path=cache_file))
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	82 continue
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	83
3209 f14eb24328d0 core (memory/cache): purge cache metadata when the referenced file doesn't exist Goffi <goffi@goffi.org> parents: 3198 diff changeset	84 filepath = self.getPath(filename)
f14eb24328d0 core (memory/cache): purge cache metadata when the referenced file doesn't exist Goffi <goffi@goffi.org> parents: 3198 diff changeset	85
f14eb24328d0 core (memory/cache): purge cache metadata when the referenced file doesn't exist Goffi <goffi@goffi.org> parents: 3198 diff changeset	86 if not filepath.exists():
f14eb24328d0 core (memory/cache): purge cache metadata when the referenced file doesn't exist Goffi <goffi@goffi.org> parents: 3198 diff changeset	87 log.warning(_(
f14eb24328d0 core (memory/cache): purge cache metadata when the referenced file doesn't exist Goffi <goffi@goffi.org> parents: 3198 diff changeset	88 "cache {cache_file!r} references an inexisting file: {filepath!r}"
f14eb24328d0 core (memory/cache): purge cache metadata when the referenced file doesn't exist Goffi <goffi@goffi.org> parents: 3198 diff changeset	89 ).format(cache_file=str(cache_file), filepath=str(filepath)))
f14eb24328d0 core (memory/cache): purge cache metadata when the referenced file doesn't exist Goffi <goffi@goffi.org> parents: 3198 diff changeset	90 log.debug("purging cache with missing file")
f14eb24328d0 core (memory/cache): purge cache metadata when the referenced file doesn't exist Goffi <goffi@goffi.org> parents: 3198 diff changeset	91 cache_file.unlink()
f14eb24328d0 core (memory/cache): purge cache metadata when the referenced file doesn't exist Goffi <goffi@goffi.org> parents: 3198 diff changeset	92 elif eol < time.time():
3185 554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	93 log.debug(
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	94 "purging expired cache {filepath!r} (expired for {time}s)"
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	95 .format(filepath=str(filepath), time=int(time.time() - eol))
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	96 )
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	97 cache_file.unlink()
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	98 try:
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	99 filepath.unlink()
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	100 except FileNotFoundError:
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	101 log.warning(
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	102 _("following file is missing while purging cache: {path}")
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	103 .format(path=filepath)
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	104 )
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	105 purged.add(cache_file)
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	106 purged.add(filepath)
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	107
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	108 def getPath(self, filename):
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	109 """return cached file URL
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	110
3185 554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	111 @param filename(str): cached file name (cache data or actual file)
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	112 @return (Path): path to the cached file
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	113 """
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2771 diff changeset	114 if not filename or "/" in filename:
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	115 log.error(
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2771 diff changeset	116 "invalid char found in file name, hack attempt? name:{}".format(filename)
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	117 )
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2771 diff changeset	118 raise exceptions.DataError("Invalid char found")
3185 554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	119 return self.cache_dir / filename
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	120
3188 a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved Goffi <goffi@goffi.org> parents: 3185 diff changeset	121 def getMetadata(self, uid, update_eol=True):
3185 554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	122 """Retrieve metadata for cached data
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	123
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	124 @param uid(unicode): unique identifier of file
3188 a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved Goffi <goffi@goffi.org> parents: 3185 diff changeset	125 @param update_eol(bool): True if eol must extended
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved Goffi <goffi@goffi.org> parents: 3185 diff changeset	126 if True, max_age will be added to eol (only if it is not already expired)
2517 cd7a53c31eb6 core (memory/cache): new getMetadata method to retrieve metadata without opening the file Goffi <goffi@goffi.org> parents: 2509 diff changeset	127 @return (dict, None): metadata with following keys:
3188 a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved Goffi <goffi@goffi.org> parents: 3185 diff changeset	128 see [cacheData] for data details, an additional "path" key is the full path to
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved Goffi <goffi@goffi.org> parents: 3185 diff changeset	129 cached file.
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	130 None if file is not in cache (or cache is invalid)
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	131 """
2517 cd7a53c31eb6 core (memory/cache): new getMetadata method to retrieve metadata without opening the file Goffi <goffi@goffi.org> parents: 2509 diff changeset	132
2116 766dbbec56f2 core (memory/cache): geFilePath now return None when uid is empty Goffi <goffi@goffi.org> parents: 2109 diff changeset	133 uid = uid.strip()
766dbbec56f2 core (memory/cache): geFilePath now return None when uid is empty Goffi <goffi@goffi.org> parents: 2109 diff changeset	134 if not uid:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2771 diff changeset	135 raise exceptions.InternalError("uid must not be empty")
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	136 cache_url = self.getPath(uid)
3188 a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved Goffi <goffi@goffi.org> parents: 3185 diff changeset	137 if not cache_url.exists():
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	138 return None
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	139
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	140 try:
3185 554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	141 with cache_url.open("rb") as f:
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	142 cache_data = pickle.load(f)
3188 a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved Goffi <goffi@goffi.org> parents: 3185 diff changeset	143 except IOError as e:
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved Goffi <goffi@goffi.org> parents: 3185 diff changeset	144 log.warning(f"can't read cache at {cache_url}: {e}")
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	145 return None
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	146 except pickle.UnpicklingError:
3188 a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved Goffi <goffi@goffi.org> parents: 3185 diff changeset	147 log.warning(f"invalid cache found at {cache_url}")
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	148 return None
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	149
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	150 try:
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	151 eol = cache_data["eol"]
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	152 except KeyError:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2771 diff changeset	153 log.warning("no End Of Life found for cached file {}".format(uid))
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	154 eol = 0
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	155 if eol < time.time():
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	156 log.debug(
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2771 diff changeset	157 "removing expired cache (expired for {}s)".format(time.time() - eol)
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	158 )
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	159 return None
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	160
3188 a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved Goffi <goffi@goffi.org> parents: 3185 diff changeset	161 if update_eol:
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved Goffi <goffi@goffi.org> parents: 3185 diff changeset	162 try:
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved Goffi <goffi@goffi.org> parents: 3185 diff changeset	163 max_age = cache_data["max_age"]
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved Goffi <goffi@goffi.org> parents: 3185 diff changeset	164 except KeyError:
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved Goffi <goffi@goffi.org> parents: 3185 diff changeset	165 log.warning(f"no max_age found for cache at {cache_url}, using default")
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved Goffi <goffi@goffi.org> parents: 3185 diff changeset	166 max_age = cache_data["max_age"] = C.DEFAULT_MAX_AGE
3198 08151c103636 core (memory/cache): added some metadata: Goffi <goffi@goffi.org> parents: 3188 diff changeset	167 now = int(time.time())
08151c103636 core (memory/cache): added some metadata: Goffi <goffi@goffi.org> parents: 3188 diff changeset	168 cache_data["last_access"] = now
08151c103636 core (memory/cache): added some metadata: Goffi <goffi@goffi.org> parents: 3188 diff changeset	169 cache_data["eol"] = now + max_age
3188 a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved Goffi <goffi@goffi.org> parents: 3185 diff changeset	170 with cache_url.open("wb") as f:
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved Goffi <goffi@goffi.org> parents: 3185 diff changeset	171 pickle.dump(cache_data, f, protocol=2)
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved Goffi <goffi@goffi.org> parents: 3185 diff changeset	172
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	173 cache_data["path"] = self.getPath(cache_data["filename"])
2517 cd7a53c31eb6 core (memory/cache): new getMetadata method to retrieve metadata without opening the file Goffi <goffi@goffi.org> parents: 2509 diff changeset	174 return cache_data
cd7a53c31eb6 core (memory/cache): new getMetadata method to retrieve metadata without opening the file Goffi <goffi@goffi.org> parents: 2509 diff changeset	175
cd7a53c31eb6 core (memory/cache): new getMetadata method to retrieve metadata without opening the file Goffi <goffi@goffi.org> parents: 2509 diff changeset	176 def getFilePath(self, uid):
3185 554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	177 """Retrieve absolute path to file
2517 cd7a53c31eb6 core (memory/cache): new getMetadata method to retrieve metadata without opening the file Goffi <goffi@goffi.org> parents: 2509 diff changeset	178
cd7a53c31eb6 core (memory/cache): new getMetadata method to retrieve metadata without opening the file Goffi <goffi@goffi.org> parents: 2509 diff changeset	179 @param uid(unicode): unique identifier of file
cd7a53c31eb6 core (memory/cache): new getMetadata method to retrieve metadata without opening the file Goffi <goffi@goffi.org> parents: 2509 diff changeset	180 @return (unicode, None): absolute path to cached file
cd7a53c31eb6 core (memory/cache): new getMetadata method to retrieve metadata without opening the file Goffi <goffi@goffi.org> parents: 2509 diff changeset	181 None if file is not in cache (or cache is invalid)
cd7a53c31eb6 core (memory/cache): new getMetadata method to retrieve metadata without opening the file Goffi <goffi@goffi.org> parents: 2509 diff changeset	182 """
cd7a53c31eb6 core (memory/cache): new getMetadata method to retrieve metadata without opening the file Goffi <goffi@goffi.org> parents: 2509 diff changeset	183 metadata = self.getMetadata(uid)
cd7a53c31eb6 core (memory/cache): new getMetadata method to retrieve metadata without opening the file Goffi <goffi@goffi.org> parents: 2509 diff changeset	184 if metadata is not None:
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	185 return metadata["path"]
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	186
3210 fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	187 def removeFromCache(self, uid, metadata=None):
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	188 """Remove data from cache
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	189
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	190 @param uid(unicode): unique identifier cache file
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	191 """
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	192 cache_data = self.getMetadata(uid, update_eol=False)
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	193 if cache_data is None:
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	194 log.debug(f"cache with uid {uid!r} has already expired or been removed")
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	195 return
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	196
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	197 try:
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	198 filename = cache_data['filename']
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	199 except KeyError:
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	200 log.warning(_("missing filename for cache {uid!r}") .format(uid=uid))
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	201 else:
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	202 filepath = self.getPath(filename)
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	203 try:
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	204 filepath.unlink()
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	205 except FileNotFoundError:
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	206 log.warning(
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	207 _("missing file referenced in cache {uid!r}: {filename}")
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	208 .format(uid=uid, filename=filename)
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	209 )
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	210
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	211 cache_file = self.getPath(uid)
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	212 cache_file.unlink()
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	213 log.debug(f"cache with uid {uid!r} has been removed")
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	214
2509 d485e9416493 core (memory/cache): common cache: Goffi <goffi@goffi.org> parents: 2506 diff changeset	215 def cacheData(self, source, uid, mime_type=None, max_age=None, filename=None):
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	216 """create cache metadata and file object to use for actual data
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	217
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	218 @param source(unicode): source of the cache (should be plugin's import_name)
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	219 @param uid(unicode): an identifier of the file which must be unique
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	220 @param mime_type(unicode): MIME type of the file to cache
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	221 it will be used notably to guess file extension
3198 08151c103636 core (memory/cache): added some metadata: Goffi <goffi@goffi.org> parents: 3188 diff changeset	222 It may be autogenerated if filename is specified
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	223 @param max_age(int, None): maximum age in seconds
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	224 the cache metadata will have an "eol" (end of life)
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	225 None to use default value
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	226 0 to ignore cache (file will be re-downloaded on each access)
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	227 @param filename: if not None, will be used as filename
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	228 else one will be generated from uid and guessed extension
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	229 @return(file): file object opened in write mode
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	230 you have to close it yourself (hint: use with statement)
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	231 """
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	232 cache_url = self.getPath(uid)
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	233 if filename is None:
2509 d485e9416493 core (memory/cache): common cache: Goffi <goffi@goffi.org> parents: 2506 diff changeset	234 if mime_type:
d485e9416493 core (memory/cache): common cache: Goffi <goffi@goffi.org> parents: 2506 diff changeset	235 ext = mimetypes.guess_extension(mime_type, strict=False)
d485e9416493 core (memory/cache): common cache: Goffi <goffi@goffi.org> parents: 2506 diff changeset	236 if ext is None:
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	237 log.warning(
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2771 diff changeset	238 "can't find extension for MIME type {}".format(mime_type)
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	239 )
2509 d485e9416493 core (memory/cache): common cache: Goffi <goffi@goffi.org> parents: 2506 diff changeset	240 ext = DEFAULT_EXT
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2771 diff changeset	241 elif ext == ".jpe":
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2771 diff changeset	242 ext = ".jpg"
2509 d485e9416493 core (memory/cache): common cache: Goffi <goffi@goffi.org> parents: 2506 diff changeset	243 else:
d485e9416493 core (memory/cache): common cache: Goffi <goffi@goffi.org> parents: 2506 diff changeset	244 ext = DEFAULT_EXT
d485e9416493 core (memory/cache): common cache: Goffi <goffi@goffi.org> parents: 2506 diff changeset	245 mime_type = None
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	246 filename = uid + ext
3198 08151c103636 core (memory/cache): added some metadata: Goffi <goffi@goffi.org> parents: 3188 diff changeset	247 elif mime_type is None:
08151c103636 core (memory/cache): added some metadata: Goffi <goffi@goffi.org> parents: 3188 diff changeset	248 # we have filename but not MIME type, we try to guess the later
08151c103636 core (memory/cache): added some metadata: Goffi <goffi@goffi.org> parents: 3188 diff changeset	249 mime_type = mimetypes.guess_type(filename, strict=False)[0]
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	250 if max_age is None:
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	251 max_age = C.DEFAULT_MAX_AGE
3198 08151c103636 core (memory/cache): added some metadata: Goffi <goffi@goffi.org> parents: 3188 diff changeset	252 now = int(time.time())
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	253 cache_data = {
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2771 diff changeset	254 "source": source,
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2771 diff changeset	255 "filename": filename,
3198 08151c103636 core (memory/cache): added some metadata: Goffi <goffi@goffi.org> parents: 3188 diff changeset	256 "creation": now,
08151c103636 core (memory/cache): added some metadata: Goffi <goffi@goffi.org> parents: 3188 diff changeset	257 "eol": now + max_age,
3188 a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved Goffi <goffi@goffi.org> parents: 3185 diff changeset	258 # we also store max_age for updating eol
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved Goffi <goffi@goffi.org> parents: 3185 diff changeset	259 "max_age": max_age,
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2771 diff changeset	260 "mime_type": mime_type,
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	261 }
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	262 file_path = self.getPath(filename)
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	263
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	264 with open(cache_url, "wb") as f:
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	265 pickle.dump(cache_data, f, protocol=2)
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	266
3185 554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	267 return file_path.open("wb")

Mercurial > libervia-backend

annotate sat/memory/cache.py @ 3415:814e118d9ef3