libervia-backend: sat/memory/cache.py annotate

annotate sat/memory/cache.py @ 3582:71516731d0aa

core (memory/sqla): database migration using Alembic: Alembic database migration tool, which is the recommended one for SQLAlchemy has been integrated. When a database is created, it will be used to stamp to current (head) revision, otherwise, DB will be checked to see if it needs to be updated, and upgrade will be triggered if necessary.

author	Goffi <goffi@goffi.org>
date	Fri, 25 Jun 2021 17:55:23 +0200
parents	b3fa179417e7
children	2863345c9bbb

rev	line source
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2771 diff changeset	1 #!/usr/bin/env python3
3137 559a625a236b fixed shebangs Goffi <goffi@goffi.org> parents: 3136 diff changeset	2
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	3
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	4 # SAT: a jabber client
3479 be6d91572633 date update Goffi <goffi@goffi.org> parents: 3210 diff changeset	5 # Copyright (C) 2009-2021 Jérôme Poisson (goffi@goffi.org)
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	6
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	7 # This program is free software: you can redistribute it and/or modify
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	8 # it under the terms of the GNU Affero General Public License as published by
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	9 # the Free Software Foundation, either version 3 of the License, or
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	10 # (at your option) any later version.
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	11
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	12 # This program is distributed in the hope that it will be useful,
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	13 # but WITHOUT ANY WARRANTY; without even the implied warranty of
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	14 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	15 # GNU Affero General Public License for more details.
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	16
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	17 # You should have received a copy of the GNU Affero General Public License
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	18 # along with this program. If not, see <http://www.gnu.org/licenses/>.
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	19
3185 554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	20 import pickle as pickle
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	21 import mimetypes
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	22 import time
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	23 from pathlib import Path
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	24 from sat.core.i18n import _
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	25 from sat.core.log import getLogger
3185 554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	26 from sat.core.constants import Const as C
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	27 from sat.core import exceptions
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	28 from sat.tools.common import regex
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	29
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	30
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	31 log = getLogger(__name__)
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	32
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	33 DEFAULT_EXT = ".raw"
2509 d485e9416493 core (memory/cache): common cache: Goffi <goffi@goffi.org> parents: 2506 diff changeset	34
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	35
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	36 class Cache(object):
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	37 """generic file caching"""
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	38
2509 d485e9416493 core (memory/cache): common cache: Goffi <goffi@goffi.org> parents: 2506 diff changeset	39 def __init__(self, host, profile):
d485e9416493 core (memory/cache): common cache: Goffi <goffi@goffi.org> parents: 2506 diff changeset	40 """
d485e9416493 core (memory/cache): common cache: Goffi <goffi@goffi.org> parents: 2506 diff changeset	41 @param profile(unicode, None): ame of the profile to set the cache for
d485e9416493 core (memory/cache): common cache: Goffi <goffi@goffi.org> parents: 2506 diff changeset	42 if None, the cache will be common for all profiles
d485e9416493 core (memory/cache): common cache: Goffi <goffi@goffi.org> parents: 2506 diff changeset	43 """
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	44 self.profile = profile
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	45 path_elts = [host.memory.getConfig("", "local_dir"), C.CACHE_DIR]
2509 d485e9416493 core (memory/cache): common cache: Goffi <goffi@goffi.org> parents: 2506 diff changeset	46 if profile:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2771 diff changeset	47 path_elts.extend(["profiles", regex.pathEscape(profile)])
2509 d485e9416493 core (memory/cache): common cache: Goffi <goffi@goffi.org> parents: 2506 diff changeset	48 else:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2771 diff changeset	49 path_elts.append("common")
3185 554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	50 self.cache_dir = Path(*path_elts)
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	51
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	52 self.cache_dir.mkdir(0o700, parents=True, exist_ok=True)
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	53 self.purge()
2509 d485e9416493 core (memory/cache): common cache: Goffi <goffi@goffi.org> parents: 2506 diff changeset	54
3185 554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	55 def purge(self):
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	56 # remove expired files from cache
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	57 # TODO: this should not be called only on startup, but at regular interval
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	58 # (e.g. once a day)
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	59 purged = set()
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	60 # we sort files to have metadata files first
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	61 for cache_file in sorted(self.cache_dir.iterdir()):
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	62 if cache_file in purged:
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	63 continue
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	64 try:
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	65 with cache_file.open('rb') as f:
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	66 cache_data = pickle.load(f)
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	67 except IOError:
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	68 log.warning(
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	69 _("Can't read metadata file at {path}")
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	70 .format(path=cache_file))
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	71 continue
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	72 except (pickle.UnpicklingError, EOFError):
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	73 log.debug(f"File at {cache_file} is not a metadata file")
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	74 continue
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	75 try:
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	76 eol = cache_data['eol']
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	77 filename = cache_data['filename']
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	78 except KeyError:
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	79 log.warning(
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	80 _("Invalid cache metadata at {path}")
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	81 .format(path=cache_file))
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	82 continue
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	83
3209 f14eb24328d0 core (memory/cache): purge cache metadata when the referenced file doesn't exist Goffi <goffi@goffi.org> parents: 3198 diff changeset	84 filepath = self.getPath(filename)
f14eb24328d0 core (memory/cache): purge cache metadata when the referenced file doesn't exist Goffi <goffi@goffi.org> parents: 3198 diff changeset	85
f14eb24328d0 core (memory/cache): purge cache metadata when the referenced file doesn't exist Goffi <goffi@goffi.org> parents: 3198 diff changeset	86 if not filepath.exists():
f14eb24328d0 core (memory/cache): purge cache metadata when the referenced file doesn't exist Goffi <goffi@goffi.org> parents: 3198 diff changeset	87 log.warning(_(
f14eb24328d0 core (memory/cache): purge cache metadata when the referenced file doesn't exist Goffi <goffi@goffi.org> parents: 3198 diff changeset	88 "cache {cache_file!r} references an inexisting file: {filepath!r}"
f14eb24328d0 core (memory/cache): purge cache metadata when the referenced file doesn't exist Goffi <goffi@goffi.org> parents: 3198 diff changeset	89 ).format(cache_file=str(cache_file), filepath=str(filepath)))
f14eb24328d0 core (memory/cache): purge cache metadata when the referenced file doesn't exist Goffi <goffi@goffi.org> parents: 3198 diff changeset	90 log.debug("purging cache with missing file")
f14eb24328d0 core (memory/cache): purge cache metadata when the referenced file doesn't exist Goffi <goffi@goffi.org> parents: 3198 diff changeset	91 cache_file.unlink()
f14eb24328d0 core (memory/cache): purge cache metadata when the referenced file doesn't exist Goffi <goffi@goffi.org> parents: 3198 diff changeset	92 elif eol < time.time():
3185 554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	93 log.debug(
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	94 "purging expired cache {filepath!r} (expired for {time}s)"
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	95 .format(filepath=str(filepath), time=int(time.time() - eol))
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	96 )
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	97 cache_file.unlink()
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	98 try:
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	99 filepath.unlink()
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	100 except FileNotFoundError:
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	101 log.warning(
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	102 _("following file is missing while purging cache: {path}")
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	103 .format(path=filepath)
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	104 )
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	105 purged.add(cache_file)
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	106 purged.add(filepath)
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	107
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	108 def getPath(self, filename):
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	109 """return cached file URL
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	110
3185 554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	111 @param filename(str): cached file name (cache data or actual file)
554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	112 @return (Path): path to the cached file
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	113 """
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2771 diff changeset	114 if not filename or "/" in filename:
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	115 log.error(
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2771 diff changeset	116 "invalid char found in file name, hack attempt? name:{}".format(filename)
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	117 )
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2771 diff changeset	118 raise exceptions.DataError("Invalid char found")
3185 554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	119 return self.cache_dir / filename
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	120
3188 a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved Goffi <goffi@goffi.org> parents: 3185 diff changeset	121 def getMetadata(self, uid, update_eol=True):
3185 554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	122 """Retrieve metadata for cached data
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	123
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	124 @param uid(unicode): unique identifier of file
3188 a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved Goffi <goffi@goffi.org> parents: 3185 diff changeset	125 @param update_eol(bool): True if eol must extended
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved Goffi <goffi@goffi.org> parents: 3185 diff changeset	126 if True, max_age will be added to eol (only if it is not already expired)
2517 cd7a53c31eb6 core (memory/cache): new getMetadata method to retrieve metadata without opening the file Goffi <goffi@goffi.org> parents: 2509 diff changeset	127 @return (dict, None): metadata with following keys:
3188 a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved Goffi <goffi@goffi.org> parents: 3185 diff changeset	128 see [cacheData] for data details, an additional "path" key is the full path to
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved Goffi <goffi@goffi.org> parents: 3185 diff changeset	129 cached file.
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	130 None if file is not in cache (or cache is invalid)
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	131 """
2517 cd7a53c31eb6 core (memory/cache): new getMetadata method to retrieve metadata without opening the file Goffi <goffi@goffi.org> parents: 2509 diff changeset	132
2116 766dbbec56f2 core (memory/cache): geFilePath now return None when uid is empty Goffi <goffi@goffi.org> parents: 2109 diff changeset	133 uid = uid.strip()
766dbbec56f2 core (memory/cache): geFilePath now return None when uid is empty Goffi <goffi@goffi.org> parents: 2109 diff changeset	134 if not uid:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2771 diff changeset	135 raise exceptions.InternalError("uid must not be empty")
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	136 cache_url = self.getPath(uid)
3188 a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved Goffi <goffi@goffi.org> parents: 3185 diff changeset	137 if not cache_url.exists():
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	138 return None
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	139
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	140 try:
3185 554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	141 with cache_url.open("rb") as f:
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	142 cache_data = pickle.load(f)
3572 b3fa179417e7 core (memory/cache): don't crash on EOFError in getMetadata Goffi <goffi@goffi.org> parents: 3479 diff changeset	143 except (IOError, EOFError) as e:
3188 a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved Goffi <goffi@goffi.org> parents: 3185 diff changeset	144 log.warning(f"can't read cache at {cache_url}: {e}")
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	145 return None
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	146 except pickle.UnpicklingError:
3188 a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved Goffi <goffi@goffi.org> parents: 3185 diff changeset	147 log.warning(f"invalid cache found at {cache_url}")
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	148 return None
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	149
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	150 try:
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	151 eol = cache_data["eol"]
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	152 except KeyError:
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2771 diff changeset	153 log.warning("no End Of Life found for cached file {}".format(uid))
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	154 eol = 0
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	155 if eol < time.time():
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	156 log.debug(
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2771 diff changeset	157 "removing expired cache (expired for {}s)".format(time.time() - eol)
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	158 )
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	159 return None
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	160
3188 a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved Goffi <goffi@goffi.org> parents: 3185 diff changeset	161 if update_eol:
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved Goffi <goffi@goffi.org> parents: 3185 diff changeset	162 try:
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved Goffi <goffi@goffi.org> parents: 3185 diff changeset	163 max_age = cache_data["max_age"]
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved Goffi <goffi@goffi.org> parents: 3185 diff changeset	164 except KeyError:
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved Goffi <goffi@goffi.org> parents: 3185 diff changeset	165 log.warning(f"no max_age found for cache at {cache_url}, using default")
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved Goffi <goffi@goffi.org> parents: 3185 diff changeset	166 max_age = cache_data["max_age"] = C.DEFAULT_MAX_AGE
3198 08151c103636 core (memory/cache): added some metadata: Goffi <goffi@goffi.org> parents: 3188 diff changeset	167 now = int(time.time())
08151c103636 core (memory/cache): added some metadata: Goffi <goffi@goffi.org> parents: 3188 diff changeset	168 cache_data["last_access"] = now
08151c103636 core (memory/cache): added some metadata: Goffi <goffi@goffi.org> parents: 3188 diff changeset	169 cache_data["eol"] = now + max_age
3188 a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved Goffi <goffi@goffi.org> parents: 3185 diff changeset	170 with cache_url.open("wb") as f:
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved Goffi <goffi@goffi.org> parents: 3185 diff changeset	171 pickle.dump(cache_data, f, protocol=2)
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved Goffi <goffi@goffi.org> parents: 3185 diff changeset	172
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	173 cache_data["path"] = self.getPath(cache_data["filename"])
2517 cd7a53c31eb6 core (memory/cache): new getMetadata method to retrieve metadata without opening the file Goffi <goffi@goffi.org> parents: 2509 diff changeset	174 return cache_data
cd7a53c31eb6 core (memory/cache): new getMetadata method to retrieve metadata without opening the file Goffi <goffi@goffi.org> parents: 2509 diff changeset	175
cd7a53c31eb6 core (memory/cache): new getMetadata method to retrieve metadata without opening the file Goffi <goffi@goffi.org> parents: 2509 diff changeset	176 def getFilePath(self, uid):
3185 554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	177 """Retrieve absolute path to file
2517 cd7a53c31eb6 core (memory/cache): new getMetadata method to retrieve metadata without opening the file Goffi <goffi@goffi.org> parents: 2509 diff changeset	178
cd7a53c31eb6 core (memory/cache): new getMetadata method to retrieve metadata without opening the file Goffi <goffi@goffi.org> parents: 2509 diff changeset	179 @param uid(unicode): unique identifier of file
cd7a53c31eb6 core (memory/cache): new getMetadata method to retrieve metadata without opening the file Goffi <goffi@goffi.org> parents: 2509 diff changeset	180 @return (unicode, None): absolute path to cached file
cd7a53c31eb6 core (memory/cache): new getMetadata method to retrieve metadata without opening the file Goffi <goffi@goffi.org> parents: 2509 diff changeset	181 None if file is not in cache (or cache is invalid)
cd7a53c31eb6 core (memory/cache): new getMetadata method to retrieve metadata without opening the file Goffi <goffi@goffi.org> parents: 2509 diff changeset	182 """
cd7a53c31eb6 core (memory/cache): new getMetadata method to retrieve metadata without opening the file Goffi <goffi@goffi.org> parents: 2509 diff changeset	183 metadata = self.getMetadata(uid)
cd7a53c31eb6 core (memory/cache): new getMetadata method to retrieve metadata without opening the file Goffi <goffi@goffi.org> parents: 2509 diff changeset	184 if metadata is not None:
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	185 return metadata["path"]
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	186
3210 fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	187 def removeFromCache(self, uid, metadata=None):
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	188 """Remove data from cache
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	189
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	190 @param uid(unicode): unique identifier cache file
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	191 """
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	192 cache_data = self.getMetadata(uid, update_eol=False)
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	193 if cache_data is None:
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	194 log.debug(f"cache with uid {uid!r} has already expired or been removed")
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	195 return
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	196
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	197 try:
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	198 filename = cache_data['filename']
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	199 except KeyError:
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	200 log.warning(_("missing filename for cache {uid!r}") .format(uid=uid))
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	201 else:
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	202 filepath = self.getPath(filename)
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	203 try:
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	204 filepath.unlink()
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	205 except FileNotFoundError:
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	206 log.warning(
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	207 _("missing file referenced in cache {uid!r}: {filename}")
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	208 .format(uid=uid, filename=filename)
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	209 )
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	210
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	211 cache_file = self.getPath(uid)
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	212 cache_file.unlink()
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	213 log.debug(f"cache with uid {uid!r} has been removed")
fedec192a83f core (memory/cache): new removeFromCache method Goffi <goffi@goffi.org> parents: 3209 diff changeset	214
2509 d485e9416493 core (memory/cache): common cache: Goffi <goffi@goffi.org> parents: 2506 diff changeset	215 def cacheData(self, source, uid, mime_type=None, max_age=None, filename=None):
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	216 """create cache metadata and file object to use for actual data
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	217
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	218 @param source(unicode): source of the cache (should be plugin's import_name)
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	219 @param uid(unicode): an identifier of the file which must be unique
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	220 @param mime_type(unicode): MIME type of the file to cache
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	221 it will be used notably to guess file extension
3198 08151c103636 core (memory/cache): added some metadata: Goffi <goffi@goffi.org> parents: 3188 diff changeset	222 It may be autogenerated if filename is specified
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	223 @param max_age(int, None): maximum age in seconds
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	224 the cache metadata will have an "eol" (end of life)
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	225 None to use default value
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	226 0 to ignore cache (file will be re-downloaded on each access)
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	227 @param filename: if not None, will be used as filename
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	228 else one will be generated from uid and guessed extension
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	229 @return(file): file object opened in write mode
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	230 you have to close it yourself (hint: use with statement)
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	231 """
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	232 cache_url = self.getPath(uid)
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	233 if filename is None:
2509 d485e9416493 core (memory/cache): common cache: Goffi <goffi@goffi.org> parents: 2506 diff changeset	234 if mime_type:
d485e9416493 core (memory/cache): common cache: Goffi <goffi@goffi.org> parents: 2506 diff changeset	235 ext = mimetypes.guess_extension(mime_type, strict=False)
d485e9416493 core (memory/cache): common cache: Goffi <goffi@goffi.org> parents: 2506 diff changeset	236 if ext is None:
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	237 log.warning(
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2771 diff changeset	238 "can't find extension for MIME type {}".format(mime_type)
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	239 )
2509 d485e9416493 core (memory/cache): common cache: Goffi <goffi@goffi.org> parents: 2506 diff changeset	240 ext = DEFAULT_EXT
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2771 diff changeset	241 elif ext == ".jpe":
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2771 diff changeset	242 ext = ".jpg"
2509 d485e9416493 core (memory/cache): common cache: Goffi <goffi@goffi.org> parents: 2506 diff changeset	243 else:
d485e9416493 core (memory/cache): common cache: Goffi <goffi@goffi.org> parents: 2506 diff changeset	244 ext = DEFAULT_EXT
d485e9416493 core (memory/cache): common cache: Goffi <goffi@goffi.org> parents: 2506 diff changeset	245 mime_type = None
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	246 filename = uid + ext
3198 08151c103636 core (memory/cache): added some metadata: Goffi <goffi@goffi.org> parents: 3188 diff changeset	247 elif mime_type is None:
08151c103636 core (memory/cache): added some metadata: Goffi <goffi@goffi.org> parents: 3188 diff changeset	248 # we have filename but not MIME type, we try to guess the later
08151c103636 core (memory/cache): added some metadata: Goffi <goffi@goffi.org> parents: 3188 diff changeset	249 mime_type = mimetypes.guess_type(filename, strict=False)[0]
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	250 if max_age is None:
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	251 max_age = C.DEFAULT_MAX_AGE
3198 08151c103636 core (memory/cache): added some metadata: Goffi <goffi@goffi.org> parents: 3188 diff changeset	252 now = int(time.time())
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	253 cache_data = {
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2771 diff changeset	254 "source": source,
ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2771 diff changeset	255 "filename": filename,
3198 08151c103636 core (memory/cache): added some metadata: Goffi <goffi@goffi.org> parents: 3188 diff changeset	256 "creation": now,
08151c103636 core (memory/cache): added some metadata: Goffi <goffi@goffi.org> parents: 3188 diff changeset	257 "eol": now + max_age,
3188 a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved Goffi <goffi@goffi.org> parents: 3185 diff changeset	258 # we also store max_age for updating eol
a15773c6c273 memory(cache): extend EOL when a file metadata is retrieved Goffi <goffi@goffi.org> parents: 3185 diff changeset	259 "max_age": max_age,
3028 ab2696e34d29 Python 3 port: Goffi <goffi@goffi.org> parents: 2771 diff changeset	260 "mime_type": mime_type,
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	261 }
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	262 file_path = self.getPath(filename)
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	263
2624 56f94936df1e code style reformatting using black Goffi <goffi@goffi.org> parents: 2562 diff changeset	264 with open(cache_url, "wb") as f:
2109 85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	265 pickle.dump(cache_data, f, protocol=2)
85f3e12e984d core (memory/cache): file caching handling, first draft: Goffi <goffi@goffi.org> parents: diff changeset	266
3185 554b3b632378 memory (cache): purge + pathlib: Goffi <goffi@goffi.org> parents: 3137 diff changeset	267 return file_path.open("wb")

Mercurial > libervia-backend

annotate sat/memory/cache.py @ 3582:71516731d0aa